I am trying to make quick test for the pub/sub mechanism to my registered device on TTN so I can build my complete solution app on the data coming to the TTN broker.
At the moment I am waiting for my loRa module to arrive, that is why I want to use a simple nodeJS script to publish dummy data, and the other to subscribe and build an app using the dummy data. I use the following code for this:
var mqtt = require('mqtt')
var options = {
port: 1883,
host: ‘mqtt://eu.thethings.network’,
username: ‘xxxx’, // here I wrote my app id
password: ‘xxxx’ // here I wrote the access key
};
var client = mqtt.connect(‘mqtt://eu.thethings.network’,options)
client.on(‘connect’, function () {
client.subscribe(‘appID/devices/MyDeviceName/down’, function (err) {
if (!err) {
client.publish(‘appID/devices/MyDeviceName/down’, ‘Hello mqtt’)
}
})
})
client.on(‘message’, function (topic, message) {
// message is Buffer
console.log(message.toString())
// client.end()
})
This is however not doing anything, I was watching the data on TTN, nothing coming in.
I also tried using mqtt explorer but it did not work.
Both methods worked fine when I played the broker on my machine, eclipse and mosquittoo on cloud.
Your help is greatly appreciated.
Thanks!
Ahmed
I have encountered a similar issue in the past. I believe the issue is with trying to use "mqtt" instead of "https". For me, it worked when I called
mqtt.connect('https://thethings.network:1883', {
"username": username,
"password": password
}
However, I wasn't using the community version of the website (The Things Stack V3), so there might be a slight difference. For example, instead of "My-App-Id" I had to use "My-App-Id#My-Company-Name".
Please, try the above and let me know if it works.
Related
Promises hang with no errors, with a Google's Pub/Sub Node client library against your project.
Example:
const { PubSub } = require("#google-cloud/pubsub");
async function start() {
const pubsubClient = new PubSub({ projectId: "my-project-id" });
try {
const [topics] = await pubsubClient.getTopics();
console.log(topics);
} catch (error) {
console.error(error);
}
}
start().catch(console.error);
would return no error and no progress would be shown. Eventually the client times out after 10 minutes. No topics would get returned. The same goes for publishing to a topic, etc.
If you used emulator pubsub for local development you have set PUBSUB_EMULATOR_HOST variable. For some reason it leads to the issue. Remove it from your environment with unset PUBSUB_EMULATOR_HOST or remove it from your .env file and restart the server.
You can check if it is set with printenv in your shell (or run exec from the node app to check)
There is a known issue and there are associated Github Issues, so if you came across this answer and it helped, feel free to let the maintainers know here:
https://github.com/googleapis/nodejs-pubsub/issues/339
or here:
https://github.com/googleapis/gax-nodejs/issues/208
as it's a won't fix as seems to not affect many people.
I started using Azure recently and It has been an overwhelming experience. I started experimenting with eventhubs and I'm basically following the official tutorials on how to send and receive messages from eventhubs using nodejs.
Everything worked perfectly so I built a small web app (static frontend app) and I connected it with a node backend, where the communication with eventhubs occurs. So basically my app is built like this:
frontend <----> node server <-----> eventhubs
As you can see it is very simple. The node server is fetching data from eventhubs and sending it forward to the frontend, where the values are shown. It is a cool experience and I'm enjoying MS Azure until this error occured:
azure.eventhub.common.EventHubError: ErrorCodes.ResourceLimitExceeded: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - nil, nil, nil, nil, nil.
This error is really confusing. Im using the default consumer group and only one app. I never tried to access this consumer group from another app. It said the limit is 5, I'm using only one app so it should be fine or am I missing something? I'm not checking what is happening here.
I wasted too much time googling and researching about this but I didn't get it. At the end, I thought that maybe every time I deploy the app (my frontend and my node server) on azure, this would be counted as one consumer and since I deployed the app more than 5 times then this error is showing up. Am I right or this is nonsense?
Edit
I'm using websockets as a communication protocol between my app (frontend) and my node server (backend). The node server is using the default consumer group ( I didn't change nothing), I just followed this official example from Microsoft. I'm basically using the code from MS docs that's why I didn't post any code snippet from my node server and since the error happens in backend and not frontend then it will not be helpful if I posted any frontend code.
So to wrap up, I'm using websocket to connect front & backend. It works perfectly for a day or two and then this error starts to happen. Sometimes I open more than one client (for example a client from the browser and client from my smartphone).
I think I don't understand the concept of this consumer group. Like is every client a consumer? so if I open my app (the same app) in 5 different tabs in my browser, do I have 5 consumers then?
I didn't quite understand the answer below and what is meant by "pooling client", therefore, I will try to post code examples here to show you what I'm trying to do.
Code snippets
Here is the function I'm using on the server side to communicate with eventhubs and receive/consume a message
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
If you notice, I'm giving the eventhub and connection string as an argument in order to be able to change that. Now in the frontend, I have a list of multiple topics and each topic have its own eventhubname but they have the same eventhub namespace.
Here is an example of two eventhubnames that I have:
{
"EventHubName": "eh-test-command"
"EventHubName": "eh-test-telemetry"
}
If the user chooses to send a command (from the frontend, I just have a list of buttons that the user can click to fire an event over websockets) then the CommandEventHubName will be sent from the frontend to the node server. The server will receive that eventhubname and switch the consumerClient in the function I posted above.
Here is the code where I'm calling that:
// io is a socket.io object
io.on('connection', socket => {
socket.on('onUserChoice', choice => {
// choice is an object sent from the frontend based on what the user choosed. e.g if the user choosed command then choice = {"EventhubName": "eh-test-command", "payload": "whatever"}
receiveEventhubMessage(socket, choice.EventHubName, choice.EventHubNameSpace)
.catch(err => console.log(`[ consumerClient ] Error while receiving eventhub messages: ${err}`));
}
}
The app I'm building will be extending in the future to a real use case in the automotive field, that's why this is important for me. Therefore, I'm trying to figure out how can I switch between eventhubs without creating a new consumerClient each time the eventhubname changes?
I must say that I didn't understand the example with the "pooling client". I am seeking more elaboration or, ideally, a minimal example just to put me on the way.
Based on the conversation in the issue, it would seem that the root cause of this is that your backend is creating a new EventHubConsumerClient for each request coming from your frontend. Because each client will open a dedicated connection to the service, if you have more than 5 requests for the same Event Hub instance using the same consumer group, you'll exceed the quota.
To get around this, you'll want to consider pooling your EventHubConsumerClient instances so that you're starting with one per Event Hub instance. You can safely use the pooled client to handle a request for your frontend by calling subscribe. This will allow you to share the connection amongst multiple frontend requests.
The key idea being that your consumerClient is not created for every request, but shares an instance among requests. Using your snippet to illustrate the simplest approach, you'd end up hoisting your client creation to outside the function to receive. It may look something like:
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
That said, the above may not be adequate for your environment depending on the architecture of the application. If whatever is hosting receiveEventHubMessage is created dynamically for each request, nothing changes. In that case, you'd want to consider something like a singleton or dependency injection to help extend the lifespan.
If you end up having issues scaling to meet your requests, you can consider increasing the number of clients for each Event Hub and/or spreading requests out to different consumer groups.
I am using this bitcore npm package.
https://bitcore.io/api/lib
And i want to monitor all the transactions over the blockchain, and read the input address, output address and amount associated with that transaction.
But i am unable to find the javascript method to invoke to accomplish this.
Even i am not able to find a example for this.
I am looking for as short as something like
var someLib = require('some-bitcore-lib')
someLib.on('transaction-found', function(){
// print everything
console.log(arguments);
// do something else;
})
Any help?
Where can i find that some-bitcore-lib or how can i create that in nodejs?
Using a third-party API, as the accepted answers suggests, will work in the short-term. But if you're looking for a long-term, reliable, not-rate-limited solution; you should run your own bitcoin node. It, of course, depends on your project's requirements.
For a robust solution to the OP's question, I suggest the following:
Run a pruned bitcoin node using bitcoind
Enable the ZeroMQ interface of bitcoind with the configuration option zmqpubrawtx=tcp://127.0.0.1:3600. This will enable streaming of raw transaction data to your node.js application
Use the ZeroMQ node.js module to subscribe to the bitcoind's ZeroMQ interface
Use bitcoinjs-lib to decode the raw transaction data
The following node.js example will use zeromq to subscribe to bitcoind's zeromq interface. Then bitcoinjs-lib is used to decode those raw transactions.
var bitcoin = require('bitcoinjs-lib');
var zmq = require('zeromq');
var sock = zmq.socket('sub');
var addr = 'tcp://127.0.0.1:3600';
sock.connect(addr);
sock.subscribe('rawtx');
sock.on('message', function(topic, message) {
if (topic.toString() === 'rawtx') {
var rawTx = message.toString('hex');
var tx = bitcoin.Transaction.fromHex(rawTx);
var txid = tx.getId();
tx.ins = tx.ins.map(function(in) {
in.address = bitcoin.address.fromOutputScript(in.script, bitcoin.networks.bitcoin);
return in;
});
tx.outs = tx.outs.map(function(out) {
out.address = bitcoin.address.fromOutputScript(out.script, bitcoin.networks.bitcoin);
return out;
});
console.log('received transaction', txid, tx);
}
});
For more details, please have a look at this guide
If you don't have your own node you can use blockchain.info APIs as described in here (https://github.com/blockchain/api-v1-client-node/tree/master/Socket)
const Socket = require('blockchain.info/Socket');
const mySocket = new Socket();
mySocket.onTransaction(function() {
console.log(arguments);
});
You can always watch transactions by running your own node without the need to depend on a service like blockchain.info... For example, if you are using btcd (Golang) (https://github.com/btcsuite/btcd) then you can get notified on transactions like in here (http://godoc.org/github.com/btcsuite/btcrpcclient#Client.NotifyNewTransactions)
I think this is what you're looking for. The tutorial helps the user set up a local btc node and demonstrates how to use a zmq subscription along with RPC comms to accomplish sending and receiving transactions as well as notifications and other functionality.
#c.hill's response is correct but leaves out the more complicated functionality described here :)
I'm using node.js to create an app that gets a PUSH from Gmail each time an email is received, checks it against a third party database in a CRM and creates a new field in the CRM if the e-mail is contained there. I'm having trouble using Google's new Cloud Pub/Sub, which seems to be the only way to get push from Gmail without constant polling.
I've gone through the instructions here: https://cloud.google.com/pubsub/prereqs but I don't understand how exactly this is supposed to work from an app on my desktop. It seems that pub/sub can connect to a verified domain, but I can't get it to connect directly toto the .js script that I have on my computer. I've saved the api key in a json file and use the following:
var gcloud = require('gcloud');
var pubsub;
// From Google Compute Engine:
pubsub = gcloud.pubsub({
projectId: 'my-project',
});
// Or from elsewhere:
pubsub = gcloud.pubsub({
projectId: 'my-project',
keyFilename: '/path/to/keyfile.json'
});
// Create a new topic.
pubsub.createTopic('my-new-topic', function(err, topic) {});
// Reference an existing topic.
var topic = pubsub.topic('my-existing-topic');
// Publish a message to the topic.
topic.publish('New message!', function(err) {});
// Subscribe to the topic.
topic.subscribe('new-subscription', function(err, subscription) {
// Register listeners to start pulling for messages.
function onError(err) {}
function onMessage(message) {}
subscription.on('error', onError);
subscription.on('message', onMessage);
// Remove listeners to stop pulling for messages.
subscription.removeListener('message', onMessage);
subscription.removeListener('error', onError);
});
However, I get errors as if it isn't connecting to server and on the API list I see only errors, no actual successes. I'm clearly doing something wrong, any idea what it might be?
Thank you in advance!
TL;DR
Your cannot subscribe to push notifications from the client side.
Set up an HTTPS server to handle the messages. Messages will be sent
to the URL endpoint that you configure, representing that server's
location. Your server must be reachable via a DNS name and must
present a signed SSL certificate. (App Engine applications are
preconfigured with SSL certificates.)
Just subscribe to the push notifications on your server, and when you get the notification, you can figure out who it concerns. The data you will get from the notifications is what user that it concerns, and the relevant historyId, like so:
// This is all the data the notifications will give you.
{"emailAddress": "user#example.com", "historyId": "9876543210"}
Then you could e.g. emit an event through Socket.io to the relevant user if he is online, and have him do a sync with the supplied historyId on the client side.
I'm having trouble using RabbitMQ with my Sails app. I'm unsure of where to place the subscriber code. What I'm trying to do is build a notifications system so that when an administrator approves a user's data request, the user's dashboard will pop a notification similar to how Facebook pops a notification. The problem is, putting the subscriber code in my dashboard controller's display route seems to never grab a published message.
Any advice would be greatly appreciated. Currently using rabbit.js package to connect to RabbitMQ.
To answer the original question, if one for some reason wanted to use Rabbit MQ instead of Sails' built-in resourceful pubsub, the best thing would be to use rabbit.js.
First, npm install rabbit.js.
Then, in your Sails project's config/sockets.js (borrowed liberally from the rabbit.js socket.io example):
var context = require('rabbit.js').createContext();
module.exports = {
onConnect: function(session, socket) {
var pub = context.socket('PUB');
var sub = context.socket('SUB');
socket.on('disconnect', function() {
pub.close();
sub.close();
});
// NB we have to adapt between the APIs
sub.setEncoding('utf8');
socket.on('message', function(msg) {
pub.write(msg, 'utf8');
});
sub.on('data', function(msg) {
socket.send(msg);
});
sub.connect('chat');
pub.connect('chat');
}
}
Here's an npm package that implements a RabbitMQ adapter for SailsJS.