I started using Azure recently and It has been an overwhelming experience. I started experimenting with eventhubs and I'm basically following the official tutorials on how to send and receive messages from eventhubs using nodejs.
Everything worked perfectly so I built a small web app (static frontend app) and I connected it with a node backend, where the communication with eventhubs occurs. So basically my app is built like this:
frontend <----> node server <-----> eventhubs
As you can see it is very simple. The node server is fetching data from eventhubs and sending it forward to the frontend, where the values are shown. It is a cool experience and I'm enjoying MS Azure until this error occured:
azure.eventhub.common.EventHubError: ErrorCodes.ResourceLimitExceeded: Exceeded the maximum number of allowed receivers per partition in a consumer group which is 5. List of connected receivers - nil, nil, nil, nil, nil.
This error is really confusing. Im using the default consumer group and only one app. I never tried to access this consumer group from another app. It said the limit is 5, I'm using only one app so it should be fine or am I missing something? I'm not checking what is happening here.
I wasted too much time googling and researching about this but I didn't get it. At the end, I thought that maybe every time I deploy the app (my frontend and my node server) on azure, this would be counted as one consumer and since I deployed the app more than 5 times then this error is showing up. Am I right or this is nonsense?
Edit
I'm using websockets as a communication protocol between my app (frontend) and my node server (backend). The node server is using the default consumer group ( I didn't change nothing), I just followed this official example from Microsoft. I'm basically using the code from MS docs that's why I didn't post any code snippet from my node server and since the error happens in backend and not frontend then it will not be helpful if I posted any frontend code.
So to wrap up, I'm using websocket to connect front & backend. It works perfectly for a day or two and then this error starts to happen. Sometimes I open more than one client (for example a client from the browser and client from my smartphone).
I think I don't understand the concept of this consumer group. Like is every client a consumer? so if I open my app (the same app) in 5 different tabs in my browser, do I have 5 consumers then?
I didn't quite understand the answer below and what is meant by "pooling client", therefore, I will try to post code examples here to show you what I'm trying to do.
Code snippets
Here is the function I'm using on the server side to communicate with eventhubs and receive/consume a message
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
If you notice, I'm giving the eventhub and connection string as an argument in order to be able to change that. Now in the frontend, I have a list of multiple topics and each topic have its own eventhubname but they have the same eventhub namespace.
Here is an example of two eventhubnames that I have:
{
"EventHubName": "eh-test-command"
"EventHubName": "eh-test-telemetry"
}
If the user chooses to send a command (from the frontend, I just have a list of buttons that the user can click to fire an event over websockets) then the CommandEventHubName will be sent from the frontend to the node server. The server will receive that eventhubname and switch the consumerClient in the function I posted above.
Here is the code where I'm calling that:
// io is a socket.io object
io.on('connection', socket => {
socket.on('onUserChoice', choice => {
// choice is an object sent from the frontend based on what the user choosed. e.g if the user choosed command then choice = {"EventhubName": "eh-test-command", "payload": "whatever"}
receiveEventhubMessage(socket, choice.EventHubName, choice.EventHubNameSpace)
.catch(err => console.log(`[ consumerClient ] Error while receiving eventhub messages: ${err}`));
}
}
The app I'm building will be extending in the future to a real use case in the automotive field, that's why this is important for me. Therefore, I'm trying to figure out how can I switch between eventhubs without creating a new consumerClient each time the eventhubname changes?
I must say that I didn't understand the example with the "pooling client". I am seeking more elaboration or, ideally, a minimal example just to put me on the way.
Based on the conversation in the issue, it would seem that the root cause of this is that your backend is creating a new EventHubConsumerClient for each request coming from your frontend. Because each client will open a dedicated connection to the service, if you have more than 5 requests for the same Event Hub instance using the same consumer group, you'll exceed the quota.
To get around this, you'll want to consider pooling your EventHubConsumerClient instances so that you're starting with one per Event Hub instance. You can safely use the pooled client to handle a request for your frontend by calling subscribe. This will allow you to share the connection amongst multiple frontend requests.
The key idea being that your consumerClient is not created for every request, but shares an instance among requests. Using your snippet to illustrate the simplest approach, you'd end up hoisting your client creation to outside the function to receive. It may look something like:
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
async function receiveEventhubMessage(socket, eventHubName, connectionString) {
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log("[ consumer ] Message received : " + event.body);
io.emit('msg-received', event.body);
}
},
processError: async (err, context) => {
console.log(`Error : ${err}`);
}
}
);
That said, the above may not be adequate for your environment depending on the architecture of the application. If whatever is hosting receiveEventHubMessage is created dynamically for each request, nothing changes. In that case, you'd want to consider something like a singleton or dependency injection to help extend the lifespan.
If you end up having issues scaling to meet your requests, you can consider increasing the number of clients for each Event Hub and/or spreading requests out to different consumer groups.
Related
I am confused as to how should the watch feature in the gmail API be implemented to recieve the push notificatons inside a node.js script. Should I call the method inside an infinite loop or something so that it doesn't stop listening for notifications for email once after the call is made?
Here's the sample code that I've written in node.js:
const getEmailNotification = () => {
return new Promise(async (resolve, reject) => {
try{
let auth = await authenticate();
const gmail = google.gmail({version: 'v1', auth});
await gmail.users.stop({
userId: '<email id>'
});
let watchResponse = await gmail.users.watch({
userId: '<email id>',
labelIds: ['INBOX'],
topicName: 'projects/<projectName>/topics/<topicName>'
})
return resolve(watchResponse);
} catch(err){
return reject(`Some error occurred`);
}
})
Thank you!
Summary
To receive push notifications through PUB/SUB you need to create a web-hook to receive them. What does this mean? You need a WEB application or any kind of service that exposes a URL where notifications can be received.
As stated in the Push subscription documentation:
The Pub/Sub server sends each message as an HTTPS request to the subscriber application at a pre-configured endpoint.
The endpoint acknowledges the message by returning an HTTP success status code. A non-success response indicates that the message should be resent.
Setup a channel for watch the notifications could be summarized in the following steps (the documentation you refer to indicates them):
Select/Create a project within the Google Cloud Console.
Create a new PUB/SUB topic
Create a subscription (PUSH) for that topic.
Add the necessary permissions, in this case add gmail-api-push#system.gserviceaccount.com as Pub/Sub Publisher.
Indicate what types of mail you want it to listen for via Users.watch() method (which is what you are doing in your script).
Example
I give you an example using Apps Script (it is an easy way to visualize it, but this could be achieved from any kind of WEB application, as you are using Node.js I suppose that you are familiarized with Express.js or related frameworks).
First I created a new Google Apps Script project, this will be my web-hook. Basically I want it to make a log of all HTTP/POST requests inside a Google Doc that I have previously created. For it I use the doPost() equal to app.post() in Express. If you want to know more about how Apps Script works, you can visit this link), but this is not the main topic.
Code.gs
const doPost = (e) => {
const doc = DocumentApp.openById(<DOC_ID>)
doc.getBody().appendParagraph(JSON.stringify(e, null, 2))
}
Later I made a new implementation as a Web App where I say that it is accessible by anyone, I write down the URL for later. This will be similar to deploying your Node.js application to the internet.
I select a project in the Cloud Console, as indicated in the Prerequisites of Cloud Pub/Sub.
Inside this project, I create a new topic that I call GmailAPIPush. After, click in Add Main (in the right bar of the Topics section ) and add gmail-api-push#system.gserviceaccount.com with the Pub/Sub Publisher role. This is a requirement that grants Gmail privileges to publish notification.
In the same project, I create a Subscription. I tell it to be of the Push type and add the URL of the Web App that I have previously created.
This is the most critical part and makes the difference of how you want your application to work. If you want to know which type of subscription best suits your needs (PUSH or PULL), you have a detailed documentation that will help you choose between these two types.
Finally we are left with the simplest part, configuring the Gmail account to send updates on the mailbox. I am going to do this from Apps Script, but it is exactly the same as with Node.
const watchUserGmail = () => {
const request = {
'labelIds': ['INBOX'],
'topicName': 'projects/my_project_name/topics/GmailAPIPush'
}
Gmail.Users.watch(request, 'me')
}
Once the function is executed, I send a test message, and voila, the notification appears in my document.
Returning to the case that you expose, I am going to try to explain it with a metaphor. Imagine you have a mailbox, and you are waiting for a very important letter. As you are nervous, you go every 5 minutes to check if the letter has arrived (similar to what you propose with setInterval), that makes that most of the times that you go to check your mailbox, there is nothing new. However, you train your dog to bark (push notification) every time the mailman comes, so you only go to check your mailbox when you know you have new letters.
I'm trying to connect my firebase cloud functions project to a third party pub/sub (a separate project). According to this thread this is not possible, so there's no traditional way to make this work. However, I've tried to manually subscribe to certain topics using the #google-cloud/pubsub client on my firebase cloud functions. I need to react to pub/sub messages and write/update certain documents.
Example (minimal):
I have a pubsub subscription on sub.ts:
const pubSubClient = new PubSub({
projectId: config.project_id,
credentials: {
client_email: config.client_email,
private_key: config.private_key
}
});
I subscribe to a certain topic to do some business logic
const subscription = pubSubClient.subscription('my-subscription');
this.subscription.on('message', async (message) => {
try {
message.ack();
const event = parseData(message.data);
await admin.firestore().collection('my-collection').add(event);
} catch (e) {
console.error(e);
}
});
Then this file is imported within the index.js where I declare most CF functions.
import * as admin from 'firebase-admin';
admin.initializeApp();
import './sub';
export { myFunction } from './modules/my-module';
export { myOtherFunction } from './modules/other-module';
It appears that my subscriptions die out after a time and messages won't go through. If I redeploy my functions it appears to be working for a time, but then it stops listening to messages. I've read that firebase cloud functions are stateless, so in this case I need a "stateful" module within my firebase project. Is this possible? Or should I manage this on another server?.
Thanks!
What you're trying to do here (subscribe to a pubsub topic from code running in Cloud Functions) won't work for two reasons:
Cloud Functions server instances scale up and down automatically. There could be 0 instances or 1000 instances concurrently running your triggers, depending on the current load.
Cloud Functions shuts down running code when the function has terminated, and there is a maximum timeout of 9 minutes for any function invocation.
So, even if you manage to subscribe to a topic, that subscription doesn't have a guaranteed duration. It will eventually be destroyed, and you will lose messages.
If you want to handle messages using Cloud Functions in "project A", but the messages come from "project B", you should consider sending them from A to B, perhaps by using pubsub function in B that does nothing other that publish each message to a topic in A. You can then write another function to handle them in A.
I am using Rascal.Js(it uses amqplib) for my messaging logic with rabbitMq on node.js app.
I am using something similar to their example on my project startup, which creates a permanent instance and "registers" all of my subscribers and redirects messages when they arrive to the queue (in the background).
My issue is with the publishers. There are http requests from outside which should trigger my publishers. A user clicks on create button of sorts which leads to certain flow of actions. At some point it reaches the point at which I need to use a publisher.
And here I am not sure about the right approach. Do I need to open a new connection every time I need to publish a message? and close it after it ends? Or maybe I should implement this in a way that it keeps the same connection open for all of the publishers? (I actually not so sure how to create it in a way that it can be accessed from other parts of my app).
At the moment I am using the following :
async publishMessage(publisherName, message) {
const dynamicSettings = setupDynamicVariablesFromConfigFiles(minimalPublishSettings);
const broker = await Rascal.BrokerAsPromised.create(Rascal.withDefaultConfig(dynamicSettings.rascal));
broker.on('error', async function(err) {
loggerUtil.writeToLog('error', 'publishMessage() broker_error_event: ' + publisherName + err + err.stack);
await broker.shutdown();
})
const publication = await broker.publish(publisherName, message);
try {
publication.on('error', async function(err) {
loggerUtil.writeToLog('error', 'publishMessage() publish_error_event: ' + err + err.stack);
await broker.shutdown();
}).on("success", async (messageId) => {
await broker.shutdown();
}).on("return", async (message) => {
loggerUtil.writeToLog('error', 'publishMessage() publish_return_event: ' + err + err.stack);
await broker.shutdown();
})
}
catch(err) {
loggerUtil.writeToLog('error', 'Something went wrong ' + err + err.stack);
await broker.shutdown();
}
}
I use this function from different parts of my app when I need to publish messages.
I thought to just add the broker.shutdown() for all of the endpoints but at some point after an error, I got an exception about closing a connection which was already closed, and this got me worried about the shutdown approach (which probably not a good one). I think it is related to this -
I tried doing that (the commented code) but I think it isnt working well in certain situations. If everything is ok it goes to "success" and then I can close it.
But one time I had an error instead of success and when I tried to use broker.shutdown() it gave me another exception which crashed the app. I think it is related to this -
https://github.com/squaremo/amqp.node/issues/111
I am not sure what might be the safest way to approach this?
Edit:
Actually now that I think about it, the exception might be related to me trying to shutdown the broker in the catch{} area as well. I will continue to investigate.
Rascal is designed to be initiated once at application startup, rather than created per HTTP request. Your application will be extremely slow if you use it in this way, and depending on how many concurrent requests you need to handle, could easily exceed max number of connections you can make to the broker. Furthermore you will get none of the benefits that Rascal provides, such as failed connection recovery.
If you can pre-determine the queue or exchange you need to publish to, then configure Rascal at application start-up (prior to your http server), and share the publisher between requests. If you are unable to determine the queue or exchange until your receive the http request, then Rascal is not an appropriate choice. Instead you're better off using amqplib directly, but should still establish a shared connection and channel. You will have to handle connection and channel errors manually though, otherwise they will crash your application.
Greetings Stackoverflow.
I've been using stackoverflow for years to find answers, and this is my first attempts to make a question myself. So feel free to tell me if I'm doing it wrong way.
Currently I'm developing a data analytical system based on microservice architecture.
It is assumed that this system will consist of a dozen self-sufficient microservices communicating with each other by RabbitMQ. Each of them is encapsulated in a docker-container and the whole system is powered by docker-swarm in the production.
In particular each microservice is a node.js application and related database, connected with some ORM interface. Its task is to manage and serve data in a CRUD manner, and to provide results of some prepared queries based on the contained data. Nothing extraordinary.
To provide microservice-microservice communication I assume to use amqplib. But the way to use it is uncertain yet.
My current question is how to make use of amqplib in a OOP manner to link inter microservice communication network with application's object-related functionality? By OOP manner, I mean the possibility to replace amqplib (and RabbitMQ itself) in the long run without the need to make changes to the data-related logic.
What I really searching for is the example of currently working microservice application utilizing AMQP. I'd pretty much appreciate that if somebody could give a link to it.
And the second part of my question.
Does it make sense to build microservice application based on event-driven principals, and just pass messages from RabbitMQ to the application's main event queue? So that each procedure would be called the same way, despite the fact that it is an internal or external event.
As for the abstract example of single microservice:
Let's say I have an event service and a listener connected to this service:
class UserManager {
constructor(eventService) {
this.eventService = eventService;
this.eventServce.on("users.user.create-request", (payload) => {
User.create(payload); // User interface is omitted in this example
}
}
}
const eventService = new EventEmmiter();
const userManager = new UserManager(eventService);
On the other hand I've got RabbitMQ connection, that is waiting for messages:
const amqp = require('amqplib');
amqp.connect('amqp-service-in-docker').then(connection => {
connection.createChannel().then(channel => {
// Here we use topic type of exchange to be able to filter only related messages
channel.assertExchange('some-exchange', 'topic');
channel.assertQueue('').then(queue => {
// And here we are waiting only the related messages
channel.bind(queue.queue, 'some-exchange', 'users.*');
channel.consume(queue.queue, message => {
// And here is the crucial part
}
}
}
}
What I'm currently think off is to just parse and forward this message to eventService and use it's routing key as the name of the event, like this:
channel.consume(query.query, message => {
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
But how about RPC's? Should I make another exchange or even a channel for them with another approach, something like:
// In RPC channel
channel.consume(query.query, message => {
eventService.once('users.user.create-response', response => {
const recipient = message.properites.replyTo;
const correlationId = msg.properties.correlationId;
// Send response to specified recipient
channel.sendToQueue(
recipient,
Buffer.from(JSON.stringify(resonse)),
{
correlationId: correlationId
}
);
channel.ack(message);
});
// Same thing
const eventName = message.fields.routingKey;
const eventPayload = JSON.parse(message.content.toString());
eventService.emit(eventName, eventPayload);
}
And then my User class should fire 'users.user.create-response' event every time it creates a new user. Isn't this a crutch?
I'm trying to implement an API that interacts with a NodeJS server for realtime messaging. Now when that NodeJS app is deployed to a scalable environment like Heroku, multiple instances of this app may be running.
Is it possible to design the node app so that all clients subscribed to a "message channel" will receive this message, although multiple node instances are running - and therefore multiple copies of this channel?
Check out zeromq, it should provide some simple, high performance IPC abstractions to do what you want. In particular, the pub/sub example will be useful.
The main challenge as I imagine it, without knowing anything about how Heroku spawns multiple server instances, will be the logic to determine who is the publisher (the rest of the instances will be subscribers). So let's say, for argument's sake, that your hosting provider gives you an environment variable called INSTANCE_NUM which is an integer in [0,1024] indicating the instance number of the process; so we'll say that instance zero is the message publisher.
var zmq = require('zeromq')
if (process.env['INSTANCE_NUM'] === '0') { // I'm the publisher.
var emitter = getEventEmitter(); // e.g. an HttpServer.
var pub = zmq.createSocket('pub');
pub.bindSync('tcp://*:5555');
emitter.on('someEvent', function(data) {
pub.send(data);
});
} else { // I'm a subscriber.
var sub = zmq.createSocket('sub');
sub.subscribe('');
sub.on('message', function(data) {
// Handle the event data...
});
sub.connect('tcp://localhost:5555');
}
Note that I'm new to zeromq and the above code is totally untested, just for demonstration.