One of the requirements of my app is that when one user makes any insert/update/delete, all users viewing a page with a list of that record type get pushed an update containing the change. The user should not be expected to repeat an API call to refresh the dozens of records that did not change, because the push should contain a short summary of the change that occurred.
I accomplished this in my small dev server using SocketIO. I can't scale this across more than one server. My target infrastructure is AWS, and I know AWS has a push notification service, but I believe it's mobile-only and not what I'm looking for. The huge number of data streams being subscribed to is the reason I haven't consider a server-less infrastructure.
I'm new to AWS and have never attempted horizontal scaling either, so please forgive me if my entire question is ignorant.
Have you taken a look at using AWS IoT MQTT messaging protocol? Each browser is a 'device' and you have javascript listening in the browser for messages published via a socket protocol. Each service pushes a message to MQTT when it has an update. There's some good POCs out there (i.e. medium.com/#jparreira/…)
Related
Background
I have a monolith Node.js + PostgreSQL app that, besides other things, needs to provide real-time in-app notifications to end users.
It is currently implemented in the following way:
there's a db table notifications which has state (pending/sent), userid (id of the notification receiver), isRead (did a user read the notification), type and body - notification data.
once specific resources get created or specific events occur, a various number of users should receive in-app notifications. When a notification is created, it gets persisted to the db and gets sent to the user using WebSockets. Notifications can also get created by a cron job.
when a user receives N number of notifications of the same type, they get collapsed into one single notification. This is done via db trigger by deleting repeated notifications and inserting a new one.
usually it works fine. But when the number of receivers exceeds several thousands, the app lags or other requests get blocked or not all notifications get sent via WebSockets.
Examples of notifications
Article published
A user is awarded with points
A user logged in multiple times but didn't perform some action
One user sends a friend request to another
One user sent a message to another
if a user receives 3+ Article published notifications, they get collapsed into the N articles published notification (N gets updated if new same notifications get received).
What I currently have doesn't seem to work very well. For example, for the Article created event the api endpoint that handles the creation, also handles notifications send-outs (which is maybe not a good approach - it creates ~5-6k notifications and sends them to users via websockets).
Question
How to correctly design such functionality?
Should I stay with a node.js + db approach or add a queuing service? Redis Pub/Sub? RabbitMQ?
We deploy to the k8s cluster, so adding another service is not a problem. More important question - is it really needed in my case?
I would love some general advice or resources to read on this topic.
I've read several articles on messaging/queuing/notifications system design but still don't quite get if this fits my case.
Should the queue store the notifications or should they be in the db? What's the correct way to notify thousands of users in real-time (websockets? SSE?)?
Also, the more I read about queues and message brokers, the more it feels like I'm overcomplicating things and getting more confused.
Consider using the Temporal open source project. It would allow modeling each user lifecycle as a separate program. The Temporal makes the code fully fault tolerant and preserves its full state (including local variables and blocking await calls) across process restarts.
I am using Angular CLI latest version and NodeJS, want to archive web push notification, we got few documentations but gives error, May I know any Documentation for web push notification which supports angular and nodejs.
The requirements are send push notification to all users who notification granted while browser closed as well.
Sample docs :
https://ciphertrick.com/2017/03/14/browser-push-notifications-in-angular-applications/ (getting error pushNotificationModule not exported)
https://www.npmjs.com/package/angular2-notifications (Not clear for CLI)
The Angular2+ directive angular2-notifications is doing only the last part showing the message in UI/UX. And that is the least of your challenges.
You say:
The requirements are send push notification to all users who
notification granted while browser closed as well
The part 'while browser closed as well' is the tricky part.
This means you need a Service Worker. A Service Worker is a script that your browser runs in the background, to which the message is being pushed when the browser is closed. For a nice introduction to Service Workers, read this. Angular has a Service Workers implemented in production version since 5.0.0. Klik here to read more about it.
I don't think setting up a WebSocket connection (like socket.io) from Node to a Service Worker is possible. It's complicated, some people say it's possible, others say no. See this for more info. I would at least say that it is not stable enough, so you need a 'real' push notification.
You can use a push notification provider to do the job. Click here for a list of them. You pay for these services.
If you want to do it yourself (free of costs) you can use Google's cross-platform messaging solution Firebase Cloud Messaging (FCM) (the successor of Google Cloud Messaging (GCM)).
To connect your NodeJS server with the FCM you can use different libraries e.g. node-pushserver and many others.
I am referring to the diagram
NodeJS is used as run time in this case and AWS Lambda is used as event notifier (updates comes from other lambda or DB).
My challenge is, the "user browser" can also be a mobile client. The "API" should acts as a service which allows client (mobile or web) to subscribe, unsubscribe, or publish data, nothing else.
Can lambda works as API that has capabilities of "pushing events notifications" to directly clients?
Is there any solution and also sample work/source code can be used as POC?
Next question is, how can I scale such architecture since it becomes stateful (requires memory to remember states of clients connections)?
Or else, how possible is it persist client connections on DB (using frameworks like websocket or socket.io)?
AWS has the SNS service to send notifications, which you can use from Lambda.
You can also directly use the relevant platform's notification system e.g for iOS, Node has an "apn" module that is used to communicate with Apple's APNS service - it's straightforward to use and can be implemented in a Lambda function.
In brief:
Your iOS app registers for APNS which responds with an APNS device token. Your app should then send this to your API / server for storage.
Your API can then send notifications to APNS, referencing any device tokens, along with the private key file you create from the Apple Developer page.
APNS will send the notifications to the registered devices.
Here is a good tutorial.
Your other queries should perhaps be separate questions.
Can lambda works as API that has capabilities of "pushing events notifications" to directly clients?
Yes! As #AndyOS mentioned, SNS is a great service that is quite literally intended to send notifications. I won't go into details here to avoid duplication of response.
Is there any solution and also sample work/source code can be used as POC?
Or else, how possible is it persist client connections on DB (using frameworks like websocket or socket.io)?
If you are looking to use websockets, I'd encourage you to take a look at IoT (https://aws.amazon.com/iot). IoT supports the MQTT protocol (http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html). This page also contains sample client-side code which might help you bootstrap your solution.
Next question is, how can I scale such architecture since it becomes stateful (requires memory to remember states of clients connections)?
You can view the service limits of IoT at http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot. You would need to decide if your app fits within these bounds, depending on the various metrics your app has (number of requests per second, number of concurrent connections, etc.).
I'm trying to get my head around MassTransit in combination with RabbitMQ.
The basic concepts are working in a test project, but what I need is the following:
My system will have one or more servers that react to real life events (telephony). These events wil, by means of MassTransit and RabbitMQ, translate into messages that will be picked up by one or more receivers via a separate server, set up as RabbitMQ host. So far so good.
However, I cannot assume that I always have a connection between the publisher and the host machines. Just assume that the publishing server will continue to consume the real life events, but now cannot publish it's messages.
So, the question is: Does MassTransit have some kind of mechanism to store messages locally some way until the connection is re-established?
Or should I install RabbitMQ on every publishing server as well, in order to create a local exchange? Then I have to make the exchanges synchronize themselves after a reconnect.
Probably you have to implement a store and forward policy. Instead of publishing directly your message through MassTransit and RabbitMQ, you can store the message in a persistence repository (a local database) and delegate to some other process the notification through Masstransit of the messages stored before. This approach is often referred as "Client High Availability". This does not substitute the standard HA (High Availability) on server like the one implemented by RabbitMQ. But it's a good approach to use in a distributed system (like the one you described) because it could help you a lot in scenarios of server failure (e.g. an issue on RabbitMQ server that causes some loss of messages that you still have inside the store of some client and therefore you can make it process again).
So i currently have a chat system running NodeJS that passes messages via rabbit and each connected user has their own unique queue that subscribed and only listening to messages (for only them). The backend can also use this chat pipeline to communicate other system messages like notifications/friend requests and other user event driven information.
Currently the backend would have to loop and publish each message 1 by 1 per user even if the payload of the message is the same for let's say 1000 users. I would like to get away from that and be able to send the same message to multiple different users but not EVERY user who's connected.
(example : notifying certain users their friend has come online).
I considered implementing a rabbit queue system where all messages are pooled into the same queue and instead of rabbit sending all user queues node takes these messages and emit's the message to the appropriate user via socket connections (to whoever is online).
Proposed - infrastructure
This way the backend does not need to loop for 100s and 1000s of users and can send a single payload containing all users this message should go to. I do plan to cluster the nodejs servers together.
I was also wondering since ive never done this in a production environment, will i need to track each socketID.
Potential pitfalls i've identified so far:
slower since 1000s of messages can pile up in a single queue.
manually storing socket IDs to manually trasmit to users.
offloading routing to NodeJS instead of RabbitMQ
Has anyone done anything like this before? If so, what are your recommendations. Is it better to scale with user unique queues, or pool all grouped messages for all users into smaller (but larger pools) of queues.
as a general rule, queue-per-user is an anti-pattern. there are some valid uses of this, but i've never seen it be a good idea for a chat app (in spite of all the demos that use this example)
RabbitMQ can be a great tool for facilitating the delivery of messages between systems, but it shouldn't be used to push messages to users.
I considered implementing a rabbit queue system where all messages are pooled into the same queue and instead of rabbit sending all user queues node takes these messages and emit's the message to the appropriate user via socket connections (to whoever is online).
this is heading down the right direction, but you have to remember that RabbitMQ is not a database (see previous link, again).
you can't randomly seek specific messages that are sitting in the queue and then leave them there. they are first in, first out.
in a chat app, i would have rabbitmq handling the message delivery between your systems, but not involved in delivery to the user.
your thoughts on using web sockets are going to be the direction you want to head for this. either that, or Server Sent Events.
if you need persistence of messages (history, search, last-viewed location, etc) then use a database for that. keep a timestamp or other marker of where the user left off, and push messages to them starting at that spot.
you're concerns about tracking sockets for the users are definitely something to think about.
if you have multiple instances of your node server running sockets with different users connected, you'll need a way to know which users are connected to which node server.
this may be a good use case for rabbitmq - but not in a queue-per-user manner. rather, in a binding-per-user. you could have each node server create a queue to receive messages from the exchange where messages are published. the node server would then create a binding between the exchange and queue based on the user id that is logged in to that particular node server
this could lead to an overwhelming number of bindings in rmq, though.
you may need a more intelligent method of tracking which server has which users connected, or just ignore that entirely and broadcast every message to every node server. in that case, each server would publish an event through the websocket based on the who the message should be delivered to.
if you're using a smart enough websocket library, it will only send the message to the people that need it. socket.io did this, i know, and i'm sure other websocket libraries are smart like this, as well.
...
I probably haven't given you a concrete answer to your situation, and I'm sure you have a lot more context to consider. hopefully this will get you down the right path, though.