I'm building an application in which we have worked on Payment gateway named flutterwave.
And now the scenario is on every success or failure of a payment, I receive a webhook and then we take further actions such as sending emails, SMS and updating the statuses of the payment in the DB.
For now, we have implemented polling in the client side and for a particular time span if the client receives a status (success or fail) we show it otherwise they can check later it in the payment history page.
Now we want to remove this polling and update users in real time about the success or failure of a payment.
What are the ways by which we can achieve this?
The questions are how we will notify a specific user about the same as we have a multiplatform app and the same user can be logged in different platforms.
What you are looking for is a real-time communication pattern with WebSockets a layer 7 protocol in the OSI model which offers bi-directional communication.
This means that you can establish communication between your servers and your user's browser (client). As a result, you can send notification data to the client and consume and react to the notification, by showing visual cues in your UI for the user to see.
Some examples of implementing WebSockets with Socket.io and Nodejs: https://dev.to/novu/sending-real-time-notifications-with-socketio-in-nodejs-1l5j
There are also paid services that can offer this functionality like Pusher, and I would actually recommend that route at the beginning so you can avoid spending too much time implementing this and focus more on the stuff that matters and is part of your roadmap.
Additionally, you can use Push Notifications as another way to notify your users even when they are not using the app.
Related
Background
I have a monolith Node.js + PostgreSQL app that, besides other things, needs to provide real-time in-app notifications to end users.
It is currently implemented in the following way:
there's a db table notifications which has state (pending/sent), userid (id of the notification receiver), isRead (did a user read the notification), type and body - notification data.
once specific resources get created or specific events occur, a various number of users should receive in-app notifications. When a notification is created, it gets persisted to the db and gets sent to the user using WebSockets. Notifications can also get created by a cron job.
when a user receives N number of notifications of the same type, they get collapsed into one single notification. This is done via db trigger by deleting repeated notifications and inserting a new one.
usually it works fine. But when the number of receivers exceeds several thousands, the app lags or other requests get blocked or not all notifications get sent via WebSockets.
Examples of notifications
Article published
A user is awarded with points
A user logged in multiple times but didn't perform some action
One user sends a friend request to another
One user sent a message to another
if a user receives 3+ Article published notifications, they get collapsed into the N articles published notification (N gets updated if new same notifications get received).
What I currently have doesn't seem to work very well. For example, for the Article created event the api endpoint that handles the creation, also handles notifications send-outs (which is maybe not a good approach - it creates ~5-6k notifications and sends them to users via websockets).
Question
How to correctly design such functionality?
Should I stay with a node.js + db approach or add a queuing service? Redis Pub/Sub? RabbitMQ?
We deploy to the k8s cluster, so adding another service is not a problem. More important question - is it really needed in my case?
I would love some general advice or resources to read on this topic.
I've read several articles on messaging/queuing/notifications system design but still don't quite get if this fits my case.
Should the queue store the notifications or should they be in the db? What's the correct way to notify thousands of users in real-time (websockets? SSE?)?
Also, the more I read about queues and message brokers, the more it feels like I'm overcomplicating things and getting more confused.
Consider using the Temporal open source project. It would allow modeling each user lifecycle as a separate program. The Temporal makes the code fully fault tolerant and preserves its full state (including local variables and blocking await calls) across process restarts.
My backend server connects to a third party API service. This third party API service sends events via a websocket to my backend, which then relays back to my frontend. Every message sent from the websocket, I pay for.
In addition, the websocket provides different data depending on which value is requested. An example: Imagine an API service that provides real-time payments for different products. If I want to receive real-time payments for books, I'd use the value "books". And if I'd want to receive real-time payments for iPhones, I'd use the value "iPhones", etc.
To minimise cost spend, I'd preferably want users connecting through my backend to share the stream of information, rather than creating a new websocket connection every time. And for connections that are not in use, to be disconnected from the third party websocket. I.e. If a user were to request "book"s through my backend, a connection would be opened to the third party API & then subsequently streamed back. If another user were to also request "book"s they'd just hop onto the existing opened connection.
Is this a possibility with GraphQL & how'd would you imagine this being implemented? Just off the top of my head, I'd imagine some sort of tracking on the backend side which checks what connections are opened & whether they're being used. Otherwise, is it possible in REST?
You could do something very similar with Apollo's polling features with a little custom logic thrown in.
Basically, you could open a connection to the third-party using polling as a socket to update the data layer. Then, when another user queries that data layer, it should already exist (or you can add a quick check on top of that).
I am looking for a solution to my problem. I have Node.js server serving my web application where user can log in. I want to handle a situation where one user A performs specific action and user B associated with this action gets real life notification. Is there a module that would help me or there is some other solution?
What you are describing is "server push" where the server proactively notifies a user on their site of some activity or event. In the web browser world these days, there are basically two underlying technology options:
webSocket (or some use socket.io, a more feature rich library built on top of webSocket)
server sent events (SSE).
For webSocket or socket.io, the basic idea is that the web page connects back to the server with a webSocket or socket.io connection. That connection stays live (unlike a typical http connection that would connect, send a request, receive a response, then close the connection). So, with that live connection, the server is free to send the client (which is the web page in a user's browser), notifications at any time. The Javascript in the web page then listens for incoming data on the connection and, based on what data it receives, then uses Javascript to update the currently displayed web page to show something to the user.
For server sent events, you open an event source on the client-side and that also creates a lasting connection to the server, but this connection is one-way only (the server can send events to the client) and it's completely built on HTTP. This is a newer technology than webSocket, but is more limited in purpose.
In both of these cases, the server has to keep track of which connection belongs to which user so when something interesting happens on the server, it can know which connection to notify of the event.
Another solution occasionally used is client-side polling. In this case, the web page just regularly sends an ajax call to the server asking if there are any new events. Anything new yet? Anything new yet? Anything new yet? While this is conceptually a bit simpler, it's typically far less efficient unless the polling intervals are spaced far apart, say 10 or 15 minutes which limits the timeliness of any notifications. This is because most polling requests (particularly when done rapidly) return no data and are just wasted cycles on your server.
If you want to notify userB, when both of you are simultaneously online during the action, then use websockets to pass message to a two-way channel to notify userB.
If you want to notify them whenever, regardless of online status, use a message queue.
I want to implement a notification system like facebook does for an Angular 7 app. I am currently using nodejs ad my backend and socket io for the socket connection.
I understand the principle of sockets, events, observables... however, I can't wrap my head around how should I create a socket for each user to receive notifications.
I thought about creating a channel called 'notifications' and then subscribe all logged in users to that channel. When a user does anything that triggers an event that requires the app to create a notification I will broadcast the notification to all users and only display it if logged in user is inside the receivers array of the new notification object.
HOWEVER I think this is a very bad approach as all users will be listening to this event and only a very small amount of them will display it. It seems like I am wasting a lot of resources with this approach. Is there a proper way to handle this or is my approach the one used by companies like Facebook and Instagram
I am working on a Chatting application (needs to connect to a server) on iPhone. The sending packet from iPhone shouldn't be a problem.
But I would like to know whether it is possible for iPhone to establish a incoming socket connection to server continuously or forever under mobile environment.
OR What do I need to do to give the connection alive ? Need to send something over it to keep it alive ?
Thanks.
Not sure why you want to have chatting app to have persisted connection... I'd better use SMS like model. Anyways, Cocoa NSStream is based on NSSocket and allows a lot of functionality. Take a look at it.
Response to the question. Here is in a nutshell, what I would do:
Get an authentication token from the server.
this will also take care of user presence if necessary but now we are talking about the state; once presence is known, the server may send out notifications to clients that are active and have a user on their contact list.
Get user's contact list and contact presence state.
When a message send, handle it according to addressee state, i.e. if online, communicate back to the other user, if offline, queue for later delivery or reject.
Once token expires, reject communication with appropriate error and make the client to request a new token.
Communication from server to client, can be based on pull or push model. In first case, client periodically makes a request and fetches all messages. This may sound not good but in reality, how often users compose and send messages? Several times a minute? That's not too much. So fetching may happen every 5-10 seconds.
For push model, client must be able to listen and accept connections.
Finally, check out SIP, session initiation protocol. No need to use full version of it though. Just basic stuff.
This is very rough and perhaps simplified. I don't know the target complexity of your chatting system. For example, the simplest thing can also be that server just enables client to client communication by distributing their end points and clients take care of everything themselves.
Good luck!
Super out of date response, but maybe it will help the next person.
I would use xmppframework and a jabber server.