A few questions about how to handle this dataflow. To notice I'm new to node and next.js but doing some starting basics and now trying to think ahead. Because I have read websockets and node being good for live-data handling.
We have an external API from a game. We have multiple moderators/clients that login to our system so they can use that API. Now they are requesting with multiple clients on that API multiple times per second which isn't a nice dataflow.
The main GET functions from that API are chat, so best would be live-data/push?? Else 1 second refresh for it. And the server info with the players. That could do with a 5 second refresh. Or is it just as easy to get that data with the 1 second/live-data from the chat ownGET flow I have in mind, as mentioned next? That's the first question sneaked in.
Then the ownGET flow.
I was thinking of making a own API that does the request to the game API. The game API doesn't support a websocket. So the idea is to let the own API do a request once a second to the game API. And then let the clients listen to the own API by websocket.
What I think websocket can do is, sending a push notification to the client when there is new data. On that push notification the client sends a get request to the own API. Instead of doing a 1 second get ping per client on the own API.
Just a quick check, is this even possible? And is this the right way of thinking or could it be optimized even better? Thanks in advance!
Related
I wonder how do I use socket.io properly with my express app.
I have a REST API written in express/node.js and I want to use socket.io to add real-time feature for my app. Consider that I want to do something I can do just by sending a request to my REST API. What should I do with socket.io? Should I send request to the REST API and send socket.io client the result of the process or handle the whole process within socket.io emitter and then send the result to socket.io client?
Thanks in advance.
Question is not that clear but from what I'm getting from it, is that you want to know what you would use it for that you cant already do with your current API?
The short answer is, well nothing really.. Websockets are just the natural progression of API's and the need for a more 'real-time' interface between systems.
Old methods (and still used and relevant for the right use case) is long polling where you keep checking back to the server for updated items and if so grab them.. This works but it can be expensive in terms of establishing a connection, performing a lookup, then closing a connection.
websockets keep that connection open, allowing both the client and server to communicate real time. So for example, lets say you make an update to your backend data and want users to get that update, using long polling you would rely on each client to ping back to the server, check if there is an update and if so grab it. This can cause lags between updates, some users have updated data while other do not etc.
Now, take the same scenario with websockets, you make an update to the backend data, hit submit, this then emits to your socket server. Socket server takes the call, performs the task ( grabs updated data ) and emits it to the users, each connected user instantly gets that update.
Socket servers are typically used for things like real time chats or polling where packets are smaller but they are also used for web games etc. Depending on the size of your payloads will determine how best to send data back and forth because the larger the payload the more resources / bandwidth it will take on the socket server so its something to consider.
I have an Express js server that supplies weather forecast information to its clients. Right now, it's basically an RPC API where the client (a python client in this case) requests different types of weather info from different endpoints. This is fine for right now, but I was thinking about implementing a subscription model. Rather than connecting to different endpoints for different information, the client could just subscribe for the info that it wants and the server would send it every so often (obviously configurable by the client).
Is there a way to do this with express? Would I have to set up a server on the client side as well to listen for the publish events?
A very good option is RabbitMQ, it's used for that right purpose.
I tried to find a good tutorial for you as a reference, however, I only find one in portuguese, but if you search around google you will find others in english.
But it's very simple, just like you want, you publish to the queue on some topic and have other clients subscribing that topic and processing the messages
Here is the link
https://medium.com/#programadriano/rabbitmq-publish-subscribe-com-node-js-9363848f58fe
I have created a bot for slack and deployed to Azure, I am making some API calls to another server from this BOT, for this bot I have a client requirement, My client wants to measure the time taken by request to reach to server and time taken by response to come back to Bot. (only time taken by request/response to reach to either side . I have been exploring Azure application insight from three days , but could not find any helpful service. I can not change my bot code , Is there any way in azure service by that I can monitor latency?
Here is simple diagram:-
Bot ----t1----> Server
<---t2-----
I don't want response time taken to process at Server side (No calculation time ) Just request/response travelling time.
Ganesh,
What you seem to be asking is how long it takes the API to process a request and return a response. Nothing to do with your bot.
My suggestion would be to create performance tests against the API directly using a tool such as Jmeter. This will give you average response times for say 10,000 requests and plot out on nice graphs etc.
If you need a help doing this, I could write it up for you in step by step instructions.
Let me know.
Thanks,
Tim
I'm working on a chat application and using socket.io / node for that. Basically I came up with the following strategies:
Send message from the client which is received by the socket server which then sends it to the receiving client. On the background I store that to the message on the DB to be retrieved later if the user wishes to seee his old conversations.
The pros of this approach is that the user gets the message almost instantly since we don't wait for the DB operation to complete, but the con is that if the DB operation failed and exactly that time the client refreshed its page to fetch the message, it won't get that.
Send message form the client to the server, the server then stores it on the DB first and then only sends it to the receiving client.
The pros is that we make sure that the message will be received to the client only if its stored in the DB. The con is that it will be no way close to real time since we'll be doing a DB operation in between slowing down the message passing.
Send message to the client which then is stored on a cache layer(redis for example) and then instantly broadcast it to the receiving client. On background keep fetching records from redis and updating DB. If the client refreshes the page, we first look into the DB and then the redis layer.
The pros is that we make the communication faster and also make sure messages are presented correctly on demand. The con is that this is quite complex as compared to above implementations, and I'm wondering if there's any easier way to achieve this?
My question is whats the way to go if you're building a serious chat application that ensures both - faster communication and data persistence. What are some strategies that app like facebook, whatsapp etc. use for the same? I'm not looking for exact example, but a few pointers will help.
Thanks.
I would go for the option number 2. I've been doing myself Chat apps in node and I found out that this is the best option. Saving in a database takes few milliseconds, which includes the 0.x milliseconds to write in the databse and the few milliseconds of latency in communication ( https://blog.serverdensity.com/mongodb-benchmarks/ ).
SO I would consider this approach realtime. The good thing with this is that if it fails, you can display a message to the sender that it failed, for whatever reason.
Facebook, whatsapp and many other big messaging apps are based on XMPP (jabber) which is a very, very big protocol for instant messaging and everything is very well documented on how to do things but it is based in XML, so you still have to parse everything etc but luckily there are very good libraries to handle with xmpp. So if you want to go the common way, using XMPP you can, but most of the big players in this area are not following anymore all the standards, since does not have all the features we are used to use today.
I would go with doing my own version, actually, I already something made (similar to slack), if you want I could give you access to it in private.
So to end this, number 2 is the way to go (for me). XMPP is cool but brings also a lot of complexity.
Today, i had the idea of the following setup. Create a nodejs server along with express and socket.io. With express, i would create a RESTful API, which is connected to a mongo. BackboneJS or similar would connect the client to that REST API.
Now every time the mongodb(ie the data in it iam interested in) changes, socket.io would fire an event to the client, which would carry a courser to the data which has changed. The client then would trigger the appropriate AJAX requests to the REST to get the new data, where it needs it.
So, the socket.io connection would behave like a synchronize trigger. It would be there for the entire visit and could also manage sessions that way. All the payload would be send over http.
Pros:
REST API for use with other clients than web
Auth could be done entirely over socket.io. Only sending token along with REST requests.
Use the benefits of REST.
Would also play nicely with pub/sub service like Redis'
Cons:
Greater overhead, than using pure socket.io.
What do you think, are there any great disadvantages i did not think of?
I agree with #CharlieKey, you should send the updated data rather than re-requesting.
This is exactly what Tower is doing:
save some data: https://github.com/viatropos/tower/blob/development/src/tower/model/persistence.coffee#L77
insert into mongodb (cursor is a query/persistence abstraction): https://github.com/viatropos/tower/blob/development/src/tower/model/cursor/persistence.coffee#L29
notify sockets: https://github.com/viatropos/tower/blob/development/src/tower/model/cursor/persistence.coffee#L68
emit updated records to client: https://github.com/viatropos/tower/blob/development/src/tower/server/net/connection.coffee#L62
The disadvantage of using sockets as a trigger to re-request with Ajax is that every connected client will have to fetch the data, so if 100 people are on your site there's going to be 100 HTTP requests every time data changes - where you could just reuse the socket connections.
I think that pushing the updated data with the socket.io event would be better than re-requesting the lastest. Even better you could only push the modified pieces of data decreasing the amount of data sent over the line. Overall though a interesting idea.
I'd look into Now.js since it does pretty much exactly what you need.
It creates a namespace which is shared among the client and server. The server can call functions on the client directly and vice versa.
That is if you insist on your current infrastructure decision to use MongoDB and Node.js, otherwise there would be CouchDB which is a full web server and document database with sophisticated replication mechanisms built-in.