I am developing a chat application using the nodejs/socket.io on the server side
Now, it is time to test how scalable it is
so, i think i can simulate a large number of soket.io clients effectively using nodejs also , but running the client code this time
the question is, How can i run the socket.io client library on nodejs? is this possible?
if so, can anyone please provide a simple example
my code is running fine on the browser, with the usual development load, the issue is not about the code is running or not, actually i am not planning to run the same client code, just openning a large number of connections , and sending thousands of messages to have a preliminary figure about scalability and resource consumption
also, any suggestion on testing socket.io server scalability will be appreciated
thanks a lot
What you're looking for isn't going to really be helpful. Even if you could simulate client-side socket.io in a node process, it wouldn't have the same dynamic properties as actual access from browsers. You'd be able to determine things like the maximum number of connections you could handle without running out of resources, but your general performance metrics would be pretty artificial and not generalizable.
Related
My client has 1000 taxi, he want to track every taxi location and see his display. My question is how to track all taxi information using driver mobile device. I am using mongoDb for database.
I plan to solve this problem using develop a api and mobile device send their location after 10 seconds. but the problem is server is very busy that time and can not working properly using api.
I saw firebase store client information realtime . I need to know its possible for me to work as like as firebase database using mongodb.
I am using nodejs for backend development . If anyone know any way how to store real time data please help
You cannot (generally speaking) track taxi in real time, the problem is your Internet connection may be poor due to low GPS signal, have really low latency sometimes, or even be down. Instead design two independent applications:
One, which will store current GPS location inside a FIFO queue locally
Second, which will flush the queue to a remote sever
This approach will ensure you will, eventually, receive all the positions without having to worry about dropped packets and other issues that may and will occur.
Instead of the TCP connection you can consider using UDP (or better DTLS) instead which is faster, but less reliable. If reliability is a must (doubt it if it is just a taxi), then go for TCP (or better TLS). How will you send or receive the data is just a detail.
Also make sure you authenticate the device before you store any data, especially if the connection between devices is not secure.
You can use Firebase as a real time database. Have a look at this link: https://github.com/firebase/geofire
But in case you want to go with MongoDB you can use MongoDB Geospatial Queries and use socket.io to enable real time behavior.
For more details: https://docs.mongodb.com/manual/geospatial-queries/
You can use https://socket.io/ for the real-time tracking system.
It is a JavaScript library for real-time web applications.
You just need to configure MongoDB.
There are many blogs which can explain to you how to set up socket.io with MongoDB.
Some of them are...
http://blog.slatepeak.com/creating-a-real-time-chat-api-with-node-express-socket-io-and-mongodb/
https://blog.feathersjs.com/building-a-rest-and-real-time-api-with-express-feathers-and-mongodb-12071e5417e1
I think you are going to implement the tracking in frontend
But that is not good way and not secure, because drivers can send the fake request real time
You can use websocket method to implement the checking real time of taxies
Please check this link
I guess this way is similar with your idea
I hope this way is good for you
Thanks
I want to develop a real time chat application and also want to show real time graph/chart.I am confused to choose technology.I heard a lot of good things about nodejs socket.io and also heard about django channel.Can you suggest me to choose from one of them for my project?
There need a special feature though.I need to run a cron job to get data from external web service and broadcast it to all clients.
Thanks in advance
Django channels is fantastic at broadcasting and creating a chat application. I have successfully used Channels with Websocket and Redis to create a chat application. I am not sure about scalability, but since you're here with this question there may be a chance that your app may not reach a large enough usage frequency to make that an issue.
I have been working on a Web App for visualizing live data. It is crucial that this data is kept up to date on the client side without such updates being invoked directly by the client (e.g. no button presses or refreshing the page). Currently, on page load, I grab the current data set from a database (DynamoDB) via Ajax, and subsequent updates are pushed to any listening clients every 5 minutes via a Websockets connection (using Socket.io).
I have overlooked the computational load of this update job. It has to mine some data, process it, update the database, and send the update out to all clients. As a result, the web server is left unresponsive for about 30 seconds with each update. Furthermore, my current architecture limits me from putting my server behind a load balancer, which is something I anticipate coming up in the future. For both these reasons, I really need to get this update job off my web server.
I am relatively inexperienced in web development, and I don't feel I am knowledgeable enough about these technologies to know the drawbacks of the solutions I have come up with. Currently, I am considering:
Break the update off into a separate process so it does not block the Node event loop. This would solve my issue in the short term, but if I ever want to load balance my application, I can't have the update running on multiple machines.
Drop Websockets entirely and just have the client query the database every 5 minutes, while a separate process (or separate server if I want load balancing) keeps the database up to date without interacting directly with the client. Will this kind of access pattern put too much load on my db?
Have a separate server run the update, and send the result via Websockets (or maybe some other protocol) to my load balanced application servers, which then push that update to all listening clients as usual. Is this even possible?
Perhaps there are other solutions. It seems like this would be a relatively common problem, so I was hoping I could find some guidance here. What are the potential issues with the solutions I have proposed, and are there other possible solutions that my suit my use case better?
It sounds like you want one process sitting somewhere which crunches the data and publishes it to a stream. Clients can then subscribe to the stream as and when they like. Redis handles streams nicely, you could process your data and push it into a redis stream. You could then create a small node service which subscribes to the redis stream and pushes the formatted data out over a websocket or via polling.
In this scenario you can then scale up either the publishing process (the one crunching the numbers) if your data load goes up, or scale up your subscribed process (which serves the data over a websocket to browsers) if you get an influx of clients watching the data.
You can also easily distribute the hosting of these services across other machines, and even write them in different languages if you decide the number crunching needs something like threading.
You're then left with the issue of clients (web browsers) consuming this data with a load balance in-between. This can be a hard problem if you use websockets and is bundled with pros and cons. But importantly you'll have separated your data crunching from your result publishing and that'll isolate out your issue to only the load balancing.
I have done pretty much the same to check ressources on some of our servers.
I have a C# service getting the information on each server that we manage, sending them to a queue (Amq).
From there, I have a stomp client fetching data from amq and emiting them to a websocket.
My main micro service is fetching the data to save them into a db.
My visualisation webapp is connected to the same ws and is fetching the data as they are sent to display them.
The Amq step isn't mandatory at all, it's just something I had to work with (historical).
I don't know what type of data your are working with, so I don't know if my solution can apply to you.
Don't hesitate if I'm not clear or you have any question.
This is a big question and I'm not going to try and give you a definitive answer.
For option 2
It really depends on how expensive your queries are. You can make DynamoDB fast if you pay for enough throughput. That said, on the face it, re-loading your whole dataset, when that sounds like its probably large, probably isn't good engineering.
For option 3
This option seems best to me if its achievable, although admittedly its hard to say with such a complex system - obviously you can't share your whole project.
Given your are already using AWS you might want to look into AWS Lambda. If you can move the update process into a stand alone job, you can host it on lambda and move the load off the web server. Lambda is essentially infinitely scalable and you only pay for the compute you use.
This really depends on you being able to split the update task off into a separate service. Its likely you would need a fair bit of refactoring to isolate it as a service. If you can break little bits off at a time, and make the move gradually, even better.
If you consider trying this, and you've not used Lambda before, I would definitely start small with some hello world examples. Then try a very simple service in your application, and build up to taking on the update service.
You might also consider looking in AWS Simple Message Queue Service to handle the comms between clients and server.
Database tuning
If a lot of your update time is spent waiting for database actions to complete, rather than server processing, you can consider tuning that side of things up. Things to consider are:
Buying more throughput
Using batch operations (as these move load to DynamoDB from your server)
Tuning keys, indexes and database access
I just started with Meteor app development and have a use case which I am not sure is good for meteor.
We have a java application that pushes data to redis at a very fast rate (data updates in less than 50 milliseconds) and we are building a web application (on NodeJS) which connects to this redis instance and sends the data to the client. For now (with native NodeJS app), we are sending data only twice a second (as we do not require such fast updates).
My question is, how can I achieve the same with Meteor? As we know Meteor has live-query which will tend to send data as soon as it changes, but this is not optimum for us. Is there a way to tune live-query to send data say only after a certain time?
Thanks
I think you are looking for ways to throttle meteors calls. This could be done with this library.
This issue has been also discussed here. Reading up on it I think they still haven't implemented it in core. This would make sense since there are no out-of-the-box throttling mechanisms in node or iojs.
Hope this was helpful.
So I have a Meteor application that is working fine on my local machine, but when I deploy to production is has terrible delays resulting in terrible performance.
Currently it talks to two services - a data processing service and mongodb. As well as the client of course.
I'm using Meteor 7.1.2, so I know it's using capped collections and op-log tailing.
I've optimized my code in every way possible but its still slow in the following ways:
Slow to load
Intermittent (bad) delays when sending requests to the data processing server.
Any suggestions would be VERY appreciated , as i've just about had it with Meteor and thinking about switching to bare-bones express and building on that.
Thanks!
Can you please try to use Kadira?
It will show you what's really happening.
Disclaimer: I maintain Kadira.