I'm writing a nodejs daemon for connecting in the background to Twitter, Facebook and other social networks. Every user can have multiple social networks connected to it's account. And say, when every user has somewhere around 5 networks connected and there are 1000 users. That would be 5000 connections. The connections have to be near realtime, like somewhere around 1 minute or something.
My idea was to start a new nodejs process for every new user that pulls in the content from those social networks instead of 1 process that pulls in every data from every user.
But my question is how to start. I allready wrote some of the script to do this, but since my knowledge of nodejs is nowhere near professional I don't know how this works.
Is there a way to automatically start a new nodejs process when a new user registers? Or is there a better way to do this?
Thanks in advance.
You want 1k processes? That's already up to 1gig overhead and a ton of work the OS.
Why not have all inside one Node.js process? Have you actually tested it? Are you sure you need multiple processes? If you want you can use the child_process module to start new child processes from within Node.js.
For keeping the main process running you should take a look at forever.
Still I would say test it with one process first, and then if you see that it doesn't perform/work as you expected it to, it should be trivial to put the stuff into multiple processes.
Related
I can't seem to get this concept right in my head. If I have a website that gets 1 million concurrent users, without any databases at all, will I need to scale? I'm Using Node.js and Socket.IO. Also is there a way I could simulate something like this on my localhost?
Having one million user, or connections, on Socke.io, doesn't mean you have to scale, but depending on what they are doing, you would probably do. Having a data base adds storage but has nothing more to do with the need for scaling the Node.JS server.
You can create a test to try to insert as much as you want using a loop to connect and then try to emit an event for each of then.
For scaling node you can use a cluster. A single instance of Node.js runs in a single thread. To take advantage of multi-core systems, the user will sometimes want to launch a cluster of Node.js processes to handle the load. https://nodejs.org/api/cluster.html#cluster_cluster
To simulate high load, there are open source tools you can use for free: http://www.opensourcetesting.org/category/performance/
So Im learning node.js and making a random player vs player cardgame. I will be using sockets and sessions.
Theoreticall ill have lots of games running at the same time.
Whats the best practice here, do I spawn a child process of my node server for every game, or simply just keep data of ongoing games and 'boards' in the main server and just grab a given game every time there is an action?
Presuming there's no intensive CPU processing going on, you should do it all in a single process. Node is very good at doing a high volume of very small operations very quickly, all on a single thread, and it sounds like that fits your application.
If you were going to be doing CPU-intensive calculations you might want to hand off to worker processes, but you still wouldn't want a separate process per game.
More than likely, you'll want to load balance however many servers you need. If you're using socket.io, then you should look into socket.io-redis (briefly covered here) to see how you can have connections running across multiple servers still talk to each other. With an approach like this, you'll need to consider the fact that you can't store some info in memory, and instead will need to persist to redis or a similarly accessible medium for all servers to access.
I am using SailsJS for a web application which basically lets the user download and process any video from the internet(say youtube). The user enters a link to the video and my sails app downloads the video if available and then starts processing the downloaded video using a shell script(Come OpenCV processing to find different frames).
This process takes a very long time to complete, and the user can navigate away from the page and do whatever he wants. Now, to check on the progress by visiting this page later I need to be able to connect with the child process that was created earlier for this video file.
I have come up with two possible solutions:
1) Using gearman to implement a job server and connect to it every time the user navigates to the page and start getting the callback events and show the progress based on them. This is the first time I'll be using gearman.
2) Somehow storing the processID of the child process in the session/db and then using it to find the process using ps-node.
Which of these is the better approach(if you think they'll work fine)? Or is there any other solution I don't know about? Any pointers in the right direction will be appreciated.
Let's start with the second option. Don't use it. Simply because this way your site users will have sort of more control over the number of processes running on your server then you will.
Number one is way better, but using a separate job server seems like a bit of overkill to me (I have to admit though that I'm not fully informed the scale of your plans).
Bottom line, I would use a message/job queue (kue seems like a perfect fit to me) and store the progress in DB or (preferably) Redis (or whatever cache you are using).
I am using Webrtc, nodejs, Expressjs as the framework to create a audio, video and chat application. I have used Forever so that the script runs continuously.
As my application deals with audio, video and chat. User presence plays an important role. We need to have the system up and running always and avoid system crashes and restart. If it happens we are going to loose all information regarding the users who were online.
Need Suggestions what are the best approaches to avoid such situations.
Also, while moving new features to the production server, what steps should we take into consideration so that the application doesn't stop and henceforth we don't loose user information.
What if the server went down or we had to make it down. What are the different techniques that can be used so that we don't loose the presence information of the online users in the system and restore them back(if necessary).
1) use node cluster to fork multiple process per core. So if one process died, another process will be auto boot up. Check out: http://nodejs.org/api/cluster.html
2) use domain to catch asyn operation instead of using try-catch or uncaught http://nodejs.org/api/domain.html. I'm not saying that try-catch or uncaught is bad thought!
3) use forever/supervisor to monitor your services
4) add daemon to run your node app: http://upstart.ubuntu.com
hope this helps!
I also answered at this post: How do I prevent node.js from crashing? try-catch doesn't work
We are having a web application build using asp.net 3.5 & SQL server as database which is quite big and used by around 300 super users for managing around 5000 staffs.
Now we are implementing SMS functionality into the application which means the users will be able to send and receive SMS. Every two minute the SMS server of the third party is pinged to check whether there are any new messages. Also SMS are hold in queue and send every time interval of 15 to 30 minutes.
I want this checking and sending process to run in the background of the application all the time, even if the user closes the browser window.
I need some advice on how do I do this?
Will using thread will achieve this or do I need to create a windows service for it or are there any other options?
More information:
I want to execute a task in a timer, what will happen if I close the browser window, the task wont be completed isn't it so.
For example I am saving 10 records to the database in a time interval of 5 minutes, which means every 5 minutes when the timer tick event fires, a record is inserted into the database.
How do I run this task if I close the browser window?
I tried looking at windows service but how do I pass a generic collection of data to it for processing.
There really is no thread or service choice, a service can (and usually is!) multi threaded, a thread can start a service.
There are three basic choices you can:-
Somehow start another thread running when a user logs in -- this is probably a very poor choice for what you want, as you cannot really keep it running once the user session is lost.
Write a fully fledged windows service which is starts on OS startup and continues running unitl the server is shutdown. You can make this dependant on the SQLserver service, so it starts after the DB is available. This is the "best" solution but may be overkill for your purposes. Aslo you need to know the services API to write it properly as you need to respond correctly to shutdown and status requests.
You can schedule your task periodically using either the Windows schedular, or, preferably the schedular which is built in to SQLServer, I think this would be the most suitable option for your needs.
Distinguish between what the browser is doing and what's happening server-side.
Your Web App is sitting server-side waiting for requests from whatever browsers may be running, and servicing those requests, in servicing those requests I guess it may well put messages on a queue and have a look in a database for any new messages.
You want the daemon processor, which talks to the third-party SMS, to be triggered by time rather than by browser function. Either of your suggestions would work:
A competely independent service could run and work against the queues and database.
Your web app, which I assume is already a service, could spawn a thread
In either case we have a few technical questions of avoiding any race conditions between the browser-request processing and the daemon - but databases and queueing systems can deal with that.
So I would decide between stand-alone daemon and background thread like this:
Which is easier to implement? I'm a Java EE developer, I know in my app server I have an API for specifying code to be run according to a timer, the API deals with the threading issues. So for me that's very easy. I don't know what you have available. Timers are not quite as trivial as they may appear - so having a reliable API is beneficial. If this was a more complex requirement, where the daemon code were gnarly and might possibly interfere with the WebApp code then I might prefer to keep it conspicuously separate.
Which is easier to deploy and administer? Deploy separate Web App and daemon, or deploy one thing. In the Java EE world we could have a single Enterprise Application with all the code, so that's a single thing to deploy, start and control.
One other thing to consider: Scaling and Resilience. You might choose to have more than one copy of your web app running, either to provide fail-over capabilities or just because you need the extra power. In which case how many daemons would you have? Would it be a problem to have two daemons running? You might need some extra code to mediate between two daemons, for example log in the database the time of last work, each daemon can say "Oh, my buddy balready did the 10:30 job, I'll go back to sleep"