I have searched for packages for Node like this: https://www.npmjs.com/package/kafka and https://www.npmjs.com/package/no-kafka
My question is: Do these packages makes the node.js subscribe to kafka all the time? or Do I need some packages like forever or pm2 to achieve that?
The purpose of something like forever is to keep your node app running all the time (if it should crash).
This is pretty much separate from what those two packages do or don't do. They run inside your node app. If you want them to run all the time, then you need them to be used in a node app that is always running.
You can either write a rock solid node app that doesn't ever crash so it runs continually or you can attempt to do that and also run something like forever so that if your app dies, forever will automatically restart it.
Do these packages makes the node.js subscribe to kafka all the time?
or Do I need some packages like forever or pm2 to achieve that?
No. forever and pm2 have no influence on what kafka does or doesn't do. They just make sure your app is restarted if it crashes or exits for some reason.
If you are using the consumer side of the kafka API, then you will have to do some research and testing to see how good the library is at keeping you connected all the time, even when the server you are connecting to temporarily restarts or there is a temporary internet glitch.
From what I can tell looking at the code for this implementation, if there's an error on an open connection, the connection is just closed and there is no reconnect logic so you would probably have to write reconnect logic by subscribing to either the error or close events and then attempting to reconnect when the connection is lost.
Related
I have a Parse Server which is a Node.js + express wrapper for a mobile app (about 100 simultaneous users every day), hosted on DigitalOcean. The app server communicates with MongoDB, which is hosted on another droplet of DigitalOcean. I'm using pm2 as a process manager and its monitoring tool, which is web-based. On the same process, we operate LiveQuery, a WebSocket server made by the Parse community as well.
The thing is, I've been having some performance issues with the server. Everything works smoothly, until the Active handles rise up uncontrollably! (see the image below) It's like after one point the server says "I'm done! Now I rest!"
Usually the active handles stay between 30 to 70. The moment I restart the process with pm2 restart everything goes back to normal!
I've been having this issue for quite some time now and I haven’t been able to figure out what’s causing it! Any help will be greatly appreciated!
EDIT: I did a stress test where I created 200 LiveQuery sockets for 1 user, instead of 2 that a user normally has and there was a spike of 300 active handles, for like 5 seconds! The moment the sockets were all created, everything went back to normal!
I usually use restart based on memory usage
pm2 start filename.js --max-memory-restart 160 --exp-backoff-restart-delay=100
pm2 has also built-in cron job or autostart script setup in case the server ever restarts, see https://pm2.keymetrics.io/docs/usage/restart-strategies/
it would be could if pm2 would provide restart options based on active connections or heap memory
I am running a nodejs server on heroku, using the throng package to spin up workers to process api requests. I am also using more than on dyno on heroku. For cron and job processing, I use bull queue to distribute the load across servers, but I came across something I am not sure how to do in this distributed environment.
I want to have only one server execute code immediately on startup. In this case, I want to open up a change stream listener for mongodb. But I only want to do this on one worker on one server, not every server.
I am not sure how to do this running in heroku, any suggestions?
Context
I've added configuration validation to some of the modules that compose my Node.js application. When they are starting, each one checks if it is properly configured and have access to the resources it needs (e.g. can write to a directory). If it detects that something is wrong it sends a SIGINT to itself (process.pid) so the application is gracefully shutdown (I close the http server, close possible connections to Redis and so on). I want the operator to realize there is a configuration and/or environment problem and fix it before starting the application.
I use pm2 to start/stop/reload the application and I like the fact pm2 will automatically restart it in case it crashes later on, but I don't want it to restart my application in the above scenario because the root cause won't be eliminated by simply restarting the app, so pm2 will keep restarting it up to max_restarts (defaults to 10 in pm2).
Question
How can I prevent pm2 from keeping restarting my application when it is aborted during startup?
I know pm2 has the --wait-ready option, but given we are talking about multiple modules with asynchronous startup logic, I find very hard to determine where/when to process.send('ready').
Possible solution
I'm considering making all my modules to emit an internal "ready" event and wire the whole thing chaining the "ready" events to finally be able to send the "ready" to pm2, but I would like to ask first if that would be a little bit of over engineering.
Thanks,
Roger
I'm trying to figure out why my nodejs app becomes unresponsive after 11h 20min. It happens every time, no matter if I run it on amazon-linux or Red Hat.
My Stack:
nodejs (v. 6.9.4)
mongodb (3.2)
pm2 process manager
AWS EC2 instance T2 medium
Every time I'm running the app it becomes unresponsive with an error returned to the browser:
net::ERR_CONNECTION_RESET
Pm2 doesn't restart the app, so I suspect it has nothing to do with nodejs, I also analysed the app and it doesn't have memory leaks. Db logs also look alright.
The only constant factor is the fact that the app crashes after it runs for 11h 20min.
I'm handling all possible errors from the nodejs app, but no errors in the log files occur so I suspect it has to be something else.
I also checked var/log/messages and /home/centos/messages but nothing related to the crash of the app there either.
/var/log/mongodb/mongo.log doesn't show anything specific either.
What would be the best way to approach the problem ?
Any clues how can I debug it or what could be the reason ?
Thanks
Copied from the comment since it apparently led to the solution:
You're leaking something other than memory is my guess, maybe file descriptors. Try using netstat or lsof to see if there are a lot more open connections or files than you expect.
I am using Webrtc, nodejs, Expressjs as the framework to create a audio, video and chat application. I have used Forever so that the script runs continuously.
As my application deals with audio, video and chat. User presence plays an important role. We need to have the system up and running always and avoid system crashes and restart. If it happens we are going to loose all information regarding the users who were online.
Need Suggestions what are the best approaches to avoid such situations.
Also, while moving new features to the production server, what steps should we take into consideration so that the application doesn't stop and henceforth we don't loose user information.
What if the server went down or we had to make it down. What are the different techniques that can be used so that we don't loose the presence information of the online users in the system and restore them back(if necessary).
1) use node cluster to fork multiple process per core. So if one process died, another process will be auto boot up. Check out: http://nodejs.org/api/cluster.html
2) use domain to catch asyn operation instead of using try-catch or uncaught http://nodejs.org/api/domain.html. I'm not saying that try-catch or uncaught is bad thought!
3) use forever/supervisor to monitor your services
4) add daemon to run your node app: http://upstart.ubuntu.com
hope this helps!
I also answered at this post: How do I prevent node.js from crashing? try-catch doesn't work