I have a nodejs application running on Linux, as we all know, whenever I restart the nodejs app it will get a new PID, suppose while the nodejs app is running, a client connects to it and running some process and the process status is processing, during that point of time, if the nodejs app restarts(on the server-side), how can we make sure the client connects back to the previous processing state.
What is happening now is, whenever the server restarts, the process stucks in processing forever.
Just direct me to a sample of how this scenario is handled in real life.
Thank You.
If I'm understanding you correctly, then the answer is you can't...
The reason for this is that, when you restart the process the event loop is restarted, meaning any processes that were running or were waiting in the event loop are gone. You are essentially clearing out the event loop when you restart.
I would say though, if you know the process is 'crashing' node then you probably want to look into that process and see why is crashing, place it in a try catch to it wont kill the server.
now with that said ( and without knowing what, processing state really means ) you could set a flag in your DB server for say 'job1' and have a status column of say 'running' when it was kicked off. When the node server restarts it can read Job status for 'running' jobs, if the 'job' is in a 'running' state you can fire off the job again and once complete update the table to 'completed'
This probably not the most efficient way as it's much better to figure out why the process if crashing, but as a fall-back this could work although in a clustered environment this could cause issues because server 1 may fail while server 2 is processing because server 1 does not know what server two is doing. With more details as to the use case, environment etc would probably allow for a better answer
Related
I've been trying to find information about Cassandra sessions relating to the Node.js cassandra-driver by Datastax. I read something which said that cassandra-driver automatically manages a session and that I don't need to call client.shutdown().
I'm looking for general information about how cassandra-driver manages sessions, how can I see all active Cassandra sessions, and do I need to call shutdown() or is that counter productive having to reopen a session every time the script is run?
Based on "pm2 info" I don't see a ton of active handles so I don't think anything wrong is going on but I may be mistaken. Ram usage does seem a bit high for a small script (85mb).
In the DataStax drivers, Session is a stateful object handling a pool of connections and aware of the status of nodes in the Cluster at any time (avoiding sending request to unavailable node). TCP sockets are opened and it is a best practice to close when you don't need it anymore. See here to get more infos : https://docs.datastax.com/en/developer/nodejs-driver-dse/2.1/features/connection-pooling/
Now session.connect() may takes a bit of time: the more nodes you have in your cluster, the longer it will be to open connections to every single one. This is the reason why, it is better to init connections in a "cold start" when you work with FAAS (avoiding to open/close for each request)
So:
Always close your connections (shutdown()) when you don't need it anymore (shutdown hook in your applications)
Keep your connections "alive" as long as you need it, do not shutdown for each request, this is NOT stateless.
yes, it is "better" to connect the client outside of the handler function. to keep it state-Full.
however, AWS Lambda with nodeJS, by default function execution continues until the event loop is empty or the function times out.
create the client outside of handler, set the context.callbackWaitsForEmptyEventLoop = false and don't call client.shutdown.
I have dynamic values in the array (say 100) all values are Neo4j database queries (cypher) each queries brings 30 values from the server.
sometimes if an error occurs in say xyz number query, to handle those error occurred, I have used process.exit() in catch() of the session of Cypher query, because I want to stop complete execution and also discard previous values, but as it also kills the running server.
I want to evaluate everything from starting,
I want to restart the server again automatically.
or
Any other way in spite of using process.exit(), if there is anything which I can call so that my current execution stops and restart current request again from scratch
In short: the server should not stop when an error occurs.
Anyone help, please?
If you want NodeJs to start again when its terminated.
Just try to use the process manager program to run your node.
For example PM2
Different users trying to access different application routes with heavy manipulation of data. At the mid time one of the request failed due to internal server error and my whole application has been crashed. Thats why other request has been failed because build has been crashed. Is there any solution to handle this situation?
If your program has thrown an uncaught error and crashed then there's nothing you can really do other than start it back up again. You could use something like pm2 to automatically restart your node process when it crashes, and then at least future requests that come in should work (although you will lose any in memory data from before the last crash).
Another thing that I think would help you would be to move your backend onto a serverless architecture where each invocation of your code is independent of the others.
And of course try to fix the code so that it handles things gracefully and doesn't actually throw errors. :)
I have a node.js app with mongodb backend going to production in a week and i have few doubts on how to handle app crashes and restart .
Say i have a simple route /followUser in which i have 2 database operations
/followUser
----->Update User1 Document.followers = User2
----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation
What happens if there is a server crash(due to power failure or maybe the remote mongodb server is down ) like this scenario :
----->Update User1 Document.followers = User2
SERVER CRASHED , FOREVER RESTARTS NODE
What happens to these operations below ? The system is now in inconsistent state and i may have error everytime i ask for User2 followers
----->Update User2 Document.followers = User1
----->Some other mongodb(via mongoose)operation
Also please recommend good logging and restart/monitor modules for apps running in linux.
Right now im using domains for to catch exceptions , doing server.close , but before process.exit() i want to make sure all database transactions are done , can i check this by testing if the event loop is empty or not (how?) and then process.exit(1) ?
You need transactions for this, and since MongoDB doesn't have them here is a workaround http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/
One way to address this problem is to add cleanup code to your application that runs whenever the application starts. You write the cleanup code to perform sanity checks on any portions of your data that can be updated in multiple steps like your example and then repairs that data in whatever way make sense for your application.
Depending on the complexity of your application/data, this may also require that you keep a log of actions the app was trying to perform, but that gets complicated real fast. Ideally it's more a matter of refreshing denormalized data and deleting partial data.
You want to do this during startup rather than shutdown as there's no guarantee your shutdown code will fully run and if you're shutting down because of an exception you don't know what the state of your system is at that point.
the solution given by vkurchatkin in this link is a workaround in case your appserver crashes, because you will be able of knowing which transactions were pending at that moment. If you implement this in your code you can create cleanup code when your system restart as suggested by JohnnyHK. The code you mention (catching exceptions, test when closing, etc) will not work because...well.. your server crashed! ;-)
That said, this is done using the database, so you will have to warrantee to a certain point that your database does not crash. I would suggest your use replication for that. It is basically a cluster of servers that recovers itself if one node fails, and also you can make some check to make sure that the data reached the servers and is safe.
Hope this helps.
I have node.js webserver using express.js and mongodb as a datastore. This server is being controlled by runit, and I am trying to implement a way to gracefully shut down the server.
I am implementing signal handlers for SIGINT and SIGTERM, and I am aware that you can stop listening for new connections by calling .close() on the object returned by createServer(). So far so good.
However, even when no more requests are forthcoming, there may be a number of requests already in the system that needs to finish before I can close the database.
I am using a mongodb ReplicaSet, and I figure that if I just call db.close() right away, that some of these requests may fail in some manner. Is there some way to close the database in a way that allows pending database queries to finish or do I have to manually have a +/- counter to see how many "active" queries are pending, and then wait to shutdown until it reaches 0?
You would have to manually ensure that you are in a clean state to shut down as the driver does not wait flush out the remaining operations before shutting down.
Feel free to log a ticket for the feature on
https://github.com/mongodb/node-mongodb-native/issues/