I have a Node application creating a number of worker processes via fork() to perform some tasks on their own. As a byproduct, the workers produce a snapshot of actions taken in the form of object. Each worker has an event listener attached to the 'exit' event, at which time they send their snapshot back to the parent process via process.send().
Here's an example of the set up:
// parent.js
const exec = require('child_process');
const worker = exec.fork('worker.js', [], {
stdio: ['pipe', 'pipe', null, 'ipc']
});
worker.on('message', (snapshot) => {
// Handle the snapshot sent from worker
});
// worker.js
process.on('exit', () => {
process.send({ /* snapshot data */ })
});
/* Arbitrary task work */
Is this an acceptable pattern for parent/worker IPC? Specifically, will this reliably about the reliability of receiving messages in the parent process given the nature of the exit event and process.send()?
Is the beforeExit event better suited to this pattern?
It is pretty clear in the nodejs doc that you should only do synchronous operations in the exit event. Here's an excerpt from the doc for the exit event:
Listener functions must only perform synchronous operations. The Node.js process will exit immediately after calling the 'exit' event listeners causing any additional work still queued in the event loop to be abandoned.
So, doing an async operation in the exit event is likely a crapshoot that will depend upon internal implementation and whether it needs the event loop to work.
You can do asynchronous things in the beforeExit event so that is where you should probably do it. Excerpt from beforeExit doc:
The 'beforeExit' event is emitted when Node.js empties its event loop and has no additional work to schedule. Normally, the Node.js process will exit when there is no work scheduled, but a listener registered on the 'beforeExit' event can make asynchronous calls, and thereby cause the Node.js process to continue.
The listener callback function is invoked with the value of process.exitCode passed as the only argument.
The 'beforeExit' event is not emitted for conditions causing explicit termination, such as calling process.exit() or uncaught exceptions.
The 'beforeExit' should not be used as an alternative to the 'exit' event unless the intention is to schedule additional work.
Related
I have a node backend taking HTTP requests using express. I am shutting down gracefully like this:
process.on( 'SIGINT', function() {
console.log("SIGINT signal received.");
server.close(function(err) {
if (err) {
console.error(err)
process.exit(1)
}
//Stop reoccurring tasks
//Close database connection
process.exit(0);
});
process.exit(0);
});
What I have is working fine, but I am concerned about my "Stop reoccurring tasks" step. Elsewhere in my code, I call a function that looks like this:
export async function launchSectionFinalizer() {
finalizeSections();
//1 hr * 60 min/hr * 60 s/min * 1,000 ms/s = 3,600,000 ms
return setInterval(finalizeSections, 3_6000_000);
}
Where finalizeSections is an async function that performs a series of database operations (postgres database).
My question is about the nature and behavior of setInterval. How can I make sure that finalizeSections isn't in the middle of its execution when I receive SIGINT? I'm worried that if my program receives SIGINT and closes the server at the wrong time it could catch finalizeSections in the middle of its operations. If that happens, I could end up with those database operations partially complete (ie if I execute a series of sql commands one after another, insert1, insert2, and insert3, I do not want to execute 1 and 2 without also executing 3).
I have done some googling and read something about how node will wait for all of its processes and events to complete before closing. Would that include waiting for my call to finalizeSections to complete?
Also, I am aware of clearInterval, but I am not sure if that function only stops the timer or if it will also cause node to wait for finalizeSections to complete.
Calling clearInterval will only cancel the timer and not wait for finalizeSections to finish.
Because your graceful shutdown calls process.exit(0) it will not wait for pending asynchronous tasks to finish and it will exit immediately:
Calling process.exit() will force the process to exit as quickly as possible even if there are still asynchronous operations pending that have not yet completed fully, including I/O operations to process.stdout and process.stderr
One way to solve this without using any packages is to save a reference to the promise returned by finalizeSections() and the intervalId returned by setInterval():
intervalId = setInterval(() => {
finalizeSectionsPromise = finalizeSections();
}, 3_6000_000)
Then in the shutdown code.
clearInterval(intervalId);
if (finalizeSectionsPromise) {
await finalizeSectionsPromise;
}
...
process.exit(0);
If you are able to use other packages I would use a job scheduling library like Agenda or Bull, or even cron jobs:
https://github.com/OptimalBits/bull
https://github.com/agenda/agenda
Also take a look a stoppable or terminus to gracefully shutdown servers without killing requests that are in-flight:
https://www.npmjs.com/package/stoppable
https://github.com/godaddy/terminus
I have the following code...
async function finish(){
console.log("Finishing");
console.time("fin");
let test = await new Promise(function(res){
setTimeout(()=>{res(test)}, 2000);
});
console.timeEnd("fin");
console.log(test);
};
process.on('exit', finish);
I would expect this to wait two second on exit and print out a timestamp close to 2s. However, when I run the timestamp is shorter and doesn't print any line after Finishing.
How do I wait for a timeout on exit?
From the node docs, you cannot use asynchronous code in the exit event.
Listener functions must only perform synchronous operations. The Node.js process will exit immediately after calling the 'exit' event listeners causing any additional work still queued in the event loop to be abandoned.
If you want to schedule additional work before exiting (e.g. your asynchronous function), you need to use beforeExit.
process.on('beforeExit', finish);
Having said that, you'll also need to recognize that beforeExit is only emitted when the process is out of work to do, so a) it'll not emit if something explicitly calls for termination (e.g. process.exit()) and b) it'll keep emitting unless that happens.
I have two different Nodejs servers in two different components, but they run in the same software together. When I close the program, i want both servers to exit gracefully, and free their respective TCP ports.
For each of them I have:
server1.js:
process.on('exit', function(){
server.close();
});
server2.js:
process.on('exit', function(){
server.close();
});
The problem is, that only one of these two functions runs with this method.
Closing them together in one function is not An option.
Is there anyway to add functions to the 'exit' event?
Node.js process exit event can have multiple listeners.
This
process
.on('exit', () => console.log('foo'))
.on('exit', () => console.log('bar'));
should result in
foo
bar
output.
Closing Node.js web servers explicitly is unnecessary because they will be closed any way on process exit.
exit event listener is not a proper place to close connections in general because this is usually asynchronous process. exit listener is supposed to be synchronous:
Listener functions must only perform synchronous operations. The Node.js process will exit immediately after calling the 'exit' event listeners causing any additional work still queued in the event loop to be abandoned.
While beforeExit event listener may be skipped in some cases but is suitable for asynchronous operations:
The 'beforeExit' event is emitted when Node.js empties its event loop
and has no additional work to schedule. Normally, the Node.js process
will exit when there is no work scheduled, but a listener registered
on the 'beforeExit' event can make asynchronous calls, and thereby
cause the Node.js process to continue.
<...>
The 'beforeExit' event is not emitted for conditions causing explicit
termination, such as calling process.exit() or uncaught exceptions.
In the following code:
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
sleep(10000);
Why NodeJS doesn't exit immediately? What causes the Promise to be waited for?
I thought I needed await sleep(10000) - but this gives an error.
Nodejs waits for timers that are still active before exiting. You can tell it to NOT wait for a timer by calling .unref() on the timer object itself.
So, it's not actually the promise it is waiting for, but rather the timer it is waiting for.
Internally, node.js keeps a reference count of the number of open timers that have not been .unref() and will not exit until that count (among other things) gets to zero.
Here's a couple excerpts from the node.js doc for timers:
Class: Timeout
By default, when a timer is scheduled using either setTimeout() or setInterval(), the Node.js event loop will continue running as long as the timer is active. Each of the Timeout objects returned by these functions export both timeout.ref() and timeout.unref() functions that can be used to control this default behavior.
timeout.unref()
When called, the active Timeout object will not require the Node.js event loop to remain active. If there is no other activity keeping the event loop running, the process may exit before the Timeout object's callback is invoked. Calling timeout.unref() multiple times will have no effect.
Take a look at the unref() function for timers in node - https://nodejs.org/api/timers.html#timers_timeout_unref
When called, the active Timeout object will not require the Node.js event loop to remain active. If there is no other activity keeping the event loop running, the process may exit before the Timeout object's callback is invoked.
You can create a timeout and call the unref() function on it - this will prevent node from staying alive if the only thing it is waiting for is the timeout.
function sleep(ms) {
return new Promise((resolve) => {
setTimeout(resolve, ms).unref();
});
}
As a side note, the same unref function can also be used for setTimeout calls.
As correctly noted by jfriend00, it is not the promise that is keeping node alive, it is the timeout.
Nodejs has a method process.nextTick(fn) which delays the execution of a function until the next tick. According to this article this can be used to emit events in the constructor of an object like this (copied from the artice):
function StreamLibrary(resourceName) {
var self = this;
process.nextTick(function() {
self.emit('start');
});
}
var stream = new StreamLibrary('fooResource');
stream.on('start', function() {
console.log('Reading has started');
});
Without process.nextTick this wouldn't have worked because the event would've been emitted before you had the chance to listen for events. However, how can you be certain that the event won't be emitted before you added an event listener? If it's fully asynchronous then there would be a chance that the next tick is executed before the end of the constructor, hence ignoring the event listener. So what are the rules behind this process? When does node executes a nextTick?
The next tick is executed on... the next tick. I think the confusion here is what you mean by "fully asynchronous".
With Node.js, all of your code runs on a single thread. No code scheduled for the next tick will be executed until the current tick is finished.
Therefore, when you emit events from the next tick within the constructor, the code that may attach handlers will all be finished executing before that next tick occurs.
The asynchronous and multi-threaded part of Node is with handling operations such as network and IO. Those function calls run while your JavaScript code is running, but when data is passed back to your code, it won't run until that next tick.