I have a job that is executed ones per day. My app is running on Heroku, and dyno is restarted ones a day.
So what can happen is that during job execution Heroku starts the restart of dyno.
That itself is not a problem as I can start job two times per day, but what is a problem is to stop the job in the middle of task when it is not in stable status.
I would like now somehow to send this signal to job function so I can break any loops and stop function execution in safe way.
I know how to get signal:
process
.on('SIGTERM', shutdown('SIGTERM'))
.on('SIGINT', shutdown('SIGINT'))
.on('uncaughtException', shutdown('uncaughtException'));
function shutdown(signal) {
console.log(`${ signal }...`);
return (err) => {
if (err) console.error(err.stack || err);
setTimeout(() => {
console.log('...waited 5s, exiting.');
process.exit(err ? 1 : 0);
}, 5000).unref();
};
}
but how to send this signal to my job function and to break from it safely?
Thank you.
So the best solution I came up with is following.
// Manage signals
let shutDownSignal = false;
process
.on('SIGTERM', shutdown('SIGTERM'))
.on('SIGINT', shutdown('SIGINT'))
.on('uncaughtException', shutdown('uncaughtException'));
function shutdown(signal) {
return (err) => {
shutDownSignal = true;
console.log(`Received signal: ${ signal }...`);
if (err) console.error(err.stack || err);
setTimeout(() => {
console.log('...waited 15s, exiting.');
process.exit(err ? 1 : 0);
}, 15000).unref();
};
}
module.exports.getShutDownSingnal = function(){ return shutDownSignal; }
then with getShutDownSingnal() anywhere I can check whether shutdown is initiated.
One more thing. It is necessary to put Procfile in app root with
web: node index.js
in it (or app.js depending what are you using).
This is necessary so that SIGTERM and SIGKILL signals are transferred correctly to node (for example if using npm, it will not transfer this signal correctly). More about this on Heroku docs
Maybe this will be useful for someone.
Related
My application terminates a process (exe file) then attempts to replace it with an updated version, I'm using process.kill with the pid of the process, I keep getting an error when trying to replace it with a newer version because the exe file is still in use and cannot be deleted, I have "resolved" this by waiting for 500ms but I wouldn't call that a good solution, I was expecting the method to be synchronous or at least have a sync counterpart just like the rest of fs methods.
Are there any other ways to do it in node.js?
The docs says the following :
Even though the name of this function is process.kill(), it is really just a signal sender, like the kill system call. The signal sent may do something other than kill the target process.
To me, it implies that process.kill will not indicate that the process is killed, but just that the signal has been sent (when it returns).
But there's another interesting line :
This method will throw an error if the target pid does not exist. As a special case, a signal of 0 can be used to test for the existence of a process. Windows platforms will throw an error if the pid is used to kill a process group.
So you can come up with something like that :
const killProcess = ({pid, signal = 'SIGTERM', timeout} = {}) => new Promise((resolve, reject) => {
process.kill(pid, signal);
let count = 0;
setInterval(() => {
try {
process.kill(pid, 0);
} catch (e) {
// the process does not exists anymore
resolve();
}
if ((count += 100) > timeout) {
reject(new Error("Timeout process kill"))
}
}, 100)
})
Is there a way of ensuring that a child process has been killed?
I currently have the following code:
let p = child_process.spawn(app, args);
...
try{
p.kill('SIGKILL');
} catch(e) {
console.error("Killing process exception:", e);
}
job = setInterval( () => {
if(p.killed || timeout === true){
clearInterval(job);
callback();
}
}, 100);
setTimeout( () => {
console.log("Killing process timeout!");
timeout = true;
}, 1000);
I check periodically (100 ms period) if the killing signal has been properly send to the process and, in that moment, I assume that the process has been killed; but, to ensure that the process is not locked, I set a timeout of 1 second.
Many times the timeout is fired, independently of waiting for 1 second or 10 seconds.
The code below is executed in linux; if working in WSL, then everything seems to work properly
I have created a small example application in node.js with unit tests and acceptance tests here
Both unit and acceptance tests are run inside mocha process. Acceptance tests start from forking the process and basically running the server on before() method. after() method stops the process and
before((initialized) => {
console.log('before script');
serverProcess = child_process.fork('server.js');
serverProcess.on('close', function (code) {
console.log('child process exited with code ' + code);
});
setTimeout(() => {
console.log('1s elapsed');
initialized();
}, 1000);
The code without any delays works on my local gitlab-runner, however on server it's not always the case, so I have added delay - wait for a while until the server will start.
Empirically I have found that 1s is enough and .5s is not.
However, I would like to know what should I do to make sure that the server is.
Are there any solutions to run server, execute the tests and shutdown the server that works on Linux, Windows, docker and outside of it?
There is a good help about how to communicate between fork processes.
The idea will be to send a message from the child saying to it's dad (I am ready!). Then the dad will continue is work.
Example :
before((initialized) => {
serverProcess = child_process.fork('server.js');
serverProcess.on('close', function(code) {
console.log('child process exited with code ' + code);
});
serverProcess.on('close', function(code) {
console.log('child process exited with code ' + code);
});
// We add a backup plan. If it takes too long to launch, throw
const timeout = setTimeout(() => {
initialized(new Error('tiemout');
}, 30000);
// Cait for the child to send a message to us
serverProcess.on('message', function(str) {
if (str === 'init done') {
clearTimeout(timeout);
// server.js got successfully initialized
initialized();
}
});
});
// To add inside of your server.js listen
if (process.send) {
process.send("init done");
}
I am have multiple micro services written in Nodejs Koa running in Docker Swarm.
Since container orchestration tools like Kubernetes or Swarm can scale up and down services instantly, I have a question on gracefully shutting down Nodejs service to prevent unfinished running process.
Below is the flow I can think of:
sending a SIGNINT signal to each worker process, Does docker swarm
send SIGNINT to worker when scaling down service?
the worker are responsible to catch the signal, cleanup or free any
used resource and finish the its process, How can I stop new api
request, wait for any running process to finish before shutting
down?
Some code below from reference:
process.on('SIGINT', () => {
const cleanUp = () => {
// How can I clean resources like DB connections using Sequelizer
}
server.close(() => {
cleanUp()
process.exit()
})
// Force close server after 5secs, `Should I do this?`
setTimeout((e) => {
cleanUp()
process.exit(1)
}, 5000)
})
I created a library (https://github.com/sebhildebrandt/http-graceful-shutdown) that can handle graceful shutdowns as you described. Works well with Express and Koa.
This package also allows you to create function (should return a promise) to additionally clean up things like DB stuff, ... here some example code how to use it:
const gracefulShutdown = require('http-graceful-shutdown');
...
server = app.listen(...);
...
// your personal cleanup function - this one takes one second to complete
function cleanup() {
return new Promise((resolve) => {
console.log('... in cleanup')
setTimeout(function() {
console.log('... cleanup finished');
resolve();
}, 1000)
});
}
// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
{
signals: 'SIGINT SIGTERM',
timeout: 30000,
development: false,
onShutdown: cleanup,
finally: function() {
console.log('Server gracefulls shutted down.....')
}
}
);
I personally would increase the final timeout from 5 secs to higher value (10-30 secs). Hope that helps.
I'm starting to learn and use node and I like it but I'm not really sure how certain features work. Maybe you can help me resolve one such issue:
I want to spawn local scripts and programs from my node server upon rest commands. looking at the fs library I saw the example below of how to spawn a child process and add some pipes/event handlers on it.
var spawn = require('child_process').spawn,
ps = spawn('ps', ['ax']),
grep = spawn('grep', ['ssh']);
ps.stdout.on('data', function (data) {
grep.stdin.write(data);
});
ps.stderr.on('data', function (data) {
console.log('ps stderr: ' + data);
});
ps.on('close', function (code) {
if (code !== 0) {
console.log('ps process exited with code ' + code);
}
grep.stdin.end();
});
grep.stdout.on('data', function (data) {
console.log('' + data);
});
grep.stderr.on('data', function (data) {
console.log('grep stderr: ' + data);
});
grep.on('close', function (code) {
if (code !== 0) {
console.log('grep process exited with code ' + code);
}
});
What's weird to me is that I don't understand how I can be guaranteed that the event handler code will be registered before the program starts to run. It's not like there's a 'resume' function that you run to start up the child. Isn't this a race condition? Granted the condition would be minisculy small and would almost never hit because its such a short snipping of code afterward but still, if it is I'd rather not code it this way out of good habits.
So:
1) if it's not a race condition why?
2) if it is a race condition how could I write it the right way?
Thanks for your time!
Given the slight conflict and ambiguity in the accepted answer's comments, the sample and output below tells me two things:
The child process (referring to the node object returned by spawn) emits no events even though the real underlying process is live / executing.
The pipes for the IPC are setup before the child process is executed.
Both are obvious. The conflict is w.r.t. interpretation of the OP's question:-
Actually 'yes', this is the epitome of a data race condition if one needs to consider the real child process's side effects. But 'no', there's no data race as far as IPC pipe plumbing is concerned. The data is written to a buffer and retrieved as a (bigger) blob as and when (as already well described) the context completes allowing the event loop to continue.
The first data event seen below pushes not 1 but 5 chunks written to stdout by the child process whilst we were blocking.. thus nothing is lost.
sample:
let t = () => (new Date()).toTimeString().split(' ')[0]
let p = new Promise(function (resolve, reject) {
console.log(`[${t()}|info] spawning`);
let cp = spawn('bash', ['-c', 'for x in `seq 1 1 10`; do printf "$x\n"; sleep 1; done']);
let resolved = false;
if (cp === undefined)
reject();
cp.on('error', (err) => {
console.log(`error: ${err}`);
reject(err);
});
cp.stdout.on('data', (data) => {
if (!resolved) {
console.log(`[${t()}|info] spawn succeeded`);
resolved = true;
resolve();
}
process.stdout.write(`[${t()}|data] ${data}`);
});
let ts = parseInt(Date.now() / 1000);
while (parseInt(Date.now() / 1000) - ts < 5) {
// waste some cycles in the current context
ts--; ts++;
}
console.log(`[${t()}|info] synchronous time wasted`);
});
Promise.resolve(p);
output:
[18:54:18|info] spawning
[18:54:23|info] synchronous time wasted
[18:54:23|info] spawn succeeded
[18:54:23|data] 1
2
3
4
5
[18:54:23|data] 6
[18:54:24|data] 7
[18:54:25|data] 8
[18:54:26|data] 9
[18:54:27|data] 10
It is not a race condition. Node.js is single threaded and handles events on a first come first serve basis. New events are put at the end of the event loop. Node will execute your code in a synchronous manner, part of which will involve setting up event emitters. When these event emitters emit events, they will be put to the end of the queue, and will not be handled until Node finishes executing whatever piece of code its currently working on, which happens to be the same code that registers the listener. Therefore, the listener will always be registered before the event is handled.