I have a node.js application that I've moved into JXcore multithreading and yet couldn't figure out how to reset a task. On the current implementation, the server creates a sub process and send the jobs one by one.
When a job takes more than X seconds, the main process kills the sub process and skips the running task and logs it. Any of the jobs shouldn't take more than X seconds.
So far, I already moved the queue system into JXcore easily and it works as expected but I couldn't figure out yet, how can I kill the running task.
Looks like the ability to kill the running task is a needed feature, since someone already asked the same question and it has been answered here: Thread lifetime management.
Upcoming JXcore release will have jxcore.tasks.killThread(). The logic is this: a task would inform the main thread, that it just has been started, and then the main thread may start counting the timeout for killing the thread, for example:
// main thread receives the message from a task
jxcore.tasks.on("message", function(threadId, obj){
if(obj.started){
//kill the task after a second
setTimeout(function(){
jxcore.tasks.killThread(threadId);
console.log("thread killed", threadId);
},1000);
}
});
// adding a task
jxcore.tasks.addTask( function() {
// informing the main thread, that task is just started
process.sendToMain({started:true});
// looping forever
while(true){};
console.log("this line will never happen.");
});
Related
I have a node application that runs long running tasks so whenever a task runs a child process is forked to run the task. The code creates a fork for the task to be run and sends a message to the child process to start.
Originally, when the task was complete, I was sending a message back to the parent process and the parent process would call .kill() on the child process. I noticed in my activity monitor that the node processes weren't being removed. All the child processes were hanging around. So, instead of sending a message to the parent and calling .kill(), I called process.exit() in the child process code once the task was complete.
The second approach seems to work fine and I see the node processes being removed from the activity monitor but I'm wondering if there is a downside to this approach that I don't know about. Is one method better than the other? What's the difference between the 2 methods?
My code looks like this for the messaging approach.
//Parent process
const forked = fork('./app/jobs/onlineConcurrency.js');
forked.send({clientId: clientData.clientId,
schoolYear: schoolYear
});
forked.on("message", (msg) => {
console.log("message", msg);
forked.kill();
});
//child Process
process.on('message', (data) => {
console.log("Message recieved");
onlineConcurrencyJob(data.clientId, data.schoolYear, function() {
console.log("Killing process");
process.send("done");
});
})
The code looks like this for the child process when just exiting
//child Process
process.on('message', (data) => {
console.log("Message received");
onlineConcurrencyJob(data.clientId, data.schoolYear, function() {
console.log("Killing process");
process.exit();
});
})
kill sends a signal to the child process. Without an argument, it sends a SIGTERM (where TERM is short for "termination"), which typically, as the name suggests, terminates the process.
However, sending a signal like that is a forced method of stopping a process. If the process is performing tasks like writing to a file, and it receives a termination signal, it might cause file corruption because the process doesn't get a chance to write all data to the file, and close it (there are mitigations for this, like installing a signal handler that can be used to "catch" signals and ignore them, or finish all tasks before exiting, but this requires explicit code to be added to the child process).
Whereas with process.exit(), the process exits itself. And typically, it does so at a point where it knows that there are no more pending tasks, so it can exit cleanly. This is generally speaking the best way to stop a (child) process.
As for why the processes aren't being removed, I'm not sure. It could be that the parent process isn't cleaning up the resources for the child processes, but I would expect that to happen automatically (I don't even think you can perform so-called "child reaping" explicitly in Node.js).
Calling process.exit(0) is the best mechanism, though there are cases where you might want to .kill from the parent (eg. A distributed search where one node returning means all nodes can stop).
.kill is probably failing due to some handling of the signal it is getting. Try .kill('SIGTERM'), or even 'SIGKILL'.
Also note that subprocesses which aren't killed when the parent process exits will be moved to the grandparent process. See here for more info and a proposed workaround: https://github.com/nodejs/node/issues/13538
In summary, this is default Unix behavior, and the workaround is to process.on("exit", () => child.kill())
I have this code periodically calls the load function which does very load work taking 10sec. Problem is when load function is being executed, it's blocking the main flow. If I send a simple GET request (like a health check) while load is being executed, the GET call is blocked until the load call is finished.
function setLoadInterval() {
var self = this;
this.interval = setInterval(function doHeavyWork() {
// this takes 10 sec
self.load();
self.emit('reloaded');
}, 20000);
I tried async.parallel but still the GET call was blocked. I tried setTimeout but got the same result. How do I make load to running on background so that it doesn't block the main flow?
this.interval = setInterval(function doHeavyWork() {
async.parallel([function(cb) {
self.load();
cb(null);
}], function(err) {
if (err) {
// log error
}
self.emit('reloaded');
})
}, 20000);
Node.js is an event driven non-blocking IO model
Anything that is IO is offloaded as a separate thread in the underlying engine and hence parallelism is achieved.
If the task is CPU intensive there is no way you can achieve parallelism as by default Javascript is a blocking sync language
However there are some ways to achieve this by offloading the CPU intensive task to a different process.
Option1:
exec or spawn a child process and execute the load() function in that spawned node app. This is okay if the interval fired is for every 20000 ms as by the time another one fired, the 10sec process will be completed.
Otherwise it is dangerous as it can spawn too many node applications eating up your Systems resources
Option2:
I dont know how much data self.load() accepts and returns. If it is trivial and network overhead is acceptable, make that task a load-balanced web service (may be 4 webservers running in parallel) which accepts (rather point to) 1M records and returns back filtered records.
NOTE
It looks like you are using node async parallel function. But keep a note of this description from the documentation.
Note: parallel is about kicking-off I/O tasks in parallel, not about parallel execution of code. If your tasks do not use any timers or perform any I/O, they will actually be executed in series. Any synchronous setup sections for each task will happen one after the other. JavaScript remains single-threaded.
This is using a sub-classed QThread based on the ideas expressed in the whitepaper "QThreads: You were not doing so wrong". It does not have an event loop, nor does it have slots. It just emits signals and stops. In fact its primary signal is the QThread finished one.
Basically I have a Qt using a background thread to monitor stuff. Upon finding what it is looking for, it records its data, and terminates.
The termination sends a signal to the main event loop part of the application, which processes it, and when done, starts the background anew. I can usually get this working for tens of seconds, but then it just seems to quit.
It seems that when the main application tries to start the thread, it doesn't really run. I base this on telemetry code that increments counters as procedures get executed.
basically
//in main application. Setup not shown.
//background points to the QThread sub-class object
void MainWindow::StartBackground()
{
background->startcount++;
background->start();
if ( background->isRunning() )
{
background->startedcount++;
}
}
//in sub-classed QThread
void Background::run()
{
runcount++;
//Do stuff until done
}
So when I notice that it seems that my background thread isn't running, by watching Process Explorer, I cause the debugger to break in, and check the counts. What I see is that startcount and startedcount are equal. And have a value of one greater than runcount
So I can only conclude that the thread didn't really run, but I have been unable to find out any evidence of why.
I have not been able to find documentation on QThreads not starting do to some error condition, or what evidence there is of such an error.
I suppose I could set up a slot to catch started from the thread. The starting code could loop on a timed-out semaphore, trying again and again until the started slot actually resets the semaphore. But it feels ugly.
EDIT - further information
So using the semaphore method, I have a way to breakpoint on failure to start.
I sampled isFinished() right before I wanted to do start(), and it was false. After my 100ms semaphore timeout it became true.
So the question seems to be evolving into 'Why does QThread sometimes emit a finished() signal before isFinished() becomes true?'
Heck of a race condition. I'd hate to spin on isFinished() before starting the next background thread.
So this may be a duplicate of
QThread emits finished() signal but isRunning() returns true and isFinished() returns false
But not exactly, because I do override run() and I have no event loop.
In particular the events 8 and 9 in that answer are not in the same order. My slot is getting a finished() before isFinished() goes true.
I'm not sure an explicit quit() is any different than letting run() return;
It sounds as if you have a race condition wherein you may end up trying to restart your thread before the previous iteration has actually finished. If that's the case then, from what I've seen, the next call to QThread::start will be silently ignored. You need to update your code so that it checks the status of the thread before restarting -- either by calling QThread::isFinished or handling the QThread::finished signal.
On the other hand... why have the thread repeatedly started/stopped. Would it not be easier to simply start the thread once? Whatever code is run within the context of QThread::run can monitor whatever it monitors and signal the main app when it finds anything of note.
Better still. Separate the monitor logic from the thread entirely...
class monitor: public QObject {
.
.
.
};
QThread monitor_thread;
monitor monitor;
/*
* Fix up any signals to/from monitor.
*/
monitor.moveToThread(&monitor_thread);
monitor_thread.start();
The monitor class can do whatever it wants and when it's time to quit the app can just call monitor_thread::quit.
There is a race condition in the version of Qt I am using. I don't know if it was reported or not before, but I do not have the latest, so it's probably moot unless I can demonstrate it in the current version.
Similar bugs were reported here long ago:
QThread.isFinished returns False in slot connected to finished() signal
(the version I use is much more recent than Qt 4.8.5)
What more important is I can workaround it with the following code
while ( isRunning() )
{
msleep(1);
}
start();
I've run a few tests, and it never seems to take more than 1ms for the race condition to settle. Probably just needs a context switch to clean up.
I have a Meteor server method that executes a function that I want to run in the background. I want the server method to continue while the function runs in a different process.
the method:
myMethod: function(arg) {
_event.emit("myEvent",args);
...do some other stuff
}
I'd like the server to do the other stuff and return to the client, and to do the stuff in _event.emit in the background no results have to be sent back to the client.
What it currently does is run the stuff in _event.emit and than the other stuff and then return to the client.
I tried to solve it by doing a call on server side with an empty callback function, but that didn't do the trick
myMethod: function (arg) {
return Meteor.call("secondMethod", _.toArray(arg), function() {})
}
secondMethod: function (arg) {
_event.emit("myEvent",args);
}
hope someone knows a solution for this.
thanks
The javascript in nodejs is single threaded. It works off an event queue where an event is popped off the queue, some callback is called to serve that event and the code behind that callback runs to completion, then the next event is pulled off the event queue and the process is repeated.
As such, there is no real "background processing". If your JS thread is executing, then nothing else can run at that time until that JS thread finishes and allows the next event in the event queue to be processed.
To truly run something else in parallel where both your task and other nodejs code can literally run at the same time, you would have to create a new process and let the new process carry out your task. This new process could be any type of process (some pre-built program running a task or another custom nodejs process).
"Running in the background" can sometimes be simulated by time slicing your "background" process work such that it does small amounts of work on timer ticks and the pauses between timer ticks allow other nodejs JS events to be processed and their code to run. But, to do this type of simulated "background", you have to write your task to execute in small chunks. It's a different (and often more cumbersome) way of writing code.
On the server you can just use Meteor.setTimeout() to run a function asynchronously. Just use an interval of zero if you want to return control immediately. Your main function will continue to run and as soon as there's nothing left in the event queue the scheduled task will run (or it will wait the amount of time you specify if non-zero).
How do "modern" cron daemons internally schedule their jobs? Some cronds used to schedule a run every so often via at. So after a crontab is written out, does crond:
Parse the crontab for all future events and the sleep for the intervals?
Poll an aggregated crontab database every minute to determine if the current time matches the schedule pattern?
Other?
Thanks,
A few crickets heard in this question. Good 'ol RTFC with some discrete event simulation papers and Wikipedia:
http://en.wikipedia.org/wiki/Cron#Multi-user_capability
The algorithm used by this cron is as
follows:
On start-up, look for a file named .crontab in the home directories of
all account holders.
For each crontab file found, determine the next time in the future
that each command is to be run.
Place those commands on the Franta-Maly event list with their
corresponding time and their "five
field" time specifier.
Enter main loop:
Examine the task entry at the head of the queue, compute how far in the
future it is to be run.
Sleep for that period of time.
On awakening and after verifying the correct time, execute the task at
the head of the queue (in background)
with the privileges of the user who
created it.
Determine the next time in the future to run this command and place
it back on the event list at that time
I wrote a blog post describing it.
Quoting the relevant text from there:
We can have a finite thread-pool which will execute all the tasks by picking them up from a PriorityBlockingQueue (thread-safe heap) prioritized on job.nextExecutionTime().
Meaning that the top element of this heap will be always be the one that will fire the soonest.
We will be following the standard threadpool producer-consumer pattern.
We will have one thread which will be running in an infinite loop and submitting new jobs to the thread pool after consuming them from the queue.
Lets call it QueueConsumerThread:
void goToSleep(job, jobQueue){
jobQueue.push(job);
sleep(job.nextExecutionTime() - getCurrentTime());
}
void executeJob(job, jobQueue){
threadpool.submit(job); // async call
if (job.isRecurring()) {
job = job.copy().setNextExecutionTime(getCurrentTime() + job.getRecurringInterval());
jobQueue.add(job);
}
}
#Override
void run(){
while(true)
{
job = jobQueue.pop()
if(job.nextExecutionTime() > getCurrentTime()){
// Nothing to do
goToSleep(job, jobQueue)
}
else{
executeJob(job, jobQueue)
}
}
}
There will be one more thread which will be monitoring the crontab file for any new job additions and will push them to the queue.
Lets call it QueueProducerThread:
#Override
void run()
{
while(true)
{
newJob = getNewJobFromCrontabFile() // blocking call
jobQueue.push(newJob)
}
}
However, there is a problem with this:
Imagine that Thread1 is sleeping and will wake up after an hour.
Meanwhile a new task arrives which is supposed to run every minute.
This new task will not be able to start executing until an hour later.
To solve this problem, we can have ProducerThread wakeup ConsumerThread from its sleep forcefully whenever the new task has to run sooner than the front task in the queue:
#Override
void run()
{
while(true)
{
newJob = getNewJobFromCrontabFile() // blocking call
jobQueue.push(newJob)
if(newJob == jobQueue.peek())
{
// The new job is the one that will be scheduled next.
// So wakeup consumer thread so that it does not oversleep.
consumerThread.interrupt()
}
}
}
Note that this might not be how cron is implemented internally.
However, this is the most optimal solution that I can think of.
It requires no polling and all threads sleep until they need to do any work.