node.js multithreading with max child count - node.js

I need to write a script, that takes an array of values and multithreaded way it (forks?) runs another script with a value from array as a param, but so max running forks would be set, so it would wait for script to finish if there are more than n running already. How do I do that?
There is a plugin named child_process, but not sure how to get it done, as it always waits for child termination.
Basically, in PHP it would be something like this (wrote it from head, may contain some syntax errors):
<php
declare(ticks = 1);
$data = file('data.txt');
$max=20;
$child=0;
function sig_handler($signo) {
global $child;
switch ($signo) {
case SIGCHLD:
$child -= 1;
}
}
pcntl_signal(SIGCHLD, "sig_handler");
foreach($data as $dataline){
$dataline = trim($dataline);
while($child >= $max){
sleep(1);
}
$child++;
$pid=pcntl_fork();
if($pid){
// SOMETHING WENT WRONG? NEVER HAPPENS!
}else{
exec("php processdata.php \"$dataline\"");
exit;
}//fork
}
while($child != 0){
sleep(1);
}
?>

After the conversation in the comments, here's how to have Node executing your PHP script.
Since you're calling an external command, there's no need to create a new thread. The Node.js runloop understands that calls to external commands are async operations, and it can execute all of them at the same time.
You can see different ways for executing an external process in this SO question (linked answer may be the best in your case).
However, since you're already moving everything to Node, you may even consider rewriting your "process.php" script to Node.js code. Since, as you explained, that script connects to remote servers and databases and uses nslookup (which you may not really need with Node.js), you won't need any separate thread: they're all async operations that Node.js excels at performing.

Related

get the output of the remote execution of the SSH command

I work with expressJs and to execute a remote SSH command I use the 'simple-ssh', this code allows to execute the command except that I could not get the result of the display outside this block.
ssh.exec('ls Documents/versions', {
out: function(stdout) {
arrayOfVersion = stdout.split("\n");}}).start();
How to get the content of arrayOfVersion and manipulate it after
Your function which creates arrayofVersion async, you won't be able access it outside of this scope without some sort of waiting process which waits until the variable has a value.
You can do this in a few ways, to begin with I would recommend researching how nodejs handles async functions as this is a big part of nodejs. Generally you would use one of the following: callbacks, promises, or async/await.
With any of those techniques, you should be able to run your SSH code and then continue on with the result of the stdout.

Limit Execution Time in Node

I am working on a node-based MUD game and I would like to limit the amount of time any one command can execute before it gets killed (e.g. 1000ms). I found a module called Tripwire which seems promising but it does not appear to be actively maintained. Tripwire does work as advertised. It manages to force an exception if someone creates an endless loop, but it does not support any resumption of the original script thread.
I am looking for either:
(1) A similar but actively maintained Node module that can interrupt and resume the original event thread, or,
(2) A working example of V8's Isolate::IsExecutionTerminating + Isolate::CancelTerminateExecution (I forked Tripwire but I haven't done any meaningful C++ in a long time and am now just beating my head against the wall).
I have only been able to find test cases so far (which is at least something). I am really hoping that someone has already tackled this, though.
Test cases:
https://chromium.googlesource.com/v8/v8/+/ad55afcb459dafda1cf48e676985717fd7eae786/test/cctest/test-thread-termination.cc
I know this is a bit vague.
I ended up instrumenting the script by passing it through acorn and generating my own final script. I am hoping that the sandbox is locked down to prevent users from escaping it. Example of "compiled" output:
createPermissions(expr) {
let __mec = __bfc(this || GameMaster, 'public', 'createPermissions', __FILE__, false); try { let parts = expr.split('/');
for (let i = 0; i < parts.length; i++) {
__ala(); let foo = parts.slice(0, i).join('/');
} } finally { __efc(__mec, 'createPermissions'); }
}
This new "language" supports public, protected, package, and private variables/methods (by maintaining its own internal call stack, execution context, etc). The directives are "reserved words" (e.g. __bfc=begin function call, __ala=assert loop alarm).
Thanks #jfriend00 for the suggestion.
For those who are curious: Transpiler Module

Is there any risk to read/write the same file content from different 'sessions' in Node JS?

I'm new in Node JS and i wonder if under mentioned snippets of code has multisession problem.
Consider I have Node JS server (express) and I listen on some POST request:
app.post('/sync/:method', onPostRequest);
var onPostRequest = function(req,res){
// parse request and fetch email list
var emails = [....]; // pseudocode
doJob(emails);
res.status(200).end('OK');
}
function doJob(_emails){
try {
emailsFromFile = fs.readFileSync(FILE_PATH, "utf8") || {};
if(_.isString(oldEmails)){
emailsFromFile = JSON.parse(emailsFromFile);
}
_emails.forEach(function(_email){
if( !emailsFromFile[_email] ){
emailsFromFile[_email] = 0;
}
else{
emailsFromFile[_email] += 1;
}
});
// write object back
fs.writeFileSync(FILE_PATH, JSON.stringify(emailsFromFile));
} catch (e) {
console.error(e);
};
}
So doJob method receives _emails list and I update (counter +1) these emails from object emailsFromFile loaded from file.
Consider I got 2 requests at the same time and it triggers doJob twice. I afraid that when one request loaded emailsFromFile from file, the second request might change file content.
Can anybody spread the light on this issue?
Because the code in the doJob() function is all synchronous, there is no risk of multiple requests causing a concurrency problem.
If you were using async IO in that function, then there would be possible concurrency issues.
To explain, Javascript in node.js is single threaded. So, there is only one thread of Javascript execution running at a time and that thread of execution runs until it returns back to the event loop. So, any sequence of entirely synchronous code like you have in doJob() will run to completion without interruption.
If, on the other hand, you use any asynchronous operations such as fs.readFile() instead of fs.readFileSync(), then that thread of execution will return back to the event loop at the point you call fs.readFileSync() and another request can be run while it is reading the file. If that were the case, then you could end up with two requests conflicting over the same file. In that case, you would have to implement some form of concurrency protection (some sort of flag or queue). This is the type of thing that databases offer lots of features for.
I have a node.js app running on a Raspberry Pi that uses lots of async file I/O and I can have conflicts with that code from multiple requests. I solved it by setting a flag anytime I'm writing to a specific file and any other requests that want to write to that file first check that flag and if it is set, those requests going into my own queue are then served when the prior request finishes its write operation. There are many other ways to solve that too. If this happens in a lot of places, then it's probably worth just getting a database that offers features for this type of write contention.

How to crash my Node app on purpose?

I've been working on a deployment work flow with Dokku and Docker and now I want to take care of continuity of my app (along the lines of Forever). To test it, I need a way to deliberately crash my app.
I created a new route '/crashme' with a function that is supposed to wreck my app.
Haven't found a way that worked locally with node/nodemon so far, I've tried:
Division by zero
Throw a new user exception
Referencing a variable that doesn't exist
None of those things crash the app to a point where it needs to be restarted.
Just how can I bring it down?
Three things come to my mind:
You could just call process.exit. This for sure brings your application to a state where it needs to be restarted.
The other option might be to run an endless loop, something such as while (true) {}. This should make Node.js use 100% of your CPU, and hence the application should be restarted as well (although this, of course, means that you / someone has to watch your application).
Create a module in C that crashes by e.g. trying to access a random place in memory. I have no such module at hand, but I'm pretty sure that it should be quite easy for someone with C skills to write such a module.
I was attempting a similar thing with a /crash route in express, but just throwing an error from within the route handler was not enough to crash it.
process.exit would stop my app but forever would not restart it. (The forever logs just said something like process self terminated.)
What did work for me was inserting this into my /crash route:
setTimeout(function () {
throw new Error('We crashed!!!!!');
}, 10);
To add to Golo answer:
C module to crash by segmentation fault:
int main ()
{
//Create a array of 1 char
char a [1];
//Create a index
int i = 0;
//Infinite loop to go around the compiler
while(1)
{
//Write on case i of a, on the second iteration, it will write in unreserved memory => crash
a[i] = 0;
i = i + 1;
}
//Should not go there
return -1;
}
And adding to DrakaSAN's answer, an even simpler C module to crash:
int main()
{
*(int*)(0) = 0;
return -1;
}
Even shorter ones are available on this page. If you don't want it to be too hard to read, you can probably go with
int main()
{
int i=1/0;
}

issue with fork() from a firebreath npapi plugin

I am trying to fork() a new process so that I can call a separate console application.
The fork does happen fine and I get a new process id but the process is in sleeping state and does not get active at all even if the browser exits.
I just took the sample plugin project and modified the echo method to do the fork.
A regular console application works fine with the fork code.
Is there something different that has to be taken into account for a firebreath plugin app?
Can someone suggest what might be the issue?
The platform is archlinux 64 bit.
FB::variant PluginTestVZAPI::echo(const FB::variant& msg)
{
static int n(0);
fire_echo("So far, you clicked this many times: ", n++);
// fork
pid_t pid = fork();
if(pid == 0) // Child
{
m_host->htmlLog("child process");
}
else if (pid < 0) // Failed to fork
{
m_host->htmlLog("Failed to fork");
m_host->htmlLog(boost::lexical_cast<std::string>(pid));
}
else // Parent
{
m_host->htmlLog("Parent process");
}
m_host->htmlLog("Child Process PID = " + boost::lexical_cast<std::string>(pid));
// end fork
// return "foobar";
return msg;
}
I can't be certain but if I were you I'd try removing the htmlLog calls -- there is no way for you to access the DOM from the child process, so htmlLog won't work at all and it is quite possible that trying to use it in a forked process is causing it to go into an inactive state while it tries (unsuccessfully) to communicate with a browser process that doesn't know about it.
I don't know for certain if this can work or not, but I'd be more than a bit nervous about forking a process that is already a child process of something else; the browser owns the plugin process and communicates with it via IPC, so if you fork that process there could be a lot of code that you don't know about still running and trying to talk to the browser through a now-defunct IPC connection.
My recommendation would be to launch a seperate process, but that's just me. At the very least, you absolutely cannot use anything FireBreath provides for communicating with the browser from the child process.

Resources