I want to spawn a child process, and then at a later time send an argument to it that will then execute. How can I do that? (NodeJS, on Mac)
For example, I have a command to execute a script file:
osascript script-test.scpt
This command works in the terminal, and it also works using exec, like so:
const { exec } = require('child_process')
var script = 'osascript script-test.scpt'
exec(script)
But how do I get it to work in an already running child process?
I've tried the following, but nothing happens (no errors, and no activity):
const { spawn } = require('child_process')
var process = spawn('osascript')
...
[at some later point (after spawned process has been created)]
process.stdin.write('script-test.scpt')
In all current operating systems, a process is spawned with a given set of arguments (also called argv, argument values) and preserves this set until execution ends. This means that you cannot change arguments on the fly.
For a program to support multiple job submissions after spawning, it needs to implement this explicitly using some form of communication - this is known as IPC, or Inter-Process Communication. A program that supports IPC will usually allow another program to control its behavior to some extent - for example, submit jobs for processing and report back on their completion.
Popular methods of implementing IPC include:
Network communication
Local calls via a "message bus" such as D-Bus
Pipes (direct communication over stdin/stdout)
Inspect the documentation for program that you're trying to call and find out if it supports any form of control, out of the ones listed above. If yes, you may be able to integrate (in a program-specific way) with it. If not, then you will need to spawn a new instance every time you need to process a new job.
Related
I've been having trouble sort of succinctly describing what I'm tying to accomplish, but what I'm attempting to do is write a program that will execute other programs/scripts as plugins, and during that execution, facilitate communication between my program and the other program/script for various requests (like asking for a userlist, for example). However, if these other programs/scripts need to communicate with the end user, I'd also like to not get in the way of that. I explored some other means of doing this (like dbus, an http api, a different file handle, etc), and all those options are either too heavy a solution for what I'm trying to do, or I didn't have much success in implementing (as is the case of the last idea), and just using normal STDIN/STDOUT seems like the cleanest path forward.
Here's what I have at the moment:
let mut child = Command::new("test.pl")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("Failed to spawn child process");
let mut stdin = child.stdin.as_mut().unwrap();
writeln!(&mut stdin, "Hey, testing!").unwrap();
let stdout = child.stdout.as_mut().unwrap();
let stdout_reader = BufReader::new(stdout);
let stdout_lines = stdout_reader.lines();
for line in stdout_lines {
match line.unwrap().as_str() {
"member_list" => {
println!("Here's a member list");
let mlist = cluster::members_list()?;
for (member, port) in mlist.iter() {
writeln!(&mut stdin, "{}:{}", member, port);
}
}
e => {
println!("{}", e);
}
};
}
child.wait().unwrap();
This works well with communication between my program and the other, passing STDOUT to the user if it doesn't match any of the keywords expected for communication. However, I can't think of a way of being able to pass information to STDIN of the other program without getting in the way of the other program requesting input from the user (such as if the program needs to ask the user, not my program, for input). I tried a silly usage of drop(stdin); and redeclaring stdin later on, which of course didn't work due to scope issues.
I'm really trying to avoid acting as an intermediary for STDIN since that seems like it would be terribly messy. I would appreciate any insight into how I might be able to accomplish this. I'm also open to other ideas on facilitating this communication other than through STDIN/STDOUT.
Thanks all!
You could use an anonymous pipes. In Linux, at a high level, that would look like this:
Your process creates the pipe using the pipe system call, which returns a pair of FDs, one for the reading end of the pipe and the other for the writing end
Your process then forks. In the child subprocess, you then call exec to start the process you want to run, passing it one of the FDs as a command line argument
The new program takes the FD and uses it to communicate with the parent process.
Both the parent and child process should take care to close the FDs representing the half of the connection they are not using.
You can do this in rust - the needed system calls are exposed in the libc crate, or by the more rust-friendly nix crate.
I also stumbled across the interprocess crate which provides an abstraction over these low level calls.
These newly created FDs are completely independant of the stdin / stdout / stderr of the process.
If bidirectional communication is required you could use two such pipes (one for each direction) - however in that case it is probably easier to use an anonymous socket, which you create using the socketpair system call. This works in a similar fashion to the pipe, returning a pair of FDs - except that in this case the FDs are bi-directional. Again, libc and nix expose this system call.
Im creating NodeJS application, that creates quite a few child processes. They are started by both spawn and exec (based on lib implementation). Some examples may be GraphicsMagick (gm) for image manipulation or Tesseract (node-tesseract) for OCR. Now I would like to gracefully end my application so I created shutdown hook:
function exitHandler() {
killer.waitForShutdown().then(function(){
logger.logInfo("Exited successfully.");
process.exit();
}).catch(function(err) {
logger.logError(err, "Error during server shutdown.");
process.exit();
});
}
process.on('exit', exitHandler);
process.on('SIGINT', exitHandler);
process.on('SIGTERM', exitHandler);
Exit handling itself works fine, it is waiting well and so on, but there is a catch. All "native" (gm, tesseract, ...) processes that run at that time are also killed. Exception messages only consists of "Command failed" and then content of command which failed e.g.
"Command failed: /bin/sh -c tesseract tempfiles/W1KwFSdz7MKdJQQnUifQFKdfTRDvBF4VkdJgEvxZGITng7JZWcyPYw6imrw8JFVv/ocr_0.png /tmp/node-tesseract-49073e55-0ef6-482d-8e73-1d70161ce91a -l eng -psm 3\nTesseract Open Source OCR Engine v3.03 with Leptonica"
So at least for me, they do not tell anything useful. I'm also queuing process execution, so PC don't get overloaded by 50 processes at one time. When running processes are killed by SIGINT, new processes that were queued are started just fine and finishes successfully. I have problem only with those few running at the time of receiving SIGINT. This behavior is same on Linux (Debian 8) and Windows (W10). From what I read here, people usually have opposite problem (to kill child processes). I tried to search if stdin gets somehow piped into child processes but I can't find it. So is this how its supposed to work? Is there any trick to prevent this behavior?
The reason this happens is because, by default, the detached option is set to false. If detached is false, the signals will also be sent to the child processes, regardless of whether you setup an event listener.
To stop this happening, you need to change your spawn calls to use the third argument in order to specify detached; for example:
spawn('ls', ['-l'], { detached: true })
From the Node documentation:
On Windows, setting options.detached to true makes it possible for the
child process to continue running after the parent exits. The child
will have its own console window. Once enabled for a child process, it
cannot be disabled.
On non-Windows platforms, if options.detached is set to true, the
child process will be made the leader of a new process group and
session. Note that child processes may continue running after the
parent exits regardless of whether they are detached or not. See
setsid(2) for more information.
I am trying to search information if inter process communication can happen with tcl threads. I am biginner on this topic so right now just collecting information. I understand that sender and receiver mechanism to be coded to pass data between processess. And tcl thread package provides send command. Also thread can be used as timer for spawn process inside the same.
Is it possible to recieve data from thread to another thread?
Thanking you.
#contains of test.tcl
puts stdout "hello from wish"
# end of file
# set cmd
set exe {wish85.exe}
set exepath [list $exe test.tcl]
# This next line is slightly magical
set f [open |$exepath r+]
# Use the next line or you'll regret it!
puts $f {fconfigure stdout -buffering line}
fileevent $f readable "getline $f"
proc getline f {
if {[gets $f line]<0} {
close $f ;
return ;
}
puts "line=$line"
}
You need to be much clearer in your mind about what you are looking for. Threads are not processes! With Tcl, every Tcl interpreter context (the thing you make commands and variables in) is bound to a single thread, and every thread is coupled to a single process.
Tcl has a Thread package for managing threads (it should be shipped with any proper distribution of Tcl 8.6) and that provides a mechanism for sending messages between threads, thread::send. Those messages? They're executable scripts, which means that they are really flexible.
For communication between processes, things are much more complicated because you have to consider both discovery of the other processes and security (because processes are a security boundary by design). Here are some of the options:
Tcl is very good at running subprocesses and talking with them via pipes. For example, you can run a subordinate interpreter in just a couple of lines using open:
# This next line is slightly magical
set mypipeline [open |[list [info nameofexecutable]] r+]
# Use the next line or you'll regret it!
puts $mypipeline {fconfigure stdout -buffering line}
It even works well with the fileevent command, so you can do asynchronous processing within each interpreter. (That's really quite uncommon in language runtimes, alas.)
The send command in Tk lets you send scripts to other processes using the same display (I'm not sure if this works on Windows) much as thread::send does with threads in the same process.
The comm package in Tcllib does something very similar, but uses general sockets as a communication fabric.
On Windows, you can use the dde command in Tcl to communicate with other processes. I don't think Tcl registers a DDE server by default, but it's pretty easy to do (provided you are running the event loop, but that's a common requirement for most of the IPC mechanisms to work at their best).
More generally, you can think in terms of running webservices and so on, but that's getting quite complicated!
I'm trying to read the output of a Python script launched by Node.js as it arrives. However, I only get access to the data once the process has finished.
var proc, args;
args = [
'./bin/build_map.py',
'--min_lon',
opts.sw.lng,
'--max_lon',
opts.ne.lng,
'--min_lat',
opts.sw.lat,
'--max_lat',
opts.ne.lat,
'--city',
opts.city
];
proc = spawn('python', args);
proc.stdout.on('data', function (buf) {
console.log(buf.toString());
socket.emit('map-creation-response', buf.toString());
});
If I launch the process with { stdio : 'inherit' } I can see the output as it happens directly in the console. But doing something like process.stdout.on('data', ...) will not work.
How do I make sure I can read the output from the child process as it arrives and direct it somewhere else?
The process doing the buffering, because it knows the terminal was redirected and not really going to the terminal, is python. You can easily tell Python not to do this buffering: Just run "python -u" instead of "python". Should be easy as that.
When a process is spawned by child_process.spawn(), the streams connected to the child process's standard output and standard error are actually unbuffered on the Nodejs side. To illustrate this, consider the following program:
const spawn = require('child_process').spawn;
var proc = spawn('bash', [
'-c',
'for i in $(seq 1 80); do echo -n .; sleep 1; done'
]);
proc.stdout
.on('data', function (b) {
process.stdout.write(b);
})
.on('close', function () {
process.stdout.write("\n");
});
This program runs bash and has it emit . characters every second for 80 seconds, while consuming this child process's standard output via data events. You should notice that the dots are emitted by the Node program every second, helping to confirm that buffering does not occur on the Nodejs side.
Also, as explained in the Nodejs documentation on child_process:
By default, pipes for stdin, stdout and stderr are established between
the parent Node.js process and the spawned child. It is possible to
stream data through these pipes in a non-blocking way. Note, however,
that some programs use line-buffered I/O internally. While that does
not affect Node.js, it can mean that data sent to the child process
may not be immediately consumed.
You may want to confirm that your Python program does not buffer its output. If you feel you're emitting data from your Python program as separate distinct writes to standard output, consider running sys.stdout.flush() following each write to suggest that Python should actually write data instead of trying to buffer it.
Update: In this commit that passage from the Nodejs documentation was removed for the following reason:
doc: remove confusing note about child process stdio
It’s not obvious what the paragraph is supposed to say. In particular,
whether and what kind of buffering mechanism a process uses for its
stdio streams does not affect that, in general, no guarantees can be
made about when it consumes data that was sent to it.
This suggests that there could be buffering at play before the Nodejs process receives data. In spite of this, care should be taken to ensure that processes within your control upstream of Nodejs are not buffering their output.
Fork()-ing a process will end up calling do_fork() inside kernel, making an exact copy of itself. When I read through books, it says that child of fork will call exec to create the new process.
example:
ls command on a shell, will create this way.
sh(Parent)
|
sh(Child)
|
ls(New Process)
But, I am not able to understand how & where the exec*() is called?
Because, All I can see is the shell(child) is what created in fork.
But, when and where will the new process be created/executed?
You have to exec() if you actually want a new program running in one of the processes (usually the child but not absolutely necessary). In your specific case where the shell executes ls, the shell first forks, then the child process execs. But it's important to realise that this is two distinct operations.
All fork() does is give you two (nearly) identical processes and you can then use the return code from fork() to decide if you're the parent (you get the positive PID of the child, or -1 if the fork() failed) or child (you get 0).
See this answer for a description on how fork() and exec() work together (under your control) and how they can be used without each other.
Similar to do_fork(), the exec stuff all boils down to calls to do_execve, located in exec.c.