I have main process which save stdout, and nth amount of child processes with stdin.
I need to share stdout of main process, with other child processes, and append all of them to hashmap
My code:
let vars: Vec<String> = Vec::from([String::from("foo"), String::from("bar")]);
let mut poll: HashMap<String, Child> = HashMap::new();
let mut t_proc: Child;
let main_proc = Command::new("Main").stdout(Stdio::piped()).spawn().expect("Failed to execute Main");
let pipe = Stdio::from(main_proc.stdout.unwrap());
for var in vars.into_iter(){
t_proc = Command::new("Child").args(var)
.stdin(pipe)
.stdout(Stdio::null()).spawn().expect("Failed to execute Child");
poll.insert(var, t_proc);
}
Related
I'm trying to use std::process::Command to run a command and stream its stdout and stderr while also capturing a copy of stdout/stderr. I found I can use spawn.
This code will capture the output, but won't stream it to stdout/stderr while it's happening:
let mut child = command
.envs(env)
.stdout(Stdio::piped()) // <=== Difference here
.spawn()
.unwrap();
let output = child
.wait_with_output().unwrap();
println!("Done {}", std::str::from_utf8(&output.stdout).unwrap());
This code will stream the output but not capture it:
let mut child = command
.envs(env)
.spawn()
.unwrap();
let output = child
.wait_with_output().unwrap();
println!("Done {}", std::str::from_utf8(&output.stdout).unwrap());
Is there a way to capture a command's output while also streaming it to the parent stdout/stderr?
There might be a less verbose way to do this, but this is the solution I came up with.
Spawn the process with a piped io for stdout and stderr. Spawn a thread for stdout and stderr. In each thread read from the pipe and output directly to stdout or stderr then write the contents to a channel.
In the main thread wait for the process to finish, then join the threads and finally read each channel to get the contents of stdout and stderr.
use std::io::BufRead;
let mut child = command
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let child_stdout = child
.stdout
.take()
.expect("Internal error, could not take stdout");
let child_stderr = child
.stderr
.take()
.expect("Internal error, could not take stderr");
let (stdout_tx, stdout_rx) = std::sync::mpsc::channel();
let (stderr_tx, stderr_rx) = std::sync::mpsc::channel();
let stdout_thread = thread::spawn(move || {
let stdout_lines = BufReader::new(child_stdout).lines();
for line in stdout_lines {
let line = line.unwrap();
println!("{}", line);
stdout_tx.send(line).unwrap();
}
});
let stderr_thread = thread::spawn(move || {
let stderr_lines = BufReader::new(child_stderr).lines();
for line in stderr_lines {
let line = line.unwrap();
eprintln!("{}", line);
stderr_tx.send(line).unwrap();
}
});
let status = child
.wait()
.expect("Internal error, failed to wait on child");
stdout_thread.join().unwrap();
stderr_thread.join().unwrap();
let stdout = stdout_rx.into_iter().collect::<Vec<String>>().join("");
let stderr = stderr_rx.into_iter().collect::<Vec<String>>().join("");
The channel isn't strictly needed. I originally wanted to mutate a string, but I'm new in Rust with threads and couldn't find any examples showing how to mutate a string in a thread and then read it back into main.
I'm accepting the other solution as it really answered my main question. I just wanted to post back to give everyone a fully-featured answer that does exactly what I originally asked
This is similar to how I stream the compilation and execution output on Rust Explorer.
To stream the output you can pipe the stdout and read it line by line using BufReader.
Playground
use std::io::BufRead;
use std::io::BufReader;
use std::process::Command;
use std::process::Stdio;
fn main() {
// Compile code.
let mut child = Command::new("bash")
.args([
"-c",
"echo 'Hello'; sleep 3s; echo 'World'"
])
.stdout(Stdio::piped())
.spawn()
.unwrap();
let stdout = child.stdout.take().unwrap();
// Stream output.
let lines = BufReader::new(stdout).lines();
for line in lines {
println!("{}", line.unwrap());
}
}
I'm writing a function that has a particular logging requirement. I want to capture the output from a Command::new() call and save it into a file. I'm just using echo here for simplicity's sake.
fn sys_command(id: &str) -> u64 {
let mut cmd = Command::new("echo")
.args(&[id])
.stdout(Stdio::piped())
.spawn()
.expect("failed to echo");
let stdout = cmd.stdout.as_mut().unwrap();
let stdout_reader = BufReader::new(stdout);
// let log_name = format!("./tmp/log/{}.log", id);
// fs::write(log_name, &stdout_reader);
println!("{:?}", stdout_reader.buffer());
cmd.wait().expect("failed to call");
id.parse::<u64>().unwrap()
}
How can I capture the output and save it to a file? I've made a playground here. My println! call returns [].
Even reading to another buffer prints the wrong value. Below, sys_command("10") prints 3. Here's an updated playground.
fn sys_command(id: &str) -> u64 {
let mut buffer = String::new();
let mut cmd = Command::new("echo")
.args(&[id])
.stdout(Stdio::piped())
.spawn()
.expect("failed to echo");
let stdout = cmd.stdout.as_mut().unwrap();
let mut stdout_reader = BufReader::new(stdout);
let result = stdout_reader.read_to_string(&mut buffer);
println!("{:?}", result.unwrap());
cmd.wait().expect("failed to draw");
id.parse::<u64>().unwrap()
}
What am I missing?
Instead of capturing the output in memory and then writing it to a file, you should just redirect the process's output to a file. This is simpler and more efficient.
let log_name = format!("./tmp/log/{}.log", id);
let log = File::create(log_name).expect("failed to open log");
let mut cmd = Command::new("echo")
.args(&[id])
.stdout(log)
.spawn()
.expect("failed to start echo");
cmd.wait().expect("failed to finish echo");
Even reading to another buffer prints the wrong value. Below, sys_command("10") prints 3.
This is unrelated — read_to_string() returns the number of bytes read, not the contents of them, and there are 3 characters: '1' '0' '\n'.
I'll try to simplify as much as possible what I'm trying to do accomplish but in a nutshell here is my problem:
I am trying to spawn the node shell as a process in Rust. I would like to pass to the process' stdin javascript code and read the nodejs output from stdout of the process. This would be an interactive usage where the node shell is spawned and keeps receiving JS instructions and executing them.
I do not wish to launch the nodejs app using a file argument.
I have read quite a bit about std::process::Command, tokio and why we can't write and read to a piped input using standard library. One of the solutions that I kept seeing online (in order to not block the main thread while reading/writing) is to use a thread for reading the output. Most solutions did not involve a continuous write/read flow.
What I have done is to spawn 2 threads, one that keeps writing to stdin and one that keeps reading from stdout. That way, I thought, I won't be blocking the main thread. However my issue is that only 1 thread can actively be used. When I have a thread for stdin, stdout does not even receive data.
Here is the code, comments should provide more details
pub struct Runner {
handle: Child,
pub input: Arc<Mutex<String>>,
pub output: Arc<Mutex<String>>,
input_thread: JoinHandle<()>,
output_thread: JoinHandle<()>,
}
impl Runner {
pub fn new() -> Runner {
let mut handle = Command::new("node")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.expect("Failed to spawn node process!");
// begin stdout thread part
let mut stdout = handle.stdout.take().unwrap();
let output = Arc::new(Mutex::new(String::new()));
let out_clone = Arc::clone(&output);
let output_thread = spawn(move || loop {
// code here never executes...why ?
let mut buf: [u8; 1] = [0];
let mut output = out_clone.lock().unwrap();
let what_i_read = stdout.read(&mut buf);
println!("reading: {:?}", what_i_read);
match what_i_read {
Err(err) => {
println!("{}] Error reading from stream: {}", line!(), err);
break;
}
Ok(bytes_read) => {
if bytes_read != 0 {
let char = String::from_utf8(buf.to_vec()).unwrap();
output.push_str(char.as_str());
} else if output.len() != 0 {
println!("result: {}", output);
out_clone.lock().unwrap().clear();
}
}
}
});
// begin stdin thread block
let mut stdin = handle.stdin.take().unwrap();
let input = Arc::new(Mutex::new(String::new()));
let input_clone = Arc::clone(&input);
let input_thread = spawn(move || loop {
let mut in_text = input_clone.lock().unwrap();
if in_text.len() != 0 {
println!("writing: {}", in_text);
stdin.write_all(in_text.as_bytes()).expect("!write");
stdin.write_all("\n".as_bytes()).expect("!write");
in_text.clear();
}
});
Runner {
handle,
input,
output,
input_thread,
output_thread,
}
}
// this function should receive commands
pub fn execute(&mut self, str: &str) {
let input = Arc::clone(&self.input);
let mut input = input.lock().unwrap();
input.push_str(str);
}
}
In the main thread I'd like use this as
let mut runner = Runner::new();
runner.execute("console.log('foo'");
println!("{:?}", runner.output);
I am still new to Rust but at least I passed the point where the borrow checker makes me bang my head against the wall, I am starting to find it more pleasing now :)
I have been trying to write an application that invokes a system command in a separate thread.
The key characteristic is that I want to be able to kill this command and invoke a new one from the main thread upon request.
My secondary thread looks like so:
let (tx, rx): (mpsc::Sender<String>, mpsc::Receiver<String>) = mpsc::channel();
let child_handle = stoppable_thread::spawn(move |stop| {
let mut child = Command::new("[the program]").arg(format!("-g {}", gain)).stdout(Stdio::piped()).spawn().expect("naw man");
let mut childout = child.stdout.as_mut().unwrap();
while !stop.get() {
let mut buffer = [0; 128];
childout.try_read(&mut buffer).unwrap(); // This part makes the code wait for the next output
// Here the buffer is sent via mpsc irrelevant to the issue
}});
The problem is when I send a stop signal (or if I used an mpsc channel to notify the thread to stop) it waits for the command to output something to stdout. This is unwanted behavior.
How can I remedy this? How can I interrupt the read() function?
You can kill the child process, which will cause it to close its output, at which point read will see an EOF and return. However, this requires sending the child to the parent thread. Something like:
let (tx, rx): (mpsc::Sender<String>, mpsc::Receiver<String>) = mpsc::channel();
let (ctx, crx) = mpsc::channel();
let child_handle = stoppable_thread::spawn(move |stop| {
let mut child = Command::new("[the program]").arg(format!("-g {}", gain)).stdout(Stdio::piped()).spawn().expect("naw man");
let mut childout = child.stdout.take().unwrap();
ctx.send (child);
while !stop.get() {
let mut buffer = [0; 128];
childout.try_read(&mut buffer).unwrap(); // This part makes the code wait for the next output
// Here the buffer is sent via mpsc irrelevant to the issue
}});
// When you want to stop:
let child = crx.recv().unwrap();
child.kill().unwrap();
I have a long-running child process to which I need to read and write a lot of data. I have a reader thread and a writer thread that manipulate the child.stdout and child.stdin respectively:
extern crate scoped_threadpool;
fn main() {
// run the subprocess
let mut child = std::process::Command::new("cat")
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.spawn()
.unwrap();
let child_stdout = child.stdout.as_mut().unwrap();
let child_stdin = std::sync::Mutex::new(child.stdin.as_mut().unwrap());
let mut pool = scoped_threadpool::Pool::new(2);
pool.scoped(|scope| {
// read all output from the subprocess
scope.execute(move || {
use std::io::BufRead;
let reader = std::io::BufReader::new(child_stdout);
for line in reader.lines() {
println!("{}", line.unwrap());
}
});
// write to the subprocess
scope.execute(move || {
for a in 0..1000 {
use std::io::Write;
writeln!(&mut child_stdin.lock().unwrap(), "{}", a).unwrap();
} // close child_stdin???
});
});
}
When the writer is done, I want to close child_stdin so that the subprocess finishes and exits, so that the reader sees EOF and pool.scoped returns. I can't do this without child.wait() and I can't call child.wait() because it's being borrowed by the two threads.
How do I make this program complete?
Amusingly, you've caused this yourself by sharing ownership using the Mutex ^_^. Instead of taking a reference to child.stdin, take complete ownership of it and pass it to the thread. When the thread is over, it will be dropped, closing it implicitly:
let mut child_stdin = child.stdin.unwrap();
// ...
scope.execute(move ||
for a in 0..1000 {
use std::io::Write;
writeln!(&mut child_stdin, "{}", a).unwrap();
}
// child_stdin has been moved into this closure and is now
// dropped, closing it.
);
If you'd like to still be able to call wait to get the ExitStatus, change the reference to stdout and the transfer of ownership of stdin to use Option::take. This means that child is not borrowed at all:
let mut child = // ...
let child_stdout = child.stdout.as_mut().unwrap();
let mut child_stdin = child.stdin.take().unwrap();
// ...
child.wait().unwrap();