I want to create a rust program that feeds in from an external program via pipe and spits to another external program via pipe, like a sandwich. To be more specific, a toy model I'm trying to create is "gzip -cd input | rust | gzip -c - > output".
Basically, I know how to do the first part (pipe in), by something like:
let child = match Command::new("zcat")
.args(&["input"])
.stdout(Stdio::piped())
.stderr(Stdio::null())
.spawn();
let filehand = BufReader::new(child.stdout.unwrap());
for line in filehand.lines() {
...
}
But I'm stuck in the second part, nor do I know how to piece both together. I know Perl can do it in an elegant manner:
open filehand_in, "gzip -cd input |";
open filehand_out, "| gzip -c - > output";
while ( <filehand_in> ) {
# processing;
print filehand_out ...;
}
By elegant I mean the external processing to in and out are transparent to the main program: you just need to treat the first pipe as stdin and the second pipe as stdout, nothing more to worry about. I wonder if anybody knows how to implement similarly in Rust, thanks.
You can do it like this:
use std::io::{BufRead, BufReader, Write};
use std::process::{Command, Stdio};
pub fn main() {
let mut left_child = Command::new("/bin/ls")
.stdout(Stdio::piped())
.spawn()
.expect("failed to execute child");
let mut right_child = Command::new("/bin/cat")
.stdin(Stdio::piped())
.spawn()
.expect("failed to execute child");
// extra scope to ensure that left_in and right_out are closed after
// copying all available data
{
let left_in = BufReader::new(left_child.stdout.take().unwrap());
let mut right_out = right_child.stdin.take().unwrap();
for line in left_in.lines() {
writeln!(&mut right_out, "{}", line.unwrap()).unwrap();
}
}
let left_ecode = left_child.wait().expect("failed to wait on child");
let right_ecode = right_child.wait().expect("failed to wait on child");
assert!(left_ecode.success());
assert!(right_ecode.success());
}
Related
I want to communicate with dart language sever process. Currently i'm experimenting with it and i don't understand one think. I made 2 very simple implementations which in my opinion should work exactly the same but one of them works and second doesn't. Could someone describe me why?
First implementation that i tried is:
use std::{process::Stdio, time::Duration};
use tokio::{io::AsyncWriteExt, process::Command, time::sleep};
#[tokio::main]
async fn main() {
let mut process = Command::new("dart")
.args([
"language-server",
"--protocol=analyzer",
"--protocol-traffic-log=logs.txt",
])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true)
.spawn()
.expect("failed to spawn process");
process
.stdin
.take()
.unwrap()
.write_all(b"test")
.await
.unwrap();
sleep(Duration::from_secs(5)).await;
}
And the log.txt look like this (I know input is invalid but important thing is that server received message):
1673866099379:Ver:1645600465579221612245:unknown.client.classic:unknown:1.33.1:2.18.0
1673866099410:Noti:{"event"::"server.connected","params"::{"version"::"1.33.1","pid"::16810}}
1673866099421:Req:test
1673866099422:Res:{"id"::"","error"::{"code"::"INVALID_REQUEST","message"::"Invalid request"}}
Second implementation look like this. The only difference is that I store process.stdin.take().unwrap() in variable.
use std::{process::Stdio, time::Duration};
use tokio::{io::AsyncWriteExt, process::Command, time::sleep};
#[tokio::main]
async fn main() {
let mut process = Command::new("dart")
.args([
"language-server",
"--protocol=analyzer",
"--protocol-traffic-log=logs.txt",
])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.kill_on_drop(true)
.spawn()
.expect("failed to spawn process");
let mut stdin = process.stdin.take().unwrap();
stdin.write_all(b"Test").await.unwrap();
sleep(Duration::from_secs(5)).await;
}
log.txt
1673866343914:Ver:1645600465579221612245:unknown.client.classic:unknown:1.33.1:2.18.0
1673866343942:Noti:{"event"::"server.connected","params"::{"version"::"1.33.1","pid"::16966}}
I'm trying to use std::process::Command to run a command and stream its stdout and stderr while also capturing a copy of stdout/stderr. I found I can use spawn.
This code will capture the output, but won't stream it to stdout/stderr while it's happening:
let mut child = command
.envs(env)
.stdout(Stdio::piped()) // <=== Difference here
.spawn()
.unwrap();
let output = child
.wait_with_output().unwrap();
println!("Done {}", std::str::from_utf8(&output.stdout).unwrap());
This code will stream the output but not capture it:
let mut child = command
.envs(env)
.spawn()
.unwrap();
let output = child
.wait_with_output().unwrap();
println!("Done {}", std::str::from_utf8(&output.stdout).unwrap());
Is there a way to capture a command's output while also streaming it to the parent stdout/stderr?
There might be a less verbose way to do this, but this is the solution I came up with.
Spawn the process with a piped io for stdout and stderr. Spawn a thread for stdout and stderr. In each thread read from the pipe and output directly to stdout or stderr then write the contents to a channel.
In the main thread wait for the process to finish, then join the threads and finally read each channel to get the contents of stdout and stderr.
use std::io::BufRead;
let mut child = command
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let child_stdout = child
.stdout
.take()
.expect("Internal error, could not take stdout");
let child_stderr = child
.stderr
.take()
.expect("Internal error, could not take stderr");
let (stdout_tx, stdout_rx) = std::sync::mpsc::channel();
let (stderr_tx, stderr_rx) = std::sync::mpsc::channel();
let stdout_thread = thread::spawn(move || {
let stdout_lines = BufReader::new(child_stdout).lines();
for line in stdout_lines {
let line = line.unwrap();
println!("{}", line);
stdout_tx.send(line).unwrap();
}
});
let stderr_thread = thread::spawn(move || {
let stderr_lines = BufReader::new(child_stderr).lines();
for line in stderr_lines {
let line = line.unwrap();
eprintln!("{}", line);
stderr_tx.send(line).unwrap();
}
});
let status = child
.wait()
.expect("Internal error, failed to wait on child");
stdout_thread.join().unwrap();
stderr_thread.join().unwrap();
let stdout = stdout_rx.into_iter().collect::<Vec<String>>().join("");
let stderr = stderr_rx.into_iter().collect::<Vec<String>>().join("");
The channel isn't strictly needed. I originally wanted to mutate a string, but I'm new in Rust with threads and couldn't find any examples showing how to mutate a string in a thread and then read it back into main.
I'm accepting the other solution as it really answered my main question. I just wanted to post back to give everyone a fully-featured answer that does exactly what I originally asked
This is similar to how I stream the compilation and execution output on Rust Explorer.
To stream the output you can pipe the stdout and read it line by line using BufReader.
Playground
use std::io::BufRead;
use std::io::BufReader;
use std::process::Command;
use std::process::Stdio;
fn main() {
// Compile code.
let mut child = Command::new("bash")
.args([
"-c",
"echo 'Hello'; sleep 3s; echo 'World'"
])
.stdout(Stdio::piped())
.spawn()
.unwrap();
let stdout = child.stdout.take().unwrap();
// Stream output.
let lines = BufReader::new(stdout).lines();
for line in lines {
println!("{}", line.unwrap());
}
}
I'd like to capture stdout/stderr of a subprocess but also stream the outputs to my process' stdout/stderr as well. In bash I'd do something like:
my_command > >(tee /tmp/capture.out) 2> >(tee /tmp/capture.err >&2)
Is there any reasonably concise or typical way to do this in Rust? Looking at std::process::Child.wait_with_output() and sys::pipe::read2 handling stdout/stderr directly seems somewhat involved (and involves an unsafe call to poll).
Another way of framing the question might be "what's the expected way to stream over a subprocess' stdout and stderr?" which would at least allow me to manually capture and print the streams' contents.
I'm currently using std::process but if it's easier with the subprocess crate I'd be willing to swap to that.
Something along those lines?
use std::{
fs::File,
io::{stderr, stdout, Read, Write},
process::{Command, Stdio},
thread,
};
fn main() {
let mut child = Command::new("/usr/bin/date")
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.expect("failed to execute child");
fn communicate(
mut stream: impl Read,
filename: &'static str,
mut output: impl Write,
) -> std::io::Result<()> {
let mut file = File::create(filename)?;
let mut buf = [0u8; 1024];
loop {
let num_read = stream.read(&mut buf)?;
if num_read == 0 {
break;
}
let buf = &buf[..num_read];
file.write_all(buf)?;
output.write_all(buf)?;
}
Ok(())
}
let child_out = std::mem::take(&mut child.stdout).expect("cannot attach to child stdout");
let child_err = std::mem::take(&mut child.stderr).expect("cannot attach to child stderr");
let thread_out = thread::spawn(move || {
communicate(child_out, "stdout.txt", stdout())
.expect("error communicating with child stdout")
});
let thread_err = thread::spawn(move || {
communicate(child_err, "stderr.txt", stderr())
.expect("error communicating with child stderr")
});
thread_out.join().unwrap();
thread_err.join().unwrap();
let ecode = child.wait().expect("failed to wait on child");
assert!(ecode.success());
}
Output:
Tue Jul 5 01:07:01 CEST 2022
stdout.txt:
Tue Jul 5 01:07:01 CEST 2022
handling stdout/stderr directly seems somewhat involved (and involves an unsafe call to poll
It's probably going to be involved either way as Rust doesn't really have abstractions over complicated pipe & fd operations and this requires a fair number of them: creating two pipes and doing a lot of dup'ing around to properly set all the input and output streams to match.
The "easy" way to do this (it's not actually easy but it avoids having to deal with libc and fds) would be to spawn threads to explicitly copy data around between the various bits.
I know how to read the command line arguments, but I am having difficulties reading the command output from a pipe.
Connect a program (A) that outputs data to my Rust program using a pipe:
A | R
The program should consume the data line by line as they come.
$ pwd | cargo run should print the pwd output.
OR
$ find . | cargo run should output the find command output which is more than 1 line.
Use BufRead::lines on a locked handle to standard input:
use std::io::{self, BufRead};
fn main() {
let stdin = io::stdin();
for line in stdin.lock().lines() {
let line = line.expect("Could not read line from standard in");
println!("{}", line);
}
}
If you wanted to reuse the allocation of the String, you could use the loop form:
use std::io::{self, Read};
fn main() {
let stdin = io::stdin();
let mut stdin = stdin.lock(); // locking is optional
let mut line = String::new();
// Could also `match` on the `Result` if you wanted to handle `Err`
while let Ok(n_bytes) = stdin.read_to_string(&mut line) {
if n_bytes == 0 { break }
println!("{}", line);
line.clear();
}
}
You just need to read from Stdin.
This is based on an example taken from the documentation:
use std::io;
fn main() {
loop {
let mut input = String::new();
match io::stdin().read_line(&mut input) {
Ok(len) => if len == 0 {
return;
} else {
println!("{}", input);
}
Err(error) => {
eprintln!("error: {}", error);
return;
}
}
}
}
It's mostly the docs example wrapped in a loop, breaking out of the loop when there is no more input, or if there is an error.
The other changes is that it's better in your context to write errors to stderr, which is why the error branch uses eprintln!, instead of println!. This macro probably wasn't available when that documentation was written.
use std::io;
fn main() {
loop {
let mut input = String::new();
io::stdin()
.read_line(&mut input)
.expect("failed to read from pipe");
input = input.trim().to_string();
if input == "" {
break;
}
println!("Pipe output: {}", input);
}
}
OUTPUT:
[18:50:29 Abhinickz#wsl -> pipe$ pwd
/mnt/d/Abhinickz/dev_work/learn_rust/pipe
[18:50:46 Abhinickz#wsl -> pipe$ pwd | cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running `target/debug/pipe`
Pipe output: /mnt/d/Abhinickz/dev_work/learn_rust/pipe
You can do it in a pretty snazzy and concise way with rust's iterator methods
use std::io::{self, BufRead};
fn main() {
// get piped input
// eg `cat file | ./program`
// ( `cat file | cargo run` also works )
let input = io::stdin().lock().lines().fold("".to_string(), |acc, line| {
acc + &line.unwrap() + "\n"
});
dbg!(input);
}
I'm an absolute Rust beginner trying to build a simple confirmation function (yes or no), but I can't get the user to type anything, the function just keeps looping without waiting for user input:
""
""
""
etc.
is the result of the simplified version below.
use std::process;
use std::io;
pub fn confirm() {
loop {
let mut answer = String::new();
io::stdin().read_line(&mut answer)
.ok()
.expect("Failed to read line");
println!("{:?}", answer);
}
}
I've built my function around the guessing game example, and the rest of my program does nothing much, just reading a file and printing text.
Perhaps is due to the way my program (a git hook) is launched?
Assuming that the problem is that your git commit hook is running in an non-interactive environment, you can follow the advice laid out in that question and directly open /dev/tty. Unlike STDIN, we don't treat it as a magical global variable and instead we pass it into the places we need:
use std::io::{self, BufRead, BufReader};
use std::fs::File;
type Tty = BufReader<File>;
fn open_tty() -> io::Result<Tty> {
let f = try!(File::open("/dev/tty"));
Ok(BufReader::new(f))
}
fn confirm(tty: &mut Tty) -> io::Result<String> {
let mut answer = String::new();
try!(tty.read_line(&mut answer));
Ok(answer)
}
fn inner_main() -> io::Result<()> {
let mut tty = try!(open_tty());
let answer = try!(confirm(&mut tty));
println!("The answer was: {}", answer);
Ok(())
}
fn main() {
inner_main().unwrap()
}
Note that this will not be platform independent. Specifically, this is very unlikely to work on Windows!
I've also gone ahead and allowed the io::Result to propagate throughout the program, only panicking at the outermost shell.
Are you testing the function on the Rust Playground? Running this program in a terminal seems to work fine. That being said, there is no guarantee that stdin will block, but you could change the function to check if the string is empty or not, and only return once it is isn't.
use std::io;
fn main() {
println!("{:?}", confirm());
}
fn confirm() -> String {
loop {
let mut answer = String::new();
io::stdin().read_line(&mut answer)
.ok()
.expect("Failed to read line");
if !answer.is_empty() && answer != "\n" && answer != "\r\n" {
return answer
}
}
}