For some reason, I am unable to read from the child process stderr the second time. Here is what I do.
I am spawning a child process for cucumber tests. In the first step I spawn the process, take its stderr, save it, and then read from it. Here is the code:
pub fn wait_process_output(
reader: &mut BufReader<ChildStderr>,
output: Vec<(String, u16)>,
) -> Result<(), String> {
let mut process_output = String::new();
loop {
match reader.read_line(&mut process_output) {
Err(e) => {
return Err(format!("Unable to read output: {}", e));
}
Ok(_) => {
// processing here
}
};
}
}
pub fn step1(world: &mut TestWorld) {
world.app_handler = Some(
Command::new(&world.app_bin)
.stderr(Stdio::piped())
.spawn()
.unwrap(),
);
let app_stderr = world.app_handler.as_mut().unwrap().stderr.take().unwrap();
world.app_reader = Some(BufReader::new(app_stderr));
wait_process_output(world.app_reader.as_mut().unwrap(), /* expected data */).ok();
}
This code works correctly: stderr is being read as expected.
In the third test step, I try to read process output one more time:
pub fn step3(world: &mut TestWorld) {
wait_process_output(world.app_reader.as_mut().unwrap(), /* expected data */).ok();
}
This time reader.read_line hangs infinitely: nothing is being read. I am sure the child process produces some output: I can see it if I run it within the same conditions separately.
Could you please suggest any ideas why the BufReader object becomes corrupted when I try to read from it the second time?
I got to the solution. The issue was that app produced an output earlier than step3 started to read it. I thought that there is some kind of buffer implemented for this kind of situation but it seems I was wrong. So I finally used the following two methods to solve my issue:
pub fn wait_process_output(
receiver: &Receiver<String>,
output: Vec<(String, u16)>,
) -> Result<(), String> {
loop {
match receiver.try_recv() {
// process output
}
}
}
pub fn start_listener(sender: Sender<String>, stream: ChildStderr) {
spawn(move || {
let mut f = BufReader::new(stream);
loop {
let mut buf = String::new();
match f.read_line(&mut buf) {
Ok(_) => {
sender.send(buf).unwrap();
continue;
}
Err(e) => {
println!("Unable to read process stderr: {:?}", e);
break;
}
}
}
});
}
Related
I am working on a program using rust-tokio for asynchronous execution. The main function periodically calls a function to append to a CSV file to log operation over time.
I would like to make the CSV creation function asynchronous and run it as a separate task so I can continue the main function if CSV creation is taking some time (like waiting for another application like Excel to release it).
Is there an elegant way to accomplish this?
LocalSet almost seems like it would do the job, but the tasks need to execute in order so the CSV is chronological. To me, the documentation doesn't seem to guarantee this.
Here's some pseudo code to illustrate the idea. Essentially, I'm thinking a queue of tasks that need to be completed.
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let local = task::LocalSetOrdered::new(); //This is a fictitious struct
let mut data: usize = 10; //For simplicity, just store a single number
loop {
// Some operations here
data = data + 1;
let data_clone = data.clone();
//Add a new task to complete after all prior tasks
local.push(async move {
match append_to_csv(data_clone).await {
Ok(_) => Ok(()),
Err(_) => Err(()),
}
});
sleep(Duration::from_secs(60)).await;
}
Ok(())
}
async fn append_to_csv(data_in: usize) -> Result<(), Box<dyn Error>> {
loop {
let file = match OpenOptions::new().write(true).append(true).open(filename) {
Ok(f) => f,
Err(_) => {
//Error opening the file, try again
sleep(Duration::from_secs(60)).await;
continue;
}
};
let wtr = csv::Writer::from_writer(file);
let date_time = Utc::now();
wtr.write_field(format!("{}", date_time.format("%Y/%m/%d %H:%M")))?;
wtr.write_field(format!("{}", data_in))?;
wtr.write_record(None::<&[u8]>)?; //Finish the line
}
}
You could use a worker task to write to the csv file, and a channel to pass data to be written
use tokio::sync::mpsc::{channel, Receiver};
#[derive(Debug)]
pub struct CsvData(i32, &'static str);
async fn append_to_csv(mut rx: Receiver<CsvData>) {
let mut wtr = csv::Writer::from_writer(std::io::stdout());
while let Some(data) = rx.recv().await {
wtr.write_record([&data.0.to_string(), data.1]).unwrap();
wtr.flush().unwrap();
}
}
#[tokio::main]
async fn main() {
let (tx, rx) = channel(10);
tokio::spawn(async {
append_to_csv(rx).await;
});
for i in 0.. {
tx.send(CsvData(i, "Hello world")).await.unwrap();
}
}
The channel sender can be cloned if you need to write data sourced from multiple tasks.
This code spawns a child process, consuming its stderr and stdout line by line, and logging each appropriately. It compiles and works.
use std::error::Error;
use std::process::{Stdio};
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::process::{Command, Child};
use tracing::{info, warn};
macro_rules! relay_pipe_lines {
($pipe:expr, $handler:expr) => {
tokio::spawn(async move {
let mut reader = BufReader::new($pipe).lines();
loop {
let line = reader
.next_line()
.await
.unwrap_or_else(|_| Some(String::new()));
match line {
None => break,
Some(line) => $handler(line)
}
}
});
};
}
pub fn start_and_log_command(mut command: Command) -> Result<Child, Box<dyn Error>> {
command.stdout(Stdio::piped()).stderr(Stdio::piped());
let mut child = command.spawn()?;
let child_stdout = child.stdout.take().unwrap(); // remove `take` from here
let child_stderr = child.stderr.take().unwrap(); // .. or from here and it fails
let child_pid = child.id().unwrap();
relay_pipe_lines!(child_stdout, |line|info!("[pid {}:stdout]: {}", child_pid, line));
relay_pipe_lines!(child_stderr, |line|warn!("[pid {}:stderr]: {}", child_pid, line));
Ok(child)
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
tracing_subscriber::fmt::init();
info!("Tracing logging initialised.");
let mut command = Command::new("ls");
command.arg("-l");
let mut child = start_and_log_command(command)?;
// Compose reading waiting concurrently.
let exit_status = child.wait().await.expect("Cannot reap child process");
dbg!(exit_status.success());
Ok(())
}
Removing the call to take() from the indicated lines fails the build, as "child.stdout partially moved due to this method call", which I mostly understand.
I'd like to understand how using take() does not partially move child.stdout.
It's a call to Option::take(), which avoids a "partial move" by leaving None in place of the moved value. As a result, child is left in a valid state and can be returned from the function.
In other words, child.stdout.take() is equivalent to std::mem::replace(&mut child.stdout, None), and means "take the current value out of the option (whatever it is), and leave None in its place."
I have a Rust program where I spawn a number of threads and store the JoinHandles. When I later join the threads that panic!ed, I'm not sure how to retrieve the message that was passed to panic!
Recovering from `panic!` in another thread covers a simple case, but when I applied it to my code it turns out it doesn't work, so I have updated my example:
If I run the following program
fn main() {
let handle = std::thread::spawn(||panic!(format!("demo {}",1)));
match handle.join() {
Ok(_) => println!("OK"),
Err(b) => println!("Err {:?}", b.downcast_ref::<&'static str>())
}
}
(rust playground)
It prints Err None . I want to access the "demo" message.
Based on the comments, I was able to come up with a working solution (rust playground):
fn main() {
let handle = std::thread::spawn(|| {
panic!("malfunction {}", 7);
if true {
Err(14)
} else {
Ok("bacon".to_string())
}
});
match handle.join() {
Ok(msg) => println!("OK {:?}", msg),
Err(b) => {
let msg = if let Some(msg) = b.downcast_ref::<&'static str>() {
msg.to_string()
} else if let Some(msg) = b.downcast_ref::<String>() {
msg.clone()
} else {
format!("?{:?}", b)
};
println!("{}", msg);
}
}
}
which prints malfunction 7 like I want. While this toy example doesn't need all the branches, a more complex piece of code might have some panic!s that give &'static str and others that give String.
I'm trying to make a Stream that would wait until a specific character is in buffer. I know there's read_until() on BufRead but I actually need a custom solution, as this is a stepping stone to implement waiting until a specific string in in buffer (or, for example, a regexp match happens).
In my project where I first encountered the problem, problem was that future processing just hanged when I get a Ready(_) from inner future and return NotReady from my function. I discovered I shouldn't do that per docs (last paragraph). However, what I didn't get, is what's the actual alternative that is promised in that paragraph. I read all the published documentation on the Tokio site and it doesn't make sense for me at the moment.
So following is my current code. Unfortunately I couldn't make it simpler and smaller as it's already broken. Current result is this:
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
<ad infinum>
Expected result is getting some Ok(Ready(_)) out of it, while printing W and W', and waiting for specific character in buffer.
extern crate futures;
extern crate tokio_core;
extern crate tokio_io;
extern crate tokio_io_timeout;
extern crate tokio_process;
use futures::stream::poll_fn;
use futures::{Async, Poll, Stream};
use tokio_core::reactor::Core;
use tokio_io::AsyncRead;
use tokio_io_timeout::TimeoutReader;
use tokio_process::CommandExt;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct Process {
child: tokio_process::Child,
stdout: Arc<Mutex<tokio_io_timeout::TimeoutReader<tokio_process::ChildStdout>>>,
}
impl Process {
fn new(
command: &str,
reader_timeout: Option<Duration>,
core: &tokio_core::reactor::Core,
) -> Self {
let mut cmd = Command::new(command);
let cat = cmd.stdout(Stdio::piped());
let mut child = cat.spawn_async(&core.handle()).unwrap();
let stdout = child.stdout().take().unwrap();
let mut timeout_reader = TimeoutReader::new(stdout);
timeout_reader.set_timeout(reader_timeout);
let timeout_reader = Arc::new(Mutex::new(timeout_reader));
Self {
child,
stdout: timeout_reader,
}
}
}
fn work() -> Result<(), ()> {
let window = Arc::new(Mutex::new(Vec::new()));
let mut core = Core::new().unwrap();
let process = Process::new("cat", Some(Duration::from_secs(20)), &core);
let mark = Arc::new(Mutex::new(b'c'));
let read_until_stream = poll_fn({
let window = window.clone();
let timeout_reader = process.stdout.clone();
move || -> Poll<Option<u8>, std::io::Error> {
let mut buf = [0; 8];
let poll;
{
let mut timeout_reader = timeout_reader.lock().unwrap();
poll = timeout_reader.poll_read(&mut buf);
}
match poll {
Ok(Async::Ready(0)) => Ok(Async::Ready(None)),
Ok(Async::Ready(x)) => {
{
let mut window = window.lock().unwrap();
println!("W: {:?}", *window);
println!("buf: {:?}", &buf[0..x]);
window.extend(buf[0..x].into_iter().map(|x| *x));
println!("W': {:?}", *window);
if let Some(_) = window.iter().find(|c| **c == *mark.lock().unwrap()) {
Ok(Async::Ready(Some(1)))
} else {
Ok(Async::NotReady)
}
}
}
Ok(Async::NotReady) => Ok(Async::NotReady),
Err(e) => Err(e),
}
}
});
let _stream_thread = thread::spawn(move || {
for o in read_until_stream.wait() {
println!("{:?}", o);
}
});
match core.run(process.child) {
Ok(_) => {}
Err(e) => {
println!("Child error: {:?}", e);
}
}
Ok(())
}
fn main() {
work().unwrap();
}
This is complete example project.
If you need more data you need to call poll_read again until you either find what you were looking for or poll_read returns NotReady.
You might want to avoid looping in one task for too long, so you can build yourself a yield_task function to call instead if poll_read didn't return NotReady; it makes sure your task gets called again ASAP after other pending tasks were run.
To use it just run return yield_task();.
fn yield_inner() {
use futures::task;
task::current().notify();
}
#[inline(always)]
pub fn yield_task<T, E>() -> Poll<T, E> {
yield_inner();
Ok(Async::NotReady)
}
Also see futures-rs#354: Handle long-running, always-ready futures fairly #354.
With the new async/await API futures::task::current is gone; instead you'll need a std::task::Context reference, which is provided as parameter to the new std::future::Future::poll trait method.
If you're already manually implementing the std::future::Future trait you can simply insert:
context.waker().wake_by_ref();
return std::task::Poll::Pending;
Or build yourself a Future-implementing type that yields exactly once:
pub struct Yield {
ready: bool,
}
impl core::future::Future for Yield {
type Output = ();
fn poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::Context<'_>) -> core::task::Poll<Self::Output> {
let this = self.get_mut();
if this.ready {
core::task::Poll::Ready(())
} else {
cx.waker().wake_by_ref();
this.ready = true; // ready next round
core::task::Poll::Pending
}
}
}
pub fn yield_task() -> Yield {
Yield { ready: false }
}
And then use it in async code like this:
yield_task().await;
I'm trying to port this Python script that sends and receives input to a helper process to Rust:
import subprocess
data = chr(0x3f) * 1024 * 4096
child = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output, _ = child.communicate(data)
assert output == data
My attempt worked fine until the input buffer exceeded 64k because presumably the OS's pipe buffer filled up before the input was written.
use std::io::Write;
const DATA: [u8; 1024 * 4096] = [0x3f; 1024 * 4096];
fn main() {
let mut child = std::process::Command::new("cat")
.stdout(std::process::Stdio::piped())
.stdin(std::process::Stdio::piped())
.spawn()
.unwrap();
match child.stdin {
Some(ref mut stdin) => {
match stdin.write_all(&DATA[..]) {
Ok(_size) => {}
Err(err) => panic!(err),
}
}
None => unreachable!(),
}
let res = child.wait_with_output();
assert_eq!(res.unwrap().stdout.len(), DATA.len())
}
Is there a subprocess.communicate equivalent in Rust? Maybe a select equivalent? Can mio be used to solve this problem? Also, there seems to be no way to close stdin.
The goal here is to make a high performance system, so I want to avoid spawning a thread per task.
Well it wasn't a small amount of code to get this done, and I needed a combination of mio and nix, because mio wouldn't set AsRawFd items to be nonblocking when they were pipes, so this had to be done first.
Here's the result
extern crate mio;
extern crate bytes;
use mio::*;
use std::io;
use mio::unix::{PipeReader, PipeWriter};
use std::process::{Command, Stdio};
use std::os::unix::io::AsRawFd;
use nix::fcntl::FcntlArg::F_SETFL;
use nix::fcntl::{fcntl, O_NONBLOCK};
extern crate nix;
struct SubprocessClient {
stdin: PipeWriter,
stdout: PipeReader,
output : Vec<u8>,
input : Vec<u8>,
input_offset : usize,
buf : [u8; 65536],
}
// Sends a message and expects to receive the same exact message, one at a time
impl SubprocessClient {
fn new(stdin: PipeWriter, stdout : PipeReader, data : &[u8]) -> SubprocessClient {
SubprocessClient {
stdin: stdin,
stdout: stdout,
output : Vec::<u8>::new(),
buf : [0; 65536],
input : data.to_vec(),
input_offset : 0,
}
}
fn readable(&mut self, _event_loop: &mut EventLoop<SubprocessClient>) -> io::Result<()> {
println!("client socket readable");
match self.stdout.try_read(&mut self.buf[..]) {
Ok(None) => {
println!("CLIENT : spurious read wakeup");
}
Ok(Some(r)) => {
println!("CLIENT : We read {} bytes!", r);
self.output.extend(&self.buf[0..r]);
}
Err(e) => {
return Err(e);
}
};
return Ok(());
}
fn writable(&mut self, event_loop: &mut EventLoop<SubprocessClient>) -> io::Result<()> {
println!("client socket writable");
match self.stdin.try_write(&(&self.input)[self.input_offset..]) {
Ok(None) => {
println!("client flushing buf; WOULDBLOCK");
}
Ok(Some(r)) => {
println!("CLIENT : we wrote {} bytes!", r);
self.input_offset += r;
}
Err(e) => println!("not implemented; client err={:?}", e)
}
if self.input_offset == self.input.len() {
event_loop.shutdown();
}
return Ok(());
}
}
impl Handler for SubprocessClient {
type Timeout = usize;
type Message = ();
fn ready(&mut self, event_loop: &mut EventLoop<SubprocessClient>, token: Token,
events: EventSet) {
println!("ready {:?} {:?}", token, events);
if events.is_readable() {
let _x = self.readable(event_loop);
}
if events.is_writable() {
let _y = self.writable(event_loop);
}
}
}
pub fn from_nix_error(err: ::nix::Error) -> io::Error {
io::Error::from_raw_os_error(err.errno() as i32)
}
fn set_nonblock(s: &AsRawFd) -> io::Result<()> {
fcntl(s.as_raw_fd(), F_SETFL(O_NONBLOCK)).map_err(from_nix_error)
.map(|_| ())
}
const TEST_DATA : [u8; 1024 * 4096] = [40; 1024 * 4096];
pub fn echo_server() {
let mut event_loop = EventLoop::<SubprocessClient>::new().unwrap();
let process =
Command::new("cat")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn().unwrap();
let raw_stdin_fd;
match process.stdin {
None => unreachable!(),
Some(ref item) => {
let err = set_nonblock(item);
match err {
Ok(()) => {},
Err(e) => panic!(e),
}
raw_stdin_fd = item.as_raw_fd();
},
}
let raw_stdout_fd;
match process.stdout {
None => unreachable!(),
Some(ref item) => {
let err = set_nonblock(item);
match err {
Ok(()) => {},
Err(e) => panic!(e),
}
raw_stdout_fd = item.as_raw_fd();},
}
//println!("listen for connections {:?} {:?}", , process.stdout.unwrap().as_raw_fd());
let mut subprocess = SubprocessClient::new(PipeWriter::from(Io::from_raw_fd(raw_stdin_fd)),
PipeReader::from(Io::from_raw_fd(raw_stdout_fd)),
&TEST_DATA[..]);
let stdout_token : Token = Token(0);
let stdin_token : Token = Token(1);
event_loop.register(&subprocess.stdout, stdout_token, EventSet::readable(),
PollOpt::level()).unwrap();
// Connect to the server
event_loop.register(&subprocess.stdin, stdin_token, EventSet::writable(),
PollOpt::level()).unwrap();
// Start the event loop
event_loop.run(&mut subprocess).unwrap();
let res = process.wait_with_output();
match res {
Err(e) => {panic!(e);},
Ok(output) => {
subprocess.output.extend(&output.stdout);
println!("Final output was {:}\n", output.stdout.len());
},
}
println!("{:?}\n", subprocess.output.len());
}
fn main() {
echo_server();
}
Basically the only way to close stdin was to call process.wait_with_output since the Stdin has no close primitive
Once this happened, the remaining input could extend the output data vector.
There's now a crate that does this
https://crates.io/crates/subprocess-communicate
In this particular example, you know that the input and output amounts are equivalent, so you don't need threads at all. You can just write a bit and then read a bit:
use std::io::{self, Cursor, Read, Write};
static DATA: [u8; 1024 * 4096] = [0x3f; 1024 * 4096];
const TRANSFER_LIMIT: u64 = 32 * 1024;
fn main() {
let mut child = std::process::Command::new("cat")
.stdout(std::process::Stdio::piped())
.stdin(std::process::Stdio::piped())
.spawn()
.expect("Could not start child");
let mut input = Cursor::new(&DATA[..]);
let mut output = Cursor::new(Vec::new());
match (child.stdin.as_mut(), child.stdout.as_mut()) {
(Some(stdin), Some(stdout)) => {
while input.position() < input.get_ref().len() as u64 {
io::copy(&mut input.by_ref().take(TRANSFER_LIMIT), stdin).expect("Could not copy input");
io::copy(&mut stdout.take(TRANSFER_LIMIT), &mut output).expect("Could not copy output");
}
},
_ => panic!("child process input and output were not opened"),
}
child.wait().expect("Could not join child");
let res = output.into_inner();
assert_eq!(res.len(), DATA.len());
assert_eq!(&*res, &DATA[..]);
}
If you didn't have that specific restriction, you will need to use select from the libc crate, which requires file descriptors for the pipes so will probably restrict your code to running on Linux / OS X.
You could also start threads, one for each pipe (and reuse the parent thread for one of the pipes), but you've already ruled that out.