I'm trying to append futures to the underlying stream of BufferUnordered. At the moment I'm pushing them directly into the underlying stream, the Fuse stream of BufferUnordered is empty, so pushing to it has no effect and the loop below does not receive the 3rd response. Changing the definition of next (1) to stream.buffer_unordered(1) seems to make it work because the underlying stream is not empty/finished.
extern crate url;
extern crate futures;
extern crate tokio_core;
extern crate reqwest;
use url::Url;
use futures::*;
use tokio_core::reactor::Core;
use reqwest::unstable::async::{Client, Response, Decoder};
fn main() {
let mut core = Core::new().unwrap();
let client = Client::new(&core.handle()).unwrap();
let hyper = client.get("https://hyper.rs").unwrap().send();
let google = client.get("https://google.com").unwrap().send();
let stream = stream::futures_unordered(vec![future::ok(hyper), future::ok(google)]);
let mut next = stream.buffer_unordered(5).into_future(); // (1)
loop {
match core.run(next) {
Ok((None, _something)) => {
println!("finished");
break;
},
Ok((Some(response), mut next_requests)) => {
{
let inner = next_requests.get_mut();
println!("{}", inner.is_empty());
println!("{}", response.status());
let yahoo = client.get("https://yahoo.com").unwrap().send();
inner.push(future::ok(yahoo)); // no effect here
}
next = next_requests.into_future();
}
Err((error, next_requests)) => {
next = next_requests.into_future();
}
}
}
}
How do I add more futures to BufferUnordered? Do I actually have to chain it or do something along these lines?
Related
This question already has answers here:
How do I create a global, mutable singleton?
(7 answers)
How can I pass a variable initialized in main to a Rocket route handler?
(1 answer)
Closed 3 years ago.
I want to respond with a different value to "the same request" depending on the state of server.
My expectation is that these responses will happen in a loop:
Someone sends a GET / request, it responds with "Hello, World!".
Then they send a GET / request, it responds with "Hello, Rust!".
Then they send a GET / request, it responds with "Hello, Rocket!".
Then they send a GET / request, it responds with "Hello, State!".
I don't want to use a variable that was initialized in main (in the Rocket route handler). I want to use a variable whose value can be changed in main (in the Rocket route handler).
This code compiles without error but the actual behavior is that it always responds "Hello, World!".
#![feature(proc_macro_hygiene)]
#![feature(decl_macro)]
#[macro_use]
extern crate rocket;
#[macro_use]
extern crate lazy_static;
use std::sync::{Arc, Mutex};
use rocket::State;
use std::thread;
use std::time::Duration;
lazy_static! {
static ref RESPONSE: Arc<Mutex<String>>
= Arc::new(Mutex::new(String::from("Hello, World!")));
}
#[get("/")]
fn get_response(state: State<Arc<Mutex<String>>>) -> String {
let response = state.lock().unwrap();
let ret: String = String::from(response.as_str());
ret
}
fn main() {
let managed_response: Arc<Mutex<String>> = RESPONSE.clone();
rocket::ignite()
.manage(managed_response)
.mount("/", routes![get_response])
.launch();
let mut server_state = 0;
loop {
// Pseudo transition of the state of server
thread::sleep(Duration::from_secs(5));
let mut response = RESPONSE.lock().unwrap();
match server_state {
0 => *response = String::from("Hello, Rust!"), // state 0
1 => *response = String::from("Hello, Rocket!"), // state 1
2 => *response = String::from("Hello, State!"), // state 2
_ => panic!(),
}
server_state += 1;
if server_state >= 3 {
server_state = 0;
}
}
}
Through trial and error, I solved it by myself.
This is the code.
If you have other better solutions, I would like you to teach me.
#![feature(proc_macro_hygiene)]
#![feature(decl_macro)]
#[macro_use]
extern crate rocket;
#[macro_use]
extern crate lazy_static;
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
lazy_static! {
static ref RESPONSE: Arc<Mutex<String>> = Arc::new(Mutex::new(String::from("Hello, World!")));
}
#[get("/")]
fn get_response() -> String {
let response = RESPONSE.lock().unwrap();
let ret: String = String::from(response.as_str());
ret
}
fn main_code() {
let mut server_state = 0;
loop {
thread::sleep(Duration::from_secs(5));
let mut response = RESPONSE.lock().unwrap();
match server_state {
0 => *response = String::from("Hello, Rust!"),
1 => *response = String::from("Hello, Rocket!"),
2 => *response = String::from("Hello, State!"),
_ => panic!(),
}
server_state += 1;
if server_state >= 3 {
server_state = 0;
}
}
}
fn main() {
let handle = thread::spawn(|| {
main_code();
});
rocket::ignite()
.mount("/", routes![get_response])
.launch();
}
I'm trying to make a Stream that would wait until a specific character is in buffer. I know there's read_until() on BufRead but I actually need a custom solution, as this is a stepping stone to implement waiting until a specific string in in buffer (or, for example, a regexp match happens).
In my project where I first encountered the problem, problem was that future processing just hanged when I get a Ready(_) from inner future and return NotReady from my function. I discovered I shouldn't do that per docs (last paragraph). However, what I didn't get, is what's the actual alternative that is promised in that paragraph. I read all the published documentation on the Tokio site and it doesn't make sense for me at the moment.
So following is my current code. Unfortunately I couldn't make it simpler and smaller as it's already broken. Current result is this:
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
<ad infinum>
Expected result is getting some Ok(Ready(_)) out of it, while printing W and W', and waiting for specific character in buffer.
extern crate futures;
extern crate tokio_core;
extern crate tokio_io;
extern crate tokio_io_timeout;
extern crate tokio_process;
use futures::stream::poll_fn;
use futures::{Async, Poll, Stream};
use tokio_core::reactor::Core;
use tokio_io::AsyncRead;
use tokio_io_timeout::TimeoutReader;
use tokio_process::CommandExt;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct Process {
child: tokio_process::Child,
stdout: Arc<Mutex<tokio_io_timeout::TimeoutReader<tokio_process::ChildStdout>>>,
}
impl Process {
fn new(
command: &str,
reader_timeout: Option<Duration>,
core: &tokio_core::reactor::Core,
) -> Self {
let mut cmd = Command::new(command);
let cat = cmd.stdout(Stdio::piped());
let mut child = cat.spawn_async(&core.handle()).unwrap();
let stdout = child.stdout().take().unwrap();
let mut timeout_reader = TimeoutReader::new(stdout);
timeout_reader.set_timeout(reader_timeout);
let timeout_reader = Arc::new(Mutex::new(timeout_reader));
Self {
child,
stdout: timeout_reader,
}
}
}
fn work() -> Result<(), ()> {
let window = Arc::new(Mutex::new(Vec::new()));
let mut core = Core::new().unwrap();
let process = Process::new("cat", Some(Duration::from_secs(20)), &core);
let mark = Arc::new(Mutex::new(b'c'));
let read_until_stream = poll_fn({
let window = window.clone();
let timeout_reader = process.stdout.clone();
move || -> Poll<Option<u8>, std::io::Error> {
let mut buf = [0; 8];
let poll;
{
let mut timeout_reader = timeout_reader.lock().unwrap();
poll = timeout_reader.poll_read(&mut buf);
}
match poll {
Ok(Async::Ready(0)) => Ok(Async::Ready(None)),
Ok(Async::Ready(x)) => {
{
let mut window = window.lock().unwrap();
println!("W: {:?}", *window);
println!("buf: {:?}", &buf[0..x]);
window.extend(buf[0..x].into_iter().map(|x| *x));
println!("W': {:?}", *window);
if let Some(_) = window.iter().find(|c| **c == *mark.lock().unwrap()) {
Ok(Async::Ready(Some(1)))
} else {
Ok(Async::NotReady)
}
}
}
Ok(Async::NotReady) => Ok(Async::NotReady),
Err(e) => Err(e),
}
}
});
let _stream_thread = thread::spawn(move || {
for o in read_until_stream.wait() {
println!("{:?}", o);
}
});
match core.run(process.child) {
Ok(_) => {}
Err(e) => {
println!("Child error: {:?}", e);
}
}
Ok(())
}
fn main() {
work().unwrap();
}
This is complete example project.
If you need more data you need to call poll_read again until you either find what you were looking for or poll_read returns NotReady.
You might want to avoid looping in one task for too long, so you can build yourself a yield_task function to call instead if poll_read didn't return NotReady; it makes sure your task gets called again ASAP after other pending tasks were run.
To use it just run return yield_task();.
fn yield_inner() {
use futures::task;
task::current().notify();
}
#[inline(always)]
pub fn yield_task<T, E>() -> Poll<T, E> {
yield_inner();
Ok(Async::NotReady)
}
Also see futures-rs#354: Handle long-running, always-ready futures fairly #354.
With the new async/await API futures::task::current is gone; instead you'll need a std::task::Context reference, which is provided as parameter to the new std::future::Future::poll trait method.
If you're already manually implementing the std::future::Future trait you can simply insert:
context.waker().wake_by_ref();
return std::task::Poll::Pending;
Or build yourself a Future-implementing type that yields exactly once:
pub struct Yield {
ready: bool,
}
impl core::future::Future for Yield {
type Output = ();
fn poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::Context<'_>) -> core::task::Poll<Self::Output> {
let this = self.get_mut();
if this.ready {
core::task::Poll::Ready(())
} else {
cx.waker().wake_by_ref();
this.ready = true; // ready next round
core::task::Poll::Pending
}
}
}
pub fn yield_task() -> Yield {
Yield { ready: false }
}
And then use it in async code like this:
yield_task().await;
I am trying to adapt the Hyper basic client example to get multiple URLs concurrently.
This is the code I currently have:
extern crate futures;
extern crate hyper;
extern crate tokio_core;
use std::io::{self, Write};
use std::iter;
use futures::{Future, Stream};
use hyper::Client;
use tokio_core::reactor::Core;
fn get_url() {
let mut core = Core::new().unwrap();
let client = Client::new(&core.handle());
let uris: Vec<_> = iter::repeat("http://httpbin.org/ip".parse().unwrap()).take(50).collect();
for uri in uris {
let work = client.get(uri).and_then(|res| {
println!("Response: {}", res.status());
res.body().for_each(|chunk| {
io::stdout()
.write_all(&chunk)
.map_err(From::from)
})
});
core.run(work).unwrap();
}
}
fn main() {
get_url();
}
It doesn't seem to be acting concurrently (it takes a long time to complete), am I giving the work to the core in the wrong way?
am I giving the work to the core in the wrong way?
Yes, you are giving one request to Tokio and requiring that it complete before starting the next request. You've taken asynchronous code and forced it to be sequential.
You need to give the reactor a single future that will perform different kinds of concurrent work.
Hyper 0.14
use futures::prelude::*;
use hyper::{body, client::Client};
use std::{
io::{self, Write},
iter,
};
use tokio;
const N_CONCURRENT: usize = 1;
#[tokio::main]
async fn main() {
let client = Client::new();
let uri = "http://httpbin.org/ip".parse().unwrap();
let uris = iter::repeat(uri).take(50);
stream::iter(uris)
.map(move |uri| client.get(uri))
.buffer_unordered(N_CONCURRENT)
.then(|res| async {
let res = res.expect("Error making request: {}");
println!("Response: {}", res.status());
body::to_bytes(res).await.expect("Error reading body")
})
.for_each(|body| async move {
io::stdout().write_all(&body).expect("Error writing body");
})
.await;
}
With N_CONCURRENT set to 1:
real 1.119 1119085us
user 0.012 12021us
sys 0.011 11459us
And set to 10:
real 0.216 216285us
user 0.014 13596us
sys 0.021 20640us
Cargo.toml
[dependencies]
futures = "0.3.17"
hyper = { version = "0.14.13", features = ["client", "http1", "tcp"] }
tokio = { version = "1.12.0", features = ["full"] }
Hyper 0.12
use futures::{stream, Future, Stream}; // 0.1.25
use hyper::Client; // 0.12.23
use std::{
io::{self, Write},
iter,
};
use tokio; // 0.1.15
const N_CONCURRENT: usize = 1;
fn main() {
let client = Client::new();
let uri = "http://httpbin.org/ip".parse().unwrap();
let uris = iter::repeat(uri).take(50);
let work = stream::iter_ok(uris)
.map(move |uri| client.get(uri))
.buffer_unordered(N_CONCURRENT)
.and_then(|res| {
println!("Response: {}", res.status());
res.into_body()
.concat2()
.map_err(|e| panic!("Error collecting body: {}", e))
})
.for_each(|body| {
io::stdout()
.write_all(&body)
.map_err(|e| panic!("Error writing: {}", e))
})
.map_err(|e| panic!("Error making request: {}", e));
tokio::run(work);
}
With N_CONCURRENT set to 1:
real 0m2.279s
user 0m0.193s
sys 0m0.065s
And set to 10:
real 0m0.529s
user 0m0.186s
sys 0m0.075s
See also:
How can I perform parallel asynchronous HTTP GET requests with reqwest?
I'm trying to build an echo server in Tokio. I've seen examples, but all of them seem to use io::copy from Tokio IO which I can't use because I want to modify the output.
However, I can't compile a server that uses writer and reader at the same time. I want to build a task based on futures that enables reading/writing in a loop (an echo server).
My actual code is this:
extern crate futures;
extern crate futures_cpupool;
extern crate tokio;
extern crate tokio_io;
use futures::prelude::*;
use futures_cpupool::CpuPool;
use tokio_io::AsyncRead;
use futures::Stream;
use futures::stream;
use tokio_io::codec::*;
use std::rc::Rc;
fn main() {
let pool = CpuPool::new_num_cpus();
use std::net::*;
let socket = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 8080);
let listener = tokio::net::TcpListener::bind(&socket).unwrap();
let server = listener.incoming().for_each(|socket| {
let (writer, reader) = socket.framed(LinesCodec::new()).split();
let writer = Rc::new(writer);
let action = reader.for_each(|line| {
println!("ECHO: {}", line);
writer.send(line);
Ok(())
});
pool.spawn(action); // std::rc::Rc<futures::stream::SplitSink<tokio_io::codec::Framed<tokio::net::TcpStream, tokio_io::codec::LinesCodec>>>` cannot be shared between threads safely
Ok(())
});
server.wait().unwrap();
}
You might say that I must use Arc because there are different threads involved. I've tried with Arc and Mutex, but another error arises and I can't figure a way to make it compile:
extern crate futures;
extern crate futures_cpupool;
extern crate tokio;
extern crate tokio_io;
use futures::prelude::*;
use std::time;
use std::thread;
use futures_cpupool::CpuPool;
use tokio_io::AsyncRead;
use futures::Stream;
use tokio_io::codec::*;
use std::sync::Arc;
use std::sync::Mutex;
fn main() {
let pool = CpuPool::new_num_cpus();
use std::net::*;
let socket = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 8080);
let listener = tokio::net::TcpListener::bind(&socket).unwrap();
let server = listener.incoming().for_each(|socket| {
let (writer, reader) = socket.framed(LinesCodec::new()).split();
let writer = Arc::new(Mutex::new(writer));
let action = reader.for_each(move |line| {
println!("ECHO: {}", line);
writer.lock().unwrap().send(line); // cannot move out of borrowed content
Ok(())
});
pool.spawn(action);
Ok(())
});
server.wait().unwrap();
}
The error it says is: cannot move out of borrowed content
I finally found that forward was the answer to my question.
extern crate tokio;
extern crate tokio_io;
extern crate futures;
use futures::prelude::*;
use tokio_io::AsyncRead;
use futures::Stream;
use tokio_io::codec::*;
struct Cancellable{
rx: std::sync::mpsc::Receiver<()>,
}
impl Future for Cancellable {
type Item = ();
type Error = std::sync::mpsc::RecvError;
fn poll(&mut self) -> Result<Async<Self::Item>,Self::Error> {
match self.rx.try_recv() {
Ok(_) => Ok(Async::Ready(())),
Err(_) => Ok(Async::NotReady)
}
}
}
fn main() {
use std::net::*;
let socket = SocketAddr::new(IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1)), 8080);
let listener = tokio::net::TcpListener::bind(&socket).unwrap();
let server = listener.incoming().for_each(|socket|{
let (writer,reader) = socket.framed(LinesCodec::new()).split();
let (tx,rx) = std::sync::mpsc::channel();
let cancel = Cancellable {
rx: rx,
};
let action = reader
.map(move |line|{
println!("ECHO: {}",line);
if line == "bye"{
println!("BYE");
tx.send(()).unwrap();
}
line
})
.forward(writer)
.select2(cancel)
.map(|_|{
})
.map_err(|err|{
println!("error");
});
tokio::executor::current_thread::spawn(action);
Ok(())
}).map_err(|err|{
println!("error = {:?}",err);
});
tokio::executor::current_thread::run(|_|{
tokio::executor::current_thread::spawn(server);
});
}
With Rust 1.9, I'd like to read from a mpsc::channel or timeout. Is there a clear idiom to make this work? I've seen the unstable approach described in mpsc::Select but this Github discussion suggests it is not a robust approach. Is there a better-recommended way for me to achieve receive-or-timeout semantics?
Rust 1.12 introduced Receiver::recv_timeout:
use std::sync::mpsc::channel;
use std::time::Duration;
fn main() {
let (.., rx) = channel::<bool>();
let timeout = Duration::new(3, 0);
println!("start recv");
let _ = rx.recv_timeout(timeout);
println!("done!");
}
I don't know how you'd do it with the standard library channels, but the chan crate provides a chan_select! macro:
#[macro_use]
extern crate chan;
use std::time::Duration;
fn main() {
let (_never_sends, never_receives) = chan::sync::<bool>(1);
let timeout = chan::after(Duration::from_millis(50));
chan_select! {
timeout.recv() => {
println!("timed out!");
},
never_receives.recv() => {
println!("Shouldn't have a value!");
},
}
}
I was able to get something working using the standard lib.
use std::sync::mpsc::channel;
use std::thread;
use std::time::{Duration, Instant};
use std::sync::mpsc::TryRecvError;
fn main() {
let (send, recv) = channel();
thread::spawn(move || {
send.send("Hello world!").unwrap();
thread::sleep(Duration::from_secs(1)); // block for two seconds
send.send("Delayed").unwrap();
});
println!("{}", recv.recv().unwrap()); // Received immediately
println!("Waiting...");
let mut resolved: bool = false;
let mut result: Result<&str, TryRecvError> = Ok("Null");
let now = Instant::now();
let timeout: u64= 2;
while !resolved {
result = recv.try_recv();
resolved = !result.is_err();
if now.elapsed().as_secs() as u64 > timeout {
break;
}
}
if result.is_ok(){
println!("Results: {:?}", result.unwrap());
}
println!("Time elapsed: {}", now.elapsed().as_secs());
println!("Resolved: {}", resolved.to_string());
}
This will spin for timeout seconds and will result in either the received value or an Err Result.