I'm stuck trying to share (read-only) data structures across async helpers. What I'm trying to accomplish is create a Hyper server where I pre-generate some data that can be used by all request handlers.
Here is the example from the Hyper getting started guide, extended with what I'm trying to do:
#[tokio::main]
async fn main() {
let address = SocketAddr::from(([127, 0, 0, 1], 3000));
let pages = generate_static_pages();
let make_service = make_service_fn(|_conn| async move {
Ok::<_, Infallible>(service_fn(|req: Request<Body>| async move {
serve(pages, req)
}))
});
let server = Server::bind(&address).serve(make_service);
if let Err(error) = server.await {
eprintln!("server error: {}", error);
}
}
In my case, generate_static_pages() returns a HashMap<&'static str, Bytes> with pre-generated pages. Unfortunately, this hash map cannot be generated at compile-time, because that would make things a lot easier. Now, I struggle because pages cannot be borrowed by the closures: "cannot move out of pages, a captured variable in an FnMut closure"
I tried to pass a reference, but that didn't work because Rust cannot infer the variable to live long enough for the closure to use. I then tried to use .clone() but that doesn't work because it would be called on the variable after it is moved, which it can't. Finally, I tried wrapping in an Arc, but that doesn't solve it, basically because of the same reason.
What would you advice me to do? Thanks!
If you only want immutable references to the pages then you should be able to use the lazy_static crate. lazy_static lets you initialize static variables at runtime - it's quite useful!
Your code would end up looking something like:
use lazy_static::lazy_static;
lazy_static! {
static ref PAGES: HashMap<&'static str, Bytes> = generate_static_pages();
}
#[tokio::main]
async fn main() {
let address = SocketAddr::from(([127, 0, 0, 1], 3000));
let make_service = make_service_fn(|_conn| async move {
Ok::<_, Infallible>(service_fn(|req: Request<Body>| async move {
serve(&PAGES, req)
}))
});
let server = Server::bind(&address).serve(make_service);
if let Err(error) = server.await {
eprintln!("server error: {}", error);
}
}
Also, here's another lazy_static example.
Related
The following code fails with
cannot move out of `my_number_vector`, a captured variable in an `FnMut` closure
move out of `my_number_vector`
[dependencies]
futures = "0.3.21"
tokio = "1.19.2"
use futures::{stream, StreamExt};
#[tokio::main]
async fn main() {
let mut my_number_vector = Vec::new();
let the_stream = stream::iter(0..=10);
let response_of_stream = stream_of_urls
.map(|number| async move {
number
})
.buffer_unordered(2);
response_of_stream
.for_each(|number| async move {
my_number_vector.push(number);
})
.await;
}
How is it possible to append to a vector in this scenario? I tried using RC and RefCells but it seems I cannot understand the error accurately in order to solve the compiler error.
The idea of Rc/RefCell was good but, since the async executor may rely on multiple threads, you need the multithreaded equivalent: Arc/Mutex.
On top of that, there is a tricky part.
The closure needs access to the vector inside the Arc/Mutex, then this closure has to be marked with move and the Arc has to be cloned.
But, this closure involves an async block which also needs access to the vector inside the Arc/Mutex, then this block also has to be marked with move and the Arc has to be cloned again!
use futures::{stream, StreamExt};
use std::sync::{Arc, Mutex};
#[tokio::main]
async fn main() {
let my_number_vector = Arc::new(Mutex::new(Vec::<i32>::new()));
let the_stream = stream::iter(0..=10);
let response_of_stream = the_stream // stream_of_urls ???
.map(|number| async move { number })
.buffer_unordered(2);
// clone the Arc to be moved into the closure
let my_vec_arc = Arc::clone(&my_number_vector);
response_of_stream
.for_each(move |number| {
// clone again the Arc to be moved into the async block
let my_vec_arc_2 = Arc::clone(&my_vec_arc);
async move {
my_vec_arc_2.lock().unwrap().push(number);
}
})
.await;
println!("{:?}", my_number_vector.lock().unwrap());
// [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
}
I am new to Rust and trying to make some code async to run a bunch of tasks in parallel. Here is a simplified example:
use futures::future::join_all;
#[tokio::main]
async fn main() {
let mut list = Vec::new();
for i in 1..10 {
let my_str = format!("Value is: {:?}", &i);
let future = do_something(&my_str);
list.push(future);
}
join_all(list).await;
}
async fn do_something(value: &str)
{
println!("value is: {:?}", value);
}
This fails with "Borrowed value does not live long enough" on the do_something(&my_str) call. I can get the code to compile by changing do_something to accept a String instead of an &str. However, it seems a bit strange to require a String when an &str would work. Is there a better pattern to use here? Thanks!
"However, it seems a bit strange to require a String when an &str would work." But an &str can't work here because it only borrows my_str which gets destroyed before the future completes:
for i in 1..10 {
// Create a new `String` and store it in `my_str`
let my_str = format!("Value is: {:?}", &i);
// Create a future that borrows `my_str`. Note that the future is not
// yet started
let future = do_something(&my_str);
// Store the future in `list`
list.push(future);
// Destroy `my_str` since it goes out of scope and wasn't moved.
}
// Run the futures from `list` until they complete. At this point each
// future will try to access the string that they have borrowed, but those
// strings have already been freed!
join_all(list).await;
Instead your do_something should take ownership of the string along with responsibility for freeing it:
use futures::future::join_all;
#[tokio::main]
async fn main() {
let mut list = Vec::new();
for i in 1..10 {
// Create a new `String` and store it in `my_str`
let my_str = format!("Value is: {:?}", &i);
// Create a future and _move_ `my_str` into it.
let future = do_something(my_str);
// Store the future in `list`
list.push(future);
// `my_str` is not destroyed since it was moved into the future.
}
join_all(list).await;
}
async fn do_something(value: String)
{
println!("value is: {:?}", value);
// Destroy `value` since it goes out of scope and wasn't moved.
}
Been learning rust and having a problem with lifetime when passing conn to the request_handler. I get an error saying
error[E0312]: lifetime of reference outlives lifetime of borrowed content...
--> src/main.rs:33:70
|
33 | let request_handler = |req: Request<Body>| async { request_handler(conn, req).await };
| ^^^^
|
= note: ...the reference is valid for the static lifetime...
but I am not sure how to handle lifetimes with closures and why/how it is static. I have a loose understanding of lifetimes and borrowing but this seems like a more complex case. I also would just not use a closure, but the return type of one of the closures has a type that is not exported by the hyper crate, so i don't know how i would create a fn without being able to declare the return type.
Also I can confirm if i remove passing conn i can get everything to work, but I want to use the conn object in the request_handler.
use hyper::server::conn::AddrStream;
use hyper::service::make_service_fn;
use hyper::Version;
use hyper::{Body, Error, Method, Request, Response, Server};
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
mod http_models;
mod utils;
use hyper::service::service_fn;
async fn request_handler(
conn: &'static AddrStream,
req: Request<Body>,
) -> Result<Response<Body>, hyper::Error> {
println!("req: {:?}", req);
if req.method() == Method::CONNECT {
println!("Connect")
}
let res: Response<Body> = Response::builder()
.status(200)
.version(Version::HTTP_11)
.body(Body::empty())
.unwrap();
return Ok(res);
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let ip = IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1));
let addr = SocketAddr::new(ip, 1337);
//let client = Client::new();
let make_service = make_service_fn(|conn: &AddrStream| async {
let request_handler = |req: Request<Body>| async { request_handler(conn, req).await };
let service = service_fn(request_handler);
Ok::<_, Error>(service)
});
let server = Server::bind(&addr).serve(make_service);
println!("Listening on http://{}", addr);
if let Err(e) = server.await {
eprintln!("server error: {}", e);
}
Ok(())
}
The conn argument of the closure passed to make_service_fn only lives as long as the closure body, but the return value of the closure (Ok(service)) references it. The closure must have the type FnMut(&Target) -> impl Future, which means it sadly is not permitted to return a value that references its argument.
The only solution is to copy/clone whatever you need from conn while setting up your request handler, since you cannot keep a reference to it once the closure returns.
I've been trying the following
Relevant imports and code shown
use std::sync::{Arc, Mutex};
use std::thread;
use hyper::rt::{self, Future, Stream};
use hyper::service::service_fn;
use hyper::{Body, Request, Response, Server, StatusCode};
pub struct ChallengeState;
pub struct ChallengeResponse;
type BoxFut<'a> = Box<Future<Item = Response<Body>, Error = hyper::Error> + Send + 'a>;
fn handle_challengeproof<'a>(
req: Request<Body>,
challenge: &Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a> {
let resp = req.into_body().concat2().map(move |body| {
let challenge_lock = challenge.lock().unwrap();
Response::builder()
.status(StatusCode::OK)
.body(Body::from("test"))
.unwrap()
});
Box::new(resp)
}
fn handle<'a>(
req: Request<Body>,
challenge: &Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a> {
handle_challengeproof(req, challenge)
}
pub fn run_listener(
challenge: Arc<Mutex<ChallengeState>>,
) -> thread::JoinHandle<()> {
let addr = ([127, 0, 0, 1], 9999).into();
let listener_service = move || {
let challenge = Arc::clone(&challenge);
service_fn(move |req: Request<Body>| {
handle(req, &challenge)
})
};
let server = Server::bind(&addr)
.serve(listener_service)
.map_err(|_| () );
thread::spawn(move || {
rt::run(server);
})
}
I've been trying to avoid an extra clone of Arc by passing a reference to the handle method but can't seem to get around this. Avoiding the lifetime on handle() got a different error regarding futures asking for static lifetime.
Code updated with only relevant stuff # https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=10ea31450e88a122455006760d7fcdd1
The whole point of an Arc is that it counts how many references there are, which happens when it is cloned. Passing around references to an Arc defeats the point.
Instead of passing references, pass the Arc itself. So handle's signature becomes:
fn handle<'a>(
req: Request<Body>,
challenge: Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a>
Passing the Arc by references from the closure isn't possible because you would be referencing something that immediately goes out of scope. Instead, move the Arc into handle:
let listener_service = move || {
service_fn(move |req: Request<Body>| handle(req, challenge))
};
I am trying to read the contents of files in a directory in parallel. I'm running into lifetime issues.
My code looks like this:
use std::io::fs;
use std::io;
use std::collections::HashMap;
use std::comm;
use std::io::File;
fn main() {
let (tx, rx) = comm::channel(); // (Sender, Receiver)
let paths = fs::readdir(&Path::new("resources/tests")).unwrap();
for path in paths.iter() {
let task_tx = tx.clone();
spawn(proc() {
match File::open(path).read_to_end() {
Ok(data) => task_tx.send((path.filename_str().unwrap(), data)),
Err(e) => fail!("Could not read one of the files! Error: {}", e)
};
});
}
let mut results = HashMap::new();
for _ in range(0, paths.len()) {
let (filename, data) = rx.recv();
results.insert(filename, data);
}
println!("{}", results);
}
The compilation error I'm getting is:
error: paths does not live long enough
note: reference must be valid for the static lifetime...
note: ...but borrowed value is only valid for the block at 7:19
I also tried to use into_iter() (or move_iter() previously) in the loop without much success.
I'm suspecting it has to do with the spawned tasks remaining alive beyond the entire main() scope, but I don't know how I can fix this situation.
The error message might be a bit confusing but what it's telling you is that you are trying to use a reference path inside of a task.
Because spawn is using proc you can only use data that you can transfer ownership of to that task (Send kind).
To solve that you can do this (you could use a move_iter but then you can't access paths after the loop):
for path in paths.iter() {
let task_tx = tx.clone();
let p = path.clone();
spawn(proc() {
match File::open(&p).read_to_end() {
The second problem is that you are trying to send &str (filename) over a channel. Same as for tasks types used must be of kind Send:
match File::open(&p).read_to_end() {
Ok(data) => task_tx.send((p.filename_str().unwrap().to_string(), data)),
Err(e) => fail!("Could not read one of the files! Error: {}", e)
};