I am trying to lazily populate a HashMap from a DB with async calls when the map does not already have an entry.
Rust compiler warns async closures are unstable but that I should try async {.
I am trying to follow that suggestion but I get the expected a FnOnce<()> closure error in the comments:
use std::collections::HashMap;
use tokio::runtime::Runtime;
async fn get_from_store(key: String) -> String {
// pretend to get this from an async sqlx db call
String::from(format!("value-for-{key}"))
}
async fn do_work() {
let mut map: HashMap<String, String> = HashMap::new();
let key = String::from("key1");
// the compiler advised async closures were unstable and...
// to use an async block, remove the `||`: `async {` (rustc E0658)
map.entry(key.clone())
.or_insert_with(async { get_from_store(key) }.await);
// the above now gives the error:
//expected a `FnOnce<()>` closure, found `impl Future<Output = String>...xpected an `FnOnce<()>` closure, found `impl Future<Output = String>`
for (key, value) in &map {
println!("{}: {}", key, value);
}
}
fn main() {
let runtime = Runtime::new().unwrap_or_else(|e| panic!("Haha: {e}"));
let result = do_work();
match runtime.block_on(result) {
_ => {}
}
}
There may be reasons not to update HashMap via async but the error above gave me hope I was just doing this wrong...
You won't be able to use .await in that position. That's because it requires that the return type of the closure is a Future, but the signature of or_insert_with does not expect that.
I would do it more simply, keeping the .await inside the async function that you already have:
use std::collections::hash_map::Entry;
async fn do_work() {
let mut map: HashMap<String, String> = HashMap::new();
let key = String::from("key1");
if let Entry::Vacant(entry) = map.entry(key.clone()) {
entry.insert(get_from_store(key).await);
}
for (key, value) in &map {
println!("{}: {}", key, value);
}
}
Related
I am trying to make two async api call and eventually get error E0597.
Here is a code:
async fn make_request() -> Result<()> {
.........
.........
.........
let mut result = client.get(uri).await?;
let some_key = result.headers().get("some_key");
let next_url = match some_key {
Some(url) => {
let some_result = client.get(Uri::from_static(url.to_str().unwrap())).await?
}
None => println!("....")
};
Ok(())
}
When I run this code the error "borrowed value does not live long enough argument requires that result is borrowed for `'static"
I have created a compile-able example based on your snipped to reproduce the error in the playground, and if you are able to do something like this in your question (for future reference), it usually helps you get more specific answers.
The Request passed into the function has no lifetime guarantees, so this will fail with the error you mentioned:
use http::{Request, Uri};
async fn make_request(result: &Request<()>) -> std::io::Result<()> {
match result.headers().get("some_key") {
// `url` is a reference to the string in the "some_key" header
Some(url) => {
let some_result = Uri::from_static(url.to_str().unwrap());
}
None => println!("....")
};
Ok(())
}
You can add that lifetime requirement, but that probably isn't what you need, and will likely give you the same error message, just in a different place:
async fn make_request_static(result: &'static Request<()>) -> std::io::Result<()> {
match result.headers().get("some_key") {
// because the request is static, so can be `url`
Some(url) => {
let some_result = Uri::from_static(url.to_str().unwrap());
}
None => println!("....")
};
Ok(())
}
Uri implements the FromStr trait, though, so you would be best off using that. There is no longer a lifetime requirement, so it can work with any string you pass in, even one which is currently borrowed:
// need to import the trait to use its methods
use std::str::FromStr;
async fn make_request_3(result: &Request<()>) -> std::io::Result<()> {
match result.headers().get("some_key") {
// because the request is static, so can be `url`
Some(url) => {
let some_result = Uri::from_str(url.to_str().unwrap());
}
None => println!("....")
};
Ok(())
}
I am trying to get an async closure working in the and_then filter from Warp.
This is the smallest example I could come up with where I am reasonably sure I didn't leave any important details out:
use std::{convert::Infallible, sync::Arc, thread, time};
use tokio::sync::RwLock;
use warp::Filter;
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });
}
async fn GetAvailableBinaries(man: &Manifest) -> Result<impl warp::Reply, Infallible> {
Ok(warp::reply::json(&man.GetAvailableBinaries().await))
}
pub struct Manifest {
binaries: Arc<RwLock<Vec<i32>>>,
}
impl Manifest {
pub fn new() -> Manifest {
let bins = Arc::new(RwLock::new(Vec::new()));
thread::spawn(move || async move {
loop {
thread::sleep(time::Duration::from_millis(10000));
}
});
Manifest { binaries: bins }
}
pub async fn GetAvailableBinaries(&self) -> Vec<i32> {
self.binaries.read().await.to_vec()
}
}
I am using:
[dependencies]
tokio = { version = "0.2", features = ["full"] }
warp = { version = "0.2", features = ["tls"] }
The error is:
error[E0525]: expected a closure that implements the `Fn` trait, but this closure only implements `FnOnce`
--> src/main.rs:9:48
|
9 | let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });
| -------- ^^^^^^^^^^^^^ ------------------------------------ closure is `FnOnce` because it moves the variable `man` out of its environment
| | |
| | this closure implements `FnOnce`, not `Fn`
| the requirement to implement `Fn` derives from here
After making Manifest implement Clone, you can fix the error by balancing when the manifest object is cloned:
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(move || {
let man = man.clone();
async move { get_available_binaries(&man).await }
});
warp::serve(check);
}
This moves man into the closure passed to and_then, then provides a clone of man to the async block each time the closure is executed. The async block then owns that data and can take a reference to it without worrying about executing the future after the data has been deallocated.
I'm not sure this is what you're going for, but this solution builds for me:
use std::{convert::Infallible, sync::Arc, thread, time};
use tokio::sync::RwLock;
use warp::Filter;
fn main() {
let man = Manifest::new();
let check = warp::path("updates").and_then(|| async { GetAvailableBinaries(&man).await });
}
async fn GetAvailableBinaries(man: &Manifest) -> Result<impl warp::Reply, Infallible> {
Ok(warp::reply::json(&man.GetAvailableBinaries().await))
}
#[derive(Clone)]
pub struct Manifest {
binaries: Arc<RwLock<Vec<i32>>>,
}
impl Manifest {
pub fn new() -> Manifest {
let bins = Arc::new(RwLock::new(Vec::new()));
thread::spawn(move || async {
loop {
thread::sleep(time::Duration::from_millis(10000));
//mutate bins here
}
});
Manifest { binaries: bins }
}
pub async fn GetAvailableBinaries(&self) -> Vec<i32> {
self.binaries.read().await.to_vec()
}
}
The move here is the reason the compiler gave a warning regarding the signature: let check = warp::path("updates").and_then(|| async move { GetAvailableBinaries(&man).await });. This means that everything referenced in this closure will be moved into the context of the closure. In this case, the compiler can't guarantee the closure to be Fn but only FnOnce meaning that the closure can only be guaranteed to execute once.
Been learning rust and having a problem with lifetime when passing conn to the request_handler. I get an error saying
error[E0312]: lifetime of reference outlives lifetime of borrowed content...
--> src/main.rs:33:70
|
33 | let request_handler = |req: Request<Body>| async { request_handler(conn, req).await };
| ^^^^
|
= note: ...the reference is valid for the static lifetime...
but I am not sure how to handle lifetimes with closures and why/how it is static. I have a loose understanding of lifetimes and borrowing but this seems like a more complex case. I also would just not use a closure, but the return type of one of the closures has a type that is not exported by the hyper crate, so i don't know how i would create a fn without being able to declare the return type.
Also I can confirm if i remove passing conn i can get everything to work, but I want to use the conn object in the request_handler.
use hyper::server::conn::AddrStream;
use hyper::service::make_service_fn;
use hyper::Version;
use hyper::{Body, Error, Method, Request, Response, Server};
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
mod http_models;
mod utils;
use hyper::service::service_fn;
async fn request_handler(
conn: &'static AddrStream,
req: Request<Body>,
) -> Result<Response<Body>, hyper::Error> {
println!("req: {:?}", req);
if req.method() == Method::CONNECT {
println!("Connect")
}
let res: Response<Body> = Response::builder()
.status(200)
.version(Version::HTTP_11)
.body(Body::empty())
.unwrap();
return Ok(res);
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let ip = IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1));
let addr = SocketAddr::new(ip, 1337);
//let client = Client::new();
let make_service = make_service_fn(|conn: &AddrStream| async {
let request_handler = |req: Request<Body>| async { request_handler(conn, req).await };
let service = service_fn(request_handler);
Ok::<_, Error>(service)
});
let server = Server::bind(&addr).serve(make_service);
println!("Listening on http://{}", addr);
if let Err(e) = server.await {
eprintln!("server error: {}", e);
}
Ok(())
}
The conn argument of the closure passed to make_service_fn only lives as long as the closure body, but the return value of the closure (Ok(service)) references it. The closure must have the type FnMut(&Target) -> impl Future, which means it sadly is not permitted to return a value that references its argument.
The only solution is to copy/clone whatever you need from conn while setting up your request handler, since you cannot keep a reference to it once the closure returns.
I am attempting to make a future that continuously finds new work to do and then maintains a set of futures for those work items. I would like to make sure neither my main future that finds work to be blocked for long periods of time and to have my work being done concurrently.
Here is a rough overview of what I am trying to do. Specifically isDone does not exist and also from what I can understand from the docs isn't necessarily a valid way to use futures in Rust. What is the idomatic way of doing this kind of thing?
use std::collections::HashMap;
use tokio::runtime::Runtime;
async fn find_work() -> HashMap<i64, String> {
// Go read from the DB or something...
let mut work = HashMap::new();
work.insert(1, "test".to_string());
work.insert(2, "test".to_string());
return work;
}
async fn do_work(id: i64, value: String) -> () {
// Result<(), Error> {
println!("{}: {}", id, value);
}
async fn async_main() -> () {
let mut pending_work = HashMap::new();
loop {
for (id, value) in find_work().await {
if !pending_work.contains_key(&id) {
let fut = do_work(id, value);
pending_work.insert(id, fut);
}
}
pending_work.retain(|id, fut| {
if isDone(fut) {
// do something with the result
false
} else {
true
}
});
}
}
fn main() {
let runtime = Runtime::new().unwrap();
let exec = runtime.executor();
exec.spawn(async_main());
runtime.shutdown_on_idle();
}
I'm attempting to learn Rust by implementing a simple in-memory URL shortener with Hyper 0.10. I'm running into an issue that I think is caused by trying to close over a mutable HashMap in my handler:
fn post(mut req: Request, mut res: Response, short_uris: &mut HashMap<&str, &str>) {
let mut body = String::new();
match req.read_to_string(&mut body) {
Ok(_) => {
let key = short_uris.len();
short_uris.insert(&key.to_string(), &body.to_string());
*res.status_mut() = StatusCode::Created;
res.start().unwrap().write(&key.to_string().into_bytes());
},
Err(_) => *res.status_mut() = StatusCode::BadRequest
}
}
fn get(req: Request, mut res: Response, short_uris: &HashMap<&str, &str>) {
match req.uri.clone() {
AbsolutePath(path) => {
match short_uris.get::<str>(&path) {
Some(short_uri) => {
*res.status_mut() = StatusCode::MovedPermanently;
res.headers_mut().set(Location(short_uri.to_string()));
},
None => *res.status_mut() = StatusCode::NotFound
}
},
_ => *res.status_mut() = StatusCode::BadRequest
}
}
fn main() {
let mut short_uris: HashMap<&str, &str> = HashMap::new();
short_uris.insert("/example", "http://www.example.com");
Server::http("0.0.0.0:3001").unwrap().handle(move |req: Request, mut res: Response| {
match req.method {
hyper::Post => post(req, res, &mut short_uris),
hyper::Get => get(req, res, &short_uris),
_ => *res.status_mut() = StatusCode::MethodNotAllowed
}
}).unwrap();
}
src/main.rs:42:40: 42:46 error: the trait bound `for<'r, 'r, 'r> [closure#src/main.rs:42:47: 48:3 short_uris:std::collections::HashMap<&str, &str>]: std::ops::Fn<(hyper::server::Request<'r, 'r>, hyper::server::Response<'r>)>` is not satisfied [E0277]
src/main.rs:42 Server::http("0.0.0.0:3001").unwrap().handle(move |req: Request, mut res: Response| {
Do I need to use an Arc to share the HashMap between threads? If so, what would that look like? Also, I could be totally wrong about the issue. The error message is very cryptic to me.
Please include all the necessary use declarations next time, thanks!
If you're using nightly Rust, the error message is a less cryptic:
expected a closure that implements the Fntrait, but this closure only implements FnMut
That means that Hyper needs the closure to be shared between threads, so the closure needs to use its environment only via immutable or shared methods – so the usage of &mut short_uris is the offender here. To provide shared threadsafe mutability in Rust, you should use Mutex or RwLock.
Please note that you don't need Arc here – Hyper manages the ownership of the closure itself (probably by wrapping the closure in Arc under the hood, or using something like scoped-threads).
There's also second issue with your code – you use HashMap<&str, &str>. &str is a borrowed reference. Each time when you have something borrowed in Rust, you should ask yourself – from where? Here you try to borrow from really short-lived strings – key.to_string() and body.to_string(). It just can't work. Just make your hashmap fully owned – HashMap<String, String>. Here's the version of your code which compiles:
extern crate hyper;
use hyper::server::{Request, Response, Server};
use std::collections::HashMap;
use hyper::status::StatusCode;
use hyper::uri::RequestUri::AbsolutePath;
use hyper::header::Location;
use std::io::prelude::*;
fn post(mut req: Request, mut res: Response, short_uris: &mut HashMap<String, String>) {
let mut body = String::new();
match req.read_to_string(&mut body) {
Ok(_) => {
let key = short_uris.len();
short_uris.insert(key.to_string(), body);
*res.status_mut() = StatusCode::Created;
res.start()
.unwrap()
.write(&key.to_string().into_bytes())
.unwrap();
}
Err(_) => *res.status_mut() = StatusCode::BadRequest,
}
}
fn get(req: Request, mut res: Response, short_uris: &HashMap<String, String>) {
match req.uri {
AbsolutePath(ref path) => match short_uris.get(path) {
Some(short_uri) => {
*res.status_mut() = StatusCode::MovedPermanently;
res.headers_mut().set(Location(short_uri.to_string()));
}
None => *res.status_mut() = StatusCode::NotFound,
},
_ => *res.status_mut() = StatusCode::BadRequest,
}
}
fn main() {
let mut short_uris: HashMap<String, String> = HashMap::new();
short_uris.insert("/example".into(), "http://www.example.com".into());
let short_uris = std::sync::RwLock::new(short_uris);
Server::http("0.0.0.0:3001")
.unwrap()
.handle(move |req: Request, mut res: Response| match req.method {
hyper::Post => post(req, res, &mut short_uris.write().unwrap()),
hyper::Get => get(req, res, &short_uris.read().unwrap()),
_ => *res.status_mut() = StatusCode::MethodNotAllowed,
})
.unwrap();
}
I've also got rid of the unnecessary .clone() in the get function.
Please note that this code, while compiles, is not perfect yet – the RwLock locks should last shorter (get and post should take &RwLock<HashMap<String,String>> as an argument and perform the locking by themselves). The .unwrap() also may be handled in a better way. You can also consider using some lockless concurrent hashmap, there should be some crates for that, but I'm not into the topic, so I won't recommend any.