Appending to a vector inside a buffered future - Rust - rust

The following code fails with
cannot move out of `my_number_vector`, a captured variable in an `FnMut` closure
move out of `my_number_vector`
[dependencies]
futures = "0.3.21"
tokio = "1.19.2"
use futures::{stream, StreamExt};
#[tokio::main]
async fn main() {
let mut my_number_vector = Vec::new();
let the_stream = stream::iter(0..=10);
let response_of_stream = stream_of_urls
.map(|number| async move {
number
})
.buffer_unordered(2);
response_of_stream
.for_each(|number| async move {
my_number_vector.push(number);
})
.await;
}
How is it possible to append to a vector in this scenario? I tried using RC and RefCells but it seems I cannot understand the error accurately in order to solve the compiler error.

The idea of Rc/RefCell was good but, since the async executor may rely on multiple threads, you need the multithreaded equivalent: Arc/Mutex.
On top of that, there is a tricky part.
The closure needs access to the vector inside the Arc/Mutex, then this closure has to be marked with move and the Arc has to be cloned.
But, this closure involves an async block which also needs access to the vector inside the Arc/Mutex, then this block also has to be marked with move and the Arc has to be cloned again!
use futures::{stream, StreamExt};
use std::sync::{Arc, Mutex};
#[tokio::main]
async fn main() {
let my_number_vector = Arc::new(Mutex::new(Vec::<i32>::new()));
let the_stream = stream::iter(0..=10);
let response_of_stream = the_stream // stream_of_urls ???
.map(|number| async move { number })
.buffer_unordered(2);
// clone the Arc to be moved into the closure
let my_vec_arc = Arc::clone(&my_number_vector);
response_of_stream
.for_each(move |number| {
// clone again the Arc to be moved into the async block
let my_vec_arc_2 = Arc::clone(&my_vec_arc);
async move {
my_vec_arc_2.lock().unwrap().push(number);
}
})
.await;
println!("{:?}", my_number_vector.lock().unwrap());
// [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
}

Related

Borrowed value does not live long enough with async funtion

I am new to Rust and trying to make some code async to run a bunch of tasks in parallel. Here is a simplified example:
use futures::future::join_all;
#[tokio::main]
async fn main() {
let mut list = Vec::new();
for i in 1..10 {
let my_str = format!("Value is: {:?}", &i);
let future = do_something(&my_str);
list.push(future);
}
join_all(list).await;
}
async fn do_something(value: &str)
{
println!("value is: {:?}", value);
}
This fails with "Borrowed value does not live long enough" on the do_something(&my_str) call. I can get the code to compile by changing do_something to accept a String instead of an &str. However, it seems a bit strange to require a String when an &str would work. Is there a better pattern to use here? Thanks!
"However, it seems a bit strange to require a String when an &str would work." But an &str can't work here because it only borrows my_str which gets destroyed before the future completes:
for i in 1..10 {
// Create a new `String` and store it in `my_str`
let my_str = format!("Value is: {:?}", &i);
// Create a future that borrows `my_str`. Note that the future is not
// yet started
let future = do_something(&my_str);
// Store the future in `list`
list.push(future);
// Destroy `my_str` since it goes out of scope and wasn't moved.
}
// Run the futures from `list` until they complete. At this point each
// future will try to access the string that they have borrowed, but those
// strings have already been freed!
join_all(list).await;
Instead your do_something should take ownership of the string along with responsibility for freeing it:
use futures::future::join_all;
#[tokio::main]
async fn main() {
let mut list = Vec::new();
for i in 1..10 {
// Create a new `String` and store it in `my_str`
let my_str = format!("Value is: {:?}", &i);
// Create a future and _move_ `my_str` into it.
let future = do_something(my_str);
// Store the future in `list`
list.push(future);
// `my_str` is not destroyed since it was moved into the future.
}
join_all(list).await;
}
async fn do_something(value: String)
{
println!("value is: {:?}", value);
// Destroy `value` since it goes out of scope and wasn't moved.
}

How do I remove `MutexGuard` around a value?

I'm trying to use ndarray as an asynchronous process to do linear algebra and such.
I used Rust's tokio and ndarray to create the following code.
use std::sync::{Arc, Mutex};
use ndarray::prelude::*;
use futures::future::join_all;
fn print_type_of<T>(_: &T) {
println!("{}", std::any::type_name::<T>())
}
#[tokio::main]
async fn main() {
let db = Arc::new(Mutex::new(array![0,0,0,0,0,0,0,0]));
let mut handels = vec![];
for i in 0..8 {
let db = db.clone();
let unchange_array = unchange_array.clone();
handels.push(tokio::spawn(async move{
print(i, db).await;
}));
}
join_all(handels).await;
let array = Arc::try_unwrap(db).unwrap();
let array = array.lock().unwrap();
print_type_of(&array); // -> std::sync::mutex::MutexGuard<ndarray::ArrayBase<ndarray::data_repr::OwnedRepr<u32>, ndarray::dimension::dim::Dim<[usize; 1]>>>
}
async fn print(i: u32, db: Arc<Mutex<Array1<u32>>>) {
let unchange = unchange.to_owned();
let mut tmp = 0;
// time-consuming process
for k in 0..100000000 {
tmp = k;
}
tmp += i;
let mut db = db.lock().unwrap();
db.fill(i);
println!("{:?}", unchange);
print_type_of(&db);
}
I would like to change the data std::sync::mutex::MutexGuard<ndarray::ArrayBase<OwnedRepr<u32>, Dim<[usize; 1]>>>
to ndarray::ArrayBase<OwnedRepr<u32>, Dim<[usize; 1]>>.
How can I do this?
You can't. That's the whole point of MutexGuard: if you could take the data out of the MutexGuard, then you would be able to make a reference that can be accessed without locking the mutex, defeating the whole purpose of having a mutex in the first place.
Depending on what you really want to do, one of the following solutions might apply to you:
Most of the time, you don't need to take the data out of the mutex: MutexGuard<T> implements Deref<Target=T> and DerefMut<Target=T>, so you can use the MutexGuard everywhere you would use a &T or a &mut T. Note that if you change your code to call print_type_of(&*array) instead of print_type_of(&array), it will print the inner type.
If you really need to, you can take the data out of the Mutex itself (but not the MutexGuard) with into_inner, which consumes the mutex, ensuring that no one else can ever access it:
let array = Arc::try_unwrap(db).unwrap();
let array = array.into_inner().unwrap();
print_type_of(&array); // -> ndarray::ArrayBase<ndarray::data_repr::OwnedRepr<u32>, ndarray::dimension::dim::Dim<[usize; 1]>>

Sharing data structures across async closures

I'm stuck trying to share (read-only) data structures across async helpers. What I'm trying to accomplish is create a Hyper server where I pre-generate some data that can be used by all request handlers.
Here is the example from the Hyper getting started guide, extended with what I'm trying to do:
#[tokio::main]
async fn main() {
let address = SocketAddr::from(([127, 0, 0, 1], 3000));
let pages = generate_static_pages();
let make_service = make_service_fn(|_conn| async move {
Ok::<_, Infallible>(service_fn(|req: Request<Body>| async move {
serve(pages, req)
}))
});
let server = Server::bind(&address).serve(make_service);
if let Err(error) = server.await {
eprintln!("server error: {}", error);
}
}
In my case, generate_static_pages() returns a HashMap<&'static str, Bytes> with pre-generated pages. Unfortunately, this hash map cannot be generated at compile-time, because that would make things a lot easier. Now, I struggle because pages cannot be borrowed by the closures: "cannot move out of pages, a captured variable in an FnMut closure"
I tried to pass a reference, but that didn't work because Rust cannot infer the variable to live long enough for the closure to use. I then tried to use .clone() but that doesn't work because it would be called on the variable after it is moved, which it can't. Finally, I tried wrapping in an Arc, but that doesn't solve it, basically because of the same reason.
What would you advice me to do? Thanks!
If you only want immutable references to the pages then you should be able to use the lazy_static crate. lazy_static lets you initialize static variables at runtime - it's quite useful!
Your code would end up looking something like:
use lazy_static::lazy_static;
lazy_static! {
static ref PAGES: HashMap<&'static str, Bytes> = generate_static_pages();
}
#[tokio::main]
async fn main() {
let address = SocketAddr::from(([127, 0, 0, 1], 3000));
let make_service = make_service_fn(|_conn| async move {
Ok::<_, Infallible>(service_fn(|req: Request<Body>| async move {
serve(&PAGES, req)
}))
});
let server = Server::bind(&address).serve(make_service);
if let Err(error) = server.await {
eprintln!("server error: {}", error);
}
}
Also, here's another lazy_static example.

Issue passing mutable Arc reference to hyper service_fn handler

I've been trying the following
Relevant imports and code shown
use std::sync::{Arc, Mutex};
use std::thread;
use hyper::rt::{self, Future, Stream};
use hyper::service::service_fn;
use hyper::{Body, Request, Response, Server, StatusCode};
pub struct ChallengeState;
pub struct ChallengeResponse;
type BoxFut<'a> = Box<Future<Item = Response<Body>, Error = hyper::Error> + Send + 'a>;
fn handle_challengeproof<'a>(
req: Request<Body>,
challenge: &Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a> {
let resp = req.into_body().concat2().map(move |body| {
let challenge_lock = challenge.lock().unwrap();
Response::builder()
.status(StatusCode::OK)
.body(Body::from("test"))
.unwrap()
});
Box::new(resp)
}
fn handle<'a>(
req: Request<Body>,
challenge: &Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a> {
handle_challengeproof(req, challenge)
}
pub fn run_listener(
challenge: Arc<Mutex<ChallengeState>>,
) -> thread::JoinHandle<()> {
let addr = ([127, 0, 0, 1], 9999).into();
let listener_service = move || {
let challenge = Arc::clone(&challenge);
service_fn(move |req: Request<Body>| {
handle(req, &challenge)
})
};
let server = Server::bind(&addr)
.serve(listener_service)
.map_err(|_| () );
thread::spawn(move || {
rt::run(server);
})
}
I've been trying to avoid an extra clone of Arc by passing a reference to the handle method but can't seem to get around this. Avoiding the lifetime on handle() got a different error regarding futures asking for static lifetime.
Code updated with only relevant stuff # https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=10ea31450e88a122455006760d7fcdd1
The whole point of an Arc is that it counts how many references there are, which happens when it is cloned. Passing around references to an Arc defeats the point.
Instead of passing references, pass the Arc itself. So handle's signature becomes:
fn handle<'a>(
req: Request<Body>,
challenge: Arc<Mutex<ChallengeState>>,
) -> BoxFut<'a>
Passing the Arc by references from the closure isn't possible because you would be referencing something that immediately goes out of scope. Instead, move the Arc into handle:
let listener_service = move || {
service_fn(move |req: Request<Body>| handle(req, challenge))
};

How to use RwLocks without scoped?

I'm trying to share a RwLock amongst several threads without using scoped threads but I can't figure out how to get the lifetimes correct. I assume that this is possible (what's the point of RwLocks otherwise?) but I can't find any examples of it.
Here is a toy example of what I'm trying to accomplish. Any advice would be appreciated.
rust playpen for this code
use std::sync::{Arc, RwLock};
use std::thread;
struct Stuff {
x: i32
}
fn main() {
let mut stuff = Stuff{x: 5};
helper(&mut stuff);
println!("done");
}
fn helper(stuff: &mut Stuff){
let rwlock = RwLock::new(stuff);
let arc = Arc::new(rwlock);
let local_arc = arc.clone();
for _ in 0..10{
let my_rwlock = arc.clone();
thread::spawn(move || {
let reader = my_rwlock.read().unwrap();
// do some stuff
});
}
let mut writer = local_arc.write().unwrap();
writer.x += 1;
}
&mut references are not safe to send to a non-scoped thread, because the thread may still run after the referenced data has been deallocated. Furthermore, after helper returns, the main thread would still be able to mutate stuff, and the spawned thread would also be able to mutate stuff indirectly, which is not allowed in Rust (there can only be one mutable alias for a variable).
Instead, the RwLock should own the data, rather than borrow it. This means helper should receive a Stuff rather than a &mut Stuff.
use std::sync::{Arc, RwLock};
use std::thread;
struct Stuff {
x: i32
}
fn main() {
let mut stuff = Stuff{x: 5};
helper(stuff);
println!("done");
}
fn helper(stuff: Stuff){
let rwlock = RwLock::new(stuff);
let arc = Arc::new(rwlock);
let local_arc = arc.clone();
for _ in 0..10{
let my_rwlock = arc.clone();
thread::spawn(move || {
let reader = my_rwlock.read().unwrap();
// do some stuff
});
}
let mut writer = local_arc.write().unwrap();
writer.x += 1;
}

Resources