How to set a lifetime to a value captured in a closure? - rust

I wrote what I think is simple code:
#![feature(plugin)]
#![plugin(rocket_codegen)]
extern crate rocket;
extern crate statsd;
use rocket::{Data, Request};
use rocket::fairing::AdHoc;
use statsd::Client;
#[get("/")]
fn index() -> &'static str {
"Hello, World"
}
fn main() {
let mut client = Client::new("127.0.0.1:9125", "miniserver-rs").unwrap();
rocket::ignite()
.attach(AdHoc::on_request(|request, data|{
client.incr("http.requests");
println!("Request URI: {}", request.uri());
}))
.mount("/", routes![index])
.launch();
client.incr("server.bootstrap");
}
I try to send some metrics on each request, but I get the following compiler error:
Compiling miniserver-rs v0.1.0 (main.rs)
error[E0373]: closure may outlive the current function, but it borrows `client`, which is owned by the current function
--> src\main.rs:19:33
|
19 | .attach(AdHoc::on_request(|request, _data|{
| ^^^^^^^^^^^^^^^^ may outlive borrowed value `client`
20 | client.incr("http.requests");
| ------ `client` is borrowed here help: to force the closure to take ownership of `client` (and any other referenced variables), use the `move` keyword
|
19 | .attach(AdHoc::on_request(move |request, _data|{
| ^^^^^^^^^^^^^^^^^^^^^
error[E0387]: cannot borrow data mutably in a captured outer variable in an `Fn` closure
--> src\main.rs:20:11
|
20 | client.incr("http.requests");
| ^^^^^^
|
help: consider changing this closure to take self by mutable reference
--> src\main.rs:19:33
|
19 | .attach(AdHoc::on_request(|request, _data|{
| _________________________________^
20 | | client.incr("http.requests");
21 | | println!("Request URI: {}", request.uri());
22 | | }))
| |_______^
I understand that client is captured in a closure and owned by another function (main) that could live less than the closure. I cannot move it because Client doesn't implement Copy, so the reference could not be used afterwards.
I also understand that I cannot borrow mutable data in a closure (Client is mutable). After a lot of search, I can conclude I need to use Arc/Rc in combination with Mutex/RwLock/RefCell, but before going further, I want to be sure it is required.

Let's look at the requirements. You want to call statsd::Client::incr(&mut client, metric) from inside the closure, so you need mutable access to client. This is a variable you close over with ||.
Now AdHoc::on_request<F>(f: F) requires F: Fn(...) + Send + Sync + 'static. Fn means you only have immutable access to your capture via &self. The 'static bound means the capture cannot be a reference itself, so it requires move ||. Finally Sync means you can't use Cell or RefCell to get a mutable reference from &self.client, since Rocket will share it between threads.
Just like you suspected, the canonical solution to have shared mutable access through a Send + Sync value is to use Arc<Mutex<_>>. This also solves the problem of "losing access by moving". Your code would look like the following (untested):
fn main() {
let client = Arc::new(Mutex::new(
Client::new("127.0.0.1:9125", "miniserver-rs").unwrap()));
// shallow-clone the Arc to move it into closure
let rocket_client = client.clone();
rocket::ignite()
.attach(AdHoc::on_request(move |request, data|{
rocket_client.lock()
.unwrap()
.incr("http.requests");
println!("Request URI: {}", request.uri());
}))
.mount("/", routes![index])
.launch();
client.lock()
.unwrap()
.incr("server.bootstrap");
}

Related

Using axum and reqwest to upload a file to cloud bucket, streaming it

I've created a small Github repo that mimics exactly what I'm trying to achieve: streaming file uploads to both file systems and the cloud (S3) using axum (and reqwest).
I can change all the code you see. There is no restriction. All signatures were invented by me who are in my early days with the awesome Rust.
If you want I can also eliminate reqwest, as long as I understand what to use instead! :smile:
Can you help me understand how to fix the only error left?
error[E0521]: borrowed data escapes outside of associated function
--> src\adapter_s3.rs:56:19
|
41 | async fn put_file<'a>(
| -- lifetime `'a` defined here
42 | &'a self,
| -------- `self` is a reference that is only valid in the associated function body
...
56 | .body(reqwest::Body::wrap_stream(ReaderStream::new(reader)))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| `self` escapes the associated function body here
| argument requires that `'a` must outlive `'static`
This is the relevant code I think:
async fn put_file<'a>(
&'a self,
filename: &'a str,
reader: Pin<Box<dyn AsyncRead + Send + Sync + 'a>>,
) -> Result<()> {
let presigned_url_duration = Duration::from_secs(60 * 60);
let action = self.bucket.put_object(Some(&self.credentials), filename);
let client = reqwest::Client::new();
client
.post(action.sign(presigned_url_duration))
// .body(reader)
// .body(reqwest::Body::wrap_stream(reader))
.body(reqwest::Body::wrap_stream(ReaderStream::new(reader)))
.send()
.await?
.error_for_status()?;
Ok(())
}
UPDATE:
I tried with hyper too and it's the same error obviously:
hyper::Client::new()
.request(
hyper::Request::builder()
.uri(action.sign(presigned_url_duration).to_string())
.body(hyper::Body::wrap_stream(ReaderStream::new(reader)))
.unwrap(),
)
.await
.unwrap();
error[E0521]: borrowed data escapes outside of associated function
|
66 | async fn put_file<'a>(
| -- lifetime `'a` defined here
67 | &'a self,
| -------- `self` is a reference that is only valid in the associated function body
...
124 | .body(hyper::Body::wrap_stream(ReaderStream::new(reader)))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| `self` escapes the associated function body here
| argument requires that `'a` must outlive `'static`

access reference in struct causes cannot infer an appropriate lifetime for borrow expression due to conflicting requirements

I'm implementing a web service using tonic grpc crate.
Here is the my code:
use std::cell::RefCell;
use std::ops::{Deref, DerefMut};
use std::sync::Mutex;
fn main() {
let my_struct = MyStruct {data_ref: RefCell::new(None), data: vec![MyData(28)]};
my_struct.my_fun();
// my_struct.my_fun_is_gone(); This is not working.
println!("{}", my_struct.data_ref.borrow().deref().unwrap().0);
}
/// AUTO GENERATED CODE OR THIRD PARTY TRAITS
///for example tonic generated Grpc services
trait TonicGeneratedGrpcService {
fn my_fun_is_gone(&self); //so I Can't change the &self lifetime parameter
}
struct MyData(u8);
struct MyStruct<'a> {
data: Vec<MyData>, //this struct is owns this property, there is no any problem
data_ref: RefCell<Option<&'a MyData>>, //And sometimes I want to save the data
}
impl<'a> TonicGeneratedGrpcService for MyStruct<'a> {
fn my_fun_is_gone(&self) { //I can't change the &self lifetime parameter because this is just an implementation of some super traits, change the lifetime parameters also requires change the super trait's method's lifetime.
//***********COMPILER ERROR is occurred HERE, how can I make this working?*************
let some_of_data = &self.data[0]; //Maybe &self.data[1], &self.data[2]...
*self.data_ref.borrow_mut() = Some(some_of_data);
}
}
impl<'a> MyStruct<'a> {
//Below code is works fine if this is myself defined method, but I can't change the lifetime of "self" in super traits if this is a implementation of tonic generated service or other third party traits
fn my_fun(&'a self) {
let some_of_data = &self.data[0]; //Maybe &self.data[1], &self.data[2]...
*self.data_ref.borrow_mut() = Some(some_of_data);
}
}
My final purpose is save the reference that one of the Vec<MyData> into data_ref field at runtime by some conditions.
Here is the compiler error:
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter in function call due to conflicting requirements
--> src/main.rs:28:29
|
28 | let some_of_data = &self.data[0]; //Maybe &self.data[1], &self.data[2]...
| ^^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime defined here...
--> src/main.rs:26:23
|
26 | fn my_fun_is_gone(&self) { //I can't change the &self lifetime parameter because this is just an implementation of some super traits,...
| ^^^^^
note: ...so that reference does not outlive borrowed content
--> src/main.rs:28:29
|
28 | let some_of_data = &self.data[0]; //Maybe &self.data[1], &self.data[2]...
| ^^^^^^^^^
note: but, the lifetime must be valid for the lifetime `'a` as defined here...
--> src/main.rs:25:6
|
25 | impl<'a> TonicGeneratedGrpcService for MyStruct<'a> {
| ^^
note: ...so that the expression is assignable
--> src/main.rs:29:39
|
29 | *self.data_ref.borrow_mut() = Some(some_of_data);
| ^^^^^^^^^^^^^^^^^^
= note: expected `Option<&'a MyData>`
found `Option<&MyData>`
I can't change the source code of TonicGeneratedGrpcService trait because it may be in the third party crate or auto generated code,
and now I cannot find any way to make work the my_fun_is_gone method implementation.
Should I avoid the lifetime and use reference counted pointers (Rc, Arc) rather then save reference (&MyData) directly in struct MyStruct?
Or there any way to make work this?
As others said, Rust's lifetime system doesn't allow self-referential struct.
To solve this, you may directly store the index integer in the data_ref if data vec is only appended. Depending on your use case, you can also consider other options here Why can't I store a value and a reference to that value in the same struct?

Multi-threaded communication with an external process in Rust

A rust newbie here.
I would like to launch an external long-running process and talk with it over pipes from multiple threads in Rust.
I am getting lifetime errors and can't figure the proper way to please the lifetimes checker. What are the ways to restructure this?
Consider the following example:
use std::process::{Command, Stdio, ChildStdin};
use std::sync::Mutex;
use std::io::{Write};
use std::thread;
struct Element {
sink: Mutex<Option<ChildStdin>>
}
impl Element {
fn launch_process(&self) {
let child =
Command::new("sed").args(&["s/foo/bar/g"])
.stdin(Stdio::piped())
.spawn()
.unwrap();
let mut sink = self.sink.lock().unwrap();
*sink = child.stdin;
}
fn tx(&self, content: &[u8]) {
let mut sink = self.sink.lock().unwrap();
sink.as_mut().unwrap().write(content);
}
fn start_tx(&self) {
thread::spawn( || {
self.tx(b"foo fighters");
});
}
}
fn main() {
let e = Element {
sink: Mutex::new(None)
};
e.launch_process();
e.start_tx();
}
If I remove the thread::spawn bit then everything works as expected. With thread::spawn in place, I get the error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:28:24
|
28 | thread::spawn( || {
| ________________________^
29 | | self.tx(b"foo fighters");
30 | | });
| |_________^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the method body at 27:5...
--> src/main.rs:27:5
|
27 | / fn start_tx(&self) {
28 | | thread::spawn( || {
29 | | self.tx(b"foo fighters");
30 | | });
31 | | }
| |_____^
= note: ...so that the types are compatible:
expected &&Element
found &&Element
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#src/main.rs:28:24: 30:10 self:&&Element]` will meet its required lifetime bounds
--> src/main.rs:28:9
|
28 | thread::spawn( || {
| ^^^^^^^^^^^^^
error: aborting due to previous error
You can't pass &self (a temporary borrow) to a thread, because the thread may keep running after the reference is no longer valid.
For using data from threads you have only two options:
Give ownership (which is exclusive) of the object to the thread, i.e. use move || closure, and don't try to use that object afterwards from the main thread, or any other thread.
Wrap the object in Arc to get shared thread-safe ownership, and send a clone to the thread (with Arc::clone it's cheap and the underlying data is shared).
When the compiler says that you need a "static lifetime", ignore that. For all practical purposes, it means "references are not allowed".

What does "borrowed data cannot be stored outside of its closure" mean?

When compiling the following code:
fn main() {
let mut fields = Vec::new();
let pusher = &mut |a: &str| {
fields.push(a);
};
}
The compiler gives me the following error:
error: borrowed data cannot be stored outside of its closure
--> src/main.rs:4:21
|
3 | let pusher = &mut |a: &str| {
| ------ --------- ...because it cannot outlive this closure
| |
| borrowed data cannot be stored into here...
4 | fields.push(a);
| ^ cannot be stored outside of its closure
And in later versions of Rust:
error[E0521]: borrowed data escapes outside of closure
--> src/main.rs:4:9
|
2 | let mut fields = Vec::new();
| ---------- `fields` declared here, outside of the closure body
3 | let pusher = &mut |a: &str| {
| - `a` is a reference that is only valid in the closure body
4 | fields.push(a);
| ^^^^^^^^^^^^^^ `a` escapes the closure body here
What does this error mean, and how can I fix my code?
It means exactly what it says: that the data you are borrowing only lives for the duration of the closure. Attempting to store it outside of the closure would expose the code to memory unsafety.
This arises because the inferred lifetime of the closure's argument has no relation to the lifetimes stored in the Vec.
Generally, this isn't a problem you experience because something has caused more type inference to happen. In this case, you can add a type to fields and remove it from the closure:
let mut fields: Vec<&str> = Vec::new();
let pusher = |a| fields.push(a);

When does a closure take ownership of its environment without the move keyword?

use std::thread;
fn test2() {
let x = "abc".to_string();
thread::spawn(|| {
foo2(x);
});
}
fn foo2(x: String) {}
fn test1() {
let x = 1;
thread::spawn(|| {
foo1(x);
});
}
fn foo1(x: i32) {}
fn main() {}
Playground
The error:
error[E0373]: closure may outlive the current function, but it borrows `x`, which is owned by the current function
--> <anon>:12:19
|
12 | thread::spawn(|| { foo1(x); });
| ^^ - `x` is borrowed here
| |
| may outlive borrowed value `x`
|
help: to force the closure to take ownership of `x` (and any other referenced variables), use the `move` keyword, as shown:
| thread::spawn(move || { foo1(x); });
Why does the closure in test1 not take ownership of x, which is specified by the signature (x: i32) of foo1? (I know I can add move to make it work.) I guess it is due to that x is copyable, but if it is copied into the closure then why do I still have the lifetime problem?
However test2 works.
Because it doesn't have to take ownership. Moving is more destructive than simply borrowing, so if the compiler thinks it can get away with not moving a captured value, it won't. It moves the String because it has no other option. It borrows the i32 because it's Copy.
But it can't get away with not borrowing it!
Aah, but the compiler doesn't know that until after it's decided if it's borrowing or moving a captured value. The heuristic it uses is just that: not always correct.
Couldn't it just work it out properly?
Probably, but no one's taught it how.

Resources