Lifetime error with Serde in a Generic Function - rust

I am trying to write a generic function that spawns a thread to deal with interaction with a server. Unfortunately, I am encountering a borrow bug that I don't understand.
This is the code of my function :
fn spawn_response_thread<
'a,
MessageT:serde::Serialize + Send + 'static,
ResponseT: serde::Deserialize<'a> + 'static,
ResponseHandlingT: Fn(Result<ResponseT, String>) + Send + 'static
>
(
message : MessageT,
server_address : Option<(String,u16)>,
response_handler: ResponseHandlingT,
max_wait_time : Option<u64>
)
{
thread::spawn(
move ||
{
// First connect to a specific or random server
/**/
// Then send the desired request
/**/
// Loop to try receiving the message for a given time
let mut nb_wait = 0;
let mut buf;
let mut received = false;
/**/
// Send a response if there is one
if !received
{
response_handler(Err(format!("No response from server")));
}
else
{
let response : bincode::Result<ResponseT> = deserialize(&buf[..]);
match response
{
Ok(rcv) =>
{
response_handler(Ok(rcv));
}
Err(_) =>
{
response_handler(Err(format!("Error when trying to read response from server")));
}
}
}
} // <- line 353
);
}
(I cut parts of the code because I feel like there were irrelevant)
With this code, I am getting the following compilation error :
error[E0597]: `buf` does not live long enough
--> src/network.rs:340:81
|
230 | 'a,
| -- lifetime `'a` defined here
...
340 | let response : bincode::Result<ResponseType> = deserialize(&buf[..]);
| -------------^^^-----
| | |
| | borrowed value does not live long enough
| argument requires that `buf` is borrowed for `'a`
...
353 | }
| - `buf` dropped here while still borrowed
I don't understand why buf would still be borrowed after deserialization.
What am I doing wrong?

The serde's Deserialize trait has a lifetime 'de. This lifetime is used to implemented zero-copy deserialization: avoiding copying e.g. strings, and instead using the already-exisiting buffer we deserialize from.
But if your object outlives the buffer, it cannot depend on it! In this case, you need a deserialization that lives forever, or for<'de> Deserialize<'de>. Serde has a trait for that, DeserializeOwned, which is essentially this: a shortcut for for<'de> Deserialize<'de>. So what you need if to use DeserializeOwned instead of Deserialize:
fn spawn_response_thread<
MessageT: serde::Serialize + Send + 'static,
ResponseT: serde::de::DeserializeOwned + 'static,
ResponseHandlingT: Fn(Result<ResponseT, String>) + Send + 'static,
>(
message: MessageT,
server_address: Option<(String, u16)>,
response_handler: ResponseHandlingT,
max_wait_time: Option<u64>,
) {
// ...
}
For more take a look at Understanding deserializer lifetimes - serde.rs.

Related

Call a C callback from a Send closure

In Rust I have a callback defined as Box<dyn Fn(MyType) + Send>. I want to be able to convert a C callback provided by FFI into that signature.
To do that, I created a Rust closure that calls the C callback with a context pointer:
let rust_closure = move |data: u32| {
callback(obj, data);
};
The problem is obj in this case is a c_void. According to the docs (https://doc.rust-lang.org/core/ffi/enum.c_void.html) it should have auto-implemented Send and Sync. However the compiler tells me:
the trait Send is not implemented for *const c_void
How can I convince Rust that my c_void is indeed Send? I've tried wrapping it in a unit struct and impl'ing Send for the struct, but that did not change the error.
I've created a minimal example in the playground.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=5737c7449b925ab83e35d2404a6bd9f8
Pointers are automatically !Send so the solution is to wrap the pointer in a custom type, that hopefully better describes its use and reasons to guarantee that the pointer is safe to use across threads. I'm assuming you made a solution similar to this:
struct UserObj(*const c_void);
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.0, data);
};
ch.add_callback(Box::new(rust_closure));
Which unfortunately, does yield the same error:
error[E0277]: `*const c_void` cannot be sent between threads safely
--> src/lib.rs:33:25
|
29 | let rust_closure = move |data: u32| {
| ____________________________-
30 | | callback(obj.0, data);
31 | | };
| |_________- within this `[closure#src/lib.rs:29:28: 31:10]`
32 |
33 | ch.add_callback(Box::new(rust_closure));
| ^^^^^^^^^^^^^^^^^^^^^^ `*const c_void` cannot be sent between threads safely
|
= help: within `[closure#src/lib.rs:29:28: 31:10]`, the trait `Send` is not implemented for `*const c_void`
= note: required because it appears within the type `[closure#src/lib.rs:29:28: 31:10]`
= note: required for the cast to the object type `dyn Fn(u32) + Send`
But why? We're moving a UserObj into the closure and we've asserted to the compiler that it is Send. Why does it still complain about *const c_void?
As of Rust 2021 edition, closures can capture disjoint fields rather than the whole object, meaning the above closure would capture obj.0 instead of obj itself, causing the error. My suggestion would be to add a method to get the pointer instead of using field access in this case:
struct UserObj(*const c_void);
impl UserObj {
fn as_ptr(&self) -> *const c_void {
self.0
}
}
// SAFETY: Any user data object must be safe to send between threads.
unsafe impl Send for UserObj {}
let obj = UserObj(obj);
let rust_closure = move |data: u32| {
callback(obj.as_ptr(), data);
};
ch.add_callback(Box::new(rust_closure));
Playground

Custom data format - `Deserializer::deserialize_str` implementation

Link to playground
I am trying to implement a custom data format with serde, I've been struggling with the deserialize_str method
pub struct Deserializer<R> {
rdr: R,
}
impl<'de, 'a, R: io::Read + 'de> de::Deserializer<'de> for &'a mut Deserializer<R> {
fn deserialize_str<V>(self, visitor: V) -> Result<V::Value>
where
V: Visitor<'de>,
{
let len = self.read_i16()?; // implemention below
if len == 0 || len == -1 {
return visitor.visit_borrowed_str("");
}
let len = len as usize;
let buf = self.read_exact(len)?; // implemention below
let out_str = std::str::from_utf8(&buf)?;
// visitor.visit_borrowed_str(out_str) doesn't compile
visitor.visit_str(out_str) // compiles but errors
}
}
impl<R: io::Read> Deserializer<R> {
fn read_exact(&mut self, len: usize) -> Result<Vec<u8>> {
let mut buf = vec![0; len];
self.rdr.read_exact(&mut buf)?;
Ok(buf)
}
fn read_i16(&mut self) -> io::Result<i8> {
self.rdr.read_i16::<byteorder::NetworkEndian>()
}
}
When using visitor.visit_borrowed_str(out_str), I get the error
|
94 | impl<'de, 'a, R: io::Read + 'de> de::Deserializer<'de> for &'a mut Deserializer<R> {
| --- lifetime `'de` defined here
...
149 | let out_str = std::str::from_utf8(&buf)?;
| ^^^^ borrowed value does not live long enough
150 |
151 | visitor.visit_borrowed_str(out_str)
| ----------------------------------- argument requires that `buf` is borrowed for `'de`
152 | }
| - `buf` dropped here while still borrowed
I understand that out_str needs to somehow live longer than its scope, but I can't find a way to go about it.
To use visit_borrowed_str, you need to hand it a reference to something that lives as long as your deserializer. Creating a new temporary Vec with read_exact won't do, you need to get access to the underlying slice, e.g. std::str::from_utf8(self.rdr.get_ref()[self.rdr.position()..][..len]) or similar. If you want to keep R a generic std::io::Read, I think you can't use visit_borrowed_str. serde_json e.g. handles this by having a special Read that returns a reference to the underlying data if it can, and then only uses visit_borrowed_str if it does have a reference to the underlying data.
Also, if you ask a deserializer to deserialize to a borrowed string when it can't, it must necessarily error. That holds true for serde_json as well. So the error from visit_str is not an error in your deserializer implementation, but an error in how you use the deserializer. You should have asked to deserialize to a String or Cow<str> instead (not that your serializer could ever give you a Cow::Borrowed, but asking for a &str just isn't a good idea with any deserializer, asking for a Cow<str> is the thing generally recommended instead).

Rust cloning HashMap<String, Object> without moving into closure

I am trying to make my own programming language in rust, and most features are done, so I thought I could add to the UnknownIdentifier error the ability to find the closest match to whatever the user wanted
However before I even got to finding the closest match I found out that cloning HashMap<String, Object> moves it into the closure
ErrorGenerator::error function:
#[allow(non_snake_case)]
mod ErrorGenerator {
pub fn error(name: &str, explanation: &str, line: usize, col: usize, file: String, after_f: Box<dyn Fn() -> ()>) -> ! {
eprintln!("\n[ERROR] {}, Line {:?}, Column {:?}", file, line, col);
eprintln!(" {}: {}", name, explanation);
after_f();
exit(1);
}
}
ErrorGenerator::error(
"UnknownIdentifier",
&format!("unknown identifier: `{}`, this identifier could not be found", tokenc.repr()),
tokenc.line,
tokenc.col,
tokenc.file,
Box::new(||{
let mut hashc: Vec<String> = hashs.clone().into_keys().collect();
hashc.sort();
}),
);
This is the error it gives:
error[E0597]: `hashs` does not live long enough
--> src/runtime/runtime.rs:960:70
|
959 | Box::new(||{
| - -- value captured here
| _____________________________________|
| |
960 | | let mut hashc: Vec<String> = hashs.clone().into_keys().collect();
| | ^^^^^ borrowed value does not live long enough
961 | | hashc.sort();
962 | | }),
| |______________________________________- cast requires that `hashs` is borrowed for `'static`
...
1203 | }
| - `hashs` dropped here while still borrowed
The problem's solution is probably either:
A way to borrow in 'static lifetime a mutable variable created in a method into a closure
or
A way to clone HashMap<String, Object> without moving it into the closure
You can find the full code in https://github.com/kaiserthe13th/tr-lang/tree/unknown-id-err-impl
What happens is that the compiler doesn't clone hashs then passes the clone to your callback; instead, it passes a reference to hashs to your callback and clones it inside the callback.
However, the callback is required to be 'static, and if it holds a reference to the containing function it is not! So the compiler is complaining.
What you want is to clone the hashmap before, then pass the clone to the callback. Like:
ErrorGenerator::error(
"UnknownIdentifier",
&format!("unknown identifier: `{}`, this identifier could not be found", tokenc.repr()),
tokenc.line,
tokenc.col,
tokenc.file,
{
let hashc = hashs.clone();
Box::new(|| {
let mut hashc: Vec<String> = hashc.into_keys().collect();
hashc.sort();
})
},
);
If you'll do that, you'll also recognize that the closure needs to be FnOnce() since you're moving out of hashc (.into_keys()). So after_f: Box<dyn FnOnce()>.

"cannot infer an appropriate lifetime" when attempting to return a chunked response with hyper

I would like to return binary data in chunks of specific size. Here is a minimal example.
I made a wrapper struct for hyper::Response to hold my data like status, status text, headers and the resource to return:
pub struct Response<'a> {
pub resource: Option<&'a Resource>
}
This struct has a build method that creates the hyper::Response:
impl<'a> Response<'a> {
pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
let mut response = hyper::Response::builder();
match self.resource {
Some(r) => {
let chunks = r.data
.chunks(100)
.map(Result::<_, std::convert::Infallible>::Ok);
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
},
None => response.body(hyper::Body::from("")),
}
}
}
There is also another struct holding the database content:
pub struct Resource {
pub data: Vec<u8>
}
Everything works until I try to create a chunked response. The Rust compiler gives me the following error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:14:15
|
14 | match self.resource {
| ^^^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the impl at 11:6...
--> src/main.rs:11:6
|
11 | impl<'a> Response<'a> {
| ^^
note: ...so that the types are compatible
--> src/main.rs:14:15
|
14 | match self.resource {
| ^^^^^^^^^^^^^
= note: expected `Option<&Resource>`
found `Option<&'a Resource>`
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the types are compatible
--> src/main.rs:19:31
|
19 | response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
| ^^^^^^^^^^^^^^^^^^^^^^^^
= note: expected `From<&[u8]>`
found `From<&'static [u8]>`
I don't know how to fulfill these lifetime requirements. How can I do this correctly?
The problem is not in the 'a itself, but in the fact that the std::slice::chunks() function returns an iterator that borrows the original slice. You are trying to create a stream future from this Chunks<'_, u8> value, but the stream requires it to be 'static. Even if your Resource did not have the 'a lifetime, you would still have the r.data borrowed, and it would still fail.
Remember that here 'static does not mean that the value lives forever, but that it can be made to live as long as necessary. That is, the future must not hold any (non-'static) borrows.
You could clone all the data, but if it is very big, it can be costly. If so, you could try using Bytes, that is just like Vec<u8> but reference counted.
It looks like there is no Bytes::chunks() function that returns an iterator of Bytes. Fortunately it is easy to do it by hand.
Lastly, remember that iterators in Rust are lazy, so they keep the original data borrowed, even if it is a Bytes. So we need to collect them into a Vec to actually own the data (playground):
pub struct Resource {
pub data: Bytes,
}
impl<'a> Response<'a> {
pub fn build(&mut self) -> Result<hyper::Response<hyper::Body>, hyper::http::Error> {
let mut response = hyper::Response::builder();
match self.resource {
Some(r) => {
let len = r.data.len();
let chunks = (0..len)
.step_by(100)
.map(|x| {
let range = x..len.min(x + 100);
Ok(r.data.slice(range))
})
.collect::<Vec<Result<Bytes, std::convert::Infallible>>>();
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))
}
None => response.body(hyper::Body::from("")),
}
}
}
UPDATE: We can avoid the call to collect() if we notice that stream::iter() takes ownership of an IntoIterator that can be evaluated lazily, as long as we make it 'static. It can be done if we do a (cheap) clone of r.data and move it into the lambda (playground):
let data = r.data.clone();
let len = data.len();
let chunks = (0..len).step_by(100)
.map(move |x| {
let range = x .. len.min(x + 100);
Result::<_, std::convert::Infallible>::Ok(data.slice(range))
});
response.body(hyper::Body::wrap_stream(stream::iter(chunks)))

How to pass immutable parameters to a thread? (about lifetimes)

Let thread state consists of immutable parameters Params and the rest of the (mutable) state State.
I am trying to mock spawning a thread that does something being controlled by parameters Params:
use std::thread;
struct Params {
x: i32,
}
struct State<'a> {
params: &'a Params,
y: i32,
}
impl<'a> State<'a> {
fn new(params: &Params) -> State {
State {
params,
y: 0,
}
}
fn start(&mut self) -> thread::JoinHandle<()> {
let params = self.params.clone();
thread::spawn(move || { params; /* ... */ })
}
}
But this does not work:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> test.rs:20:34
|
20 | let params = self.params.clone();
| ^^^^^
|
note: first, the lifetime cannot outlive the lifetime `'a` as defined on the impl at 12:6...
--> test.rs:12:6
|
12 | impl<'a> State<'a> {
| ^^
note: ...so that the types are compatible
--> test.rs:20:34
|
20 | let params = self.params.clone();
| ^^^^^
= note: expected `&&Params`
found `&&'a Params`
= note: but, the lifetime must be valid for the static lifetime...
note: ...so that the type `[closure#test.rs:21:23: 21:42 params:&Params]` will meet its required lifetime bounds
--> test.rs:21:9
|
21 | thread::spawn(move || { params; /* ... */ })
| ^^^^^^^^^^^^^
I understand why it does not work: The thread could run indefinitely long, and the params could be destroyed before it is terminated. That's clearly an error.
Now explain what is the proper way to make params long at least as long as the thread. In other words, help to correct the above code. What should I do with lifetimes?
You got the right idea with using clone and a move lambda, but you forgot one detail: Params isn't Clone! Therefore the compiler did the best it could when it saw self.params.clone() and cloned… the reference.
That's why the error messages have two & here:
= note: expected `&&Params`
found `&&'a Params`
Your issue is solved by using #[derive(Clone)] struct Params { /* … */ }.

Resources