I am writing a generic function to deserialize a type from a JSON file using serde_json. Unfortunately, I am struggling with lifetimes. The compiler complains that the string in which I temporarily store the data before I deserialize to the generic type does not live long enough.
This is my function:
use std::path::Path;
use serde::Deserialize;
fn load_data<'a, T: Deserialize<'a>>(location: &str) -> Result<T, &'a str> {
let data_path = Path::new(&location);
let mut data = String::new();
let mut f = File::open(&data_path).expect("Unable to open file stream.");
f.read_to_string(&mut data).expect("Unable to read file to string.");
let result: T = serde_json::from_str(&data).expect("Unable to deserialize data.");
Ok(result)
}
And this is the error message from the compiler:
error[E0597]: `data` does not live long enough
--> src/templates/card_service_templates.rs:21:42
|
14 | fn load_data<'a, T: Deserialize<'a>>(location: &str) -> Result<T, &'a str> {
| -- lifetime `'a` defined here
...
21 | let result: T = serde_json::from_str(&data).expect("Unable to deserialize data.");
| ---------------------^^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `data` is borrowed for `'a`
22 | Ok(result)
23 | }
| - `data` dropped here while still borrowed
Could the issue be that data needs to outlive the function?
Using T: DeserializeOwned as suggested worked out well. This is the adjusted version of my code:
use std::path::Path;
use serde::de::DeserializeOwned;
fn load_data<'a, T: DeserializeOwned>(location: &str) -> Result<T, &'a str> {
let data_path = Path::new(&location);
let mut data = String::new();
let mut f = File::open(&data_path).expect("Unable to open file stream.");
f.read_to_string(&mut data).expect("Unable to read file to string.");
let result: T = serde_json::from_str(&data).expect("Unable to deserialize data.");
Ok(result)
}
The actual type only needs to derive Deserialize, e.g.
use serde::Deserialize;
#[derive(Deserialize, Debug)]
pub struct MyType {
...
}
Related
I'm trying to implement parts of the minecraft protocol (https://wiki.vg/).
I've successfully implemented a decoder to decode packets. But I'm stuck with the encoding part. The thing is that the minecraft protocol uses "minecraft varints" (https://wiki.vg/Protocol#VarInt_and_VarLong) and I want to have methods to write data as varints.
So my goal is to have a trait named Encoder with these methods:
fn write_var_int(&mut self, value: i32) -> Result<(), error::EncodeError>;
fn write_var_long(&mut self, value: i64) -> Result<(), error::EncodeError>;
fn write_string(&mut self, value: &str) -> Result<(), error::EncodeError>;
At the moment, I've only written the code for the first method:
fn write_var_int(&mut self, mut value: i32) -> Result<(), error::EncodeError> {
loop {
let mut byte = (value & 0b01111111) as u8;
if byte == 0 {
self.write_u8(byte).unwrap();
break;
}
self.write_u8(byte | 0b10000000).unwrap();
value = value >> 7;
}
Ok(())
}
In main.rs I import the module Encoder and I try to use it on a cursor :
let test = [0; 17];
let mut wrt = Cursor::new(test);
wrt.write_var_int(packet.id);
But I get these compilation errors:
error[E0599]: the method `write_var_int` exists for struct `std::io::Cursor<[{integer}; 17]>`, but its trait bounds were not satisfied
--> src/main.rs:57:37
|
57 | ... wrt.write_var_int(packet.id);
| ^^^^^^^^^^^^^ method cannot be called on `std::io::Cursor<[{integer}; 17]>` due to unsatisfied trait bounds
|
::: /home/clement/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/cursor.rs:75:1
|
75 | pub struct Cursor<T> {
| --------------------
| |
| doesn't satisfy `std::io::Cursor<[{integer}; 17]>: Encoder`
| doesn't satisfy `std::io::Cursor<[{integer}; 17]>: std::io::Write`
|
note: the following trait bounds were not satisfied because of the requirements of the implementation of `Encoder` for `_`:
`std::io::Cursor<[{integer}; 17]>: std::io::Write`
--> src/protocol/encoder.rs:12:16
|
12 | impl<W: Write> Encoder for W {
| ^^^^^^^ ^
warning: unused import: `protocol::encoder::Encoder`
--> src/main.rs:11:5
|
11 | use protocol::encoder::Encoder;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0599`.
warning: `eupim` (bin "eupim") generated 2 warnings
error: could not compile `eupim` due to previous error; 2 warnings emitted
I don't understand why the import of the Encoder module is marked as unused, and how to fix these errors.
I'll be pleased to have hints on how to fix this.
Thanks!
I tried to turn the code in your question into a minumal reproducible example:
use std::io::{Cursor, Write};
use std::slice;
mod error {
#[derive(Debug)]
pub enum EncodeError { }
}
trait Encoder {
fn write_u8(&mut self, value: u8) -> Result<(), error::EncodeError>;
fn write_var_int(&mut self, value: i32) -> Result<(), error::EncodeError>;
}
impl<W: Write> Encoder for W {
fn write_u8(&mut self, value: u8) -> Result<(), error::EncodeError> {
self.write(slice::from_ref(&value)).unwrap();
Ok(())
}
fn write_var_int(&mut self, mut value: i32) -> Result<(), error::EncodeError> {
loop {
let mut byte = (value & 0b01111111) as u8;
if byte == 0 {
self.write_u8(byte).unwrap();
break;
}
self.write_u8(byte | 0b10000000).unwrap();
value = value >> 7;
}
Ok(())
}
}
fn main() {
let test = [0; 17];
let mut wrt = Cursor::new(test);
wrt.write_var_int(3);
}
This produces the error:
Compiling playground v0.0.1 (/playground)
error[E0599]: the method `write_var_int` exists for struct `std::io::Cursor<[{integer}; 17]>`, but its trait bounds were not satisfied
--> src/main.rs:39:9
|
39 | wrt.write_var_int(3);
| ^^^^^^^^^^^^^ method cannot be called on `std::io::Cursor<[{integer}; 17]>` due to unsatisfied trait bounds
|
note: the following trait bounds were not satisfied because of the requirements of the implementation of `Encoder` for `_`:
`std::io::Cursor<[{integer}; 17]>: std::io::Write`
--> src/main.rs:14:16
|
14 | impl<W: Write> Encoder for W {
| ^^^^^^^ ^
For more information about this error, try `rustc --explain E0599`.
error: could not compile `playground` due to previous error
The problem here is that we've got a Cursor<[{integer}; 17]>, but Write is only implemented for a Cursor<&mut [u8]>, where &mut [u8] is a mutable reference to a u8 slice.
This makes sense if you think about it: Cursor wraps an in-memory buffer, but there's no need for it to take ownership of it.
So let's make sure we're passing a mutable slice to our Cursor:
fn main() {
let mut test = [0u8; 17];
let mut wrt = Cursor::new(&mut test[..]);
wrt.write_var_int(3);
}
This compiles as expected.
I want to flatten a vector containing &[u8]:
fn main() {
let x: &[u8] = &b"abc"[..];
let y: &[u8] = &b"def"[..];
let z: Vec<&[u8]> = vec![x, y];
println!("z: {:?}", z);
let z_flat: &[u8] = z.into_iter().flatten().collect();
println!("z_flat: {:?}", z_flat);
}
I get the compiler error:
error[E0277]: a value of type `&[u8]` cannot be built from an iterator over elements of type `&u8`
--> src/main.rs:9:49
|
9 | let z_flat: &[u8] = z.into_iter().flatten().collect();
| ^^^^^^^ value of type `&[u8]` cannot be built from `std::iter::Iterator<Item=&u8>`
|
= help: the trait `FromIterator<&u8>` is not implemented for `&[u8]`
I have also tried the following:
let z_flat: &[u8] = &z.into_iter().map(|x| x.to_vec()).flatten().collect();
Compile error:
error[E0277]: a value of type `[u8]` cannot be built from an iterator over elements of type `u8`
--> src/main.rs:9:70
|
9 | let z_flat: &[u8] = &z.into_iter().map(|x| x.to_vec()).flatten().collect();
| ^^^^^^^ value of type `[u8]` cannot be built from `std::iter::Iterator<Item=u8>`
|
= help: the trait `FromIterator<u8>` is not implemented for `[u8]`
The following does work (playground):
let z_flat: &[u8] = &z.concat();
I am not able to use concat in my real-world application because I am using another struct when retrieving the nested bytes (x and y) and this struct cannot implement Copy (only Clone) because it contains a Vec<u8> field. I have something similar to this (playground):
#[derive(Clone, Debug)]
struct Foo {
Bar: Vec<u8>,
}
impl Foo {
fn baz(&self) -> Vec<u8> {
self.Bar
}
}
fn main() {
let x = Foo { Bar: vec![8u8] };
let y = Foo { Bar: vec![18u8] };
let z: Vec<Foo> = vec![x, y];
println!("z: {:?}", z);
let z_flat = z
.into_iter()
.map(|x| &x.baz()[..])
.collect::<Vec<&[u8]>>()
.concat();
println!("z_flat: {:?}", z_flat);
}
Compile error:
error[E0507]: cannot move out of `self.Bar` which is behind a shared reference
--> src/main.rs:8:9
|
8 | self.Bar
| ^^^^^^^^ move occurs because `self.Bar` has type `Vec<u8>`, which does not implement the `Copy` trait
error[E0515]: cannot return value referencing temporary value
--> src/main.rs:22:18
|
22 | .map(|x| &x.baz()[..])
| ^-------^^^^
| ||
| |temporary value created here
| returns a value referencing data owned by the current function
No, it is not possible. By definition, a slice is one contiguous region of memory. You don't have that to start with, so you can't create a slice.
Your workaround copies everything into one contiguous chunk of memory, which you can then get a slice of. I'd write it something like:
let z: Vec<&[u8]> = vec![b"abc", b"def"];
let z_alloc: Vec<u8> = z.iter().flat_map(|&x| x).copied().collect();
let z_flat: &[u8] = &z_alloc;
println!("z_flat: {:?}", z_flat);
For your larger example:
// fn baz(self) -> Vec<u8>
let z_flat: Vec<u8> = z.into_iter().flat_map(Foo::baz).collect();
// fn baz(&self) -> &[u8]
let z_flat: Vec<u8> = z.iter().flat_map(Foo::baz).copied().collect();
See also:
How to get a slice from an Iterator?
I am in a &mut self function, I have a member field that is a Vec<u8>. I want to call a windows xxxA function (via the excellent winapi crate). I have no choice on the fn signature, it is implementing a trait.
I think I need to make a CString. So I tried
fn flush(&mut self) -> std::io::Result<()> {
unsafe {
let str = CString::new(self.buf).unwrap();
OutputDebugStringA(str.as_ptr());
}
Ok(())
}
this doesn't work.
error[E0507]: cannot move out of `self.buf` which is behind a mutable reference
--> src\windbg.rs:51:36
|
51 | let str = CString::new(self.buf).unwrap();
| ^^^^^^^^ move occurs because `self.buf` has type `std::vec::Vec<u8>`, which does not implement the `Copy` trait
I've read the explain of this but none of the 3 suggested solutions seem possible.
Here is the whole struct:
struct WinDbgWriter {
buf: Vec<u8>,
}
use std::io::Write;
impl std::io::Write for WinDbgWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.buf.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
unsafe {
let str = CString::new(self.buf).unwrap();
OutputDebugStringA(str.as_ptr());
}
Ok(())
}
}
If we take your minimized case and try to borrow self.mut to avoid moving it (transferring its ownership), we get a new error which has a suggestion:
error[E0277]: the trait bound `Vec<u8>: From<&Vec<u8>>` is not satisfied
--> src/lib.rs:9:33
|
9 | let _str = CString::new(&self.buf).unwrap();
| ^^^^^^^^^
| |
| the trait `From<&Vec<u8>>` is not implemented for `Vec<u8>`
| help: consider adding dereference here: `&*self.buf`
|
= help: the following implementations were found:
<Vec<T> as From<&[T]>>
<Vec<T> as From<&mut [T]>>
<Vec<T> as From<BinaryHeap<T>>>
<Vec<T> as From<Box<[T]>>>
and 6 others
= note: required because of the requirements on the impl of `Into<Vec<u8>>` for `&Vec<u8>`
If we follow that suggestion, or explicitly coerce self.buf to a &[u8], then the code compiles.
CString::new takes an argument of some type Into<Vec<u8>>, but self.buf is, at this point of type &Vec<u8> because of it being accessed through a &self borrow, and there's no impl Into<Vec<u8>> for &Vec<u8>.
Here's a simple application that duplicates 2 times on stdout the contents of stdin:
use std::{
io,
io::{stdin, stdout, Read, Write},
num::NonZeroUsize,
};
fn dup_input(
input: &mut Box<dyn Read>,
output: &mut Box<dyn Write>,
count: NonZeroUsize,
) -> io::Result<()> {
let mut buf = Vec::new();
input.read_to_end(&mut buf)?;
for _idx in 0..count.get() {
output.write_all(&buf)?;
}
Ok(())
}
fn main() {
let mut input: Box<dyn Read> = Box::new(stdin());
let mut output: Box<dyn Write> = Box::new(stdout());
dup_input(&mut input, &mut output, NonZeroUsize::new(2).unwrap())
.expect("Failed to duplicate input");
}
This part works fine. I want to put a unit test on top of this and this is where the problem lies. The closest I've got to build is with the following attempt:
#[cfg(test)]
mod tests {
use super::*;
use std::{any::Any, io::Cursor};
#[test]
fn test() {
let mut input: Box<dyn Read> = Box::new(Cursor::new([b't', b'e', b's', b't', b'\n']));
let mut output: Box<dyn Write + Any> = Box::new(Vec::<u8>::new());
assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
assert_eq!(output.downcast::<Vec<u8>>().unwrap().len(), 15);
}
}
but rust 1.41.0 doesn't agree:
$ cargo test
Compiling unbox-example v0.1.0 (/XXX/unbox-example)
error[E0225]: only auto traits can be used as additional traits in a trait object
--> src/main.rs:39:41
|
39 | let mut output: Box<dyn Write + Any> = Box::new(Vec::<u8>::new());
| ----- ^^^
| | |
| | additional non-auto trait
| | trait alias used in trait object type (additional use)
| first non-auto trait
| trait alias used in trait object type (first use)
error[E0308]: mismatched types
--> src/main.rs:41:39
|
41 | assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
| ^^^^^^^^^^^ expected trait `std::io::Write`, found a different trait `std::io::Write`
|
= note: expected mutable reference `&mut std::boxed::Box<(dyn std::io::Write + 'static)>`
found mutable reference `&mut std::boxed::Box<(dyn std::io::Write + 'static)>`
error[E0599]: no method named `downcast` found for type `std::boxed::Box<(dyn std::io::Write + 'static)>` in the current scope
--> src/main.rs:43:27
|
43 | assert_eq!(output.downcast::<Vec<u8>>().unwrap().len(), 15);
| ^^^^^^^^ method not found in `std::boxed::Box<(dyn std::io::Write + 'static)>`
error: aborting due to 3 previous errors
Some errors have detailed explanations: E0225, E0308, E0599.
For more information about an error, try `rustc --explain E0225`.
error: could not compile `unbox-example`.
To learn more, run the command again with --verbose.
Is there a way to change the unit test without changing the main code? Note: I could have used generics while implementing dup_input and make the problem significantly easier to solve but this code is part of a broader application and I have to use Read/Write trait objects.
Passing a mutable reference to a Box to dup_input is unnecessarily complicated. You can simply pass a mutable reference to the trait object.
use std::{
io,
io::{stdin, stdout, Read, Write},
num::NonZeroUsize,
};
fn dup_input(
input: &mut dyn Read,
output: &mut dyn Write,
count: NonZeroUsize,
) -> io::Result<()> {
let mut buf = Vec::new();
input.read_to_end(&mut buf)?;
for _idx in 0..count.get() {
output.write_all(&buf)?;
}
Ok(())
}
fn main() {
let mut input = stdin();
let mut output = stdout();
dup_input(&mut input, &mut output, NonZeroUsize::new(2).unwrap())
.expect("Failed to duplicate input");
}
With this version, the test can be written like this:
#[cfg(test)]
mod tests {
use super::*;
use std::io::Cursor;
#[test]
fn test() {
let mut input = Cursor::new([b't', b'e', b's', b't', b'\n']);
let mut output = Vec::<u8>::new();
assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
assert_eq!(output.len(), 15);
}
}
We don't need to use Any at all here: output is simply a Vec<u8>.
I'm implementing a small utility and the compiler is telling me that a variable (a TcpStream) does not live long enough and is advising me to find a way to make it live exactly as long as it is currently living.
Error message
error[E0597]: `stream` does not live long enough
--> src/main.rs:47:35
|
47 | match handle_request(&mut stream){
| ^^^^^^ borrowed value does not live long enough
...
54 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the function body at 43:1...
--> src/main.rs:43:1
|
43 | / fn handle_array(stream: &mut BufReader<TcpStream>) -> Result<Data,Errors>
44 | | {
45 | | let mut array: Vec<Data> = Vec::with_capacity(50);//arbitrary size, picked differently in the complete program
46 | | for _x in 0..50 {
... |
53 | | Ok(Data::Array(array))
54 | | }
| |_^
Code
Rust playground snippet with the exact problem
use std::collections::HashMap;
use std::io::BufReader;
use std::io::Read;
use std::net::TcpStream;
static TOKEN: &[u8; 2] = b"\r\n";
fn main() {}
#[derive(Debug, Clone)]
pub enum Data {
String(Vec<u8>),
Error(Vec<u8>),
Integer(i64),
Binary(Vec<u8>),
Array(Vec<Data>),
Dictionary(HashMap<String, Data>),
}
#[derive(Debug, Clone)]
pub enum Errors<'a> {
CommandError(&'a str),
EndOfConnection,
NotImplemented,
}
pub fn handle_request(stream: &mut BufReader<TcpStream>) -> Result<Data, Errors> {
//TODO handle the empty stream
let mut buff: [u8; 1] = *b"0";
stream.read_exact(&mut buff); //TODO: handle error here
match &buff {
/* part skipped, not relevant */
b"*" => handle_array(stream),
&[_] => Err(Errors::CommandError("Bad request")),
}
}
/*part skipped, not relevant */
fn handle_array(stream: &mut BufReader<TcpStream>) -> Result<Data, Errors> {
let mut array: Vec<Data> = Vec::with_capacity(50); //arbitrary size, picked differently in the complete program
for _x in 0..50 {
match handle_request(&mut stream) {
Ok(x) => array.push(x.clone()),
Err(x) => return Err(x.clone()),
}
}
Ok(Data::Array(array))
}
I'm really stuck on this one.
It seems that I can't use the value of the Err. If I replace
match handle_request(&mut stream){
Ok(x) => array.push(x.clone()),
Err(x) => return Err(x.clone()),
}
with
match handle_request(&mut stream){
Ok(x) => array.push(x.clone()),
Err(_) => return Err(Errors::NotImplemented),
}
the problem is solved, but I don't know why.
Your problem can be reduced to this:
struct Reader;
struct Data;
struct Errors<'a>(&'a str);
fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
for _ in 0..0 {
handle_request(&mut stream)?;
}
unimplemented!();
}
fn handle_request(_stream: &mut Reader) -> Result<Data, Errors> {
unimplemented!()
}
fn main() {}
error[E0597]: `stream` does not live long enough
--> src/main.rs:7:29
|
7 | handle_request(&mut stream)?;
| ^^^^^^ borrowed value does not live long enough
...
11 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the anonymous lifetime #1 defined on the function body at 5:1...
--> src/main.rs:5:1
|
5 | / fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
6 | | for _ in 0..0 {
7 | | handle_request(&mut stream)?;
8 | | }
9 | |
10 | | unimplemented!();
11 | | }
| |_^
In the body of handle_array, stream is of type &mut Reader. However, when calling handle_request, you take another reference to it, creating a &mut &mut Reader.
Adding some explicit lifetimes to the code (for educational purposes, this doesn't compile), the code would look kind of like this:
fn handle_array<'stream>(stream: &'stream mut Reader) -> Result<Data, Errors> {
let tmp: &'tmp mut &'stream mut Reader = &mut stream;
if let Err(x) = handle_request(tmp)
handle_request requires a &mut Reader, so the compiler inserts some code to align these two types for you. The compiler has to be conservative about how it performs this conversion, so it picks the shorter lifetime:
fn handle_array<'stream>(stream: &'stream mut Reader) -> Result<Data, Errors> {
let tmp: &'tmp mut &'stream mut Reader = &mut stream;
let tmp2: &'tmp mut Reader = tmp;
if let Err(x) = handle_request(tmp2)
The next aspect of the problem is that lifetime elision has kicked in for both functions. Their expanded forms look like:
fn handle_array<'stream>(stream: &'stream mut Reader) -> Result<Data, Errors<'stream>>
fn handle_request<'_stream>(_stream: &_stream mut Reader) -> Result<Data, Errors<'_stream>>
This means that the lifetime of the returned Errors is tied to the lifetime of the argument, but in your case, the argument to handle_request has the shorter 'tmp lifetime, not the lifetime of 'stream. This shows why you get the compiler error: You are trying to return an Errors that can only live inside the function (the lifetime of the variable stream itself), but you are trying to return a reference that needs to live longer.
We can solve this by only passing stream to handle_request:
handle_request(stream)?;
Unfortunately, this only changes the error:
error[E0499]: cannot borrow `*stream` as mutable more than once at a time
--> src/main.rs:9:40
|
9 | if let Err(x) = handle_request(stream) {
| ^^^^^^ mutable borrow starts here in previous iteration of loop
...
15 | }
| - mutable borrow ends here
This part is much harder to explain. See:
"Problem Case #3" of the Non-Lexical Lifetimes RFC (RFC 2094)
Cannot borrow `x` as mutable more than once at a time
Cannot borrow node as mutable more than once while implementing a binary search tree
This is a really rough edge of Rust's right now, but it's getting closer and closer to being fixed! Right now, however, you have two main choices:
Call the function twice
This probably won't work because you can't read from the stream twice, but in other cases it might be useful:
fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
let mut array = vec![];
for _ in 0..0 {
if handle_request(stream).is_err() {
return handle_request(stream);
}
if let Ok(r) = handle_request(stream) {
array.push(r);
};
}
unimplemented!();
}
Remove the references
Give up on trying to have references in this case for now.
struct Errors(String);
fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
let mut array = vec![];
for _ in 0..0 {
array.push(handle_request(stream)?);
}
unimplemented!();
}
Which I'd write using iterators for efficiency:
fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
let array = (0..0)
.map(|_| handle_request(stream))
.collect::<Result<Vec<_>, _>>()?;
unimplemented!();
}
The future?
With the unstable NLL feature and the experimental "Polonius" implementation, this code works:
struct Errors<'a>(&'a str);
fn handle_array(stream: &mut Reader) -> Result<Data, Errors> {
let mut array = vec![];
for _ in (0..0) {
array.push(handle_request(stream)?);
}
unimplemented!();
}
It will just be a while before this is generally available...