I'm trying to implement deserializer for a BERT data which comes from an another program via sockets. For the following code:
use std::io::{self, Read};
#[derive(Clone, Copy)]
pub struct Deserializer<R: Read> {
reader: R,
header: Option<u8>,
}
impl<R: Read> Read for Deserializer<R> {
#[inline]
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
self.reader.read(buf)
}
}
impl<R: Read> Deserializer<R> {
/// Creates the BERT parser from an `std::io::Read`.
#[inline]
pub fn new(reader: R) -> Deserializer<R> {
Deserializer {
reader: reader,
header: None,
}
}
#[inline]
pub fn read_string(&mut self, len: usize) -> io::Result<String> {
let mut string_buffer = String::with_capacity(len);
self.reader.take(len as u64).read_to_string(&mut string_buffer);
Ok(string_buffer)
}
}
The Rust compiler generates an error, when I'm trying to read a string from a passed data:
error: cannot move out of borrowed content [E0507]
self.reader.take(len as u64).read_to_string(&mut string_buffer);
^~~~
help: run `rustc --explain E0507` to see a detailed explanation
How can I fix this even if my Deserializer<R> struct has had Clone/Copy traits?
The take method takes self:
fn take(self, limit: u64) -> Take<Self> where Self: Sized
so you cannot use it on anything borrowed.
Use the by_ref method. Replace the error line with this:
{
let reference = self.reader.by_ref();
reference.take(len as u64).read_to_string(&mut string_buffer);
}
Related
I have a struct that has an iterator over a borrowed slice, and I want to create a method that returns an iterator that does some computation on the contained iterator
Basically this:
#[derive(Clone)]
struct Foo<'a> {
inner: ::std::iter::Cycle<::std::slice::Iter<'a, u8>>,
}
impl<'a> Foo<'a> {
pub fn new<T: AsRef<[u8]> + ?Sized>(vals: &'a T) -> Foo<'a> {
Self {
inner: vals.as_ref().iter().cycle(),
}
}
pub fn iter(&mut self) -> impl Iterator<Item = u8> + 'a {
self.inner.by_ref().map(Clone::clone) // simple case but this could be any computation
}
}
Which yields a cryptic error.
Here's my understanding: I need to bind the returned iterator's lifetime to the &mut self of iter() so that &mut self is only dropped when .collect() is called on the returned iterator.
BUT, changing to fn iter(&'a mut self) isn't good enough because &'a mut self now lives for the entire duration of the struct instance (meaning that calling collect() doesn't drop that reference). Which prevents something like this
#[test]
fn test(){
let mut foo = Foo::new("hello, there");
foo.iter().zip(0..4).collect::<Vec<_>>(); // call once, ok
foo.iter().zip(0..4).collect::<Vec<_>>(); // error because 2 &mut references exist at the same
}
rust playground link
The way I would do this is to make a struct just for iterating over those items. It makes things much easier.
...
pub fn iter(&mut self) -> ClonedFooIter<'a, '_> {
ClonedFooIter {
inner: &mut self.inner,
}
}
}
pub struct ClonedFooIter<'a, 'b> {
inner: &'b mut std::iter::Cycle<::std::slice::Iter<'a, u8>>,
}
impl<'a, 'b> Iterator for ClonedFooIter<'a, 'b> {
type Item = u8;
fn next(&mut self) -> Option<Self::Item> {
self.inner.next().cloned()
}
}
The idea is that the returned struct only lives till the mutable borrow of self lives.
Using a struct makes naming the iterator type easier too.
I'm trying to implement an abstraction that allows me to read from either a directory or a zip file. I start by implementing something of this sort:
pub trait FileOpener<'a> {
type ReaderType: Read;
fn open(&'a self, file_name: &str) -> Result<Self::ReaderType, Box<dyn Error>>;
}
pub struct DirectoryFileOpener<'a> {
root: &'a Path
}
impl<'a> DirectoryFileOpener<'a> {
pub fn new(root: &'a Path) -> Self {
DirectoryFileOpener { root }
}
}
impl<'a> FileOpener<'a> for DirectoryFileOpener<'a> {
type ReaderType = File;
fn open(&'a self, file_name: &str) -> Result<File, Box<dyn Error>> {
Ok(File::open(self.root.join(file_name))?)
}
}
But then I realize that the zip-rs package's zip::ZipFile is constructed from a mutable reference to the zip::ZipArchive which it is located in, so I end up with the following code:
use std::path::Path;
use std::error::Error;
use std::fs::File;
use std::io::prelude::*;
use zip::{ZipArchive, read::ZipFile};
use std::marker::PhantomData;
pub trait FileOpener<'a> {
type ReaderType: Read;
fn open(&'a mut self, file_name: &str) -> Result<Self::ReaderType, Box<dyn Error>>;
}
pub struct DirectoryFileOpener<'a> {
root: &'a Path
}
impl<'a> DirectoryFileOpener<'a> {
pub fn new(root: &'a Path) -> Self {
DirectoryFileOpener { root }
}
}
impl<'a> FileOpener<'a> for DirectoryFileOpener<'a> {
type ReaderType = File;
fn open(&'a mut self, file_name: &str) -> Result<File, Box<dyn Error>> {
Ok(File::open(self.root.join(file_name))?)
}
}
pub struct ZipFileOpener<'a, R: Read + Seek> {
zip: ZipArchive<R>,
phantom: PhantomData<&'a Self>
}
impl<'a, R: Read + Seek> ZipFileOpener<'a, R> {
pub fn new(zip: ZipArchive<R>) -> Self {
ZipFileOpener { zip, phantom: PhantomData }
}
}
impl<'a, R: Read + Seek> FileOpener<'a> for ZipFileOpener<'a, R> {
type ReaderType = ZipFile<'a>;
fn open(&'a mut self, file_name: &str) -> Result<ZipFile<'a>, Box<dyn Error>> {
Ok(self.zip.by_name(file_name)?)
}
}
I'm not sure if that's the most optimal way to write that, but at least it compiles. Then I try to use it as such:
fn load(root: &Path) -> Result<...> {
let mut opener = io::DirectoryFileOpener::new(root);
let a = Self::parse_a(opener.open("a.txt")?)?;
let b = Self::parse_b(opener.open("b.txt")?, a)?;
}
and I get cannot borrow 'opener' as mutable more than once at a time. This does not surprise me much, as I indeed use open(), which borrows opener as mutable, twice - although a is only a u64, and from my point of view it is unrelated to the lifetime of opener.open(), from the compiler's point of view it has to be in the same lifetime of the line below it, and thus we attempt to borrow opener as mutable twice.
However, I then look at the following code, which compiles and works well and which I started this whole thing by trying to improve:
fn load_zip(root: &Path) -> Result<...> {
let file = File::open(root)?;
let mut zip = ZipArchive::new(file)?;
let a = Self::parse_a(zip.by_name("a.txt")?)?;
let b = Self::parse_b(zip.by_name("b.txt")?, a)?;
}
This throws me off completely, because the function by_name() also borrows zip as mutable, and is also called twice! Why is it allowed to borrow zip as mutable twice here but not in the previous case?
After researching the issue and Rust's semantics deeper, and building on top of the notes by trentcl, I came to realize that the problem essentially boils down to defining the FileOpener trait where the lifetime argument is bound to the associated type and not to the trait itself, e.g.
pub trait FileOpener {
type ReaderType: Read;
fn open(&'a mut self, file_name: &str) -> Result<Self::ReaderType, Box<dyn Error>>;
}
impl<'a, R: Read + Seek> FileOpener for ZipFileOpener<R> {
type ReaderType = ZipFile<'a>;
...
}
However, this is known as generic associated types (GAT), and is not yet supported in Rust. The GAT RFC does however mention that in some cases the problem can be circumvented by binding the lifetime to the trait itself and using higher-rank trait bounds (HRTB) in the receiving function, which yields the following working solution to this question:
pub trait FileOpener<'a> {
type ReaderType: Read;
fn open(&'a self, file_name: &str) -> Result<Self::ReaderType, Box<dyn Error>>;
}
...
fn load<T: for<'a> FileOpener<'a>>(opener: T) -> ... {
let a = parse_a(opener.open("a.txt")?)?;
let b = parse_b(opener.open("b.txt")?, a)?;
}
This is because the HRTB allows us to bind T to a FileOpener without binding a specific lifetime to it, which enables the late binding of different lifetimes for each call to opener.open()
Not sure what I am missing here, the lifetime is declared, therefore the struct should use the path to create the file and return a Struct with the mutable File reference for me to be able to call "write" wrapper later...
use std::path::Path;
use std::fs::File;
// use std::io::Write;
#[derive(Debug)]
pub struct Foo<'a> {
file: &'a mut File,
}
impl<'a> Foo<'a> {
pub fn new(path: &'a Path) -> Result<Self, std::io::Error> {
let mut f: &'a File = &File::create(path)?;
Ok(Self { file: &mut f })
}
//pub fn write(&self, b: [u8]) {
// self.file.write(b);
//}
}
Error:
| impl<'a> Foo<'a> {
| -- lifetime `'a` defined here
11 | pub fn new(path: &'a Path) -> Result<Self, std::io::Error> {
12 | let mut f: &'a File = &File::create(path)?;
| -------- ^^^^^^^^^^^^^^^^^^^ creates a temporary which is freed while still in use
| |
| type annotation requires that borrow lasts for `'a`
...
15 | }
| - temporary value is freed at the end of this statement
As #E_net4 mentioned, I don't want a mutable reference, yet I want to own the value. Rather than trying to play with lifetimes, I can basically just own the file and handle the whole struct as mutable when trying to write to the file!
use std::path::{ PathBuf };
use std::fs::File;
use std::io::Write;
use std::env;
#[derive(Debug)]
pub struct Foo {
file: File,
}
impl Foo {
pub fn new(path: PathBuf) -> Self {
Self {
file: File::create(path).unwrap(),
}
}
pub fn write(&mut self, b: &[u8]) -> Result<usize, std::io::Error> {
self.file.write(b)
}
}
fn main() {
let mut tmp_dir = env::temp_dir();
tmp_dir.push("foo23");
let mut f = Foo::new(tmp_dir);
f.write(b"test2").unwrap();
}
This is a simplified example of my code:
#[derive(Debug, Clone, Copy)]
enum Data<'a> {
I32(&'a [i32]),
F64(&'a [f64]),
}
impl<'a> From<&'a [i32]> for Data<'a> {
fn from(v: &'a [i32]) -> Data<'a> {
Data::I32(v)
}
}
impl<'a> From<&'a [f64]> for Data<'a> {
fn from(v: &'a [f64]) -> Data<'a> {
Data::F64(v)
}
}
#[derive(Debug, Clone, Copy)]
struct DataVar<'a> {
name: &'a str,
data: Data<'a>,
}
impl<'a> DataVar<'a> {
fn new<T>(name: &'a str, data: T) -> Self
where
T: Into<Data<'a>>,
{
Self {
name,
data: data.into(),
}
}
}
First of all, considering that I need to cast different DataVars to the same vector, and I would like to avoid using trait objects, do you think my implementation is correct or do you have suggestions for improvement?
Now my main question. I can define new DataVars passing a slice, for instance as follows:
let x = [1, 2, 3];
let xvar = DataVar::new("x", &x[..]);
How can I modify my constructor so that it works not only with a slice, but also with a reference to array or vector? For instance I would like the following to work as well:
let x = [1, 2, 3];
let xvar = DataVar::new("x", &x);
EDIT:
Now I tried implementing the same code using a trait object instead of an enum, but the result is even worse... isn't there really any solution to this?
trait Data: std::fmt::Debug {}
impl Data for &[i32] {}
impl Data for &[f64] {}
#[derive(Debug, Clone, Copy)]
struct DataVar<'a> {
name: &'a str,
data: &'a dyn Data,
}
impl<'a> DataVar<'a> {
fn new<T>(name: &'a str, data: &'a T) -> Self
where
T: Data,
{
Self { name, data }
}
}
let x = [1, 2, 3];
let xvar = DataVar::new("x", &&x[..]);
To me, AsRef doesn't seem to be the right abstraction for two reasons: first, because it's possible (if unlikely) for a type to implement both AsRef<[i32]> and AsRef<[f64]>, and it's not clear what should happen in that case; and second, because there's already a built-in language feature (coercion) that can turn Vec<T> or &[T; n] into &[T], and you're not taking advantage of it.
What I'd like is to write a new function that looks basically like this:
fn new<T>(name: &'a str, data: &'a [T]) -> Self
where
// what goes here?
This will automatically work with &[T; n], &Vec<T>, &Cow<T>, etc. if we can tell the compiler what to do with T. It makes sense that you could make a trait that knows how to convert &'a [Self] to Data and is implemented for i32 and f64, so let's do that:
trait Item: Sized {
fn into_data<'a>(v: &'a [Self]) -> Data<'a>;
}
impl Item for i32 {
fn into_data<'a>(v: &'a [i32]) -> Data<'a> {
Data::I32(v)
}
}
impl Item for f64 {
fn into_data<'a>(v: &'a [f64]) -> Data<'a> {
Data::F64(v)
}
}
The trait bound on new becomes trivial:
impl<'a> DataVar<'a> {
fn new<T>(name: &'a str, data: &'a [T]) -> Self
where
T: Item,
{
Self {
name,
data: T::into_data(data),
}
}
}
I find this more readable than the version with From and AsRef, but if you still want From, you can easily add it with a generic impl:
impl<'a, T> From<&'a [T]> for Data<'a>
where
T: Item,
{
fn from(v: &'a [T]) -> Self {
T::into_data(v)
}
}
We can use the AsRef trait to convert references to arrays or vectors to slices. AsRef is a generic trait, so we need to introduce a second type parameter to represent the "intermediate type" (the slice type). After calling as_ref, we've got a slice that can be converted to a Data using into.
impl<'a> DataVar<'a> {
fn new<T, U>(name: &'a str, data: &'a T) -> Self
where
T: AsRef<U> + ?Sized,
U: ?Sized + 'a,
&'a U: Into<Data<'a>>,
{
Self {
name,
data: data.as_ref().into(),
}
}
}
Note however that the data parameter is now a reference: this is necessary because the lifetime of the reference returned by as_ref is bound by the lifetime of the self parameter passed to as_ref. If we changed the parameter back to data: T, then data.as_ref() now implicitly references data in order to call as_ref, which expects a shared reference to self (&self). But data here is a local parameter, which means that the lifetime of the reference created by this implicit referencing operation is limited to the local function, and so is the reference returned by data.as_ref(). This lifetime is shorter than 'a, so we can't store it in the DataVar and return it.
If you need to handle data values that are not references in addition to values that are references, this solution cannot support that, unfortunately.
This is actually the best solution for my case:
impl<'a> DataVar<'a> {
fn new<T, U>(name: &'a str, data: &'a T) -> Self
where
T: AsRef<[U]> + ?Sized,
U: 'a,
&'a [U]: Into<Data<'a>>,
{
Self {
name,
data: data.as_ref().into(),
}
}
}
It works with slices, references to vectors, and references to arrays up to length 32 which implement AsRef<[T]> https://doc.rust-lang.org/beta/std/convert/trait.AsRef.html
Thanks #Francis for your hints!
Actually, this is IMHO the best solution... so similar to my initial code, I just needed a small fix in the new constructor:
#[derive(Debug, Clone, Copy)]
enum Data<'a> {
I32(&'a [i32]),
F64(&'a [f64]),
}
impl<'a> From<&'a [i32]> for Data<'a> {
fn from(data: &'a [i32]) -> Data<'a> {
Data::I32(data)
}
}
impl<'a> From<&'a [f64]> for Data<'a> {
fn from(data: &'a [f64]) -> Data<'a> {
Data::F64(data)
}
}
#[derive(Debug, Clone, Copy)]
struct DataVar<'a> {
name: &'a str,
data: Data<'a>,
}
impl<'a> DataVar<'a> {
fn new<T>(name: &'a str, data: &'a [T]) -> Self
where
&'a [T]: Into<Data<'a>>,
{
Self {
name,
data: data.into(),
}
}
}
#trentcl your solution is brilliant! Now I see how to leverage coercion.
However I tweaked it a little bit as follows, I will finally use this code unless you see any drawbacks in it, thanks!
#[derive(Debug, Clone, Copy)]
enum Data<'a> {
I32(&'a [i32]),
F64(&'a [f64]),
}
trait IntoData<'a>: Sized {
fn into_data(&self) -> Data<'a>;
}
impl<'a> IntoData<'a> for &'a [i32] {
fn into_data(&self) -> Data<'a> {
Data::I32(&self)
}
}
impl<'a> IntoData<'a> for &'a [f64] {
fn into_data(&self) -> Data<'a> {
Data::F64(&self)
}
}
#[derive(Debug, Clone, Copy)]
struct DataVar<'a> {
name: &'a str,
data: Data<'a>,
}
impl<'a> DataVar<'a> {
fn new<T>(name: &'a str, data: &'a [T]) -> Self
where
&'a [T]: IntoData<'a>,
{
Self {
name,
data: data.into_data(),
}
}
}
pub struct Storage<T>{
vec: Vec<T>
}
impl<T: Clone> Storage<T>{
pub fn new() -> Storage<T>{
Storage{vec: Vec::new()}
}
pub fn get<'r>(&'r self, h: &Handle<T>)-> &'r T{
let index = h.id;
&self.vec[index]
}
pub fn set(&mut self, h: &Handle<T>, t: T){
let index = h.id;
self.vec[index] = t;
}
pub fn create(&mut self, t: T) -> Handle<T>{
self.vec.push(t);
Handle{id: self.vec.len()-1}
}
}
struct Handle<T>{
id: uint
}
I am currently trying to create a handle system in Rust and I have some problems. The code above is a simple example of what I want to achieve.
The code works but has one weakness.
let mut s1 = Storage<uint>::new();
let mut s2 = Storage<uint>::new();
let handle1 = s1.create(5);
s1.get(handle1); // works
s2.get(handle1); // unsafe
I would like to associate a handle with a specific storage like this
//Pseudo code
struct Handle<T>{
id: uint,
storage: &Storage<T>
}
impl<T> Handle<T>{
pub fn get(&self) -> &T;
}
The problem is that Rust doesn't allow this. If I would do that and create a handle with the reference of a Storage I wouldn't be allowed to mutate the Storage anymore.
I could implement something similar with a channel but then I would have to clone T every time.
How would I express this in Rust?
The simplest way to model this is to use a phantom type parameter on Storage which acts as a unique ID, like so:
use std::kinds::marker;
pub struct Storage<Id, T> {
marker: marker::InvariantType<Id>,
vec: Vec<T>
}
impl<Id, T> Storage<Id, T> {
pub fn new() -> Storage<Id, T>{
Storage {
marker: marker::InvariantType,
vec: Vec::new()
}
}
pub fn get<'r>(&'r self, h: &Handle<Id, T>) -> &'r T {
let index = h.id;
&self.vec[index]
}
pub fn set(&mut self, h: &Handle<Id, T>, t: T) {
let index = h.id;
self.vec[index] = t;
}
pub fn create(&mut self, t: T) -> Handle<Id, T> {
self.vec.push(t);
Handle {
marker: marker::InvariantLifetime,
id: self.vec.len() - 1
}
}
}
pub struct Handle<Id, T> {
id: uint,
marker: marker::InvariantType<Id>
}
fn main() {
struct A; struct B;
let mut s1 = Storage::<A, uint>::new();
let s2 = Storage::<B, uint>::new();
let handle1 = s1.create(5);
s1.get(&handle1);
s2.get(&handle1); // won't compile, since A != B
}
This solves your problem in the simplest case, but has some downsides. Mainly, it depends on the use to define and use all of these different phantom types and to prove that they are unique. It doesn't prevent bad behavior on the user's part where they can use the same phantom type for multiple Storage instances. In today's Rust, however, this is the best we can do.
An alternative solution that doesn't work today for reasons I'll get in to later, but might work later, uses lifetimes as anonymous id types. This code uses the InvariantLifetime marker, which removes all sub typing relationships with other lifetimes for the lifetime it uses.
Here is the same system, rewritten to use InvariantLifetime instead of InvariantType:
use std::kinds::marker;
pub struct Storage<'id, T> {
marker: marker::InvariantLifetime<'id>,
vec: Vec<T>
}
impl<'id, T> Storage<'id, T> {
pub fn new() -> Storage<'id, T>{
Storage {
marker: marker::InvariantLifetime,
vec: Vec::new()
}
}
pub fn get<'r>(&'r self, h: &Handle<'id, T>) -> &'r T {
let index = h.id;
&self.vec[index]
}
pub fn set(&mut self, h: &Handle<'id, T>, t: T) {
let index = h.id;
self.vec[index] = t;
}
pub fn create(&mut self, t: T) -> Handle<'id, T> {
self.vec.push(t);
Handle {
marker: marker::InvariantLifetime,
id: self.vec.len() - 1
}
}
}
pub struct Handle<'id, T> {
id: uint,
marker: marker::InvariantLifetime<'id>
}
fn main() {
let mut s1 = Storage::<uint>::new();
let s2 = Storage::<uint>::new();
let handle1 = s1.create(5);
s1.get(&handle1);
// In theory this won't compile, since the lifetime of s2
// is *slightly* shorter than the lifetime of s1.
//
// However, this is not how the compiler works, and as of today
// s2 gets the same lifetime as s1 (since they can be borrowed for the same period)
// and this (unfortunately) compiles without error.
s2.get(&handle1);
}
In a hypothetical future, the assignment of lifetimes may change and we may grow a better mechanism for this sort of tagging. However, for now, the best way to accomplish this is with phantom types.