I have a type that represents a file. For simplicity lets say the type holds a buffer with the contents of the file.
There is also a method for building an iterator which holds a reference to the internal buffer.
The file-type is created on the main thread, but I need to "send" the iterator into the thread.
Let me show what I am trying to do
struct FileType {...}
let mut my_file_type = FileType::new("some_filename");
let mut my_iterator = my_file_type.iter();
external_library_object.start_process(move|_| {
for _ in (0..10) {
println!(my_iterator.next().unwrap());
}
}
external_library_object (which by the way is cpal's device) is... well an object from a library I can't change. The closure parameter implements Send trait though.
This doesn't work because my_file_type doesn't live long enough.
So I tried a combination of Mutex (so the iterator is mutable) and Arc(so we share the iterator between the 2 threads)
...
let mut my_file_type = FileType::new("some_filename");
let mut my_iterator = Arc::new(Mutex::new(my_file_type.iter()));
external_library_object.start_process(move|_| {
let mut cloned = my_iterator.clone();
for _ in (0..10) {
println!(cloned.lock().unwrap().next().unwrap());
}
}
But again, this doesn't work. I get an error similar to this:
xx | let mut my_iterator = Arc::new(Mutex::new(my_file_type.iter()));
| ^^^^^^^^^^^^-------
| |
| borrowed value does not live long enough
| argument requires that `my_file_type` is borrowed for `'static`
...
xx | }
| - `my_file_type` dropped here while still borrowed
I am kind of stuck here. Is there a way to use the iterator which has a reference to other object(a buffer in this case) inside a different thread?
Edit: In my specific case the itertor is infinite and the closure executes multiple times. That is the reason I can't move the entire FileType inside the thread.
This is tricky, because the process borrows an iterator, which in turn borrows a struct. Putting just the iterator into an Arc<Mutex<>> is not sufficient, because this pointer might still outlive the FileType it borrows. However, we can't put both the FileType and the iterator into a reference-counted struct, since that struct would then be self-referential.
The easiest solution to this that I can think of (short of using scoped threads) is to create an iterator that owns the FileType, and then put it into a Arc<Mutex<>> (minimal playground example):
let my_file_type = FileType::new("test!");
let my_iterator = Arc::new(Mutex::new(my_file_type.into_iter()));
external_library_object.start_process(move |_| {
for _ in 0..10 {
println!("{:?}", my_iterator.lock().unwrap().next().unwrap());
}
});
You just have to implement an owned iterator for your FileType struct, if it doesn't exist yet.
Related
An example of how a range gets consumed is:
let coll = 1..10;
for i in coll {
println!("i is {}", &i);
}
println!("coll length is {}", coll.len());
This will fail with
error[E0382]: borrow of moved value: `coll`
--> src/main.rs:6:35
|
2 | let coll = 1..10;
| ---- move occurs because `coll` has type `std::ops::Range<i32>`, which does not implement the `Copy` trait
3 | for i in coll {
| ----
| |
| `coll` moved due to this implicit call to `.into_iter()`
| help: consider borrowing to avoid moving into the for loop: `&coll`
...
6 | println!("coll length is {}", coll.len());
| ^^^^ value borrowed here after move
|
note: this function consumes the receiver `self` by taking ownership of it, which moves `coll`
The usual way to fix this is to borrow the coll, but that doesn't work here:
error[E0277]: `&std::ops::Range<{integer}>` is not an iterator
--> src/main.rs:3:14
|
3 | for i in &coll {
| -^^^^
| |
| `&std::ops::Range<{integer}>` is not an iterator
| help: consider removing the leading `&`-reference
|
= help: the trait `std::iter::Iterator` is not implemented for `&std::ops::Range<{integer}>`
= note: required by `std::iter::IntoIterator::into_iter`
Why is that? Why is a borrowed range not an iterator, but the range is? Is it interpreting it differently?
To understand what is happening here it is helpful to understand how for loops work in Rust.
Basically a for loop is a short hand for using an iterator, so:
for item in some_value {
// ...
}
is basically a short-hand for
let mut iterator = some_value.into_iter();
while let Some(item) = iterator.next() {
// ... body of for loop here
}
So we can see that whatever we loop over with the for loop, Rust calls the into_iter method from the IntoIterator trait on. The IntoIterator trait looks (approximately) like this:
trait IntoIterator {
// ...
type IntoIter;
fn into_iter(self) -> Self::IntoIter;
}
So into_iter takes self by value and returns Self::IntoIter which is the type of the iterator. As Rust moves any arguments which are taken by value, the thing .into_iter() was called on is no longer available after the call (or after the for loop). That's why you can't use coll in your first code snippet.
So far so good, but why can we still use a collection if we loop over a reference of it as in the following?
for i in &collection {
// ...
}
// can still use collection here ...
The reason is that for a lot of collections C, the IntoIterator trait is implemented not just for the collection, but also for a shared reference to the collection &C and this implementation produces shared items. (Sometimes it is also implemented for mutable references &mut C which produces mutable references to items).
Now coming back to the example with the Range we can check how it implements IntoIterator.
Looking at the reference docs for Range, Range strangely does not seem to implement IntoIterator directly... but if we check the Blanket Implementations section on doc.rust-lang.org, we can see that every iterator implements the IntoIterator trait (trivially, by just returning itself):
impl<I> IntoIterator for I
where
I: Iterator
How does this help? Well, checking further up (under trait implementations) we see that Range does implement Iterator:
impl<A> Iterator for Range<A>
where
A: Step,
And thus Range does implement IntoIterator via the indirection of Iterator. However, there is no implementation of either Iterator for &Range<A> (this would be impossible) or of IntoIterator for &Range<A>. Therefore, we can use a for loop by passing Range by value, but not by reference.
Why can &Range not implement Iterator? An iterator needs to keep track of "where it is", which requires some kind of mutation, but we cannot mutate a &Range because we only have a shared reference. So this cannot work. (Note that &mut Range can and does implement Iterator - more on this later).
It would technically be possible to implement IntoIterator for &Range as that could produce a new iterator. But the likelihood that this would clash with the blanket iterator implementation of Range would be very high and things would be even more confusing. Besides, a Range is at most two integers and copying this is very cheap, so there is really no big value in implementing IntoIterator for &Range.
If you still want to use collection, you can clone it
for i in coll.clone() { /* ... */ }
// `coll` still available as the for loop used the clone
This brings up another question: If we can clone the range and it is (as claimed above) cheap to copy it, why doesn't Range implement the Copy trait? Then the .into_iter() call would copy the range coll (instead of moving it) and it could still be used after the loop. According to this PR the Copy trait implementation actually existed but was removed because the following was considered a footgun (hat tip to Michael Anderson for pointing this out):
let mut iter = 1..10;
for i in iter {
if i > 2 { break; }
}
// This doesn't work now, but if `Range` implemented copy,
// it would produce `[1,2,3,4,5,6,7,8,9]` instead of
// `[4,5,6,7,8,9]` as might have been expected
let v: Vec<_> = iter.collect();
Also note that &mut Range does implement iterator, so you can do
let mut iter = 1..10;
for i in &mut iter {
if i > 2 { break; }
}
// `[4,5,6,7,8,9]` as expected
let v: Vec<_> = iter.collect();
Finally, for completeness, it might be instructive to see which methods are actually called when we loop over a Range:
for item in 1..10 { /* ... */ }
is translated to
let mut iter = 1..10.into_iter();
// ˆˆˆˆˆˆˆˆˆ--- which into_iter() is this?
while let Some(item) = iter.next() { /* ... */ }
we can make this explicit using qualified method syntax:
let mut iter = std::iter::Iterator::into_iter(1..10);
// it's `Iterator`s method! ------^^^^^^^^^
while let Some(item) = iter.next() { /* ... */ }
Ranges are iterators that modify themselves to generate elements. Therefore, to loop over a range, it is necessary to modify it (or a copy of it, as shown below).
Vectors, on the other hand, are not iterators themselves. .into_iter() is called to create an iterator when a vector is looped over; the vector itself doesn't need to be consumed.
The solution here is to use clone to create a new iterator that can be looped over:
for i in coll.clone() {
println!("i is {}", i);
}
(Incidentally, the println! family of macros take references automatically.)
Let's say you have a vector:
let v = vec![1, 2, 3];
The method iter on Vec returns something that implements the Iterator trait. With a vector, there is also an implementation of the trait Borrow (and BorrowMut), that does not return a &Vec though. Instead, you get a slice &[T]. This slice can then be used to iterate over the elements of the vector.
However, the range (e.g. 1..10) implements IntoIterator already and does not need to be transformed into a slice or some other view into it. Therefore, you can consume the range itself by calling into_iter() (which you do implicitly). Now, it is as if you moved the range into some function and you cannot use your variable coll anymore. The borrowing syntax won't help, since this is only some special functionality of Vec.
In this case, you could construct a Vec from your range (with the collect method), clone the range when iterating over it or get the length before iterating (since getting the length doesn't consume the range itself).
Some references:
https://doc.rust-lang.org/std/vec/struct.Vec.html
https://doc.rust-lang.org/std/primitive.slice.html
https://doc.rust-lang.org/std/ops/struct.Range.html
I'm working on the Rust Book (Chapter 4) and I am surprised that code like this compiles:
fn main() {
let mut s = String::from("hello");
let r1 = &s;
let r2 = &s;
println!("{}, {}", r1, r2);
// this line silences the warning: 'variable does not need to be mutable'
s.push_str(" world");
}
Why does Rust allow an immutable reference to a mutable variable? This would seem to weaken the safety guarantees. If I have a mutable variable, and I pass immutable references to some threads, those threads are assuming the value will not change, but I could mutate the value through the original variable.
I haven't reached threading yet, but found this strange, and in this case, no different from C++:
void doNotChangeMyString(const std::string& myConstString) {
// ... assume that myConstString cannot change and use it on a thread
// return immediately even though some worker thread is still
// using myConstString
}
void main() {
std::string s = "hello" // not const!
doNotChangeMyString(s);
s = "world"; // oops
}
Edit: I fixed the Rust code so that it compiles. Please reconsider the downvotes and close votes. The accepted answer explains a concept that I did not get from Rust Book's chapter on borrowing, was very helpful to me, and could help others who are at the same point in learning Rust.
An item's mutability is essentially part of the name of the variable in rust. Take for example this code:
let mut foo = String::new();
let foo = foo;
let mut foo = foo;
foo suddenly becomes immutable, but it does not mean that the first two foos don't exist.
On the other hand, a mutable reference is attached to the lifetime of the object and is therefore type-bound, and will exist for its own lifetime, disallowing any kind of access to the original object if it is not through the reference.
let mut my_string = String::new();
my_string.push_str("This is ok! ");
let foo: &mut String = &mut my_string;
foo.push_str("This goes through the mutable reference, and is therefore ok! ");
my_string.push_str("This is not ok, and will not compile because `foo` still exists");
println!("We use foo here because of non lexical lifetimes: {:?}", foo);
The second call to my_string.push_str will not compile because foo can (in this case it is guaranteed to) be used afterwards.
Your specific question asks something similar to the following, but you don't even need multithreading to test this:
fn immutably_use_value(x: &str) {
println!("{:?}", x);
}
let mut foo = String::new();
let bar = &foo; //This now has immutable access to the mutable object.
let baz = &foo; //Two points are allowed to observe a value at the same time. (Ignoring `Sync`)
immutably_use_value(bar); //Ok, we can observe it immutably
foo.push_str("Hello world!"); //This would be ok... but we use the immutable references later!
immutably_use_value(baz);
This does not compile. If you could annotate the lifetimes, they'd look something similar to this:
let mut foo = String::new(); //Has lifetime 'foo
let bar: &'foo String = &foo; //Has lifetime 'bar: 'foo
let baz: &'foo String = &foo; //Has lifetime 'baz: 'foo
//On the other hand:
let mut foo = String::new(); //Has lifetime 'foo
let bar: &'foo mut String = &mut foo; //Has lifetime 'bar: mut 'foo
let baz: &'foo mut String = &mut foo; //Error, we cannot have overlapping mutable borrows for the same object!
A few extra notes:
Due to NLL (Non Lexical Lifetimes), the following code will compile:
let mut foo = String::new();
let bar = &foo;
foo.push_str("Abc");
Because bar is not used after the mutable use of foo.
You mention threading, which has its own constraints and traits involved:
The Send trait will allow you to give ownership of a variable across a thread.
The Sync trait will allow you to share a reference to a variable across a thread. This includes mutable references, as long as the original thread does not use the object for the duration of the borrow.
A few examples:
Type T is Send + Sync, it can be sent across threads and be shared between them
Type T is !Send + Sync, it can be shared across threads, but not sent between them. An example is a window handle that can only be destroyed on the original thread.
Type T is Send + !Sync, it can be sent across threads, but not shared between them. An example is RefCell, which will can only use its runtime borrow-checking on a single thread due to it not using atomics (Multithreading safe components).
Type T is !Send + !Sync, it can only live on the thread it was created on. An example is Rc, which cannot send a copy of itself across threads because it cannot count references atomically (Look at Arc to do that) and since it carries no lifetimes to force a single copy of itself to exist when sending across a thread boundary, it therefore cannot be sent across threads.
I use &str instead of &String in my third example, this is because String: Deref<str> (You may need to scroll down to see it), and therefore anywhere I need a &str I can chuck a &String in because the compiler will autoderef.
This question already has an answer here:
Mutating the same data in multiple 'static closures
(1 answer)
Closed 4 years ago.
I'm using the libpulse_binding library, and I'm trying to obtain a sequence of SinkInputInfos from the get_sink_input_info_list function:
pub fn get_sink_input_info_list<F>(
&self,
callback: F,
) -> Operation<dyn FnMut(ListResult<&SinkInputInfo>)>
where
F: FnMut(ListResult<&SinkInputInfo>) + 'static,
The function takes a callback and calls it once for each SinkInputInfo it produces. I'm trying to collect all those SinkInputInfos into a single list so I can get a clearer view of the state of the world. Irritatingly, SinkInputInfo doesn't implement Copy or Clone, so I made a custom struct and implemented From to get the useful information out of the SinkInputInfo:
struct StreamInfo {
readable_name: String,
binary_name: String,
pid: String,
}
impl From<&pulse::context::introspect::SinkInputInfo<'_>> for StreamInfo {
fn from(info: &pulse::context::introspect::SinkInputInfo) -> Self {
let name = info.proplist.gets("application.name").unwrap();
let binary = info.proplist.gets("application.process.binary").unwrap();
let pid = info.proplist.gets("application.process.id").unwrap();
StreamInfo {
readable_name: name,
binary_name: binary,
pid: pid,
}
}
}
However, this doesn't seem to work. I have the following code:
let mut sink_infos: Vec<StreamInfo> = Vec::new();
let op = introspector.get_sink_input_info_list(|result| match result {
pulse::callbacks::ListResult::Item(info) => sink_infos.push(info.into()),
pulse::callbacks::ListResult::End => {},
pulse::callbacks::ListResult::Error => panic!("Error getting sink input info"),
});
but it doesn't compile:
error[E0373]: closure may outlive the current function, but it borrows `sink_infos`, which is owned by the current function
--> src/bin/play-pause.rs:49:52
|
49 | let op = introspector.get_sink_input_info_list(|result| match result {
| ^^^^^^^^ may outlive borrowed value `sink_infos`
50 | pulse::callbacks::ListResult::Item(info) => sink_infos.push(info.into()),
| ---------- `sink_infos` is borrowed here
|
note: function requires argument type to outlive `'static`
--> src/bin/play-pause.rs:49:14
|
49 | let op = introspector.get_sink_input_info_list(|result| match result {
| ______________^
50 | | pulse::callbacks::ListResult::Item(info) => sink_infos.push(info.into()),
51 | | pulse::callbacks::ListResult::End => {},
52 | | pulse::callbacks::ListResult::Error => panic!("Error getting sink input info"),
53 | | });
| |______^
help: to force the closure to take ownership of `sink_infos` (and any other referenced variables), use the `move` keyword
|
49 | let op = introspector.get_sink_input_info_list(move |result| match result {
| ^^^^^^^^^^^^^
tldr: the closure has to have the 'static lifetime because libpulse_binding says so (presumably because it gets handed to the PulseAudio C API, which can then do whatever it likes with it), but sink_infos is not 'static, and the closure has to borrow sink_infos in order to append to it, which makes the closure not 'static, IIUC.
How can I make a Vec (or any container, I'm not fussy) of SinkInputInfos given a 'static closure that gets called repeatedly with a &SinkInputInfo?
The basic problem here is that you're running into Rust's borrowing rules:
Given an object T, it is only possible to have one of the following:
Having several immutable references (&T) to the object (also known as aliasing).
Having one mutable reference (&mut T) to the object (also known as mutability).
You are trying to keep a reference &Vec to your Vec (so you can use it later) whilst simultaneously trying to add things to it in a closure (i.e. an &mut Vec). Rust doesn't know that you won't try to use the &Vec while the closure is using the &mut Vec, so it doesn't let you create a &mut Vec in the closure while there's still a &Vec hanging around outside the closure.
The next best thing you can do is to use an Rc. This will allow you to sidestep the compiler's borrow-checking and instead defer it to runtime. However: this means that if you try to violate the borrowing rules while your program is running, it will panic instead of being a compile-time error!
For the most part, you can treat the Rc<Vec<_>> the same as a normal Vec, since Rc implements Deref.
Since you also want to be able to mutate the Vec in order to add things to it, you also need to put it in a RefCell. This will put a lock around the Vec, making sure that you only have one &mut Vec available at once, and that if you have an &Vec, you can't have a &mut Vec (again, if you try to violate the rules, your program will panic). You can use the .borrow() and .borrow_mut() methods on RefCell to get shared and mutable references to the Vec (there are also try_* variants of these methods, if you can do something sensible if a borrow is not possible).
If you didn't use a RefCell, you would only be able to obtain immutable/shared references (&Vec) from the Rc (unless you only had one Rc, but then you wouldn't need an Rc!)
Try something like the following:
use std::cell::RefCell;
use std::rc::Rc;
let sink_infos: Rc<RefCell<Vec<StreamInfo>>> = Rc::new(RefCell::new(Vec::new()));
let sink_infos2 = sink_infos.clone(); // Create a new Rc which points to the same data.
let op = introspector.get_sink_input_info_list(move |result| match result {
pulse::callbacks::ListResult::Item(info) => sink_infos2.borrow_mut().push(info.into()),
pulse::callbacks::ListResult::End => {},
pulse::callbacks::ListResult::Error => panic!("Error getting sink input info"),
});
I'm using tcod-rs. Each method used to draw to the RootConsole takes a mutable reference. The central loop is a while loop that waits for the window to close, and clears the screen, draws, and then flushes. The "check for window close" method also takes a mutable reference. clear and flush do too. I have a custom drawing function that needs a mutable reference so it can draw:
fn draw(root: &mut RootConsole) { /* Do state-y stuff */ }
while !root.window_closed() /* Mutable borrow 1 */ {
root.clear(); // Mutable borrow 2
draw(root); // Mutable borrow 3
root.flush(); // Mutable borrow 4
}
The compiler has a nervous breakdown screaming about aliens stealing RootConsoles. I've managed to cut it down to one error by just making the changes it suggests. I can't change the tcod-rs library, or at least, I'd rather not, and I would prefer to keep using it. Replacing the draw(root) function with a root.put_char(...) for instance, works.
Here is the error I get on attempting to cargo run. The tcod::console::Root type has documentation here.
error[E0596]: cannot borrow immutable argument `root` as mutable
--> src/draw/mod.rs:27:33
|
9 | pub fn draw_map(root: &mut RootConsole, world: WorldState) {
| ---- consider changing this to `mut root`
...
27 | .draw_char(&mut root,
| ^^^^ cannot borrow mutably
My actual draw_map code is pretty simple:
pub trait DrawChar {
fn draw_char(self, root: &mut RootConsole, pos: (i32, i32));
}
pub fn draw_map(root: &mut RootConsole, world: WorldState) {
let screen_start_y = world.screen.1;
let screen_end_y = screen_start_y + root.height();
let screen_start_x = world.screen.0;
let screen_end_x = screen_start_x + root.width();
let x = screen_start_x;
let y = screen_start_y;
for scan_line in &world.map[screen_start_y as usize..
screen_end_y as usize]
{
y += 1;
for unit in &scan_line[screen_start_x as usize..
screen_end_x as usize]
{
x += 1;
unit.tiles[world.level as usize]
.draw_char(&mut root,
(x as i32 - screen_start_x,
y as i32 - screen_start_y));
}
}
}
Each tile implements DrawChar, which, since it uses functions from tcod-rs on a root, requires a mutable reference. My main loop isn't much different from the while loop above, except that it handles key commands using some native functions.
The problem is at the draw_char call in draw_map. root is already a mutable reference to a RootConsole; you are trying to pass to draw_char a mutable reference to root, which essentially becomes a mutable reference to a mutable reference to a RootConsole. Simply pass root as the parameter and you are done.
I suppose this question is about lifetimes in general, but I'm having difficulty with closures specifically because you can't write out their type.
This example is a bit contrived - I'm just starting to learn Rust, and this is something I've been hung up on.
This program won't compile:
fn main () {
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
list.push(&|| 1);
}
}
Because:
src/main.rs:5:25: 5:24 error: borrowed value does not live long enough
src/main.rs:5 list.push(&|| 1);
^~~~
src/main.rs:2:50: 7:2 note: reference must be valid for the block suffix following statement 0 at 2:49...
src/main.rs:2 let mut list: Vec<&Fn() -> i32> = Vec::new();
src/main.rs:3
src/main.rs:4 {
src/main.rs:5 list.push(&move || 1);
src/main.rs:6 }
src/main.rs:7 }
src/main.rs:5:9: 5:26 note: ...but borrowed value is only valid for the statement at 5:8
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
src/main.rs:5:9: 5:26 help: consider using a `let` binding to increase its lifetime
src/main.rs:5 list.push(&|| 1);
^~~~~~~~~~~~~~~~~
What I gather from this error is that the closure's lifetime is limited to the
statement inside the block, but it needs to live for the entire body of main.
I know (or, I think) that passing the closure to push as a reference means that push is only borrowing the closure, and that ownership will be returned to the block. This code would work if I could just give the closure to push (i.e. if push took ownership of the closure), but since a closure isn't sized, I must pass it as a reference.
Is that right? How can I make this code work?
There are two things you are asking about:
specifying a typename for something that has no specifyable typename
letting a closure live longer than the block where it's defined.
The first issue is fixed by NOT specifying the typename, and letting rust's type inference do the work.
let mut list: Vec<_> = Vec::new();
The second issue is fixed by not trying to make the closure live longer, but by making it "by value" so you can move it. This enforces that your closure does not reference anything, but owns all the captured values.
for i in 0..10 {
list.push(move || i);
}
Now this gives us a new problem. If we add a different closure to the Vec, the types won't match. Therefore to achieve that, we need to box the closures.
fn main () {
let mut list: Vec<Box<Fn() -> i32>> = Vec::new();
for i in 0..10 {
list.push(Box::new(move|| i));
}
{
list.push(Box::new(move|| 42));
}
}
Borrows do not own the thing they point to. Your problem is that you're borrowing a temporary which is going to cease to exist right after it's borrowed because you haven't stored it anywhere. If it helps, consider that borrows don't borrow values, they borrow storage, and a temporary has only transient storage.
If you want a borrow to something to last for any given period, you must borrow from storage that will last at least that long. In this case, because you want to store the borrow in a Vec, this means that whatever storage you borrow from must outlive the Vec as well. Thus:
fn main () {
let closure;
let mut list: Vec<&Fn() -> i32> = Vec::new();
{
closure = || 1;
list.push(&closure);
}
}
Note that closure is defined before list is. In Rust, values are dropped in reverse lexical order at the end of their scope, so any variable defined after list will necessarily be dropped before it, thus leading to list containing invalid pointers.
If you want to push multiple closures, you will need a separate variable for each one.
To forestall a possible "my actual problem isn't this simple" addendum (:P): f you need to return list or in some way persist it beyond a single function call, note that there is no way to extend a borrow. In that case, what you need to do is change list to a vector of owned, boxed closures (i.e. Vec<Box<Fn() -> i32>>).