This question already has an answer here:
Lifetime troubles sharing references between threads
(1 answer)
Closed 7 years ago.
I have a threading operation in Rust which requires the use of a variable passed as an argument to the function the threads are spawned within, however I am seeing the following compile time error:
Compiling Test v0.0.1 (file:///Users/clarkj84/Desktop/RustTest)
main.rs:9:22: 9:35 error: captured variable `test` does not outlive the enclosing closure
main.rs:9 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
main.rs:7:20: 16:2 note: captured variable is valid for the anonymous lifetime #1 defined on the block at 7:19
main.rs:7 fn test(test: &str){
main.rs:8 for _x in 0..2 {
main.rs:9 let handle = thread::spawn(move || {
main.rs:10 for _y in 0..2 {
main.rs:11 println!("{}", test);
main.rs:12 }
...
note: closure is valid for the static lifetime
error: aborting due to previous error
Could not compile `Test`.
To learn more, run the command again with --verbose.
This is my code implementation:
use std::thread;
fn main() {
test("test");
}
fn test(test: &str){
for _x in 0..2 {
let handle = thread::spawn(move || {
for _y in 0..2 {
println!("{}", test);
}
});
}
}
Checkout #Shepmaster's answer for an explanation. There are two solutions to your problem depending on your use-case.
If your use case is just string literals (of type &'static str) and not arbitrary strings that are created during the program's execution, you can modify your function's signature to
fn test(test: &'static str)
This will allow you to pass any string literal to this function and move the string literal to another thread.
If on the other hand you want to create new strings by e.g. taking them from program arguments or environment variables, then you can create one String object per thread and pass that.
use std::thread;
use std::env;
fn main() {
test("test");
test(std::env::var("HOME").unwrap()); // pass the current home-directory
}
fn test(test: &str) {
for _x in 0..2 {
let string_object = test.to_string();
let handle = thread::spawn(move || {
for _y in 0..2 {
println!("{}", string_object);
}
});
}
}
Related
Consider the following Rust code:
use std::thread;
fn main() {
bar();
loop {}
}
fn bar() {
let var = b"foo";
thread::spawn(|| {
write(var);
});
}
fn write(d: &[u8]) {
println!("{:?}", d)
}
To my understanding, var is on the stack of function bar, which no longer exists after it returns.
Still, the new thread accesses it afterwards and successfully writes the data.
Why does this work?
Why doesn't the Rust compiler complain?
To my understanding, var is on the stack of function bar, which no longer exists after it returns.
var is just a reference with type &'static [u8; 3]. This reference value is what is on the stack, not the string literal.
The owner of the byte string literal b"foo" is the program binary, which results in the string literal having a 'static lifetime because it exists for the entire lifetime of the running program.
The b"foo" value doesn't live on the stack actually. It is stored in the read-only memory of the compiled binary and has a 'static lifetime.
Consider this alternative example:
fn bar() {
let var = format!("foo");
thread::spawn(|| {
write(&var);
});
}
fn write(d: &str) {
println!("{:?}", d)
}
That won't work (unless you add move before the closure), because var (of type String) is allocated on the stack.
I am trying to fire up a new thread using some heap data in Rust and I am getting a bunch of errors that stem from the need of the data to have 'static lifetime. I've worked my way backwards up my program but hit a problem.
use std::sync::Arc;
use std::thread;
struct ThreadData {
vector_of_strings: Vec<String>,
terms: Vec<&'static str>,
quotient: usize,
}
fn perform_search(slice: &[String], terms: &[&str]) {
/* ... */
}
fn threaded_search(td_arc: &Arc<ThreadData>) {
let no_of_lines = td_arc.vector_of_strings.len();
let new_tda1 = td_arc.clone();
let strings_as_slice1 = new_tda1.vector_of_strings.as_slice();
thread::spawn(move || {
perform_search(&strings_as_slice1[0..td_arc.quotient], &new_tda1.terms);
});
}
fn main() {
let td = ThreadData {
vector_of_strings: Vec::new(),
terms: Vec::new(),
quotient: 0,
};
let td_arc = Arc::new(td);
threaded_search(&td_arc);
}
Error:
error[E0621]: explicit lifetime required in the type of `td_arc`
--> src/main.rs:20:5
|
14 | fn threaded_search(td_arc: &Arc<ThreadData>) {
| ---------------- help: add explicit lifetime `'static` to the type of `td_arc`: `&'static std::sync::Arc<ThreadData>`
...
20 | thread::spawn(move || {
| ^^^^^^^^^^^^^ lifetime `'static` required
The error about 'static is because the new thread created within thread::spawn may outlive the invocation of threaded_search during which the thread is initially created, which means the thread must not be permitted to use any local variables from threaded_search with a lifetime shorter than 'static.
In your code the new thread is referring to strings_as_slice1 and td_arc.
Generally with thread::spawn and Arc you will want to move ownership of one reference count into the thread and have the thread access whatever it needs through that reference counted pointer rather than from the enclosing short-lived scope directly.
fn threaded_search(td_arc: &Arc<ThreadData>) {
// Increment reference count that we can move into the new thread.
let td_arc = td_arc.clone();
thread::spawn(move || {
perform_search(&td_arc.vector_of_strings[0..td_arc.quotient], &td_arc.terms);
});
}
I've been playing around with AudioUnit via Rust and the Rust library coreaudio-rs. Their example seems to work well:
extern crate coreaudio;
use coreaudio::audio_unit::{AudioUnit, IOType};
use coreaudio::audio_unit::render_callback::{self, data};
use std::f32::consts::PI;
struct Iter {
value: f32,
}
impl Iterator for Iter {
type Item = [f32; 2];
fn next(&mut self) -> Option<[f32; 2]> {
self.value += 440.0 / 44_100.0;
let amp = (self.value * PI * 2.0).sin() as f32 * 0.15;
Some([amp, amp])
}
}
fn main() {
run().unwrap()
}
fn run() -> Result<(), coreaudio::Error> {
// 440hz sine wave generator.
let mut samples = Iter { value: 0.0 };
//let buf: Vec<[f32; 2]> = vec![[0.0, 0.0]];
//let mut samples = buf.iter();
// Construct an Output audio unit that delivers audio to the default output device.
let mut audio_unit = try!(AudioUnit::new(IOType::DefaultOutput));
// Q: What is this type?
let callback = move |args| {
let Args { num_frames, mut data, .. } = args;
for i in 0..num_frames {
let sample = samples.next().unwrap();
for (channel_idx, channel) in data.channels_mut().enumerate() {
channel[i] = sample[channel_idx];
}
}
Ok(())
};
type Args = render_callback::Args<data::NonInterleaved<f32>>;
try!(audio_unit.set_render_callback(callback));
try!(audio_unit.start());
std::thread::sleep(std::time::Duration::from_millis(30000));
Ok(())
}
However, changing it up a little bit to load via a buffer doesn't work as well:
extern crate coreaudio;
use coreaudio::audio_unit::{AudioUnit, IOType};
use coreaudio::audio_unit::render_callback::{self, data};
fn main() {
run().unwrap()
}
fn run() -> Result<(), coreaudio::Error> {
let buf: Vec<[f32; 2]> = vec![[0.0, 0.0]];
let mut samples = buf.iter();
// Construct an Output audio unit that delivers audio to the default output device.
let mut audio_unit = try!(AudioUnit::new(IOType::DefaultOutput));
// Q: What is this type?
let callback = move |args| {
let Args { num_frames, mut data, .. } = args;
for i in 0..num_frames {
let sample = samples.next().unwrap();
for (channel_idx, channel) in data.channels_mut().enumerate() {
channel[i] = sample[channel_idx];
}
}
Ok(())
};
type Args = render_callback::Args<data::NonInterleaved<f32>>;
try!(audio_unit.set_render_callback(callback));
try!(audio_unit.start());
std::thread::sleep(std::time::Duration::from_millis(30000));
Ok(())
}
It says, correctly so, that buf only lives until the end of run and does not live long enough for the audio unit—which makes sense, because "borrowed value must be valid for the static lifetime...".
In any case, that doesn't bother me; I can modify the iterator to load and read from the buffer just fine. However, it does raise some questions:
Why does the Iter { value: 0.0 } have the 'static lifetime?
If it doesn't have the 'static lifetime, why does it say the borrowed value must be valid for the 'static lifetime?
If it does have the 'static lifetime, why? It seems like it would be on the heap and closed on by callback.
I understand that the move keyword allows moving inside the closure, which doesn't help me understand why it interacts with lifetimes. Why can't it move the buffer? Do I have to move both the buffer and the iterator into the closure? How would I do that?
Over all this, how do I figure out the expected lifetime without trying to be a compiler myself? It doesn't seem like guessing and compiling is always a straightforward method to resolving these issues.
Why does the Iter { value: 0.0 } have the 'static lifetime?
It doesn't; only references have lifetimes.
why does it say the borrowed value must be valid for the 'static lifetime
how do I figure out the expected lifetime without trying to be a compiler myself
Read the documentation; it tells you the restriction:
fn set_render_callback<F, D>(&mut self, f: F) -> Result<(), Error>
where
F: FnMut(Args<D>) -> Result<(), ()> + 'static, // <====
D: Data
This restriction means that any references inside of F must live at least as long as the 'static lifetime. Having no references is also acceptable.
All type and lifetime restrictions are expressed at the function boundary — this is a hard rule of Rust.
I understand that the move keyword allows moving inside the closure, which doesn't help me understand why it interacts with lifetimes.
The only thing that the move keyword does is force every variable directly used in the closure to be moved into the closure. Otherwise, the compiler tries to be conservative and move in references/mutable references/values based on the usage inside the closure.
Why can't it move the buffer?
The variable buf is never used inside the closure.
Do I have to move both the buffer and the iterator into the closure? How would I do that?
By creating the iterator inside the closure. Now buf is used inside the closure and will be moved:
let callback = move |args| {
let mut samples = buf.iter();
// ...
}
It doesn't seem like guessing and compiling is always a straightforward method to resolving these issues.
Sometimes it is, and sometimes you have to think about why you believe the code to be correct and why the compiler states it isn't and come to an understanding.
Given
fn greet(peeps: &str) {
println!("Hello, {}", peeps);
}
I can do:
fn main() {
let a = "World";
thread::spawn(move || greet(a)).join().unwrap();
}
The compiler understands that the thread does not outlive the borrowed string, but this is merely a special case when the lifetime of the &str is known to be 'static. When I try to do the same with a function argument, it does not compile:
fn indirect(peeps: &str) {
thread::spawn(move || greet(&peeps)).join().unwrap();
// Does not compile, for fear that the thread may outlive peeps
}
However, to a human reader, it is obviously the case that the thread cannot outlive the borrowed string.
I have found two workarounds:
Make a copy of the string, which can be moved into the thread:
fn indirect(peeps: &str) {
let peeps = peeps.to_string();
thread::spawn(move || greet(&peeps)).join().unwrap();
}
or, make use of the famously deprecated thread::scoped:
#![feature(scoped)]
fn indirect_scoped(peeps: &str) {
thread::scoped(move || greet(&peeps)).join();
}
I don't want to specify 'static lifetime for the function parameter, I'd prefer not to make an unnecessary copy (workaround 1) and I'd prefer not to use deprecated features (workaround 2).
What should I do in this situation?
The approach with scoped() is the correct way when you want to pass borrowed data to child threads. While thread::scoped() itself is deprecated due to its unsoundness, an alternative sound APIs like crossbeam or scoped_threadpool provide a way to do this on stable Rust:
extern crate crossbeam;
fn indirect(peeps: &str) {
crossbeam::scope(|scope| {
scope.spawn(|| greet(peeps));
});
}
I am attempting to write a simpler unit test runner for my Rust project. I have created a TestFixture trait that my test fixture structs will implement, similar to inheriting from the unit test base class in other testing frameworks. The trait is fairly simple. This is my test fixture
pub trait TestFixture {
fn setup(&mut self) -> () {}
fn teardown(&mut self) -> () {}
fn before_each(&mut self) -> () {}
fn after_each(&mut self) -> () {}
fn tests(&mut self) -> Vec<Box<Fn(&mut Self)>>
where Self: Sized {
Vec::new()
}
}
My test running function is as follows
pub fn test_fixture_runner<T: TestFixture>(fixture: &mut T) {
fixture.setup();
let _r = fixture.tests().iter().map(|t| {
let handle = thread::spawn(move || {
fixture.before_each();
t(fixture);
fixture.after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
});
fixture.teardown();
}
I get the error
src/tests.rs:73:22: 73:35 error: the trait `core::marker::Send` is not implemented for the type `T` [E0277]
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 note: `T` cannot be sent between threads safely
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 error: the trait `core::marker::Sync` is not implemented for the type `for<'r> core::ops::Fn(&'r mut T)` [E0277]
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
src/tests.rs:69:41: 84:6 note: expansion site
src/tests.rs:73:22: 73:35 note: `for<'r> core::ops::Fn(&'r mut T)` cannot be shared between threads safely
src/tests.rs:73 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
I have tried adding Arcs around the types being sent to the thread, no dice, same error.
pub fn test_fixture_runner<T: TestFixture>(fixture: &mut T) {
fixture.setup();
let fix_arc = Arc::new(Mutex::new(fixture));
let _r = fixture.tests().iter().map(|t| {
let test_arc = Arc::new(Mutex::new(t));
let fix_arc_clone = fix_arc.clone();
let test_arc_clone = test_arc.clone();
let handle = thread::spawn(move || {
let thread_test = test_arc_clone.lock().unwrap();
let thread_fix = fix_arc_clone.lock().unwrap();
(*thread_fix).before_each();
(*thread_test)(*thread_fix);
(*thread_fix).after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
});
fixture.teardown();
}
A sample test fixture would be something like
struct BuiltinTests {
pwd: PathBuf
}
impl TestFixture for BuiltinTests {
fn setup(&mut self) {
let mut pwd = env::temp_dir();
pwd.push("pwd");
fs::create_dir(&pwd);
self.pwd = pwd;
}
fn teardown(&mut self) {
fs::remove_dir(&self.pwd);
}
fn tests(&mut self) -> Vec<Box<Fn(&mut BuiltinTests)>> {
vec![Box::new(BuiltinTests::cd_with_no_args)]
}
}
impl BuiltinTests {
fn new() -> BuiltinTests {
BuiltinTests {
pwd: PathBuf::new()
}
}
}
fn cd_with_no_args(&mut self) {
let home = String::from("/");
env::set_var("HOME", &home);
let mut cd = Cd::new();
cd.run(&[]);
assert_eq!(env::var("PWD"), Ok(home));
}
#[test]
fn cd_tests() {
let mut builtin_tests = BuiltinTests::new();
test_fixture_runner(&mut builtin_tests);
}
My whole intention of using threads is isolation from the test runner. If a test fails an assertion it causes a panic which kills the runner. Thanks for any insight, I'm willing to change my design if that will fix the panic problem.
There are several problems with your code, I'll show you how to fix them one by one.
The first problem is that you're using map() to iterate over an iterator. It won't work correctly because map() is lazy - unless you consume the iterator, the closure you passed to it won't run. The correct way is to use for loop:
for t in fixture().tests().iter() {
Second, you're iterating the vector of closures by reference:
fixture.tests().iter().map(|t| {
iter() on a Vec<T> returns an iterator yielding items of type &T, so your t will be of type &Box<Fn(&mut Self)>. However, Box<Fn(&mut T)> does not implement Sync by default (it is a trait object which have no information about the underlying type except that you specified explicitly), so &Box<Fn(&mut T)> can't be used across multiple threads. That's what the second error you see is about.
Most likely you don't want to use these closures by reference; you probably want to move them to the spawned thread entirely. For this you need to use into_iter() instead of iter():
for t in fixture.tests().into_iter() {
Now t will be of type Box<Fn(&mut T)>. However, it still can't be sent across threads. Again, it is a trait object, and the compiler does not know if the type contained inside is Send. For this you need to add Send bound to the type of the closure:
fn tests(&mut self) -> Vec<Box<Fn(&mut Self)+Send>>
Now the error about Fn is gone.
The last error is about Send not being implemented for T. We need to add a Send bound on T:
pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
And now the error becomes more comprehensible:
test.rs:18:22: 18:35 error: captured variable `fixture` does not outlive the enclosing closure
test.rs:18 let handle = thread::spawn(move || {
^~~~~~~~~~~~~
note: in expansion of closure expansion
test.rs:18:5: 28:6 note: expansion site
test.rs:15:66: 31:2 note: captured variable is valid for the anonymous lifetime #1 defined on the block at 15:65
test.rs:15 pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
test.rs:16 fixture.setup();
test.rs:17
test.rs:18 for t in fixture.tests().into_iter() {
test.rs:19 let handle = thread::spawn(move || {
test.rs:20 fixture.before_each();
...
note: closure is valid for the static lifetime
This error happens because you're trying to use a reference in a spawn()ed thread. spawn() requires its closure argument to have 'static bound, that is, its captured environment must not contain references with non-'static lifetimes. But that's exactly what happens here - &mut T is not 'static. spawn() design does not prohibit avoiding joining, so it is explicitly written to disallow passing non-'static references to the spawned thread.
Note that while you're using &mut T, this error is unavoidable, even if you put &mut T in Arc, because then the lifetime of &mut T would be "stored" in Arc and so Arc<Mutex<&mut T>> also won't be 'static.
There are two ways to do what you want.
First, you can use the unstable thread::scoped() API. It is unstable because it is shown to allow memory unsafety in safe code, and the plan is to provide some kind of replacement for it in the future. However, you can use it in nightly Rust (it won't cause memory unsafety by itself, only in specifically crafted situations):
pub fn test_fixture_runner<T: TestFixture+Send>(fixture: &mut T) {
fixture.setup();
let tests = fixture.lock().unwrap().tests();
for t in tests.into_iter() {
let f = &mut *fixture;
let handle = thread::scoped(move || {
f.before_each();
t(f);
f.after_each();
});
handle.join();
}
fixture.teardown();
}
This code compiles because scoped() is written in such a way that it guarantees (in most cases) that the thread won't outlive all captured references. I had to reborrow fixture because otherwise (because &mut references aren't copyable) it would be moved into the thread and fixture.teardown() would be prohibited. Also I had to extract tests variable because otherwise the mutex will be locked by the main thread for the duration of the for loop which would naturally disallow locking it in the child threads.
However, with scoped() you can't isolate the panic in the child thread. If the child thread panics, this panic will be rethrown from join() call. This may or may not be a problem in general, but I think it is a problem for your code.
Another way is to refactor your code to hold the fixture in Arc<Mutex<..>> from the beginning:
pub fn test_fixture_runner<T: TestFixture + Send + 'static>(fixture: Arc<Mutex<T>>) {
fixture.lock().unwrap().setup();
for t in fixture.lock().unwrap().tests().into_iter() {
let fixture = fixture.clone();
let handle = thread::spawn(move || {
let mut fixture = fixture.lock().unwrap();
fixture.before_each();
t(&mut *fixture);
fixture.after_each();
});
if let Err(_) = handle.join() {
println!("Test failed!")
}
}
fixture.lock().unwrap().teardown();
}
Note that now T has also to be 'static, again, because otherwise it couldn't be used with thread::spawn() as it requires 'static. fixture inside the inner closure is not &mut T but a MutexGuard<T>, and so it has to be explicitly converted to &mut T in order to pass it to t.
This may seem overly and unnecessarily complex, however, such design of a programming language does prevent you from making many errors in multithreaded programming. Each of the above errors we have seen is valid - each of them would be a potential cause of memory unsafety or data races if it was ignored.
As stated in the Rust HandBook's Concurrency section:
When a type T implements Send, it indicates to the compiler that something of this type is able to have ownership transferred safely between threads.
If you do not implement Send, ownership cannot be transfered between threads.