Deferred initialization and mutable static borrowing [Rust] - multithreading

My question is the following:
Would it be possible to make a deferred initialization of an object and then borrow it as mutable for a 'static lifetime?
Some context
First of all, I am going to show an example of what I mean by deferred initialization. The deferred initialization model that I have come up with is to use the crate lazy_static with a standard library Mutex holding an Option<T>. Here there is an example of a deferred initialization.
use lazy_static::lazy_static;
use std::thread::JoinHandle;
use std::sync::Mutex;
pub struct DeferredStruct {
internal_field: u32,
}
impl DeferredStruct {
pub fn new(internal_field: u32) -> Self {
Self {
internal_field,
}
}
pub fn regular_function(&mut self) {
self.internal_field += 1;
println!("{}", self.internal_field);
}
}
lazy_static! {
static ref MY_DEFERRED: Mutex<Option<DeferredStruct>> = Mutex::new(None);
}
fn main() {
// Initial processing
// ...
// ...
// The value 10 would be obtained from a configuration file on runtime.
let deferred_struct = DeferredStruct::new(10);
let mut lock = MY_DEFERRED.lock().unwrap();
lock.replace(deferred_struct);
std::mem::drop(lock); // In this example we drop the lock to avoid a deadlock when calling another_function.
// More processing
// ...
// ...
let thread_handle = another_function();
thread_handle.join().unwrap();
}
// From another part of the program and possibly from another thread we
// lock MY_DEFERRED and call regular_funcion on it.
fn another_function() -> JoinHandle<()> {
std::thread::spawn(|| {
let mut lock = MY_DEFERRED.lock().unwrap();
if let Some(deferred) = lock.as_mut() {
deferred.regular_function();
}
})
}
You can execute the above code in Rust Playground and check that it prints 11 correctly.
Introducing a struct function that requires a static lifetime
Now, let's say I add a function inside DeferredStruct that will create a worker thread in order to execute some computations that will take a long time:
pub struct DeferredStruct {
internal_field: u32,
}
impl DeferredStruct {
pub fn new(internal_field: u32) -> Self {
Self {
internal_field,
}
}
pub fn regular_function(&mut self) {
self.internal_field += 1;
println!("{}", self.internal_field);
}
pub fn static_function(&'static mut self) {
std::thread::spawn(move || {
// Do something really long.
// Finally I make some changes on self
self.internal_field += 100;
});
}
}
In this case is required for &mut self to have a 'static lifetime:
error[E0759]: `self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement.
Due to this, the function will take a &'static mut self.
The problem comes when trying to borrow the DeferredStruct inside the Option in MY_DEFERRED as 'static and mut.
If I cannot borrow for 'static and mut then I cannot call deferred.static_function(). As the value will not live for long enough.
error[E0597]: `lock` does not live long enough
--> src/main.rs:57:33
|
57 | if let Some(deferred) = lock.as_mut() {
| ^^^^---------
| |
| borrowed value does not live long enough
| argument requires that `lock` is borrowed for `'static`
...
60 | })
| - `lock` dropped here while still borrowed
Here there is a minimal reproducible example in Rust Playground.
TL;DR
Is it possible to borrow an object created at runtime (not necessarily created at the immediate start of the program) inside a Mutex<Option<T>> as mutable and for a 'static lifetime?
Any help is appreciated.

Related

borrowed data escapes outside of closure

I created two structs, Car and CarModifier. Both Car and CarModifier are instantiated in main.rs. CarModifier modifies one of the fields of Car using data from curl request. Here is the code:
main.rs
use sample::{car::Car, car_modifier::CarModifier};
fn main() {
println!("Hello, world!");
let mut car = Car::new();
let mut car_modifier = CarModifier::new();
car_modifier.update_car(&mut car);
println!("{:?}", car.driver);
}
Car.rs
#[derive(Debug)]
pub struct Car<'a> {
pub driver: &'a [u8]
}
impl<'a> Car<'a> {
pub fn new() -> Self {
Self {
driver: &[26]
}
}
pub fn update_driver(&mut self, new_driver: &'a [u8]) {
self.driver = new_driver;
}
}
CarModifier.rs
use curl::easy::Easy;
use crate::{car::Car};
pub struct CarModifier;
impl CarModifier {
pub fn new() -> Self {
Self {
}
}
pub fn update_car(&mut self, car: &mut Car) {
let mut easy = Easy::new();
easy.url("https://www.rust-lang.org/").unwrap();
let mut transfer = easy.transfer();
transfer.write_function(|data: &[u8]| {
car.update_driver(data);
Ok(data.len())
}).unwrap();
transfer.perform().unwrap();
}
}
This is the error which I get when trying to run
error[E0521]: borrowed data escapes outside of closure
--> src/car_modifier.rs:19:13
|
13 | pub fn update_car(&mut self, car: &mut Car) {
| --- `car` declared here, outside of the closure body
...
18 | transfer.write_function(|data: &[u8]| {
| ---- `data` is a reference that is only valid in the closure body
19 | car.update_driver(data);
| ^^^^^^^^^^^^^^^^^^^^^^^ `data` escapes the closure body here
For more information about this error, try `rustc --explain E0521`.
error: could not compile `sample` due to previous error
I seem to understand from the error that data lives only in the closure body and the code tries to refer that in car.driver which will outlive the closure body, hence the error. Is this the right understanding?
I am also aware that instead of using &[u8], I could use Vec<u8> to resolve the error. But how can I keep using &[u8]?
Basically data belongs to the curl library. It may be a slice of some library internal buffer pool, and may be re-used for some other curl api calls later on, or freed after this call. You need to copy data into the Car structure instead of just having a pointer to the underlying buffer. Rust compiler helps you to catch that problem. In C++ the compiler won't even tell you this error and the C++ code will have runtime problems.

Parameterized lifetime causing trouble in chains of function

playground link
I'm having trouble trouble understanding why this code can't compile:
struct Ctx<'a> {
n: u64,
phantom: std::marker::PhantomData<&'a u8>,
}
fn process<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>) {
ctx.n += 1;
}
fn main() {
let mut ctx = Ctx { n: 0, phantom: std::marker::PhantomData };
let ref_ctx = &mut ctx;
process(ref_ctx);
process(ref_ctx);
}
The code is silly, but the error is the same as on the "real" code:
Compiling playground v0.0.1 (/playground)
error[E0499]: cannot borrow `*ref_ctx` as mutable more than once at a time
--> src/main.rs:14:13
|
13 | process(ref_ctx);
| ------- first mutable borrow occurs here
14 | process(ref_ctx);
| ^^^^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
For more information about this error, try `rustc --explain E0499`.
error: could not compile `playground` due to previous error
If change the process signature to
fn process(ctx: &mut Ctx<'a>) {
Then it compiles, but I need the lifetimes annotations because in practice Ctx contains references that needs to be annotated. I feel like the problem is 'b: 'a because is could be telling that the borrow MUST outlive the references in Ctx, but it feels weird that just calling a function that returns nothing would extend the borrow,
That feels like a beginner's question, but it definitely has me stumped.
edit:
The code above is minimal because I wanted to avoid explaining the whole context (pun intended), but it might be necessary to find a practical solution. Sorry, that's going to be a bit long.
The real Ctx is here, it's struct that wraps a few Varnish C pointers and gets passed to plugins so they can read it and possibly change it, notably manipulating HTTP objects.
An important part of Ctx is the WS contained in it. It's a workspace that can return non-overlapping slices of writable data that will be deleted automatically once a task has been completed. For the compiler, the best we can tell it is "any slice allocated in the WS will survive for as long as the Ctx wil live".
WS can be used in two ways:
either directly by a plugin to request a slice it'll fill in
or by the boilerplate that's currently python-generated to convert Varnish types into C types. Typically, the plugin returns a String and the boilerplate will allocate a WS slice and copy the data for storage. And example of generated code looks like this:
// the C-matching function that needs to exist for the plugin frame work to pick it up
unsafe extern "C" fn vmod_c_unset_hdr(vrt_ctx: * mut varnish_sys::vrt_ctx, arg1: varnish_sys::VCL_STRING) {
// create the rust Ctx from the pointer
let mut _ctx = Ctx::new(vrt_ctx);
// call the plugin function
match vmod::unset_hdr(
&mut _ctx,
&*arg1.into_rust()
).into_result() { // make sure we have a Result by wrapping lone values
Err(ref e) => { _ctx.fail(e); }, // yell if we didn't got and Err
Ok(v) => v.into_vcl(&mut _ctx.ws), // convert the Ok value into a C type, possibly by copying it into the workspace
}
}
Now, so far, the code at the commit I shared works (hurray), but I get into trouble when I try to implement WS_ReserveAll and WS_Release: basically, grab all the free space you can, write what you need, and then tell Varnish how much you used so the rest is reclaimed and can be used by future calls.
Of course, you can only WS_ReserveAll() once before you need to WS_Release so that felt like a perfect job for Rust.
I though I'd create a new type ReservedBuf, by adding this to the code:
impl<'a> struct WS<'a> {
// some lines omitted
pub fn reserve(&'a mut self) -> ReservedBuf<'a> {
let wsp = unsafe { self.raw.as_mut().unwrap() };
assert_eq!(wsp.magic, varnish_sys::WS_MAGIC);
unsafe {
let sz = varnish_sys::WS_ReserveAll(wsp) as usize;
let buf = from_raw_parts_mut(wsp.f as *mut u8, sz as usize);
ReservedBuf {
buf,
wsp: self.raw,
b: wsp.f as *const u8,
len: 0,
}
}
}
}
pub struct ReservedBuf<'a> {
pub buf: &'a mut [u8],
wsp: *mut varnish_sys::ws,
b: *const u8,
len: usize,
}
impl<'a> ReservedBuf<'a> {
pub fn seal(mut self, sz: usize) -> &'a [u8] {
unsafe {
self.len = std::cmp::min(sz, self.buf.as_ptr().add(sz).offset_from(self.b) as usize);
from_raw_parts_mut(self.buf.as_mut_ptr(), self.len)
}
}
}
impl<'a> Drop for ReservedBuf<'a> {
fn drop(&mut self) {
unsafe {
let wsp = self.wsp.as_mut().unwrap();
assert_eq!(wsp.magic, varnish_sys::WS_MAGIC);
varnish_sys::WS_Release(wsp, self.len as u32);
}
}
}
And it appears to work well enough in the limited tests I ran. My plugin function can be this for example:
pub fn ws_reserve<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>, s: &str) -> Result<varnish_sys::VCL_STRING, String> {
let mut rbuf = ctx.ws.reserve();
let s_buf = s.as_bytes();
let vcl_string = rbuf.buf.as_ptr() as *const i8;
rbuf.buf.write(s_buf);
rbuf.buf.write(b" ");
rbuf.buf.write(s_buf);
rbuf.buf.write(b" ");
rbuf.buf.write(s_buf);
rbuf.buf.write(b"\0");
rbuf.seal(0);
Ok(vcl_string)
}
but I need the lifetime annotations, otherwise I get:
--> src/vmod.rs:21:27
|
20 | pub fn ws_reserve(ctx: &mut Ctx, s: &str) -> Result<varnish_sys::VCL_STRING, String> {
| --------
| |
| these two types are declared with different lifetimes...
21 | let mut rbuf = ctx.ws.reserve();
| ^^^^^^^ ...but data from `ctx` flows into `ctx` here
but with the lifetime annotations, the generated-code calling ws_reserve complains:
unsafe extern "C" fn vmod_c_ws_reserve(vrt_ctx: * mut varnish_sys::vrt_ctx, arg1: varnish_sys::VCL_STRING) -> varnish_sys::VCL_STRING {
let mut _ctx = Ctx::new(vrt_ctx);
match vmod::ws_reserve(
&mut _ctx,
&*arg1.into_rust()
).into_result() {
Err(ref e) => { _ctx.fail(e); ptr::null() },
Ok(v) => v.into_vcl(&mut _ctx.ws),
}
}
error[E0499]: cannot borrow `_ctx` as mutable more than once at a time
--> /home/gquintard/project/varnish-rs/vmod_test/target/debug/build/vmod_example-7ff3d57b642f2bfa/out/generated.rs:69:29
|
66 | &mut _ctx,
| --------- first mutable borrow occurs here
...
69 | Err(ref e) => { _ctx.fail(e); ptr::null() },
| ^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
error[E0499]: cannot borrow `_ctx.ws` as mutable more than once at a time
--> /home/gquintard/project/varnish-rs/vmod_test/target/debug/build/vmod_example-7ff3d57b642f2bfa/out/generated.rs:70:33
|
66 | &mut _ctx,
| --------- first mutable borrow occurs here
...
70 | Ok(v) => v.into_vcl(&mut _ctx.ws),
| ^^^^^^^^^^^^
| |
| second mutable borrow occurs here
| first borrow later used here
For more information about this error, try `rustc --explain E0499`.
error: could not compile `vmod_example` due to 2 previous errors
I guess I sort of understand the compiler qualms about the issues, and I might have painted myself into a corner, but I don't see how to solve this.
When we take a closer look at your code we see you write this:
fn process<'a, 'b: 'a>(ctx: &'b mut Ctx<'a>) {
This 'b: 'a reads as 'b outlives 'a. So you're basically writing a method which borrows ctx longer than what ctx is holding internally with 'a.
With that knowledge it seems very logical why this won't work.
let mut ctx = Ctx { n: 0, phantom: std::marker::PhantomData };
let ref_ctx = &mut ctx;
process(ref_ctx); // Borrow ref_ctx for at least the lifetime of `Ctx::<'a>`
process(ref_ctx); // Can't borrow it again since `'a` has not expired yet
What you're trying to do is probably this:
fn process<'a: 'b, 'b>(ctx: &'b mut Ctx<'a>) {
Borrow ctx for the lifetime 'b which is less long than 'a
If that's the case you can just write this:
fn process<'a>(ctx: &mut Ctx<'a>) {
Or even this:
fn process(ctx: &mut Ctx) {
If this isn't what you're trying to do I hope it still makes sense that using the constraint 'b: 'a creates an impossible borrow.

Putting `self` into an `Arc` in Rust

I have a Rust struct with a method that is designed to parallelise over multiple threads. Each thread needs to access that struct. So naturally I want to put it into an Arc. However I can't work out a way to do this.
Here's a simplified example:
use std::sync::{Arc};
use std::thread;
struct Foo {
field: usize
}
impl Foo {
fn parallel(&self){
let arc = Arc::new(&self);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::spawn(move || {
i + arc.field
});
}
}
}
And the compiler says:
fn parallel(&self){
^^^^^ this data with an anonymous lifetime `'_`...
let arc = Arc::new(&self);
^^^^^ ...is captured here...
thread::spawn(move || {
^^^^^^^^^^^^^ ...and is required to live as long as `'static` here
`self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
So it would seem that I'm actually not allowed to put self into an Arc (not quite sure why, though).
I've also tried cloning self but that gives the same error:
impl Foo {
fn parallel(&self){
let clone = self.clone();
let arc = Arc::new(&clone);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::spawn(move || {
i + arc.field
});
}
}
}
fn parallel(&self){
^^^^^ this data with an anonymous lifetime `'_`...
let clone = self.clone();
^^^^^ ...is captured here...
thread::spawn(move || {
^^^^^^^^^^^^^ ...and is required to live as long as `'static` here
`self` has an anonymous lifetime `'_` but it needs to satisfy a `'static` lifetime requirement
Similar things happen if I try to move the clone into the Arc, or if I remove the move in front of the closure.
What is going on here, and how can I fix it? Also, in general, is there any way to do this: ie putting self into an Arc, where self is a struct?
If you put self or &self in the arc, you have to take it back at the end of the function.
It means you can't let it to the other threads.
The crossbeam crate includes a facility to deal with exactly this: ensure the threads release what you gave them:
Creates a new scope for spawning threads.
All child threads that haven’t been manually joined will be
automatically joined just before this function invocation ends. If all
joined threads have successfully completed, Ok is returned with the
return value of f. If any of the joined threads has panicked, an Err
is returned containing errors from panicked threads.
Example:
use std::sync::{Arc};
use crossbeam::thread;
struct Foo {
field: usize
}
impl Foo {
fn parallel(&self){
let arc = Arc::new(&self);
for i in 1..5 {
let arc = Arc::clone(&arc);
thread::scope(|s| {
s.spawn(|_| {
i + arc.field
});
});
}
}
}
You can wrap Foo in Arc and define methods for Arc<Foo>
use std::{sync::Arc, time::Duration};
use tokio::task::JoinSet;
struct Foo {
field: usize
}
impl Foo {
async fn add(&self, n: usize) {
println!("calculation {} + {n}...", self.field);
tokio::time::sleep(Duration::from_secs(1)).await;
println!("{} + {n} = {}", self.field, self.field + n);
}
async fn parallel(self: Arc<Self>){
let mut set = JoinSet::new();
for i in 1..5 {
let foo = self.clone();
set.spawn(async move {
foo.add(i).await
});
}
// wait for tasks...
while let Some(_) = set.join_next().await {}
}
}
#[tokio::main]
async fn main() {
println!("Start");
let foo = Foo { field: 23 };
foo.add(10).await;
// foo.parallel().await; // error: Foo should be wrapped in Arc
println!("Non parallel done");
let foo = Arc::new(foo);
{
let foo = foo.clone();
tokio::spawn(async move { foo.parallel().await });
}
foo.parallel().await;
println!("End");
}

Convert boxed trait to mutable trait reference in Rust

I am having some trouble with the dynamic dispatch pointer types in Rust. I want to convert a value of type Box<MyTrait> to &mut MyTrait to pass to a function. For example, I tried:
use std::borrow::BorrowMut;
trait MyTrait {
fn say_hi(&mut self);
}
struct MyStruct { }
impl MyTrait for MyStruct {
fn say_hi(&mut self) {
println!("hi");
}
}
fn invoke_from_ref(value: &mut MyTrait) {
value.say_hi();
}
fn main() {
let mut boxed_trait: Box<MyTrait> = Box::new(MyStruct {});
invoke_from_ref(boxed_trait.borrow_mut());
}
This fails with the following error:
error: `boxed_trait` does not live long enough
--> <anon>:22:5
|
21 | invoke_from_ref(boxed_trait.borrow_mut());
| ----------- borrow occurs here
22 | }
| ^ `boxed_trait` dropped here while still borrowed
|
= note: values in a scope are dropped in the opposite order they are created
Strangely enough, this works for &MyTrait but not for &mut MyTrait. Is there any way I can get this conversion to work in the mutable case?
I think you're running into a limitation of the current compiler's lifetime handling. borrow_mut, being a function, imposes stricter lifetime requirements than necessary.
Instead, you can take a mutable borrow to the box's interior by first dereferencing the box, like this:
fn main() {
let mut boxed_trait: Box<MyTrait> = Box::new(MyStruct {});
invoke_from_ref(&mut *boxed_trait);
}

Why doesn't my struct live long enough?

In Rust, I get the following error:
<anon>:14:9: 14:17 error: `mystruct` does not live long enough
<anon>:14 mystruct.update();
^~~~~~~~
<anon>:10:5: 17:6 note: reference must be valid for the lifetime 'a as defined on the block at 10:4...
<anon>:10 {
<anon>:11 let initial = vec![Box::new(1), Box::new(2)];
<anon>:12 let mystruct = MyStruct { v : initial, p : &arg };
<anon>:13
<anon>:14 mystruct.update();
<anon>:15
...
<anon>:12:59: 17:6 note: ...but borrowed value is only valid for the block suffix following statement 1 at 12:58
<anon>:12 let mystruct = MyStruct { v : initial, p : &arg };
<anon>:13
<anon>:14 mystruct.update();
<anon>:15
<anon>:16 mystruct
<anon>:17 }
error: aborting due to previous error
for the following code:
struct MyStruct<'a>
{
v : Vec<Box<i32>>,
p : &'a i32
}
impl<'a> MyStruct<'a>
{
fn new(arg : &'a i32) -> MyStruct<'a>
{
let initial = vec![Box::new(1), Box::new(2)];
let mystruct = MyStruct { v : initial, p : &arg };
mystruct.update();
mystruct
}
fn update(&'a mut self)
{
self.p = &self.v.last().unwrap();
}
}
fn main() {
let x = 5;
let mut obj = MyStruct::new(&x);
}
(Playground)
I don't understand why mystruct does not live enough. If I comment out the mystruct.update() line it works fine though. What's more is, if I comment out the body of update the code still fails. Why does calling an empty function which borrows a mutable self changes things?
I don't understand which reference is the one the error talks about. Can somebody explain this?
The reference this error talks about is the one which is implicitly created when you call update(). Because update() takes &'a mut self, it means that it accepts a value of type &'a mut MyStruct<'a>. It means that in theory you should call update() like this:
(&mut mystruct).update();
It would be very inconvenient to write this everywhere, and so Rust is able to automatically insert necessary &s, &muts and *s in order to call a method. This is called "autoreference", and the only place it happens is method invocations/field access.
The problem is the definition of update() method:
impl<'a> MyStruct<'a> {
...
fn update(&'a mut self) { ... }
...
}
Here you are requesting that update() receives the value it is called at via a reference with lifetime 'a, where 'a is the lifetime of the reference stored in the structure.
However, when you have a structure value you're calling this method on, there should be already a reference to i32 you stored in this structure. Hence the lifetime of the structure value is strictly smaller than the lifetime designated by the lifetime parameter, so it is just impossible to construct &'a mut MyStruct<'a> with local variables (as in your case).
The solution is to use &mut self instead of &'a mut self:
fn update(&mut self) { ... }
// essentially equivalent to
fn update<'b>(&'b mut self) where 'a: 'b { ... }
// `'b` is a fresh local lifetime parameter
This way the lifetime of the structure in this method call is not tied to the reference this structure contains and can be smaller.
More in-depth explanation follows below.
By itself your definition is not nonsense. For example:
struct IntRefWrapper<'a> {
value: &'a i32
}
static X: i32 = 12345;
static Y: IntRefWrapper<'static> = IntRefWrapper { value: &X };
impl<'a> IntRefWrapper<'a> {
fn update(&'a self) { ... }
}
Y.update();
Here update() invocation won't cause compilation errors because both lifetimes (of Y and of X, reference to which is contained in Y) are 'static.
Let's consider your example, for comparison:
impl<'a> MyStruct<'a> {
fn new(arg : &'a i32) -> MyStruct<'a> {
let initial = vec![Box::new(1), Box::new(2)];
let mystruct = MyStruct { v : initial, p : &arg };
mystruct.update();
mystruct
}
}
Here we have a lifetime parameter, 'a, which is supplied by the caller of the function. For example, the caller could call this function with a static reference:
static X: i32 = 12345;
MyStruct::new(&X); // here &X has static lifetime
However, when update() method is invoked, mystruct lifetime is bounded by the block it is called in:
{
let initial = vec![Box::new(1), Box::new(2)];
let mystruct = MyStruct { v : initial, p : &arg }; // +
// |
mystruct.update(); // |
// |
mystruct // |
}
Naturally, the borrow checker can't prove that this lifetime is the same as the lifetime provided by the caller (and for any possible "external" lifetime it is indeed impossible for them to match), so it throws an error.
When update is defined like this:
fn update(&mut self) { ... }
// or, equivalently
fn update<'b>(&'b mut self) where 'a: 'b { ... }
then when you call it, it is no longer required that the value you call this method on must live exactly as long as 'a - it is sufficient for it to live for any lifetime which is smaller than or equal to 'a - and the lifetime inside the function perfectly matches these requirements. Thus you can call such method on your value, and the compiler won't complain.
Additionally (as noticed in the comments) the following line is indeed invalid and there is no way around it:
self.p = &self.v.last().unwrap();
The borrow check fails here because you're trying to store a reference with lifetime of the structure into the structure itself. In general this can't be done because it has nasty soundness issues. For example, suppose you were indeed able to store this reference into the structure. But now you can't mutate Vec<Box<i32>> in the structure because it may destroy an element which the previously stored references points at, making the code memory unsafe.
It is impossible to check for such things statically, and so it is disallowed on the borrow checking level. In fact, it is just a nice consequence of general borrow checking rules.

Resources