I'm trying to figure out how to send a function through a channel, and how to avoid extra cloning in order to execute the function at the other end. If I remove the extra cloning operation inside the closure, I get the following error:
error: cannot move out of captured outer variable in an 'Fn' closure
Ignoring the fact that this code does absolutely nothing, and makes use of a global mutable static Sender<T>, it represents what I'm trying to achieve while giving the proper compiler errors. This code is not meant to be ran, just compiled.
use std::ops::DerefMut;
use std::sync::{Arc, Mutex};
use std::collections::LinkedList;
use std::sync::mpsc::{Sender, Receiver};
type SafeList = Arc<Mutex<LinkedList<u8>>>;
type SendableFn = Arc<Mutex<(Fn() + Send + Sync + 'static)>>;
static mut tx: *mut Sender<SendableFn> = 0 as *mut Sender<SendableFn>;
fn main() {
let list: SafeList = Arc::new(Mutex::new(LinkedList::new()));
loop {
let t_list = list.clone();
run(move || {
foo(t_list.clone());
});
}
}
fn run<T: Fn() + Send + Sync + 'static>(task: T) {
unsafe {
let _ = (*tx).send(Arc::new(Mutex::new(task)));
}
}
#[allow(dead_code)]
fn execute(rx: Receiver<SendableFn>) {
for t in rx.iter() {
let mut guard = t.lock().unwrap();
let task = guard.deref_mut();
task();
}
}
#[allow(unused_variables)]
fn foo(list: SafeList) { }
Is there a better method to getting around that error and/or another way I should be sending functions through channels?
The problem with Fn() is that you can call it multiple times. If you moved out of a captured value, that value would not be available anymore at the next call. You need a FnOnce() to make sure calling the closure also moves out of it, so it's gone and can't be called again.
There's no way to have an Arc<Mutex<(FnOnce() + Send + Sync + 'static)>>. This would again require that you statically guarantee that after you call the function, noone else can call it again. Which you cannot, since someone else might have another Arc pointing to your FnOnce. What you can do is box it and send it as Box<FnOnce() + Send + Sync + 'static>. There's only ever one owner of a Box.
The trouble with FnOnce() is, is that you can't really call it while it's in the Box, because that would require moving it out of the Box and calling it. But we don't know the size of it, so we cannot move it out of the Box. In the future Box<FnOnce()> closures might become directly usable.
"Luckily" this problem occurred more often, so there's FnBox. Sadly this requires nightly to work. Also I couldn't figure out how to use the function call syntax that is described in the docs, but you can manually call call_box on the Box<FnBox()>. Try it out in the Playground
Related
I'm attempting to write a generic set_interval function helper:
pub fn set_interval<F, Fut>(mut f: F, dur: Duration)
where
F: Send + 'static + FnMut() -> Fut,
Fut: Future<Output = ()> + Send + 'static,
{
let mut interval = tokio::time::interval(dur);
tokio::spawn(async move {
// first tick is at 0ms
interval.tick().await;
loop {
interval.tick().await;
tokio::spawn(f());
}
});
}
This works fine until it's called from inside a class:
fn main() {}
struct Foo {}
impl Foo {
fn bar(&self) {
set_interval(|| self.task(), Duration::from_millis(1000));
}
async fn task(&self) {
}
}
self is not 'static, and we can't restrict lifetime parameter to something that is less than 'static because of tokio::task.
Is it possible to modify set_interval implementation so it works in cases like this?
Link to playground
P.S. Tried to
let instance = self.clone();
set_interval(move || instance.task(), Duration::from_millis(1000));
but I also get an error: error: captured variable cannot escape FnMut closure body
Is it possible to modify set_interval implementation so it works in cases like this?
Not really. Though spawn-ing f() really doesn't help either, as it precludes a simple "callback owns the object" solution (as you need either both callback and future to own the object, or just future).
I think that leaves two solutions:
Convert everything to shared mutability Arc, the callback owns one Arc, then on each tick it clones that and moves the clone into the future (the task method).
Have the future (task) acquire the object from some external source instead of being called on one, this way the intermediate callback doesn't need to do anything. Or the callback can do the acquiring and move that into the future, same diff.
Incidentally at this point it could make sense to just create the future directly, but allow cloning it. So instead of taking a callback set_interval would take a clonable future, and it would spawn() clones of its stored future instead of creating them anew.
As mentioned by #Masklinn, you can clone the Arc to allow for this. Note that cloning the Arc will not clone the underlying data, just the pointer, so it is generally OK to do so, and should not have a major impact on performance.
Here is an example. The following code will produce the error async block may outlive the current function, but it borrows data, which is owned by the current function:
fn main() {
// 🛑 Error: async block may outlive the current function, but it borrows data, which is owned by the current function
let data = Arc::new("Hello, World".to_string());
tokio::task::spawn(async {
println!("1: {}", data.len());
});
tokio::task::spawn(async {
println!("2: {}", data.len());
});
}
Rust unhelpfully suggests adding move to both async blocks, but that will result in a borrowing error because there would be multiple ownership.
To fix the problem, we can clone the Arc for each task and then add the move keyword to the async blocks:
fn main() {
let data = Arc::new("Hello, World".to_string());
let data_for_task_1 = data.clone();
tokio::task::spawn(async move {
println!("1: {}", data_for_task_1.len());
});
let data_for_task_2 = data.clone();
tokio::task::spawn(async move {
println!("2: {}", data_for_task_2.len());
});
}
I am trying to run multiple functions on different threads. I wrote this (minimal) code that works
use std::thread;
fn f1(count: usize) {
for i in 0..count { /*do something*/ }
}
fn f2(count: usize) {
for i in 0..count { /*do something*/ }
}
fn run(ops: &Vec<impl Fn(usize) + Sync + Send + Copy + 'static>) {
for i in 0..ops.len() {
let func = ops[i];
thread::spawn(move || func(1000));
}
}
fn main() {
let ops = vec![f1, f1]; // putting the same function twice
run(&ops);
}
However, as soon as I send different functions
let ops = vec![f1, f2];
Compiling fails. As I understand it, I cannot hold different types in the same vector (I guess, functions have different memory requirements, though I'm not quite sure how function pointers work).
I tried browsing similar questions on SO, and I tried this solution
let ops: Vec<&dyn Fn(usize)> = vec![&f1, &f2];
and I get this error
`dyn Fn(usize)` cannot be shared between threads safely
the trait `Sync` is not implemented for `dyn Fn(usize)`
And I'm stuck as I'm struggling to understand the base issue. Do you have insights on how I should understand this problem, and any pointers on how to solve it ?
Thank you!
Just as you can add the Sync marker with impl, you can also add it with &dyn, but you may need parenthesis to disambiguate:
fn run(ops: &Vec<&'static (dyn Fn(usize) + Sync)>)
Two minor comments:
Generally, using &[…] instead of &Vec<…> is more flexible and to be preferred.
No need to do for i in 0…foos.len() in rust, for foo in foos will do fine. (You may occasionally have to use for foo in &foos or for foo in foos.iter(). If you do need the index, .enumerate() is convenient.)
If the 'static is a bother (e.g. because you want to pass in some closure that captures local variables and can't be 'static), you could either use scoped threads (example), or pass owned Fns as Vec<Box<dyn Fn(usize) + Sync>>.
I'm working on an app that optionally uses a GUI to display video data that's roughly structured like this:
fn main() {
let (window_tx, window_rx) =
MainContext::channel::<MyStruct>(PRIORITY_DEFAULT);
let some_thread = thread::spawn(move || -> () {
// send data to window_tx
});
let application =
gtk::Application::new(Some("com.my.app"), Default::default());
application.connect_activate(move |app: >k::Application| {
build_ui(app, window_rx);
});
application.run();
some_thread.join().unwrap();
}
fn build_ui(application: >k::Application, window_rx: Receiver<MyStruct>) {
window_rx.attach( ... );
}
The gtk rust library requires a Fn callback passed to application.connect_activate on startup, so I can't use a FnOnce or FnMut closure to move the glib::Receiver in the callback. The compiler throws this error:
error[E0507]: cannot move out of `window_rx`, a captured variable in an `Fn` closure
I've tried to avoid the move by wrapping window_rx in a Rc, ie:
let r = Rc::new(RefCell::new(window_rx));
application.connect_activate(move |app: >k::Application| {
build_ui(app, Rc::clone(&r));
});
But upon dereferencing the Rc in my build_ui function, I get this error:
error[E0507]: cannot move out of an `Rc`
The fallback I've used thus far is to just move the channel creation and thread creation into my build_ui function, but because the GUI is not required, I was hoping to avoid using GTK and the callback entirely if GUI is not used. Is there some way I can either safely move window_rx within a closure or otherwise dereference it in the callback without causing an error?
When you need to move a value out from code that, by the type system but not in practice, could be called more than once, the simple tool to reach for is Option. Wrapping the value in an Option allows it to be swapped with an Option::None.
When you need something to be mutable even though you're inside a Fn, you need interior mutability; in this case, Cell will do. Here's a complete compilable program that approximates your situation:
use std::cell::Cell;
// Placeholders to let it compile
use std::sync::mpsc;
fn wants_fn_callback<F>(_f: F) where F: Fn() + 'static {}
struct MyStruct;
fn main() {
let (_, window_rx) = mpsc::channel::<MyStruct>();
let window_rx: Cell<Option<mpsc::Receiver<MyStruct>>> = Cell::new(Some(window_rx));
wants_fn_callback(move || {
let _: mpsc::Receiver<MyStruct> = window_rx.take().expect("oops, called twice");
});
}
Cell::take() removes the Option<Receiver> from the Cell, leaving None in its place. The expect then removes the Option wrapper (and handles the possibility of the function being called twice by panicking in that case).
Applied to your original problem, this would be:
let window_rx: Option<Receiver<MyStruct>> = Cell::new(Some(window_rx));
application.connect_activate(move |app: >k::Application| {
build_ui(app, window_rx.take().expect("oops, called twice"));
});
However, be careful: if the library requires a Fn closure, there might be some condition under which the function could be called more than once, in which case you should be prepared to do something appropriate in that circumstance. If there isn't such a condition, then the library's API should be improved to take a FnOnce instead.
Suppose I have an object video_source: Option<Arc<Mutex<Box<dyn GetVideo>>>> and I pass it to a thread:
std::thread::spawn(||{
loop {
if let Some(video_source) = video_source {
let video_frame = video_source.lock().unwrap().get();
}
}
})
where
trait GetVideo {
fn get() -> Vec<u8>
}
What if I want to change the video source on the fly? Well, I'd do this on another thread:
video_frame.unwrap().lock().unwrap() = Box::new(other_source);
I want to make this idea more generic. I want a type that permits such thing. Here's my sketch:
use std::sync::{Arc, Mutex};
pub type OnTheFlyInner<T> = Box<T + Send + Sync>;
pub type OnTheFly<T> = Arc<Mutex<OnTheFlyInner<T>>>;
//I'd like this to be a method of `OnTheFly`
pub fn on_the_fly_substitute(on_the_fly: &mut Option<OnTheFly>, substitute_by: Option<OnTheFlyInner>) {
if let Some(substitute_by) = substitute_by {
if let Some(on_the_fly) = on_the_fly {
*on_the_fly.lock().unwrap() = substitute_by;
}
} else {
on_the_fly.take();
}
}
However, I cannot make something generic over T where T is a trait, it should be a type.
Any ideas?
Bounty
This is solved by #user4815162342. But what if I want to make one OnTheFly object point to the same thing as the other one?
First, you are correct that T cannnot be a trait like GetVideo; traits are not types. However, T can be dyn GetVideo.
Second, your aliases have generic parameters, so they should be reflected as such in the function signature:
pub fn on_the_fly_substitute<T>(on_the_fly: &mut Option<OnTheFly<T>>, substitute_by: Option<OnTheFlyInner<T>>)
^^^ ^^^ ^^^
Third, your alias looks like an attempt to constrain T to be Send + Sync, but aliases cannot define additional bounds. You would instead put them on the function (with ?Sized since you want to allow trait objects):
pub fn on_the_fly_substitute<T: ?Sized>(on_the_fly: &mut Option<OnTheFly<T>>, substitute_by: Option<OnTheFlyInner<T>>)
where
T: ?Sized + Send + Sync
{
...
}
Note: your function body does not require Send and Sync so these bounds should probably not be included.
Fourth, Option<Arc<Mutex<Box<dyn GetVideo>>>> is not thread safe. You'll need to constrain that the trait object is at least Send:
Option<Arc<Mutex<Box<dyn GetVideo + Send>>>>
^^^^^^
Fifth, a complete example is lacking, but you appear to be wanting multiple threads to modify the same video_source. This would likely not compile since you would need multiple threads to keep a &mut _ in order to change it.
If you want shared ownership of a value that might not exist, move the option into the Mutex and adjust your function and aliases accordingly:
video_source: Arc<Mutex<Option<Box<dyn GetVideo>>>>
Sixth, your comment "I'd like this to be a method of OnTheFly" is misguided. Aliases are just aliases, you'd need a method on the aliased Option/Arc type. Keep it as a free function, introduce an extension trait for it, or create it as a wrapper type instead of an alias if you want more fine-grained control.
I'm trying to pass a non-static closure into tokio. Obviously this doesn't work. Is there a way to make sure the lifetimes are appropriate at runtime? Here's what I tried:
Attempt with Arc
In order to not pass the closure directly into tokio, I put it into the struct that manages our timers:
type Delays<'l, K: Eq + Hash + Debug + Copy + Send> = HashMap<K, Box<dyn FnOnce() + 'l + Send>>;
pub struct Timers<'l, K: Eq + Hash + Debug + Clone + Send> {
delays: Arc<Mutex<Delays<'l, K>>>,
}
The impl for that struct lets us easily add and remove timers. My plan was to somehow pass a static closure into tokio, by only moving a Weak reference
to the mutexed hashmap:
// remember handler function
delays.insert(key.clone(), Box::new(func));
// create a weak reference to the delay map to pass into the closure
let weak_handlers = Arc::downgrade(&self.delays);
// task that runs after a delay
let task = Delay::new(Instant::now() + delay)
.map_err(|e| warn!("Tokio timer error: {}", e)) // Map the error type to ()
.and_then(move |_| {
// get the handler from the table, of which we have only a weak ref.
let handler = Weak::upgrade(&weak_handlers)
.ok_or(())? // If the Arc dropped, return an error and thus aborting the future
.lock()
.remove(&key)
.ok_or(())?; // If the handler isn't there anymore, we can abort aswell.
// call the handler
handler();
Ok(())
});
So with the Weak we make sure that we abort, if the hash table was dropped.
It's important to know that the lifetime 'l is the same as that of the Timers struct, but how can I tell the compiler? Also, I think the real problem is that Weak<T>: 'static is not satisfied.
Writing it myself using unsafe
I tried building something similar to Sc to achieve this. First, is Sc going to work here? I read the code and understand it. I can't see any obvious problems - though it was kind of hard to come to the conclusion that the map method is actually safe, because the reference will definitely be dropped at the end of the map and not stored somewhere.
So I tried to adapt Sc for my needs. This is only a rough outline and I know there are some issues with this, but I believe something like this should be possible:
Have a struct Doa<T> that will own T
Doa::ref(&self) -> DoaRef<T> will produce a opaque object that internally contain a *const u8 to the owned object.
DoaRef doesn't contain references with non-static lifetimes and thus can be passed to tokio.
Have impl<T> Drop for Doa<T> that sets that *const u8 to null
So the DoaRef can now check if the value still exists and get a reference to it.
I also tried to make sure that the lifetime of &self in ref must be longer than the lifetimes of references in T, to ensure this works only if Doa really lives longer than the object the pointer points to.
struct Doa<'t, T: 'l> { ... }
pub fn ref(&'s self) -> DoaRef<T> where 't: 'a
But then T is lifetime-contrained and since DoaRef is parameterized over it DoaRef: 'static doesn't hold anymore.
Or is there some crate, or maybe even something in std that can do this?