Lifetime and closures and continuation passing style - rust

Question

I'm playing with Rust closures, trying to get a feeling for them, and one task I set for myself was to implement an in-order traversal with continuation-passing-style. Usually something I can do in about an hour in other languages, but here I've managed to get stuck on how to manage the lifetime of a closure I use in callback.
I've defined a binary tree like this, which is pretty straightforward:
// A node is a tuple of (left, value, right)
struct Node<T>(pub Tree<T>, pub T, pub Tree<T>);
// A tree is an optional pointer to a node
type Tree<T> = Option<Box<Node<T>>>;
and I can do a direct recursive traversal the way I would expect:
// Direct recursion.
// The <T: Copy> is needed because we copy values into
// the result vector. For other usages we might not need it
fn inorder<T: Copy>(t: &Tree<T>, res: &mut Vec<T>) {
// If t is not empty it is Some(node)
if let Some(node) = t {
// Pull out the values of the node.
// In the development branch of Rust there is
// a nicer syntax, but this is how it is now
let Node(left, val, right) = node.as_ref();
inorder(left, res);
res.push(*val);
inorder(right, res);
}
}
To do it with continuations, however, I need to define closures to manage the rest of the traversal and pass those along with a tail-recursion, and my immediate solution looked not much different from my current one:
// Ideally, we wouldn't need to pass the vector along with the closures
// but Rust won't let us hold on to a mutable reference in a closure if
// another closure also has a mutable reference, so we pass the vector
// from continuation to continuation instead.
fn cps_rec<T: Copy>(t: &Tree<T>, res: &mut Vec<T>,
k: Box<dyn FnOnce(&mut Vec<T>)>)
{
match t {
None => { k(res) },
Some(node) => {
let Node(left, val, right) = node.as_ref();
let after_left = |v: &mut Vec<T>| {
// After we have gone left, add v and go right,
// calling k when we are done
v.push(*val);
cps_rec(right, v, k);
};
cps_rec(left, res, Box::new(after_left));
}
}
}
fn cps<T: Copy>(t: &Tree<T>) -> Vec<T> {
let mut res = vec![];
cps_rec(t, &mut res, Box::new(|_| ()));
res
}
Playground
The problem is in the recursion cps_rec(left, res, after_left) where the lifetime of the closure doesn't match the constraint. (I think, I can't say I fully understand exactly what Rust is complaining about, but the problem goes away as soon as I make the closure's body empty, so it is something in the inferred lifetime there that is messing things up). I first suspected that it was the lifetime of val or right that was fighting me, but it doesn't help to put move in front of the closure after dereferencing them
fn cps_rec<T: Copy>(t: &Tree<T>, res: &mut Vec<T>,
k: Box<dyn FnOnce(&mut Vec<T>)>)
{
match t {
None => { k(res) },
Some(node) => {
let Node(left, val, right) = node.as_ref();
let val = *val;
let right = *right;
let after_left = move |v: &mut Vec<T>| {
// After we have gone left, add v and go right,
// calling k when we are done
v.push(val);
cps_rec(&right, v, k);
};
cps_rec(left, res, Box::new(after_left));
}
}
}
Playground
so I can't work out what the problem is, or how to tell Rust that this closure will survive long enough to complete the recursion...
I then tried using a generic type for the continuation, seeing if that would make Rust resolve the issue for me:
fn cps_rec<'a, T: Copy, Cont>(t: &'a Tree<T>, res: &'a mut Vec<T>, k: Cont)
where Cont: FnOnce(&'a mut Vec<T>)
{
match t {
None => { k(res) },
Some(node) => {
let Node(left, val, right) = node.as_ref();
let after_left = |v: &'a mut Vec<T>| {
// After we have gone left, add v and go right,
// calling k when we are done
v.push(*val);
cps_rec(right, v, k);
};
cps_rec(left, res, Box::new(after_left));
}
}
}
Playground
but that breaks the compilation when the type checker has to recurse on the type inference. Maybe not surprising since the type of the closure probably depends on the inferred type of the recursion.
Is there something I can do to fix this, or is the idea dead on arrival and I have to try a completely different approach if I want to make something along those lines?

Answer 1

The lifetime problem is becaue Box<dyn FnOnce(&mut Vec<T>)> is actually, with lifetimes explicitly specified, Box<dyn FnOnce(&mut Vec<T>) + 'static>. That is, it requires the closures to be 'static. A closure is static if everything it captures is 'static. Of course an empty closure fulfills this condition - it captures nothing, so it worked if you emptied its body.
However, your closure captures over non-'static references - val and right. I'll soon explain why copying them doesn't work.
The solution is simple: we don't really need the closure to be 'static; we can be fine even if it won't be - we don't store it or something. To express that to the compiler, we need to change Box<dyn FnOnce(&mut Vec<T>)> to Box<dyn FnOnce(&mut Vec<T>) + '_> (playground). That's it. '_ is the explicitly elided lifetime: it tells the compiler "figure the right lifetime out yourself". Usually the compiler does that automatically, but with dyn Trait he sometimes gives up and just uses 'static. The lifetime is inferred according to the lifetime elision rules; in this case, it introduces a new lifetime, like if we wrote:
fn cps_rec<'elided_lifetime, T: Copy>(
t: &Tree<T>,
res: &mut Vec<T>,
k: Box<dyn FnOnce(&mut Vec<T>) + 'elided_lifetime>,
) { ... }
What confuses here is the compiler's error message. It'd be clearer if you'd followed the compiler's suggestion and put T: 'static (playground):
error: lifetime may not live long enough
--> src/lib.rs:17:32
|
6 | fn cps_rec<T: Copy + 'static>(t: &Tree<T>, res: &mut Vec<T>, k: Box<dyn FnOnce(&mut Vec<T>)>) {
| - let's call the lifetime of this reference `'1`
...
17 | cps_rec(left, res, Box::new(after_left));
| ^^^^^^^^^^^^^^^^^^^^ cast requires that `'1` must outlive `'static`
This is what I said. 'static is required because of the closure's implied 'static bound.
But the compiler is has something more important to tell you (or rather, yell at you). It thinks: "given that the closure requires 'static, those references are obviously 'static. This user isn't lying to me, he's a good citizen!" Only later it'll realize it was wrong, but it will never get to this "later" because it will stop compilation due to errors earlier. But as of now it thinks, "given that those references are obviously 'static, they're &'static T (for val, or &'static Option<Box<T>> for right). But for &'static T to be valid, T: 'static must hold (imagine a reference like &'long &'short i32: you'll be able to access it for 'long, but as long as 'short is over it's invalid, and you will use a dangling reference!). Can I prove that T: 'static always hold? Nope! Well, should emit an error." (You can fill a bug report at https://github.com/rust-lang/rust/issues, the Rust developers will appreciate it even though I'm not sure how much they can do to improve the situation).
Why copying them didn't help? First, the way the compiler thought about it hasn't changed, and so the same error is still emitted. Second, even if we could solve it (for example by specifying T: 'static), there is another error (playground):
error[E0507]: cannot move out of `*right` which is behind a shared reference
--> src/lib.rs:12:34
|
12 | let right: Tree<T> = *right;
| ^^^^^^ move occurs because `*right` has type `Option<Box<Node<T>>>`, which does not implement the `Copy` trait
|
help: consider borrowing the `Option`'s content
|
12 | let right: Tree<T> = *right.as_ref();
| +++++++++
help: consider borrowing here
|
12 | let right: Tree<T> = &*right;
| ~~~~~~~
Hopefully this one is self-explanatory.
About the generic version, your assumption is indeed correct: no language can do recursive CPS with monomorphization, they all use dynamic dispatch (which is equivalent to Box<dyn FnOnce()> (or some GCed version), or to &mut dyn FnMut() or something similar). We will have to instantiate it ad infinitum.