I'm working with a friend to define a safe public API for lifetimes of a "scoped" garbage collector. The lifetimes are either overly constrained and correct code does not compile or the lifetimes are too loose and they may allow invalid behavior. After trying multiple approaches, we are still stuck getting a correct API. This is especially frustrating because Rust's lifetimes can help avoid bugs in this situation but right now it just looks stubborn.
Scoped garbage collection
I am implementing an ActionScript interpreter and need a garbage collector. I studied rust-gc but it did not suit my needs. The main reason is that it requires the garbage collected values to have a static lifetime because the GC state is a thread-local static variable. I need to get garbage-collected bindings to a dynamically created host object. The other reason to avoid globals is that it is easier for me to handle multiple independent garbage-collected scopes, control their memory limits or serialize them.
A scoped garbage collector is similar to a typed-arena. You can use it to allocate values and they are all freed once the garbage collector is dropped. The difference is that you can also trigger garbage collection during its lifetime and it will clean-up the unreachable data (and is not limited to a single type).
I have a working implementation implemented (mark & sweep GC with scopes), but the interface is not yet safe to use.
Here is a usage example of what I want:
pub struct RefNamedObject<'a> {
pub name: &'a str,
pub other: Option<Gc<'a, GcRefCell<NamedObject<'a>>>>,
}
fn main() {
// Initialize host settings: in our case the host object will be replaced by a string
// In this case it lives for the duration of `main`
let host = String::from("HostConfig");
{
// Create the garbage-collected scope (similar usage to `TypedArena`)
let gc_scope = GcScope::new();
// Allocate a garbage-collected string: returns a smart pointer `Gc` for this data
let a: Gc<String> = gc_scope.alloc(String::from("a")).unwrap();
{
let b = gc_scope.alloc(String::from("b")).unwrap();
}
// Manually trigger garbage collection: will free b's memory
gc_scope.collect_garbage();
// Allocate data and get a Gc pointer, data references `host`
let host_binding: Gc<RefNamed> = gc_scope
.alloc(RefNamedObject {
name: &host,
other: None,
})
.unwrap();
// At the end of this block, gc_scope is dropped with all its
// remaining values (`a` and `host_bindings`)
}
}
Lifetime properties
The basic intuition is that Gc can only contain data that lives as long (or longer) than the corresponding GcScope. Gc is similar to Rc but supports cycles. You need to use Gc<GcRefCell<T>> to mutate values (similar to Rc<RefCell<T>>).
Here are the properties that must be satisfied by the lifetimes of my API:
Gc cannot live longer than its GcScope
The following code must fail because a outlives gc_scope:
let a: Gc<String>;
{
let gc_scope = GcScope::new();
a = gc_scope.alloc(String::from("a")).unwrap();
}
// This must fail: the gc_scope was dropped with all its values
println("{}", *a); // Invalid
Gc cannot contain data that lives shorter than its GcScope
The following code must fail because msg does not live as long (or longer) as gc_scope
let gc_scope = GcScope::new();
let a: Gc<&string>;
{
let msg = String::from("msg");
a = gc.alloc(&msg).unwrap();
}
It must be possible to allocate multiple Gc (no exclusion on gc_scope)
The following code must compile
let gc_scope = GcScope::new();
let a = gc_scope.alloc(String::from("a"));
let b = gc_scope.alloc(String::from("b"));
It must be possible to allocate values containing references with lifetimes longer than gc_scope
The following code must compile
let msg = String::from("msg");
let gc_scope = GcScope::new();
let a: Gc<&str> = gc_scope.alloc(&msg).unwrap();
It must be possible to create cycles of Gc pointers (that's the whole point)
Similarly to the Rc<Refcell<T>> pattern, you can use Gc<GcRefCell<T>> to mutate values and create cycles:
// The lifetimes correspond to my best solution so far, they can change
struct CircularObj<'a> {
pub other: Option<Gc<'a, GcRefCell<CircularObj<'a>>>>,
}
let gc_scope = GcScope::new();
let n1 = gc_scope.alloc(GcRefCell::new(CircularObj { other: None }));
let n2 = gc_scope.alloc(GcRefCell::new(CircularObj {
other: Some(Gc::clone(&n1)),
}));
n1.borrow_mut().other = Some(Gc::clone(&n2));
Solutions so far
Automatic lifetime / lifetime tag
Implemented on the auto-lifetime branch
This solution is inspired by neon's handles.
This lets any valid code compile (and allowed me to test my implementation) but is too loose and allows invalid code. It allows Gc to outlive the gc_scope that created it. (Violates the first property)
The idea here is that I add a single lifetime 'gc to all my structs. The idea is that this lifetime represents "how long gc_scope lives".
// A smart pointer for `T` valid during `'gc`
pub struct Gc<'gc, T: Trace + 'gc> {
pub ptr: NonNull<GcBox<T>>,
pub phantom: PhantomData<&'gc T>,
pub rooted: Cell<bool>,
}
I call it automatic lifetimes because the methods never mix these struct lifetimes with the lifetime of the references they receive.
Here is the impl for gc_scope.alloc:
impl<'gc> GcScope<'gc> {
// ...
pub fn alloc<T: Trace + 'gc>(&self, value: T) -> Result<Gc<'gc, T>, GcAllocErr> {
// ...
}
}
Inner/outer lifetimes
Implemented on the inner-outer branch
This implementation tries to fix the previous issue by relating Gc to the lifetime of GcScope. It is overly constrained and prevents the creation of cycles. This violates the last property.
To constrain Gc relative to its GcScope, I introduce two lifetimes: 'inner is the lifetime of GcScope and the result is Gc<'inner, T>. 'outer represents a lifetime longer than 'inner and is used for the allocated value.
Here is the alloc signature:
impl<'outer> GcScope<'outer> {
// ...
pub fn alloc<'inner, T: Trace + 'outer>(
&'inner self,
value: T,
) -> Result<Gc<'inner, T>, GcAllocErr> {
// ...
}
// ...
}
Closure (context management)
Implemented on the with branch
Another idea was to not let the user create a GcScope manually with GcScope::new but instead expose a function GcScope::with(executor) providing a reference to the gc_scope. The closure executor corresponds to the gc_scope. So far, it either prevents the use of external references or allows to leak data to external Gc variables (first and fourth properties).
Here is the alloc signature:
impl<'gc> GcScope<'gc> {
// ...
pub fn alloc<T: Trace + 'gc>(&self, value: T) -> Result<Gc<'gc, T>, GcAllocErr> {
// ...
}
}
Here is a usage example showing the violation of the first property:
let message = GcScope::with(|scope| {
scope
.alloc(NamedObject {
name: String::from("Hello, World!"),
})
.unwrap()
});
println!("{}", message.name);
What I'd like
From what I understand, the alloc signature I'd like is:
impl<'gc> GcScope<'gc> {
pub fn alloc<T: Trace + 'gc>(&'gc self, value: T) -> Result<Gc<'gc, T>, GcAllocErr> {
// ...
}
}
Where everything lives as long or longer than self (the gc_scope). But this blows up with the most simple tests:
fn test_gc() {
let scope: GcScope = GcScope::new();
scope.alloc(String::from("Hello, World!")).unwrap();
}
causes
error[E0597]: `scope` does not live long enough
--> src/test.rs:50:3
|
50 | scope.alloc(String::from("Hello, World!")).unwrap();
| ^^^^^ borrowed value does not live long enough
51 | }
| - `scope` dropped here while still borrowed
|
= note: values in a scope are dropped in the opposite order they are created
I have no idea what happens here. Playground link
Edit: As explained to me on IRC, this is because I implement Drop which requires &mut self, but the scope is already borrowed in read-only mode.
Overview
Here is a quick overview of the main components of my library.
GcScope contains a RefCell to its mutable state. This was introduced to not require &mut self for alloc because it "locked" the gc_scope and violated property 3: allocate multiple values.
This mutable state is GcState. It keeps track of all the allocated values. The values are stored as a forward-only linked list of GcBox. This GcBox is heap-allocated and contains the actual value with some metadata (how many active Gc pointers have it as their root and a boolean flag used to check if the value is reachable from the root (see rust-gc). The value here must outlive its gc_scope so GcBox uses a lifetime, and in turn GcState must then use a lifetime as well as GcScope: this is always the same lifetime meaning "longer than gc_scope". The fact that GcScope has a RefCell (interior mutability) and lifetime is maybe the reason why I can't get my lifetimes to work (it causes invariance?).
Gc is a smart pointer to some gc_scope-allocated data. You can only get it through gc_scope.alloc or by cloning it.
GcRefCell is most likely fine, it's just a RefCell wrapper adding metadata and behavior to properly support borrows.
Flexibility
I'm fine with the following requirements to get a solution:
unsafe code
nightly features
API changes (see for example my with approach). What matters is that I can create a temporary zone where I can manipulate garbage-collected values and that they are all dropped after this. These garbage-collected values need to be able to access longer-lived (but not static) variables outside of the scope.
The repository has a few tests in scoped-gc/src/lib.rs (compile-fail) as scoped-gc/src/test.rs.
I found a solution, I'll post it once redacted.
This is one of the hardest problems I had with lifetimes with Rust so far, but I managed to find a solution. Thank you to panicbit and mbrubeck for having helped me on IRC.
What helped me to move forward was the explanation of the error I posted at the end of my question:
error[E0597]: `scope` does not live long enough
--> src/test.rs:50:3
|
50 | scope.alloc(String::from("Hello, World!")).unwrap();
| ^^^^^ borrowed value does not live long enough
51 | }
| - `scope` dropped here while still borrowed
|
= note: values in a scope are dropped in the opposite order they are created
I did not understand this error because it wasn't clear to me why scope was borrowed, for how long, or why it needs to no longer be borrowed at the end of the scope.
The reason is that during the allocation of the value, the scope is immutably borrowed for the duration of the allocated value. The issue now is that the scope contains a state object that implements "Drop": custom implementations of drop use &mut self -> it is not possible to get a mutable borrow for the drop while the value is already immutably borrowed.
Understanding that drop requires &mut self and that it is incompatible with immutable borrows unlocked the situation.
It turns out that the inner-outer approach described in the question above had the correct lifetimes with alloc:
impl<'outer> GcScope<'outer> {
// ...
pub fn alloc<'inner, T: Trace + 'outer>(
&'inner self,
value: T,
) -> Result<Gc<'inner, T>, GcAllocErr> {
// ...
}
// ...
}
The returned Gc lives as long as GcScope and the allocated values must live longer than the current GcScope. As mentioned in the question, the issue with this solution is that it did not support circular values.
The circular values failed to work not because of the lifetimes of alloc but due to the custom drop. Removing drop allowed all the tests to pass (but leaked memory).
The explanation is quite interesting:
The lifetime of alloc expresses the properties of the allocated values. The allocated values cannot outlive their GcScope but their content must live as long or longer than GcScope. When creating a cycle, the value is subject to both of these constraints: it is allocated so must live as long or shorter than GcScope but also referenced by another allocated value so it must live as long or longer than GcScope. Because of this there is only one solution: the allocated value must live exactly as long as its scope.
It means that the lifetime of GcScope and its allocated values is exactly the same. When two lifetimes are the same, Rust does not guarantee the order of the drops. The reason why this happens is that the drop implementations could try to access each other and since there's no ordering it would be unsafe (the value might already have been freed).
This is explained in the Drop Check chapter of the Rustonomicon.
In our case, the drop implementation of the state of the garbage collected does not dereference the allocated values (quite the opposite, it frees their memory) so the Rust compiler is overly cautious by preventing us from implementing drop.
Fortunately, the Nomicon also explains how to work around these check of values with the same lifetimes. The solution is to use the may_dangle attribute on the lifetime parameter of the Drop implementation.
This is as unstable attribute that requires to enable the generic_param_attrs and dropck_eyepatch features.
Concretely, my drop implementation became:
unsafe impl<'gc> Drop for GcState<'gc> {
fn drop(&mut self) {
// Free all the values allocated in this scope
// Might require changes to make sure there's no use after free
}
}
And I added the following lines to lib.rs:
#![feature(generic_param_attrs)]
#![feature(dropck_eyepatch)]
You can read more about these features:
generic_param_attrs
may_dangle
I updated my library scoped-gc with the fix for this issue if you want to take a closer look at it.
Related
In How does Rust prevent data races when the owner of a value can read it while another thread changes it?, I understand I need &mut self, when we want to mutate an object, even when the method is called with an owned value.
But how about primitive values, like i32? I ran this code:
fn change_aaa(bbb: &mut i32) {
*bbb = 3;
}
fn main() {
let mut aaa: i32 = 1;
change_aaa(&mut aaa); // somehow run this asynchronously
aaa = 2; // ... and will have data race here
}
My questions are:
Is this safe in a non concurrent situation?
According to The Rust Programming Language, if we think of the owned value as a pointer, it is not safe according the following rules, however, it compiles.
Two or more pointers access the same data at the same time.
At least one of the pointers is being used to write to the data.
There’s no mechanism being used to synchronize access to the data.
Is this safe in a concurrent situation?
I tried, but I find it hard to put change_aaa(&mut aaa) into a thread, according to Why can't std::thread::spawn accept arguments in Rust? and How does Rust prevent data races when the owner of a value can read it while another thread changes it?. However, is it designed to be hard or impossible to do this, or just because I am unfamiliar with Rust?
The signature of change_aaa doesn't allow it to move the reference into another thread. For example, you might imagine a change_aaa() implemented like this:
fn change_aaa(bbb: &mut i32) {
std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_secs(1));
*bbb = 100; // ha ha ha - kaboom!
});
}
But the above doesn't compile. This is because, after desugaring the lifetime elision, the full signature of change_aaa() is:
fn change_aaa<'a>(bbb: &'a mut i32)
The lifetime annotation means that change_aaa must support references of any lifetime 'a chosen by the caller, even a very short one, such as one that invalidates the reference as soon as change_aaa() returns. And this is exactly how change_aaa() is called from main(), which can be desugared to:
let mut aaa: i32 = 1;
{
let aaa_ref = &mut aaa;
change_aaa(aaa_ref);
// aaa_ref goes out of scope here, and we're free to mutate
// aaa as we please
}
aaa = 2; // ... and will have data race here
So the lifetime of the reference is short, and ends just before the assignment to aaa. On the other hand, thread::spawn() requires a function bound with 'static lifetime. That means that the closure passed to thread::spawn() must either only contain owned data, or references to 'static data (data guaranteed to last until the end of the program). Since change_aaa() accepts bbb with with lifetime shorter than 'static, it cannot pass bbb to thread::spawn().
To get a grip on this you can try to come up with imaginative ways to write change_aaa() so that it writes to *bbb in a thread. If you succeed in doing so, you will have found a bug in rustc. In other words:
However, is it designed to be hard or impossible to do this, or just because I am unfamiliar with Rust?
It is designed to be impossible to do this, except through types that are explicitly designed to make it safe (e.g. Arc to prolong the lifetime, and Mutex to make writes data-race-safe).
Is this safe in a non concurrent situation? According to this post, if we think owned value if self as a pointer, it is not safe according the following rules, however, it compiles.
Two or more pointers access the same data at the same time.
At least one of the pointers is being used to write to the data.
There’s no mechanism being used to synchronize access to the data.
It is safe according to those rules: there is one pointer accessing data at line 2 (the pointer passed to change_aaa), then that pointer is deleted and another pointer is used to update the local.
Is this safe in a concurrent situation? I tried, but I find it hard to put change_aaa(&mut aaa) into a thread, according to post and post. However, is it designed to be hard or impossible to do this, or just because I am unfamiliar with Rust?
While it is possible to put change_aaa(&mut aaa) in a separate thread using scoped threads, the corresponding lifetimes will ensure the compiler rejects any code trying to modify aaa while that thread runs. You will essentially have this failure:
fn main(){
let mut aaa: i32 = 1;
let r = &mut aaa;
aaa = 2;
println!("{}", r);
}
error[E0506]: cannot assign to `aaa` because it is borrowed
--> src/main.rs:10:5
|
9 | let r = &mut aaa;
| -------- borrow of `aaa` occurs here
10 | aaa = 2;
| ^^^^^^^ assignment to borrowed `aaa` occurs here
11 | println!("{}", r);
| - borrow later used here
I have the following code which you can also see in the playground:
struct Node<'a> {
parent: Option<&'a Node<'a>>,
name: &'a str,
}
fn looper(nodes: &mut Vec<Node>) {
for i in 0..nodes.len() {
let n = &nodes[i];
let next_node = Node {
parent: Some(n),
name: "please compile",
};
nodes.push(next_node);
}
}
It's supposed to take a Vec of Nodes and append Nodes to it that reference the Nodes already in the Vec. Unfortunately I get this error:
error[E0623]: lifetime mismatch
--> src/main.rs:13:20
|
6 | fn looper(nodes: &mut Vec<Node>) {
| -------------- these two types are declared with different lifetimes...
...
13 | nodes.push(next_node);
| ^^^^^^^^^ ...but data from `nodes` flows into `nodes` here
How can I make this compile? An explanation of how to think about lifetimes in this context would be useful. I understand this may be an issue due to variance but am not sure how to rephrase my problem to give me the functionality I would like.
A general rule: when some value's type, or trait it implements, has a lifetime parameter, that lifetime is always longer than the life of the value itself — the value has a guarantee that the references it contains won't go invalid before they are dropped.
In fact, in your example we can see that if this were not the case, the lifetime checking would be unsound; you're adding values to nodes, but nothing would prevent looper from instead removing values from nodes, which would then invalidate any parent references that referred to those values.
The only practical way to build a tree or linked list of references (without special-purpose unsafe code that has a particular plan to keep things sound) is to write a recursive function; each function call frame may refer to nodes constructed by its caller (and its caller's caller and so on). This is accepted by the borrow checker because each frame has its own new, shorter lifetime for the references (a reference can always be taken as having a shorter lifetime than it started with).
To illustrate the necessity of Rc<T>, the Book presents the following snippet (spoiler: it won't compile) to show that we cannot enable multiple ownership without Rc<T>.
enum List {
Cons(i32, Box<List>),
Nil,
}
use crate::List::{Cons, Nil};
fn main() {
let a = Cons(5, Box::new(Cons(10, Box::new(Nil))));
let b = Cons(3, Box::new(a));
let c = Cons(4, Box::new(a));
}
It then claims (emphasis mine)
We could change the definition of Cons to hold references instead, but then we would have to specify lifetime parameters. By specifying lifetime parameters, we would be specifying that every element in the list will live at least as long as the entire list. The borrow checker wouldn’t let us compile let a = Cons(10, &Nil); for example, because the temporary Nil value would be dropped before a could take a reference to it.
Well, not quite. The following snippet compiles under rustc 1.52.1
enum List<'a> {
Cons(i32, &'a List<'a>),
Nil,
}
use crate::List::{Cons, Nil};
fn main() {
let a = Cons(5, &Cons(10, &Nil));
let b = Cons(3, &a);
let c = Cons(4, &a);
}
Note that by taking a reference, we no longer need a Box<T> indirection to hold the nested List. Furthermore, I can point both b and c to a, which gives a multiple conceptual owners (which are actually borrowers).
Question: why do we need Rc<T> when immutable references can do the job?
With "ordinary" borrows you can very roughly think of a statically proven order-by-relationship, where the compiler needs to prove that the owner of something always comes to life before any borrows and always dies after all borrows died (a owns String, it comes to life before b which borrows a, then b dies, then a dies; valid). For a lot of use-cases, this can be done, which is Rust's insight to make the borrow-system practical.
There are cases where this can't be done statically. In the example you've given, you're sort of cheating, because all borrows have a 'static-lifetime; and 'static items can be "ordered" before or after anything out to infinity because of that - so there actually is no constraint in the first place. The example becomes much more complex when you take different lifetimes (many List<'a>, List<'b>, etc.) into account. This issue will become apparent when you try to pass values into functions and those functions try to add items. This is because values created inside functions will die after leaving their scope (i.e. when the enclosing function returns), so we cannot keep a reference to them afterwards, or there will be dangling references.
Rc comes in when one can't prove statically who is the original owner, whose lifetime starts before any other and ends after any other(!). A classic example is a graph structure derived from user input, where multiple nodes can refer to one other node. They need to form a "born after, dies before" relationship with the node they are referencing at runtime, to guarantee that they never reference invalid data. The Rc is a very simple solution to that because a simple counter can represent these relationships. As long as the counter is not zero, some "born after, dies before" relationship is still active. The key insight here is that it does not matter in which order the nodes are created and die because any order is valid. Only the points on either end - where the counter gets to 0 - are actually important, any increase or decrease in between is the same (0=+1+1+1-1-1-1=0 is the same as 0=+1+1-1+1-1-1=0) The Rc is destroyed when the counter reaches zero. In the graph example this is when a node is not being referred to any longer. This tells the owner of that Rc (the last node referring) "Oh, it turns out I am the owner of the underlying node - nobody knew! - and I get to destroy it".
Even single-threaded, there are still times the destruction order is determined dynamically, whereas for the borrow checker to work, there must be a determined lifetime tree (stack).
fn run() {
let writer = Rc::new(std::io::sink());
let mut counters = vec![
(7, Rc::clone(&writer)),
(7, writer),
];
while !counters.is_empty() {
let idx = read_counter_index();
counters[idx].0 -= 1;
if counters[idx].0 == 0 {
counters.remove(idx);
}
}
}
fn read_counter_index() -> usize {
unimplemented!()
}
As you can see in this example, the order of destruction is determined by user input.
Another reason to use smart pointers is simplicity. The borrow checker does incur some code complexity. For example, using smart pointer, you are able to maneuver around the self-referential struct problem with a tiny overhead.
struct SelfRefButDynamic {
a: Rc<u32>,
b: Rc<u32>,
}
impl SelfRefButDynamic {
pub fn new() -> Self {
let a = Rc::new(0);
let b = Rc::clone(&a);
Self { a, b }
}
}
This is not possible with static (compile-time) references:
struct WontDo {
a: u32,
b: &u32,
}
I have read the concept of Rust lifetimes from many different resources and I'm still not able to figure out the intuition behind it. Consider this code:
#[derive(Debug)]
struct Example<'a> {
name: &'a str,
other_name: String,
}
fn main() {
let a: &'static str = "hello world";
println!("{}", a);
let b: Example = Example {
name: "Hello",
other_name: "World".into(),
};
println!("{:?}", b);
}
In my understanding, all things in Rust have a lifetime attached to them. In the line let a: &'static str = "hello world"; the variable a is kept alive till the end of the program and the'static is optional that is let a: &str = "hello world"; is also valid. My confusion is when we add custom lifetime to others such as struct Example.
struct Example<'a> {
name: &'a str,
other_name: String,
}
Why do we need to attach a lifetime 'a to it? What is a simplified and intuitive reasoning why we use lifetimes in Rust?
If you come from a background in garbage-collected languages (I see you're familiar with Python), the whole notion of lifetimes can feel very alien indeed. Even quite high level memory management concepts, such as the difference between stack and heap or when allocations and deallocations occur, can be difficult to grasp: because these are details that garbage collection hides from you (at a cost).
On the other hand, if you come from a language where you've had to manage memory yourself (like C++, for example), these are concepts with which you'll already be quite comfortable. My understanding is that Rust was primarily designed to compete in this "systems language" space whilst at the same time introducing strategies (like the borrow checker) to help avoid most memory management errors. Hence much of the documentation has been written with this audience in mind.
Before you can really understand "lifetimes", you should get to grips with the stack and the heap. Lifetime issues mostly arise with things that are (or might be) on the heap. Rust's ownership model is ultimately about associating each heap allocation with a specific stack item (perhaps via other intermediate heap items), such that when an item is popped from the stack all its associated heap allocations are freed.
Then ask yourself, whenever you have a reference to (i.e. the memory address of) something: will that something still be at the expected location in memory when the reference is used? One reason it might not be is because it was on the heap and its owning item has been popped from the stack, causing it to be dropped and its memory allocation freed; another might be because it has relocated to some other location in memory (for example, it's a Vec that outgrew the space available in its previous allocation). Even mere mutations of the data can violate expectations about what’s held there, so they’re not allowed to happen from under you either.
The most important thing to grasp is that Rust's lifetimes have no impact whatsoever on this question: that is, they never affect how long something remains at a memory location—they are merely assertions that we make about the answers to that question, and the code won't compile if those assertions cannot be verified.
So, on to your example:
struct Example<'a>{
name: &'a str,
other_name: String,
}
Let's imagine we create an instance of this struct:
let foo = Example { name: "eggyal", other_name: String::from("Eka") };
Now suppose this foo, a stack item, is at address 0x1000. Delving into the implementation details for a typical 64-bit system, our memory might look something like this:
...
0x1000 foo.name#ptr = 0xabcd
0x1008 foo.name#len = 6
0x1010 foo.other_name#ptr = 0x5678
0x1018 foo.other_name#cap = 3
0x1020 foo.other_name#len = 3
...
0x5678 'E'
0x5679 'k'
0x567a 'a'
...
0xabcd 'e'
0xabce 'g'
0xabcf 'g'
0xabd0 'y'
0xabd1 'a'
0xabd2 'l'
...
Notice that, in foo, name is comprised of just a pointer and a length; whereas other_name additionally has a capacity (which, in this example, is the same as its length). So what's the difference between &str and String? It's all about where responsibility for managing the associated memory allocation lies.
Since String is an owned, heap-allocated string, foo.other_name "owns" (is responsible for) its associated memory allocation—and hence, when foo is dropped (e.g. because it is popped from the stack), Rust will ensure that those three bytes at address 0x5678 are freed and returned to the allocator (which ultimately happens through an implementation of std::ops::Drop). Owning the allocation also means that String can safely mutate the memory, relocate the value to another address, etc (provided that it's not currently on loan somewhere else).
By contrast, the memory allocation at 0xabcd is not "owned" by foo.name—we say that it's "borrowing" the allocation instead—but if foo.name does not manage the allocation, how can it be sure that it contains what it's supposed to? Well, we programmers promise Rust that we will keep the contents valid for the duration of the borrow (which we give a name, in your case 'a: &'a str means that the memory holding the str is being borrowed for lifetime 'a), and the borrow checker ensures that we keep our promise.
But how long are we promising that lifetime 'a will be? Well, it’ll be different for every instance of Example: the period of time for which we promise "eggyal" will be at 0xabcd for foo will in all likelihood be completely different to the period of time that we promise the name value of some other instance will be at its address. So our lifetime 'a is a parameter of Example: this is why it’s declared as Example<'a>.
Fortunately, we don’t ever need to explicitly define how long our lifetimes will actually last as the compiler knows everything's actual lifetime and merely needs to check that our assertions hold: in our example, the compiler determines that the provided value, "eggyal" is a string literal and therefore of type &'static str, so will be at its address 0xabcd for the 'static lifetime; thus in the case of foo, 'a is allowed to be "any lifetime up to and including 'static"; in #Aloso's answer you can see an example with a different lifetime. Then wherever foo is used, any lifetime assertions at that usage site can be checked and verified against this determined bound.
It takes some getting used to, but I find picturing the memory layout like this and asking myself "when does the memory allocation get freed?" helps me to understand the lifetimes in my code (sometimes I need to think about when the value might be relocated or mutated instead, but merely considering deallocation is often enough—and is usually a little bit easier to grasp).
In this line let a:&'static str = "hello world"; the variable a is kept alive till the end of the program
No, that's not what happens. a is a reference, i.e. it refers to some string data. That string data is 'static, which means that it is alive util the end of the program. a however doesn't need to be alive util the end of the program (it happens to be in this case, because it is the first value declared in the main function, but that's just coincidence).
When a struct has a lifetime, that usually means that it borrows another value, and it can only be used while that value is alive. For example:
struct Example<'a> {
name: &'a str,
other_name: String,
}
fn main() {
let s: String = "hello world".to_string();
let example = Example {
name: &s[..],
other_name: "".to_string(),
};
drop(s); // the lifetime of s ends here
// however, example borrows s, therefore its lifetime
// is tied to s. This means that example can't be used
// after s was dropped. Therefore, the following line
// will trigger a compiler error:
println!("{}", example.name);
}
The reasoning in the actual error message is slightly different, but I think it's still easy to understand:
error[E0505]: cannot move out of `s` because it is borrowed
--> src/main.rs:12:10
|
9 | name: &s[..],
| - borrow of `s` occurs here
...
12 | drop(s); // the lifetime of s ends here
| ^ move out of `s` occurs here
...
18 | println!("{}", example.name);
| ------------ borrow later used here
The error message points out that example, which borrows s, is used after s was dropped. This is forbidden because example has a lifetime that can't outlive s.
I hope I gave you a better understanding how this works. I also recommend you to read this and this chapter of the Rust book.
Writing a simple interpreter has lead me to this battle with the borrow checker.
#[derive(Clone, Debug)]
struct Context<'a> {
display_name: &'a str,
parent: Option<Box<Context<'a>>>,
parent_entry_pos: Position<'a>,
}
// --snip--
#[derive(Copy, Clone, Debug)]
pub enum BASICVal<'a> {
Float(f64, Position<'a>, Position<'a>, &'a Context<'a>),
Int(i64, Position<'a>, Position<'a>, &'a Context<'a>),
Nothing(Position<'a>, Position<'a>, &'a Context<'a>),
}
// --snip--
pub fn run<'a>(text: &'a String, filename: &'a String) -> Result<(Context<'a>, BASICVal<'a>), BASICError<'a>> {
// generate tokens
let mut lexer = Lexer::new(text, filename);
let tokens = lexer.make_tokens()?;
// parse program to AST
let mut parser = Parser::new(tokens);
let ast = parser.parse();
// run the program
let context: Context<'static> = Context {
display_name: "<program>",
parent: None,
parent_entry_pos: Position::default(),
};
Ok((context, interpreter_visit(&ast?, &context)?))
}
The error is "cannot return value referencing local variable `context`" and (secondary) the "borrow of moved value: `context`":
error[E0515]: cannot return value referencing local variable `context`
--> src\basic.rs:732:2
|
732 | Ok((context, interpreter_visit(&ast?, &context)?))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^--------^^^^
| | |
| | `context` is borrowed here
| returns a value referencing data owned by the current function
error[E0382]: borrow of moved value: `context`
--> src\basic.rs:732:40
|
727 | let context: Context<'static> = Context {
| ------- move occurs because `context` has type `basic::Context<'_>`, which does not implement the `Copy` trait
...
732 | Ok((context, interpreter_visit(&ast?, &context)?))
| ------- value moved here ^^^^^^^^ value borrowed here after move
As far as I understand it: The context references several lifetime-dependent structs. The values of these structs are static in this case, as I can see by explicitly setting the lifetime parameter to 'static and the compiler not complaining. The interpreter_visit function needs to borrow the context because it gets passed to several independent functions, including itself recursively. In addition, the interpreter_visit returns BASICVals that reference the context themselves. For this reason, the context needs to outlive the run return. I try to achieve that by passing the context itself as part of the return value, thereby giving the caller control over its life. But now, I move the context to the return value before actually using it? This makes no sense. I should be able to reference one part of the return value in another part of the return value because both values make it out of the function "alive". I have tried:
boxing the context, thereby forcing it off the stack onto the heap, but that seems to only complicate things.
switching the order of the tuple, but that doesn't help.
storing interpreter_visit's result in an intermediate variable, which as expected doesn't help.
cloning the interpreter_visit result or the context itself
The issue may lie with the result and the error. The error doesn't reference a context but giving it a separate lifetime in interpreter_visit breaks the entire careful balance I have been able to achieve until now.
Answering this so that people don't have to read the comment thread.
This is a problem apparently not solvable by Rust's borrow checker. The borrow checker cannot understand that a Box of context will live on the heap and therefore last longer than the function return, therefore being "legally" referencable by the return value of interpreter_visit which itself escapes the function. The solution in this case is to circumvent borrow checking via unsafe, namely a raw pointer. Like this:
let context = Box::new(Context {
display_name: "<program>",
parent: None,
parent_entry_pos: Position::default(),
});
// Obtain a pointer to a location on the heap
let context_ptr: *const Context = &*context;
// outsmart the borrow checker
let result = interpreter_visit(&ast?, unsafe { &*context_ptr })?;
// The original box is passed back, so it is destroyed safely.
// Because the result lives as long as the context as required by the lifetime,
// we cannot get a pointer to uninitialized memory through the value and its context.
Ok((context, result))
I store a raw pointer to the context in context_ptr. The borrowed value passed to interpreter_visit is then piped through a (completely memory-safe) raw pointer dereference and borrow. This will (for some reason, only the Rust gods know) disable the borrow check, so the context data given to interpreter_visit is considered to have a legal lifetime. As I am however still passing back the very safe Box around the context data, I can avoid creating memory leaks by leaving the context with no owner. It might be possible now to pass around the interpreter_visit return value with having the context destroyed, but because both values are printed and discarded immediately, I see no issues arising from this in the future.
If you have a deeper understanding of Rust's borrow checker and would consider this a fixable edge case that doesn't have more "safe" solutions I couldn't come up with, please do comment and I will report this to the Rust team. I'm however not that certain especially because my experience with and knowledge of Rust is limited.