I have some code which looks like this (greatly simplified version). A function takes two function arguments of type LoadClient and CheckApproval and returns either an error or a string.
pub struct Client {
pub id: String,
}
pub enum MyErr {
RequiresApproval(Client, String),
LoadFailed,
}
pub fn authorize<LoadClient, CheckApproval>(load_client: LoadClient, check_approval: CheckApproval) -> Result<String, MyErr>
where
LoadClient: FnOnce(String) -> Result<Client, String>,
CheckApproval: for<'a> FnOnce(&'a Client, &str) -> Result<&'a str, ()>,
{
let client = load_client("hello".to_string()).map_err(|_| MyErr::LoadFailed)?;
let permission = "something";
// This doesn't compile
// let authorized = check_approval(&client, permission).map_err(|_| MyErr::RequiresApproval(client, permission.to_string()))?;
// Ok(authorized.to_string())
// This version does
match check_approval(&client, permission) {
Err(_) => Err(MyErr::RequiresApproval(client, permission.to_string())),
Ok(authorized) => Ok(authorized.to_string()),
}
}
I'd like to used ? with the check_approval call (as the commented out code shows) for simpler code and to avoid the extra nesting - the Ok branch in the final match is actually a much longer block.
Unfortunately that doesn't compile:
error[E0505]: cannot move out of `client` because it is borrowed
--> src/lib.rs:19:66
|
19 | let authorized = check_approval(&client, permission).map_err(|_| MyErr::RequiresApproval(client, permission.to_string()))?;
| ------- ------- ^^^ ------ move occurs due to use in closure
| | | |
| | | move out of `client` occurs here
| | borrow later used by call
| borrow of `client` occurs here
These seem similar (to my untrained eye). Hasn't the borrowed reference to client been returned by the time map_err is called?
My main question: Is there a way to get round this and write the code without using match?
rust playground link.
While your two versions of the code are semantically equivalent, they are actually quite different for the compiler.
The failing one calls Result::map_err() with a closure that captures the value of client. That is, client is moved into the closure, but it is borrowed when calling check_approval(). And here lies the error, a borrowed value cannot be moved.
You may think that this borrow should finish when the function returns, but that is not the case because of its return type Result<&'a str, ()>, being 'a precisely the lifetime of that borrow. The borrow of client is extended for as long as this 'a exists. And that is why your second version works: when you match your Result, the 'a does not extend to the Err(()) branch, only to the Ok(&'a str) so the Err(()) is able to move client freely.
Is there a way to get round this and write the code without using match?
Well, you are calling authorized.to_string() in the returned &'a str and converting it to an owned String. So, if you can change your CheckApproval constraint to:
CheckApproval: FnOnce(&Client, &str) -> Result<String, ()>,
the problem just goes away.
If you cannot change that, another option is to do the to_string() before moving the client into the closure, finishing the borrow before it can do harm:
let authorized = check_approval(&client, permission)
.map(|a| a.to_string())
.map_err(|_| MyErr::RequiresApproval(client, permission.to_string()))?;
Ok(authorized)
Related
Suppose I have several structures like in the following example, and in the next() method I need to pull the next event using a user-provided buffer, but if this event is a comment, and ignore comments flag is set to true, I need to pull the next event again:
struct Parser {
ignore_comments: bool,
}
enum XmlEvent<'buf> {
Comment(&'buf str),
Other(&'buf str),
}
impl Parser {
fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
let result = loop {
buffer.clear();
let temp_event = self.parse_outside_tag(buffer);
match temp_event {
XmlEvent::Comment(_) if self.ignore_comments => {}
_ => break temp_event,
}
};
result
}
fn parse_outside_tag<'buf>(&mut self, _buffer: &'buf mut String) -> XmlEvent<'buf> {
unimplemented!()
}
}
This code, however, gives a double borrow error, even when I have #![feature(nll)] enabled:
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:14:13
|
14 | buffer.clear();
| ^^^^^^ second mutable borrow occurs here
15 |
16 | let temp_event = self.parse_outside_tag(buffer);
| ------ first mutable borrow occurs here
|
note: borrowed value must be valid for the lifetime 'buf as defined on the method body at 12:5...
--> src/main.rs:12:5
|
12 | fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:16:53
|
16 | let temp_event = self.parse_outside_tag(buffer);
| ^^^^^^ mutable borrow starts here in previous iteration of loop
|
note: borrowed value must be valid for the lifetime 'buf as defined on the method body at 12:5...
--> src/main.rs:12:5
|
12 | fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error: aborting due to 2 previous errors
I can (approximately at least) understand why an error could happen here with the NLL feature turned off, but I don't understand why it happens with NLL.
Anyway, my end goal is to implement this without flags, so I also tried doing this (it is recursive, which is really unfortunate, but all non-recursive versions I came up with could not possibly work without NLL):
fn next<'buf>(&mut self, buffer: &'buf mut String) -> XmlEvent<'buf> {
buffer.clear();
{
let temp_event = self.parse_outside_tag(buffer);
match temp_event {
XmlEvent::Comment(_) if self.ignore_comments => {}
_ => return temp_event,
}
}
self.next(buffer)
}
Here I tried to confine the borrow inside a lexical block, and nothing from this block leaks to the outside. However, I'm still getting an error:
error[E0499]: cannot borrow `*buffer` as mutable more than once at a time
--> src/main.rs:23:19
|
15 | let temp_event = self.parse_outside_tag(buffer);
| ------ first mutable borrow occurs here
...
23 | self.next(buffer)
| ^^^^^^ second mutable borrow occurs here
24 | }
| - first borrow ends here
error: aborting due to previous error
And again, NLL does not fix it.
It has been a long time since I encountered a borrow checking error which I don't understand, so I'm hoping it is actually something simple which I'm overlooking for some reason :)
I really suspect that the root cause is somehow connected with the explicit 'buf lifetime (in particular, errors with the NLL flag turned on have these notes about it), but I can't understand what exactly is wrong here.
This is a limitation of the current implementation of non-lexical lifetimes This can be shown with this reduced case:
fn next<'buf>(buffer: &'buf mut String) -> &'buf str {
loop {
let event = parse(buffer);
if true {
return event;
}
}
}
fn parse<'buf>(_buffer: &'buf mut String) -> &'buf str {
unimplemented!()
}
fn main() {}
This limitation prevents NLL case #3: conditional control flow across functions
In compiler developer terms, the current implementation of non-lexical lifetimes is "location insensitive". Location sensitivity was originally available but it was disabled in the name of performance.
I asked Niko Matsakis about this code:
In the context of your example: the value event only has to have the lifetime 'buf conditionally — at the return point which may or may not execute. But when we are "location insensitive", we just track the lifetime that event must have anywhere, without considering where that lifetime must hold. In this case, that means we make it hold everywhere, which is why you get a compilation failure.
One subtle thing is that the current analysis is location sensitive in one respect — where the borrow takes place. The length of the borrow is not.
The good news is that adding this concept of location sensitivity back is seen as an enhancement to the implementation of non-lexical lifetimes. The bad news:
That may or may not be before the [Rust 2018] edition.
(Note: it did not make it into the initial release of Rust 2018)
This hinges on a (even newer!) underlying implementation of non-lexical lifetimes that improves the performance. You can opt-in to this half-implemented version using -Z polonius:
rustc +nightly -Zpolonius --edition=2018 example.rs
RUSTFLAGS="-Zpolonius" cargo +nightly build
Because this is across functions, you can sometimes work around this by inlining the function.
I posted a question (A borrow checker problem with a loop and non-lexical lifetimes) that was answered by the answer of this question.
I'll document here a workaround that also answers the question. Let's say you have code like this, that only compiles with Polonius:
struct Inner;
enum State<'a> {
One,
Two(&'a ()),
}
fn get<'s>(_inner: &'s mut Inner) -> State<'s> {
unimplemented!()
}
struct Outer {
inner: Inner,
}
impl Outer {
pub fn read<'s>(&'s mut self) -> &'s () {
loop {
match get(&mut self.inner) {
State::One => (), // In this case nothing happens, the borrow should end and the loop should continue
State::Two(a) => return a, // self.inner ought to be borrowed for 's, that's just to be expected
}
}
}
}
As it was said in the another answer:
One subtle thing is that the current analysis is location sensitive in one respect — where the borrow takes place. The length of the borrow is not.
Indeed, borrowing the needed reference again inside the conditional branch makes it compile! Of course, this makes the assumption that get is referentially transparent, so your mileage may vary, but borrowing again seems like an easy enough workaround.
struct Inner;
enum State<'a> {
One,
Two(&'a ()),
}
fn get<'s>(_inner: &'s mut Inner) -> State<'s> {
unimplemented!()
}
struct Outer {
inner: Inner,
}
impl Outer {
pub fn read<'s>(&'s mut self) -> &'s () {
loop {
match get(&mut self.inner) {
State::One => (), // In this case nothing happens, the borrow should end and the loop should continue
State::Two(a) => {
return match get(&mut self.inner) { // Borrowing again compiles!
State::Two(a) => a,
_ => unreachable!(),
}
}, // self.inner ought to be borrowed for 's, that's just to be expected
}
}
}
}
fn main() {
println!("Hello, world!");
}
I am calling closures with a fold function inside another closure. While I am intending for nothing in this some_closure function to live outside the closure environment, I am receiving an error that I am dropping a value while it is still borrowed.
I have tried removing all lifetime specifiers from some_closure, because I find the compiler is much smarter than myself at figuring out lifetimes, but I'm also finding no success in this (the compiler will always ask for lifetime specifiers leading up to the point of the shown example).
What I would desire to do here is to specify a lifetime restricted to the length of the closure inside the function, rather than the function itself. But I have a feeling that what I think is the problem may not actually be my problem, and that there is some gap in my understanding of lifetimes in closures.
I've tried to minimize the example as much as possible:
struct HoldStr<'a>(&'a str);
fn clone_slice_borrows_into_vec<'a>() -> impl Fn(&[&'a HoldStr]) -> Vec<&'a HoldStr<'a>> {
|slice| {
let mut temp = vec![];
temp.clone_from_slice(slice);
temp
}
}
fn clone_slice_borrows_into_vec_same<'a>() -> impl Fn(&[&'a HoldStr]) -> Vec<&'a HoldStr<'a>> {
// Same as last function for the sake of example, but one can assume it does something else
}
fn some_closure<'a>() -> impl Fn() {
|| {
let my_vec = vec![HoldStr("one"), HoldStr("two")];
let my_vec_holding_borrow: Vec<&'a HoldStr> = my_vec.iter().collect();
let called_closures: [Box<dyn Fn(&[&'a HoldStr]) -> Vec<&'a HoldStr<'a>>>; 2] = [
Box::new(clone_slice_borrows_into_vec()),
Box::new(clone_slice_borrows_into_vec_same())
];
let _result = called_closures
.iter()
.fold(my_vec_holding_borrow, |acc, closure| closure(&acc));
}
}
I would expect everything to be dropped by the end of the closure inside some_closure and for this to be fine, especially since I am specifying that lifetime 'a does not relate to anything the function itself returns. But it seems that the borrowed value is expected to live until the end of the function itself. I get this error:
error[E0597]: `my_vec` does not live long enough
--> src/lib.rs:61:51
|
## | fn some_closure<'a>() -> impl Fn() {
| -- lifetime `'a` defined here
...
## | let my_vec_holding_borrow: Vec<&'a HoldStr> = my_vec.iter().collect();
| ---------------- ^^^^^^ borrowed value does not live long enough
| |
| type annotation requires that `my_vec` is borrowed for `'a`
...
## | }
| - `my_vec` dropped here while still borrowed
I'd be happy to hear anything from how to resolve the error, to that I've been going about this the wrong way in the first place.
You need higher-rank trait bounds for your closure types:
fn clone_slice_borrows_into_vec() -> impl for<'a> Fn(&[&'a HoldStr]) -> Vec<&'a HoldStr<'a>> {
...
(Full code in the playground)
The lifetime 'a isn't fixed for your closure. It should return a vector of references with lifetime 'a for any input slice with references of this lifetime. Your code used an externally fixed lifetime instead, which could be chosen by the caller of clone_slice_borrows_into_vec().
If you have a funciton definition like
fn foo<'a>() -> &'a Foo
then it's basically always a mistake. This lets the caller request an arbitrary lifetime, and the function promises to create a reference of this lifetime out of thin air, which is only possible if it gets static references from some global storage, in which case it should simply return &'static Foo.
I have this Rust program using nom 4.2.2. (I have taken the liberty of expanding the nom parser function.)
extern crate failure;
extern crate nom;
use failure::Error;
use std::fs::File;
use std::io::Read;
fn nom_parser(i: &[u8]) -> ::nom::IResult<&[u8], String, u32> {
{ ::nom::lib::std::result::Result::Ok((i, ("foo".to_owned()))) }
}
fn my_parser(buf: &[u8]) -> Result<(&[u8], String), Error> {
Ok((buf, "foo".to_owned()))
}
fn main() -> Result<(), Error> {
let handler = |mut entries: String| { entries.clear() };
loop {
let mut buf = Vec::new();
File::open("/etc/hosts")?.read_to_end(&mut buf)?;
let res = nom_parser(&buf)?.1;
// let res = my_parser(&buf)?.1;
handler(res);
}
}
Compiling this program with rustc 1.33.0 (2aa4c46cf 2019-02-28) yields the following issue:
error[E0597]: `buf` does not live long enough
--> nom-parsing/src/main.rs:21:26
|
21 | let res = nom_parser(&buf)?.1;
| -----------^^^^-
| | |
| | borrowed value does not live long enough
| argument requires that `buf` is borrowed for `'static`
...
24 | }
| - `buf` dropped here while still borrowed
Switching to the commented out version of the parser compiles just fine. How are my_parser and nom_parser different? Who is borrowing buf? How should I change the program to placate the borrow checker?
let res = nom_parser(&buf)?.1;
^ here
You are using the ? operator to propagate the error out of main. The IResult<&[u8], String, u32> = Result<(&[u8], String), nom::Err<&[u8], u32>>. So in case of error the &buf is returned as part of it, so it must stay alive even after main function exits, but it won't because buf is local variable inside main.
In your case the nom_parser never returns error, but the validation only cares about the types and function signatures.
To fix it, you should process the error somehow before propagating it up. For example:
let res = nom_parser(&buf).map_err(|_| failure::format_err!("Parsing failed!"))?.1;
Note that Err in the IResult is not always hard error. It could be nom::Err::Incomplete, meaning that the parsing may succeed if more data is supplied, or nom::Err::Error meaning that the input was not matched by the parser (so perhaps another parser in alt! could succeed), or nom::Err::Failure, meaning that something went really wrong during parsing. Depending on the situation, you may consider them all as failure, or handle them differently.
The problem appears to be in IResult<I, O, E = u32>, which expends to Result<(I, O), Err<I, E>>
As you can see, when you use the ?, the Err that you may return can still contain a reference to the type I, which is your &[u8], and return from your function.
The only way for the function to return this reference would be that the reference has a lifetime that doesn't end with the function, 'static
A simple solution to your problem would be to change the &[u8] to a Vec<u8>, even if I'm not sure what you're trying to do with it.
Why doesn't this code compile:
fn use_cursor(cursor: &mut io::Cursor<&mut Vec<u8>>) {
// do some work
}
fn take_reference(data: &mut Vec<u8>) {
{
let mut buf = io::Cursor::new(data);
use_cursor(&mut buf);
}
data.len();
}
fn produce_data() {
let mut data = Vec::new();
take_reference(&mut data);
data.len();
}
The error in this case is:
error[E0382]: use of moved value: `*data`
--> src/main.rs:14:5
|
9 | let mut buf = io::Cursor::new(data);
| ---- value moved here
...
14 | data.len();
| ^^^^ value used here after move
|
= note: move occurs because `data` has type `&mut std::vec::Vec<u8>`, which does not implement the `Copy` trait
The signature of io::Cursor::new is such that it takes ownership of its argument. In this case, the argument is a mutable reference to a Vec.
pub fn new(inner: T) -> Cursor<T>
It sort of makes sense to me; because Cursor::new takes ownership of its argument (and not a reference) we can't use that value later on. At the same time it doesn't make sense: we essentially only pass a mutable reference and the cursor goes out of scope afterwards anyway.
In the produce_data function we also pass a mutable reference to take_reference, and it doesn't produce a error when trying to use data again, unlike inside take_reference.
I found it possible to 'reclaim' the reference by using Cursor.into_inner(), but it feels a bit weird to do it manually, since in normal use-cases the borrow-checker is perfectly capable of doing it itself.
Is there a nicer solution to this problem than using .into_inner()? Maybe there's something else I don't understand about the borrow-checker?
Normally, when you pass a mutable reference to a function, the compiler implicitly performs a reborrow. This produces a new borrow with a shorter lifetime.
When the parameter is generic (and is not of the form &mut T), the compiler doesn't do this reborrowing automatically1. However, you can do it manually by dereferencing your existing mutable reference and then referencing it again:
fn take_reference(data: &mut Vec<u8>) {
{
let mut buf = io::Cursor::new(&mut *data);
use_cursor(&mut buf);
}
data.len();
}
1 — This is because the current compiler architecture only allows a chance to do a coercion if both the source and target types are known at the coercion site.
use std::thread;
fn test2() {
let x = "abc".to_string();
thread::spawn(|| {
foo2(x);
});
}
fn foo2(x: String) {}
fn test1() {
let x = 1;
thread::spawn(|| {
foo1(x);
});
}
fn foo1(x: i32) {}
fn main() {}
Playground
The error:
error[E0373]: closure may outlive the current function, but it borrows `x`, which is owned by the current function
--> <anon>:12:19
|
12 | thread::spawn(|| { foo1(x); });
| ^^ - `x` is borrowed here
| |
| may outlive borrowed value `x`
|
help: to force the closure to take ownership of `x` (and any other referenced variables), use the `move` keyword, as shown:
| thread::spawn(move || { foo1(x); });
Why does the closure in test1 not take ownership of x, which is specified by the signature (x: i32) of foo1? (I know I can add move to make it work.) I guess it is due to that x is copyable, but if it is copied into the closure then why do I still have the lifetime problem?
However test2 works.
Because it doesn't have to take ownership. Moving is more destructive than simply borrowing, so if the compiler thinks it can get away with not moving a captured value, it won't. It moves the String because it has no other option. It borrows the i32 because it's Copy.
But it can't get away with not borrowing it!
Aah, but the compiler doesn't know that until after it's decided if it's borrowing or moving a captured value. The heuristic it uses is just that: not always correct.
Couldn't it just work it out properly?
Probably, but no one's taught it how.