I am trying to execute an async function using a vector containing arguments, then collect the futures in another vector to join later.
I am getting a compiler error due to passing the ldap argument as a mutable reference ldap was mutably borrowed here in the previous iteration of the looprustc(E0499). I don't get this error when calling the function synchronously with await().
How can fix the lifetime of the ldap argument in run_query()?
Also if anyone has a more idiomatic method of executing a asynchronous function with a vector of arguments please let me know, any help is appreciated thanks!
code:
use ldap3::{Ldap, LdapConnAsync, Scope, SearchEntry, result::Result};
#[derive(Serialize, Deserialize, Debug)]
struct Query{
name: String,
base_dn: String,
query: String,
attr: Vec<String>,
}
async fn run_query(ldap: &mut Ldap, query: Query) -> Result<String>{
// run query
let (rs, _res) = ldap.search(
&query.base_dn,
Scope::Subtree,
&query.query,
&query.attr
).await?.success()?;
// output results
let output = resultentries_to_string(rs).unwrap();
return Ok(output);
}
#[tokio::main]
async fn main() -> Result<()> {
let query = ...Vector of Query struct...
let (conn, mut ldap) = LdapConnAsync::new("ldap://localhost:389").await?;
let mut futures:Vec<_> = Vec::new();
for query in queries{
futures.push(run_query(&mut ldap, query));
}
join_all(futures);
}
Full Error:
error[E0499]: cannot borrow `ldap` as mutable more than once at a time
--> src/main.rs:174:32
|
174 | futures.push(run_query(&mut ldap, query));
| ^^^^^^^^^ `ldap` was mutably borrowed here in the previous iteration of the loop
The reason you get the error is that inisde the for loop you borrow ldap multiple times. In the synchronous case it works because you wait for every iteration to end (no multiple mutable borrows of ldap at the same time, which os not the async case).
To overcome this issue, if it is not too expensive, you can clone ldap every time you use a query:
for query in queries{
let mut ldap_clone = ldap.clone()
futures.push(run_query(&mut ldap_clone, query));
}
You also need to change the signature to accept the actual object, not the reference
Related
I have a simple struct like below called TopicTree:
#[derive(Debug, PartialEq, Eq, Hash, Default, Clone)]
// #[allow(dead_code)]
pub struct TopicTree {
topic_name: String,
child: Option<Vec<Box<TopicTree>>>,
data: Option<Vec<String>>
}
And another struct called App which has an impl block as below:
struct App {
// Topic Tree
topic_tree_root:TopicTree,
}
impl App {
pub fn parse_topic_to_tree(& mut self, topic: &str){
let mut temp_node = & mut self.topic_tree_root;
let mut found = false;
for text in topic.split("/") {
for item in temp_node.child.as_mut().unwrap() {
if item.topic_name == text {
temp_node = item.as_mut();
found = true;
break;
}
}
}
}
}
When I try to compile the code, rustc gives me this error:
error[E0499]: cannot borrow `temp_node.child` as mutable more than once at a time
--> src/pub/lib/app.rs:22:26
|
22 | for j in temp_node.child.as_mut().unwrap() {
| ^^^^^^^^^^^^^^^^^^^^^^^^ `temp_node.child` was mutably borrowed here in the previous iteration of the loop
So my question is, isn't variable item local scoped? if it is not as so, how can I iterate over temp_node.child in a nested loop, it is necessary because temp_node is also mutable.
For the inner loop to execute, the compiler has to 1) create an implicit borrow on temp_node in order to 2) borrow temp_node.child, in order to call as_mut() (which takes &mut self) and then bind the result to item. The lifetime of item depends on temp_node being alive, because of this borrow-chain.
In a subsequent iteration of the outer loop, a conflict occurs: If temp_node = item.as_mut() has executed, you need to mutably borrow temp_node in the for item = ... line. But it is already being borrowed to keep temp_item alive, which came from item, which came from temp_node... Here, the circular logic might become apparent: There can be no guarantee - as the code is written, notwithstanding that the data structure wouldn't support this - that temp_node and item end up being the same object, which would cause two mutable borrows on the same value.
There might be some confusion with respect to mut and &mut here. temp_node needs to be mut (as in let mut, because you change temp_node), but it does not need to be a &mut (as in "mutable borrow", because you are not modifying the data behind the reference).
Running into an ownership issue when attempting to reference multiple values from a HashMap in a struct as parameters in a function call. Here is a PoC of the issue.
use std::collections::HashMap;
struct Resource {
map: HashMap<String, String>,
}
impl Resource {
pub fn new() -> Self {
Resource {
map: HashMap::new(),
}
}
pub fn load(&mut self, key: String) -> &mut String {
self.map.get_mut(&key).unwrap()
}
}
fn main() {
// Initialize struct containing a HashMap.
let mut res = Resource {
map: HashMap::new(),
};
res.map.insert("Item1".to_string(), "Value1".to_string());
res.map.insert("Item2".to_string(), "Value2".to_string());
// This compiles and runs.
let mut value1 = res.load("Item1".to_string());
single_parameter(value1);
let mut value2 = res.load("Item2".to_string());
single_parameter(value2);
// This has ownership issues.
// multi_parameter(value1, value2);
}
fn single_parameter(value: &String) {
println!("{}", *value);
}
fn multi_parameter(value1: &mut String, value2: &mut String) {
println!("{}", *value1);
println!("{}", *value2);
}
Uncommenting multi_parameter results in the following error:
28 | let mut value1 = res.load("Item1".to_string());
| --- first mutable borrow occurs here
29 | single_parameter(value1);
30 | let mut value2 = res.load("Item2".to_string());
| ^^^ second mutable borrow occurs here
...
34 | multi_parameter(value1, value2);
| ------ first borrow later used here
It would technically be possible for me to break up the function calls (using the single_parameter function approach), but it would be more convenient to pass the
variables to a single function call.
For additional context, the actual program where I'm encountering this issue is an SDL2 game where I'm attempting to pass multiple textures into a single function call to be drawn, where the texture data may be modified within the function.
This is currently not possible, without resorting to unsafe code or interior mutability at least. There is no way for the compiler to know if two calls to load will yield mutable references to different data as it cannot always infer the value of the key. In theory, mutably borrowing both res.map["Item1"] and res.map["Item2"] would be fine as they would refer to different values in the map, but there is no way for the compiler to know this at compile time.
The easiest way to do this, as already mentioned, is to use a structure that allows interior mutability, like RefCell, which typically enforces the memory safety rules at run-time before returning a borrow of the wrapped value. You can also work around the borrow checker in this case by dealing with mut pointers in unsafe code:
pub fn load_many<'a, const N: usize>(&'a mut self, keys: [&str; N]) -> [&'a mut String; N] {
// TODO: Assert that keys are distinct, so that we don't return
// multiple references to the same value
keys.map(|key| self.load(key) as *mut _)
.map(|ptr| unsafe { &mut *ptr })
}
Rust Playground
The TODO is important, as this assertion is the only way to ensure that the safety invariant of only having one mutable reference to any value at any time is upheld.
It is, however, almost always better (and easier) to use a known safe interior mutation abstraction like RefCell rather than writing your own unsafe code.
I am writing a small program to calculate the critical path on a PERT diagram (https://en.wikipedia.org/wiki/Program_evaluation_and_review_technique) in rust.
I am storing Task objects inside a hashmap. The hashmap is owned by an object called Pert. Each Task object owns two Vec<String> objects, identifying the task's prerequisites and successors, and has an i32 to specify its duration. The tasks are created inside main.rs and added to the Pert object through an add function.
task.rs:
pub struct Task {
name: String,
i32: duration,
followers: Vec<String>,
prerequisites: Vec<String>
// Additional fields, not relevant for the example
}
impl Task {
pub fn new(name: &str, duration: i32) -> Task {
Task {
name: String::from(name),
duration: duration,
followers: Vec::new(),
prerequisites: Vec::new(),
}
}
pub fn name(&self) -> &str {
&self.name
}
pub fn duration(&self) -> i32 {
self.duration
}
pub fn get_prerequisites(&self) -> & Vec<String> {
&self.prerequisites
}
pub fn get_followers(&self) -> & Vec<String> {
&self.followers
}
}
To evaluate the critical path it is necessary to calculate the max sum of duration of all tasks and record the earliest start and finish times of each task, as well as the latest start and finish times. The way that can be done is to add a "begin" and an "end" task that mark the beginning and the end to the graph respectively. Starting from the "begin" task, a BFS is performed on the graph until we get to the "end" task. The BFS is done inside the method completion_time of the Pert object.
In my current implementation I hit a problem with the borrow checker, since I am borrowing mutably the hashmap containing the tasks more than once. I don't see an alternative way to do this other than borrowing twice, but I am very new to rust and have no functional programming experience, so if there is an easy way to do this with functional programming, I can't see it either.
pert.rs:
pub struct Pert {
tasks: HashMap<String, Task>
}
impl Pert {
pub fn completion_time(&mut self) -> i32 {
let mut time = 0;
let mut q = VecDeque::<&mut Task>::new();
// put "begin" task at the top of the queue, first mutable borrow of self.tasks
q.push_back(self.tasks.get_mut("begin").unwrap());
while !q.is_empty() {
let old_time = time;
let mut curr_task = q.pop_front().unwrap();
for x in curr_task.get_followers() {
// second mutable borrow of self.tasks happens here
let task = self.tasks.get_mut(x).unwrap();
// additional piece of code here modifying other task properties
time = std::cmp::max(old_time, old_time + task.duration())
}
}
time
}
}
Building the project with an empty main.rs should be enough to trigger the following error message:
error[E0499]: cannot borrow `self.tasks` as mutable more than once at a time
--> src/pert.rs:84:28
|
79 | q.push_back(self.tasks.get_mut("begin").unwrap());
| ---------- first mutable borrow occurs here
80 | while !q.is_empty() {
| - first borrow later used here
...
84 | let task = self.tasks.get_mut(x).unwrap();
| ^^^^^^^^^^ second mutable borrow occurs here
The issue here is that you're trying to get multiple mutable references from the HashMap, which owns the tasks and can only safely give out one mutable reference at a time. By changing the VecDeque to take a &Task, and using .get() instead of .get_mut() on the hashmap in completion_time(), the program will compile.
It doesn't look like you're mutating the task in this example, but assuming that you want to modify this example to mutate the task, the best way is to use interior mutability within the Task struct itself, which is usually achieved with the RefCell type. Any value inside of the Task struct that you want to mutate, you can wrap in a RefCell<>, and when you need to mutate the value, you can call .borrow_mut() on the struct field to get a temporarily mutable reference. This answer explains it in a bit more detail: Borrow two mutable values from the same HashMap
I want my method of struct to perform in a synchronized way. I wanted to do this by using Mutex (Playground):
use std::sync::Mutex;
use std::collections::BTreeMap;
pub struct A {
map: BTreeMap<String, String>,
mutex: Mutex<()>,
}
impl A {
pub fn new() -> A {
A {
map: BTreeMap::new(),
mutex: Mutex::new(()),
}
}
}
impl A {
fn synchronized_call(&mut self) {
let mutex_guard_res = self.mutex.try_lock();
if mutex_guard_res.is_err() {
return
}
let mut _mutex_guard = mutex_guard_res.unwrap(); // safe because of check above
let mut lambda = |text: String| {
let _ = self.map.insert("hello".to_owned(),
"d".to_owned());
};
lambda("dd".to_owned());
}
}
Error message:
error[E0500]: closure requires unique access to `self` but `self.mutex` is already borrowed
--> <anon>:23:26
|
18 | let mutex_guard_res = self.mutex.try_lock();
| ---------- borrow occurs here
...
23 | let mut lambda = |text: String| {
| ^^^^^^^^^^^^^^ closure construction occurs here
24 | if let Some(m) = self.map.get(&text) {
| ---- borrow occurs due to use of `self` in closure
...
31 | }
| - borrow ends here
As I understand when we borrow anything from the struct we are unable to use other struct's fields till our borrow is finished. But how can I do method synchronization then?
The closure needs a mutable reference to the self.map in order to insert something into it. But closure capturing works with whole bindings only. This means, that if you say self.map, the closure attempts to capture self, not self.map. And self can't be mutably borrowed/captured, because parts of self are already immutably borrowed.
We can solve this closure-capturing problem by introducing a new binding for the map alone such that the closure is able to capture it (Playground):
let mm = &mut self.map;
let mut lambda = |text: String| {
let _ = mm.insert("hello".to_owned(), text);
};
lambda("dd".to_owned());
However, there is something you overlooked: since synchronized_call() accepts &mut self, you don't need the mutex! Why? Mutable references are also called exclusive references, because the compiler can assure at compile time that there is only one such mutable reference at any given time.
Therefore you statically know, that there is at most one instance of synchronized_call() running on one specific object at any given time, if the function is not recursive (calls itself).
If you have mutable access to a mutex, you know that the mutex is unlocked. See the Mutex::get_mut() method for more explanation. Isn't that amazing?
Rust mutexes do not work the way you are trying to use them. In Rust, a mutex protects specific data relying on the borrow-checking mechanism used elsewhere in the language. As a consequence, declaring a field Mutex<()> doesn't make sense, because it is protecting read-write access to the () unit object that has no values to mutate.
As Lukas explained, your call_synchronized as declared doesn't need to do synchronization because its signature already requests an exclusive (mutable) reference to self, which prevents it from being invoked from multiple threads on the same object. In other words, you need to change the signature of call_synchronized because the current one does not match the functionality it is intended to provide.
call_synchronized needs to accept a shared reference to self, which will signal to Rust that it can be called from multiple threads in the first place. Inside call_synchronized a call to Mutex::lock will simultaneously lock the mutex and provide a mutable reference to the underlying data, carefully scoped so that the lock is held for the duration of the reference:
use std::sync::Mutex;
use std::collections::BTreeMap;
pub struct A {
synced_map: Mutex<BTreeMap<String, String>>,
}
impl A {
pub fn new() -> A {
A {
synced_map: Mutex::new(BTreeMap::new()),
}
}
}
impl A {
fn synchronized_call(&self) {
let mut map = self.synced_map.lock().unwrap();
// omitting the lambda for brevity, but it would also work
// (as long as it refers to map rather than self.map)
map.insert("hello".to_owned(), "d".to_owned());
}
}
I do not know Rust well enough to understand lifetimes and closures yet...
Trying to collect the downloaded data into a vector using tokio-curl:
extern crate curl;
extern crate futures;
extern crate tokio_core;
extern crate tokio_curl;
use std::io::{self, Write};
use std::str;
use curl::easy::Easy;
use tokio_core::reactor::Core;
use tokio_curl::Session;
fn main() {
// Create an event loop that we'll run on, as well as an HTTP `Session`
// which we'll be routing all requests through.
let mut lp = Core::new().unwrap();
let mut out = Vec::new();
let session = Session::new(lp.handle());
// Prepare the HTTP request to be sent.
let mut req = Easy::new();
req.get(true).unwrap();
req.url("https://www.rust-lang.org").unwrap();
req.write_function(|data| {
out.extend_from_slice(data);
io::stdout().write_all(data).unwrap();
Ok(data.len())
})
.unwrap();
// Once we've got our session, issue an HTTP request to download the
// rust-lang home page
let request = session.perform(req);
// Execute the request, and print the response code as well as the error
// that happened (if any).
let mut req = lp.run(request).unwrap();
println!("{:?}", req.response_code());
println!("out: {}", str::from_utf8(&out).unwrap());
}
Produces an error:
error[E0373]: closure may outlive the current function, but it borrows `out`, which is owned by the current function
--> src/main.rs:25:24
|
25 | req.write_function(|data| {
| ^^^^^^ may outlive borrowed value `out`
26 | out.extend_from_slice(data);
| --- `out` is borrowed here
|
help: to force the closure to take ownership of `out` (and any other referenced variables), use the `move` keyword, as shown:
| req.write_function(move |data| {
Investigating further, I see that Easy::write_function requires the 'static lifetime, but the example of how to collect output from the curl-rust docs uses Transfer::write_function instead:
use curl::easy::Easy;
let mut data = Vec::new();
let mut handle = Easy::new();
handle.url("https://www.rust-lang.org/").unwrap();
{
let mut transfer = handle.transfer();
transfer.write_function(|new_data| {
data.extend_from_slice(new_data);
Ok(new_data.len())
}).unwrap();
transfer.perform().unwrap();
}
println!("{:?}", data);
The Transfer::write_function does not require the 'static lifetime:
impl<'easy, 'data> Transfer<'easy, 'data> {
/// Same as `Easy::write_function`, just takes a non `'static` lifetime
/// corresponding to the lifetime of this transfer.
pub fn write_function<F>(&mut self, f: F) -> Result<(), Error>
where F: FnMut(&[u8]) -> Result<usize, WriteError> + 'data
{
...
But I can't use a Transfer instance on tokio-curl's Session::perform because it requires the Easy type:
pub fn perform(&self, handle: Easy) -> Perform {
transfer.easy is a private field that is directly passed to session.perform.
It this an issue with tokio-curl? Maybe it should mark the transfer.easy field as public or implement new function like perform_transfer? Is there another way to collect output using tokio-curl per transfer?
The first thing you have to understand when using the futures library is that you don't have any control over what thread the code is going to run on.
In addition, the documentation for curl's Easy::write_function says:
Note that the lifetime bound on this function is 'static, but that is often too restrictive. To use stack data consider calling the transfer method and then using write_function to configure a callback that can reference stack-local data.
The most straight-forward solution is to use some type of locking primitive to ensure that only one thread at a time may have access to the vector. You also have to share ownership of the vector between the main thread and the closure:
use std::sync::Mutex;
use std::sync::Arc;
let out = Arc::new(Mutex::new(Vec::new()));
let out_closure = out.clone();
// ...
req.write_function(move |data| {
let mut out = out_closure.lock().expect("Unable to lock output");
// ...
}).expect("Cannot set writing function");
// ...
let out = out.lock().expect("Unable to lock output");
println!("out: {}", str::from_utf8(&out).expect("Data was not UTF-8"));
Unfortunately, the tokio-curl library does not currently support using the Transfer type that would allow for stack-based data.