I would like to make a variant of this program where my goal is to spawn a new thread calling a callback inside another spawned thread and the cb is declared outside the handle and keeping full parallelisation :
let counter = Arc::new(Mutex::new(0));
let cb = || {
let counter = Arc::clone(&counter);
for _ in 0..9 {
let mut num = counter.lock().unwrap();
*num += 1;
}
};
let handle = thread::spawn(move || {
for _ in 0..9 {
thread::spawn(cb);
}
})
.join();
println!("Result: {}", *counter.lock().unwrap());
Unfortunately, I got this error :
`counter` does not live long enough
borrowed value does not live long enoughrustcE0597
tests_spawn.rs(28, 5): `counter` dropped here while still borrowed
tests_spawn.rs(12, 18): value captured here
tests_spawn.rs(22, 17): argument requires that `counter` is borrowed for `'static`
How can change the lifetime of the counter variable to make it static ?
A cleaner option is probably to use the scoped threads API, which automatically joins the child threads. Plus, you don't even need Arc.
use std::sync::Mutex;
use std::thread;
fn main() {
let counter = Mutex::new(0);
let cb = || {
for _ in 0..9 {
let mut num = counter.lock().unwrap();
*num += 1;
}
};
thread::scope(|s| {
for _ in 0..9 {
s.spawn(cb);
}
});
println!("Result: {}", *counter.lock().unwrap());
}
playground
Related
i'm new to rust.
I'm trying to write a file_sensor that will start a counter after a file is created. The plan is that after an amount of time, if a second file is not received the sensor will exit with a zero exit code.
I could write the code to continue that work but i feel the code below illustrates the problem (i have also missed for example the post function referred to)
I have been struggling with this problem for several hours, i've tried Arc and mutex's and even global variables.
The Timer implementation is Ticktock-rs
I need to be able to either get heartbeat in the match body for EventKind::Create(CreateKind::Folder) or file_count in the loop
The code i've attached here runs but file_count is always zero in the loop.
use std::env;
use std::path::Path;
use std::{thread, time};
use std::process::ExitCode;
use ticktock::Timer;
use notify::{
Watcher,
RecommendedWatcher,
RecursiveMode,
Result,
event::{EventKind, CreateKind, ModifyKind, Event}
};
fn main() -> Result<()> {
let now = time::Instant::now();
let mut heartbeat = Timer::apply(
|_, count| {
*count += 1;
*count
},
0,
)
.every(time::Duration::from_millis(500))
.start(now);
let mut file_count = 0;
let args = Args::parse();
let REQUEST_SENSOR_PATH = env::var("REQUEST_SENSOR_PATH").expect("$REQUEST_SENSOR_PATH} is not set");
let mut watcher = notify::recommended_watcher(move|res: Result<Event>| {
match res {
Ok(event) => {
match event.kind {
EventKind::Create(CreateKind::File) => {
file_count += 1;
// do something with file
}
_ => { /* something else changed */ }
}
println!("{:?}", event);
},
Err(e) => {
println!("watch error: {:?}", e);
ExitCode::from(101);
},
}
})?;
watcher.watch(Path::new(&REQUEST_SENSOR_PATH), RecursiveMode::Recursive)?;
loop {
let now = time::Instant::now();
if let Some(n) = heartbeat.update(now){
println!("Heartbeat: {}, fileCount: {}", n, file_count);
if n > 10 {
heartbeat.set_value(0);
// This function will reset timer when a file arrives
}
}
}
Ok(())
}
Your compiler warnings show you the problem:
warning: unused variable: `file_count`
--> src/main.rs:31:25
|
31 | file_count += 1;
| ^^^^^^^^^^
|
= note: `#[warn(unused_variables)]` on by default
= help: did you mean to capture by reference instead?
The problem here is that you use file_count inside of a move || closure. file_count is an i32, which is Copy. Using it in a move || closure actually creates a copy of it, which does no longer update the original variable if you assign to it.
Either way, it's impossible to modify a variable in main() from an event handler. Event handlers require 'static lifetime if they reference things, because Rust cannot guarantee that the event handler lives shorter than main.
One solution for this problem is to use reference counters and interior mutability. In this case, I will use Arc for reference counters and AtomicI32 for interior mutability. Note that notify::recommended_watcher requires thread safety, otherwise instead of an Arc<AtomicI32> we could have used an Rc<Cell<i32>>, which is the same thing but only for single-threaded environments, with a little less overhead.
use notify::{
event::{CreateKind, Event, EventKind},
RecursiveMode, Result, Watcher,
};
use std::time;
use std::{env, sync::atomic::Ordering};
use std::{path::Path, sync::Arc};
use std::{process::ExitCode, sync::atomic::AtomicI32};
use ticktock::Timer;
fn main() -> Result<()> {
let now = time::Instant::now();
let mut heartbeat = Timer::apply(
|_, count| {
*count += 1;
*count
},
0,
)
.every(time::Duration::from_millis(500))
.start(now);
let file_count = Arc::new(AtomicI32::new(0));
let REQUEST_SENSOR_PATH =
env::var("REQUEST_SENSOR_PATH").expect("$REQUEST_SENSOR_PATH} is not set");
let mut watcher = notify::recommended_watcher({
let file_count = Arc::clone(&file_count);
move |res: Result<Event>| {
match res {
Ok(event) => {
match event.kind {
EventKind::Create(CreateKind::File) => {
file_count.fetch_add(1, Ordering::AcqRel);
// do something with file
}
_ => { /* something else changed */ }
}
println!("{:?}", event);
}
Err(e) => {
println!("watch error: {:?}", e);
ExitCode::from(101);
}
}
}
})?;
watcher.watch(Path::new(&REQUEST_SENSOR_PATH), RecursiveMode::Recursive)?;
loop {
let now = time::Instant::now();
if let Some(n) = heartbeat.update(now) {
println!(
"Heartbeat: {}, fileCount: {}",
n,
file_count.load(Ordering::Acquire)
);
if n > 10 {
heartbeat.set_value(0);
// This function will reset timer when a file arrives
}
}
}
}
Also, note that the ExitCode::from(101); gives you a warning. It does not actually exit the program, it only creates an exit code variable and then discards it again. You probably intended to write std::process::exit(101);. Although I would discourage it, because it does not properly clean up (does not call any Drop implementations). I'd use panic here, instead. This is the exact usecase panic is meant for.
I am having issues with the concept of lifetimes in rust. I am trying to use the crate bgpkit_parser to read in a bz2 file via url link and then create a radix trie.
One field extracted from the file is the AS Path which I have named path in my code within the build_routetable function. I am having trouble as to why rust does not like let origin = clean_path.last() which takes the last element in the vector.
fn as_parser(element: &BgpElem) -> Vec<u32> {
let x = &element.as_path.as_ref().unwrap().segments[0];
let mut as_vec = &Vec::new();
let mut as_path: Vec<u32> = Vec::new();
if let AsPathSegment::AsSequence(value) = x {
as_vec = value;
}
for i in as_vec {
as_path.push(i.asn);
}
return as_path;
}
fn prefix_parser(element: &BgpElem) -> String {
let subnet_id = element.prefix.prefix.ip().to_string().to_owned();
let prefix_id = element.prefix.prefix.prefix().to_string().to_owned();
let prefix = format!("{}/{}", subnet_id, prefix_id);//.as_str();
return prefix;
}
fn get_aspath(raw_aspath: Vec<u32>) -> Vec<u32> {
let mut as_path = Vec::new();
for i in raw_aspath {
if i < 64511 {
if as_path.contains(&i) {
continue;
}
else {
as_path.push(i);
}
}
else if 65535 < i && i < 4000000000 {
if as_path.contains(&i) {
continue;
}
else {
as_path.push(i);
}
}
}
return as_path;
}
fn build_routetable(mut trie4: Trie<String, Option<&u32>>, mut trie6: Trie<String, Option<&u32>>) {
let url: &str = "http://archive.routeviews.org/route-views.chile/\
bgpdata/2022.06/RIBS/rib.20220601.0000.bz2";
let parser = BgpkitParser::new(url).unwrap();
let mut count = 0;
for elem in parser {
if elem.elem_type == bgpkit_parser::ElemType::ANNOUNCE {
let record_timestamp = &elem.timestamp;
let record_type = "A";
let peer = &elem.peer_ip;
let prefix = prefix_parser(&elem);
let path = as_parser(&elem);
let clean_path = get_aspath(path);
// Issue is on the below line
// `clean_path` does not live long enough
// borrowed value does not live long
// enough rustc E0597
// main.rs(103, 9): `clean_path` dropped
// here while still borrowed
// main.rs(77, 91): let's call the
// lifetime of this reference `'1`
// main.rs(92, 17): argument requires
// that `clean_path` is borrowed for `'1`
let origin = clean_path.last(); //issue line
if prefix.contains(":") {
trie6.insert(prefix, origin);
}
else {
trie4.insert(prefix, origin);
}
count+=1;
if count >= 10000 {
println!("{:?} | {:?} | {:?} | {:?} | {:?}",
record_type, record_timestamp, peer, prefix, path);
count=0
}
};
}
println!("Trie4 size: {:?} prefixes", trie4.len());
println!("Trie6 size: {:?} prefixes", trie6.len());
}
Short answer: you're "inserting" a reference. But what's being referenced doesn't outlive what it's being inserted into.
Longer: The hint is your trie4 argument, the signature of which is this:
mut trie4: Trie<String, Option<&u32>>
So that lives beyond the length of the loop where things are declared. This is all in the loop:
let origin = clean_path.last(); //issue line
if prefix.contains(":") {
trie6.insert(prefix, origin);
}
While origin is a Vec<u32> and that's fine, the insert method is no doubt taking a String and either an Option<&u32> or a &u32. Obviously a key/value pair. But here's your problem: the value has to live as long as the collection, but your value is the last element contained in the Vec<u32>, which goes away! So you can't put something into it that will not live as long as the "container" object! Rust has just saved you from dangling references (just like it's supposed to).
Basically, your containers should be Trie<String, Option<u32>> without the reference, and then this'll all just work fine. Your problem is that the elements are references, and not just contained regular values, and given the size of what you're containing, it's actually smaller to contain a u32 than a reference (pointer size (though actually, it'll likely be the same either way, because alignment issues)).
Also of note: trie4 and trie6 will both be gone at the end of this function call, because they were moved into this function (not references or mutable references). I hope that's what you want.
Not much to explain. I don't understand what is even having the lifetime designated 1 and 2 by the compiler error message.
All posts I have checked so far just say use crossbeam for scopped threads, but this hasn't fixed my issue at all and I dont think I even understand the finer issue here.
Any help is appreciated.
use crossbeam_utils::thread;
struct TestStruct {
s: f64,
}
impl TestStruct {
fn new() -> Self {
Self {
s: -1.,
}
}
fn fubar(&'static self) -> f64 {
let thread_return_value = thread::scope(|scope|
// lifetime may not live long enough
// returning this value requires that `'1` must outlive `'2`
// Question: what are the two lifetimes even of? I am probably just
// a noob here.
scope.spawn(move |_| { // same error with or without move
// I have found that it doesnt matter what I put in this scope,
// but the following is the closest to what I have in my actual
// code.
let mut psum = 0.;
for _ in 0..10 { psum += self.s; }
psum
})
).unwrap();
// do anything with thread_return_value
return 0.; // just so its explicitly not the problem here, return 0.
}
}
fn main() {
let test_item = TestStruct::new();
// rustcE0597
let stored_value = test_item.fubar();
println!("{}", &stored_value);
return;
}
Edit after marking for correct answer, working minimal example:
#![feature(let_chains)]
use crossbeam_utils::thread;
struct TestStruct {
s: f64,
}
impl TestStruct {
fn new() -> Self {
Self {
s: -1.,
}
}
fn fubar(&self) -> f64 {
let thread_return_value = thread::scope(|scope| {
let th = scope.spawn(move |_| {
let mut psum = 0.;
for _ in 0..10 { psum += self.s; }
psum
});
let psum = th.join().unwrap();
psum
}
).unwrap();
return thread_return_value;
}
}
fn main() {
let test_item = TestStruct::new();
// rustcE0597
let stored_value = test_item.fubar();
println!("{}", &stored_value);
return;
}
The most obvious problem in your code is the &'static self lifetime. If you do so, you will only be able to call this function with static (that is, global) values of this type. So just remove that 'static and write &self.
Then the real problem is because you are trying to return your scoped thread handle from the crossbeam::scoped, the value returned by scope.spawn(), and that is not allowed. That is why they are called scoped threads: they are limited to the enclosing scope.
Remember that in Rust, when a block ends without a ; the value of the last expression is returned as the value of the block itself.
You probably want to return the psum. If so you need to wait for the handle to finish:
fn fubar(& self) -> f64 {
let thread_return_value = thread::scope(|scope| {
let th = scope.spawn(move |_| {
let mut psum = 0.;
for _ in 0..10 { psum += self.s; }
psum
}); // <--- here, add a ;
let psum = th.join().unwrap(); //get the inner result
psum //forward it to the outer scope
}).unwrap();
return 0.;
}
I'm new to Rust and still reading the Rust book. Below is my program.
use clap::{App, Arg};
type GenericError = Box<dyn std::error::Error + Send + Sync + 'static>;
type GenericResult<T> = Result<T, GenericError>;
fn main() -> GenericResult<()> {
let matches = App::new("test")
.arg(Arg::new("latency")
.takes_value(true))
.get_matches();
let latency_present = matches.is_present("latency");
let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
let latency_pairs: Vec<&str> = latency.split(",").collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
checker.join().unwrap()?;
Ok(())
}
When I run it, it tells me this:
error[E0597]: `matches` does not live long enough
--> src\main.rs:14:19
|
14 | let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
| ^^^^^^^--------------------
| |
| borrowed value does not live long enough
| argument requires that `matches` is borrowed for `'static`
...
30 | }
| - `matches` dropped here while still borrowed
I don't quite understand the error messages here. But I guess it's because I use latency_pairs in the checker thread and latency_pairs could get dropped while checker is still executing. Is my understanding correct? How to fix the error? I tried for (i, latency_pair) in latency_pairs.clone().iter().enumerate() { in order to pass a cloned value for the thread, but it doesn't help.
latency_pairs holds references into latency which in turn references matches. Thus cloning latency_pairs just clones the references into latency and matches.
Your code would require that latency's type is &'static str but it's actually &'a str where 'a is bound to matches' lifetime.
You can call to_owned() on latency to get an owned value and split the string inside the closure or you can call to_owned() on each of the splits collected in latency_pairs and move that Vec<String> into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
If you need to use latency_pairs outside of the closure, you can clone it before moving it into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let latency_pairs_ = latency_pairs.clone();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs_.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
println!("{:?}", latency_pairs);
I am new to Rust and I am struggling with the concept of borrowing.
I want to load a Vec<Vec<f64>> matrix and then process it in parallel. However when I try to compile this piece of code I get error: capture of moved value: `matrix` [E0382] at the let _ = line.
This matrix is supposed to be readonly for the threads, they won't modify it. How can I pass it readonly and make the "moved value" error go away?
fn process(matrix: &Vec<Vec<f64>>) {
// do nothing for now
}
fn test() {
let filename = "matrix.tsv";
// loads matrix into a Vec<Vec<f64>>
let mut matrix = load_matrix(filename);
// Determine number of cpus
let ncpus = num_cpus::get();
println!("Number of cpus on this machine: {}", ncpus);
for i in 0..ncpus {
// In the next line the "error: capture of moved value: matrix" happens
let _ = thread::spawn(move || {
println!("Thread number: {}", i);
process(&matrix);
});
}
let d = Duration::from_millis(1000 * 1000);
thread::sleep(d);
}
Wrap the object to be shared into an Arc which stands for atomically reference counted (pointer). For each thread, clone this pointer and pass ownership of the clone to the thread. The wrapped object will be deallocated when it's no longer used by anything.
fn process(matrix: &Vec<Vec<f64>>) {
// do nothing for now
}
fn test() {
use std::sync::Arc;
let filename = "matrix.tsv";
// loads matrix into a Vec<Vec<f64>>
let mut matrix = Arc::new(load_matrix(filename));
// Determine number of cpus
let ncpus = num_cpus::get();
println!("Number of cpus on this machine: {}", ncpus);
for i in 0..ncpus {
let matrix = matrix.clone();
let _ = thread::spawn(move || {
println!("Thread number: {}", i);
process(&matrix);
});
}
let d = Duration::from_millis(1000 * 1000);
thread::sleep(d);
}