I am trying to read the contents of files in a directory in parallel. I'm running into lifetime issues.
My code looks like this:
use std::io::fs;
use std::io;
use std::collections::HashMap;
use std::comm;
use std::io::File;
fn main() {
let (tx, rx) = comm::channel(); // (Sender, Receiver)
let paths = fs::readdir(&Path::new("resources/tests")).unwrap();
for path in paths.iter() {
let task_tx = tx.clone();
spawn(proc() {
match File::open(path).read_to_end() {
Ok(data) => task_tx.send((path.filename_str().unwrap(), data)),
Err(e) => fail!("Could not read one of the files! Error: {}", e)
};
});
}
let mut results = HashMap::new();
for _ in range(0, paths.len()) {
let (filename, data) = rx.recv();
results.insert(filename, data);
}
println!("{}", results);
}
The compilation error I'm getting is:
error: paths does not live long enough
note: reference must be valid for the static lifetime...
note: ...but borrowed value is only valid for the block at 7:19
I also tried to use into_iter() (or move_iter() previously) in the loop without much success.
I'm suspecting it has to do with the spawned tasks remaining alive beyond the entire main() scope, but I don't know how I can fix this situation.
The error message might be a bit confusing but what it's telling you is that you are trying to use a reference path inside of a task.
Because spawn is using proc you can only use data that you can transfer ownership of to that task (Send kind).
To solve that you can do this (you could use a move_iter but then you can't access paths after the loop):
for path in paths.iter() {
let task_tx = tx.clone();
let p = path.clone();
spawn(proc() {
match File::open(&p).read_to_end() {
The second problem is that you are trying to send &str (filename) over a channel. Same as for tasks types used must be of kind Send:
match File::open(&p).read_to_end() {
Ok(data) => task_tx.send((p.filename_str().unwrap().to_string(), data)),
Err(e) => fail!("Could not read one of the files! Error: {}", e)
};
Related
I have simple client/server application. I am receiving message on the server side from client but I want to send that response to the channel from server to other file and I am receiving error "borrowed value does not live long enough".
I have searched in the stack overflow for similar previous questions but not getting enough understanding of lifetime. Is there a good documentation or if simple example available on this topic?
For now if someone can help me to fix this code (may be edit the portion of code which needs to fix) that would be helpful.
Thanks in advance.
Server side:
use std::os::unix::net::UnixDatagram;
use std::path::Path;
fn unlink_socket (path: impl AsRef<Path>) {
let path = path.as_ref();
if Path::new(path).exists() {
let result = std::fs::remove_file(path);
match result {
Err(e) => {
println!("Couldn't remove the file: {:?}", e);
},
_ => {}
}
}
}
pub fn tcp_datagram_server() {
pub static FILE_PATH: &'static str = "/tmp/datagram.sock";
let (tx, rx) = mpsc::channel();
let mut buf = vec![0; 1024];
unlink_socket(FILE_PATH);
let socket = match UnixDatagram::bind(FILE_PATH) {
Ok(socket) => socket,
Err(e) => {
println!("Couldn't bind: {:?}", e);
return;
}
};
println!("Waiting for client to connect...");
loop {
let received_bytes = socket.recv(buf.as_mut_slice()).expect("recv function failed");
println!("Received {:?}", received_bytes);
let received_message = from_utf8(buf.as_slice()).expect("utf-8 convert failed");
tx.clone().send(received_message);
}
}
fn main() {
tcp_datagram_server();
}
client side:
use std::sync::mpsc;
use std::os::unix::net::UnixDatagram;
use std::path::Path;
use std::io::prelude::*;
pub fn tcp_datagram_client() {
pub static FILE_PATH: &'static str = "/tmp/datagram.sock";
let socket = UnixDatagram::unbound().unwrap();
match socket.connect(FILE_PATH) {
Ok(socket) => socket,
Err(e) => {
println!("Couldn't connect: {:?}", e);
return;
}
};
println!("TCP client Connected to TCP Server {:?}", socket);
loop {
socket.send(b"Hello from client to server").expect("recv function failed");
}
}
fn main() {
tcp_datagram_client();
}
Error I am getting
error[E0597]: `buf` does not live long enough
--> src/unix_datagram_server.rs:38:42
|
38 | let received_message = from_utf8(buf.as_slice()).expect("utf-8 convert failed");
| ^^^ borrowed value does not live long enough
...
41 | }
| -
| |
| `buf` dropped here while still borrowed
| borrow might be used here, when `tx` is dropped and runs the `Drop` code for type `std::sync::mpsc::Sender`
|
= note: values in a scope are dropped in the opposite order they are defined
error: aborting due to previous error; 8 warnings emitted
For now if someone can help me to fix this code (may be edit the portion of code which needs to fix) that would be helpful.
Well the message seems rather clear. send does exactly what it says it does, it sends the parameter through the channel. This means the data must live long enough and remain valid "forever" (it needs to be alive and valid in the channel, as well as when fetched from it by the receiver).
That is not the case here. rustc can't understand that the function never returns, and it can panic anyway which will end up the same: the function will terminate, which will invalidate buf. Since received_message borrows buf, that means received_message can't be valid after the function has terminated. But at that point the message would still be in the channel waiting to be read (or retrieved by the receiver doing who knows what).
Therefore your construction is not allowed.
A second issue is that you're overwriting the buffer data on every loop, which has the same effect of breaking the message you sent during the previous iteration, and thus is not correct either. Though Rust won't let you do that either: if you work around the first error it will tell you that there's an outstanding shared borrow (the message sent through the channel) so you can't modify the backing buffer in the following iteration.
The solution is quite simple: have each iteration create an owned string (copying the current iteration's message) and send that through the channel:
tx.clone().send(received_message.to_string());
Also, these are more style / inefficiency remarks but:
The clone() on tx is completely redundant. The point of having a sender that is Clone is being able to send from multiple threads (hence mp in the channel name, that's for multiple producers). Here you have a single thread, the original sender works fine.
.as_slice() and .as_mut_slice() are rarely used unless necessary, which they aren't here: array references coerce to slices, so you can just use &mut buf and &buf. And why are you calling Path::new on something that's already a path? It doesn't do anything but it's not useful either.
It is rather annoying that your snippet is missing multiple imports and thus doesn't even compile as is.
From more of a unixy perspective, errors are usually printed on stderr. In Rust, eprintln does that for you (otherwise working in the same way println does). And I don't understand the purpose of marking a lexically nested static pub. Since the static is inside the function it's not even visible to the function's siblings, to say nothing of external callers. As a result I'd end up with this:
use std::os::unix::net::UnixDatagram;
use std::path::Path;
use std::sync::mpsc;
use std::str::from_utf8;
fn unlink_socket (path: impl AsRef<Path>) {
let path = path.as_ref();
if path.exists() {
if let Err(e) = std::fs::remove_file(path) {
eprintln!("Couldn't remove the file: {:?}", e);
}
}
}
static FILE_PATH: &'static str = "/tmp/datagram.sock";
pub fn tcp_datagram_server() {
unlink_socket(FILE_PATH);
let socket = match UnixDatagram::bind(FILE_PATH) {
Ok(socket) => socket,
Err(e) => {
eprintln!("Couldn't bind: {:?}", e);
return;
}
};
let (tx, _) = mpsc::channel();
let mut buf = vec![0; 1024];
println!("Waiting for client to connect...");
loop {
let received_bytes = socket.recv(&mut buf).expect("recv function failed");
println!("Received {:?}", received_bytes);
let received_message = from_utf8(&buf).expect("utf-8 convert failed");
tx.send(received_message.to_string());
}
}
There's a hint in the compiler message, that values in a scope are dropped in the opposite order they are defined in, and in the example, buf is defined after tx, which means it will be dropped before tx. Since a reference to buf (in the form of received_message) is passed to tx.send(), then buf should live longer that tx, and therefore switching the definition order will fix this particular error (ie. switch lines 19 and 20).
I'm creating a small application that explores variable lifetimes and threads. I want to load in a file once, and then use its contents (in this case an audio file) in a separate channel. I am having issues with value lifetimes.
I'm almost certain the syntax is wrong for what I have so far (for creating a static variable), but I can't find any resources for File types and lifetimes. What I have thus far produces this error:
let file = &File::open("src/censor-beep-01.wav").unwrap();
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ creates a temporary which is freed while still in use
let x: &'static File = file;
------------- type annotation requires that borrow lasts for `'static`
The code I currently have is:
#![allow(dead_code)]
#![allow(unused_imports)]
#![allow(unused_must_use)]
#![allow(unused_variables)]
use std::io::{self, BufRead, BufReader, stdin, Read};
use std::sync::mpsc::{self, TryRecvError};
use std::thread;
use std::time::Duration;
use std::fs::File;
use std::rc::Rc;
use rodio::Source;
fn main() {
let file = &File::open("src/censor-beep-01.wav").unwrap();
let x: &'static File = file;
loop {
let (tx, rx) = mpsc::channel();
thread::spawn(move || loop {
let tmp = x;
let (stream, stream_handle) = rodio::OutputStream::try_default().unwrap();
let source = rodio::Decoder::new(BufReader::new(tmp)).unwrap();
stream_handle.play_raw(source.convert_samples());
match rx.try_recv() {
Ok(_) | Err(TryRecvError::Disconnected) => {
break;
}
Err(TryRecvError::Empty) => {
println!("z");
thread::sleep(Duration::from_millis(1000));
}
}
});
let mut line = String::new();
let stdin = io::stdin();
let _ = stdin.lock().read_line(&mut line);
let _ = tx.send(());
return;
}
}
You need to wrap the file with Arc and Mutex like Arc::new(Mutex::new(file)) and then clone the file before passing it to the thread.
Arc is used for reference counting, which is needed to share the target object (in your case it is a file) across the thread and Mutex is needed to access the target object synchronously.
sample code (I have simplified your code to make it more understandable):
let file = Arc::new(Mutex::new(File::open("src/censor-beep-01.wav").unwrap()));
loop {
let file = file.clone();
thread::spawn(move || loop {
let mut file_guard = match file.lock() {
Ok(guard) => guard,
Err(poison) => poison.into_inner()
};
let file = file_guard.deref();
// now you can pass above file object to BufReader like "BufReader::new(file)"
});
}
reason for creates a temporary which is freed while still in use error:
You have only stored the reference of the file without the actual file object. so, the object will be droped in that line itself.
This question already has answers here:
How do I share a mutable object between threads using Arc?
(1 answer)
Lifetime troubles sharing references between threads
(1 answer)
Closed 5 years ago.
With hyper, I need to make an HTTP connection and read the results.
I want to wrap the whole thing in a timeout,
so I start a thread
and use recv_timeout to wait for it.
Wrapping just the send works,
but I want to also wrap read_to_string.
Here is the code:
fn send_request(url: &str) -> Result<Response, MyAppError> {
let mut c = Client::new();
let mut req = c.get(url);
req.send().map_err(|e| MyAppError::TcpError(e))
}
fn get_url(url: &str, mut buf: &mut String) -> Result<u16, MyAppError> {
let mut resp = send_request(url)?;
resp.read_to_string(&mut buf).map_err(|e| MyAppError::ReadError(e))?;
Ok(resp.status.to_u16())
}
fn get_url_with_timeout_2(url: &str, mut buf: &mut String) -> Result<u16, MyAppError> {
let (tx, rx) = mpsc::channel();
let url = url.to_owned();
let t = thread::spawn(move || {
match tx.send(get_url(&url, &mut buf)) {
Ok(()) => {} // everything good
Err(_) => {} // we have been released, no biggie
}
});
match rx.recv_timeout(Duration::from_millis(5000)) {
Ok(resp) => resp,
Err(_) => Err(MyAppError::Timeout),
}
}
Unfortunately I get a compiler error:
error[E0477]: the type `[closure#src/main.rs:53:25: 58:4 tx:std::sync::mpsc::Sender<std::result::Result<u16, MyAppError>>, url:std::string::String, buf:&mut std::string::String]` does not fulfill the required lifetime
--> src/main.rs:53:11
|
53 | let t = thread::spawn(move || {
| ^^^^^^^^^^^^^
|
= note: type must outlive the static lifetime
How can I pass the buffer to the thread,
let it fill it,
and then print out the buffer back on the main thread?
(This is Rust 1.15.1.)
This repository gives a complete main.rs and shows three examples for getting the webpage:
With no timeout.
With a timeout just on send.
With a timeout on the whole thing.
If you take out 3, it all compiles and runs.
What can I change about 3 to make that work too?
By the way, making a web request is really just the "occasion" for this question. I've already seen this question about doing a timeout. My own interest is not the timeout per se, but about how to fill up a buffer on one thread and read it on another.
Share mutable object between threads suggests using Arc and Mutex to safely share data between threads. Here is an attempt at that:
fn get_url_with_timeout_3(url: &str) -> Result<(u16, String), MyAppError> {
let (tx, rx) = mpsc::channel();
let url = url.to_owned();
let shbuf = Arc::new(Mutex::new(String::new()));
let shbuf2 = shbuf.clone();
let t = thread::spawn(move || {
let mut c = Client::new();
let mut req = c.get(&url);
let mut ret = match req.send() {
Ok(mut resp) => {
let mut buf2 = shbuf2.lock().unwrap();
match resp.read_to_string(&mut buf2) {
Ok(_) => Ok(resp.status.to_u16()),
Err(e) => Err(MyAppError::ReadError(e)),
}
}
Err(e) => Err(MyAppError::TcpError(e)),
};
match tx.send(ret) {
Ok(()) => {} // everything good
Err(_) => {} // we have been released, no biggie
}
});
match rx.recv_timeout(Duration::from_millis(5000)) {
Ok(maybe_status_code) => {
let buf2 = shbuf.lock().unwrap();
Ok((maybe_status_code?, *buf2))
}
Err(_) => Err(MyAppError::Timeout),
}
}
This gives me the error cannot move out of borrowed content for trying to return *buf2 (which makes sense, since it would be leaking data out of the Mutex), but I'm not sure how to express this in a structure where Rust can see the pattern.
If the request thread times out, it holds the lock forever — but I never try to read the buffer, so who cares. If the request thread finishes, it releases the lock and goes away, and the main thread will hold the only reference. I can reason that it is safe, but I'm not sure how to convince the compiler.
I'm not allowed to answer this question since it is supposedly a duplicate, but I've added 3 working implementations to my GitHub repository using the ideas from the comments.
I am reading a string from a file, splitting it by lines into a vector and then I want to do something with the extracted lines in separate threads. Like this:
use std::fs::File;
use std::io::prelude::*;
use std::thread;
fn main() {
match File::open("data") {
Ok(mut result) => {
let mut s = String::new();
result.read_to_string(&mut s);
let k : Vec<_> = s.split("\n").collect();
for line in k {
thread::spawn(move || {
println!("nL: {:?}", line);
});
}
}
Err(err) => {
println!("Error {:?}",err);
}
}
}
Of course this throws an error, because s will go out of scope before the threads are started:
s` does not live long enough
main.rs:9 let k : Vec<_> = s.split("\n").collect();
^
What can I do now? I've tried many things like Box or Arc, but I couldn't get it working. I somehow need to create a copy of s which also lives in the threads. But how do I do that?
The problem, fundamentally, is that line is a borrowed slice into s. There's really nothing you can do here, since there's no way to guarantee that each line will not outlive s itself.
Also, just to be clear: there is absolutely no way in Rust to "extend the lifetime of a variable". It simply cannot be done.
The simplest way around this is to go from line being borrowed to owned. Like so:
use std::thread;
fn main() {
let mut s: String = "One\nTwo\nThree\n".into();
let k : Vec<String> = s.split("\n").map(|s| s.into()).collect();
for line in k {
thread::spawn(move || {
println!("nL: {:?}", line);
});
}
}
The .map(|s| s.into()) converts from &str to String. Since a String owns its contents, it can be safely moved into each thread's closure, and will live independently of the thread that created it.
Note: you could do this in nightly Rust using the new scoped thread API, but that is still unstable.
I read a file given as an argument, but when I try to pass it to handle_client in a task at the bottom so that it can be written to the tcp stream when someone connects I get error: capture of moved value: html... what am I missing?
fn get_file_string(path_str: &String) -> String{
let path = Path::new(path_str.as_bytes());
let file = File::open(&path);
let mut reader = BufferedReader::new(file);
reader.read_to_string().unwrap()
}
fn main() {
let listener = TcpListener::bind("127.0.0.1:8001");
let mut acceptor = listener.listen();
let ref file_to_host = os::args()[1];
let html = get_file_string(file_to_host).clone();
fn handle_client(mut stream: TcpStream, html: String) {
let write = stream.write_str(html.as_slice());
}
for stream in acceptor.incoming() {
match stream {
Err(e) => { println!("{}", e) }
Ok(stream) => spawn(proc() {
handle_client(stream,html)
})
}
}
}
html is a String. Just like everything in Rust, a String is owned in exactly one place (if it satisfied Copy, it would be able to just duplicate it implicitly, but as it involves a heap allocation it’s definitely not). At present, you’re passing html to the handle_client function by value; therefore, when you call handle_client(stream, html), both stream and html are moved into that function and are no longer accessible. In the case of stream, that doesn’t matter as it’s a variable from inside the loop, but html comes from outside the loop; if it let you do it, it would take it the first time and work fine, but then it would be freed; second time through the loop, you would have an invalid String being passed through.
The solution in this case, seeing as you’re passing it through spawn and so can’t pass a reference (the slice, &str) is to clone the value so that that value can be moved into the proc and into the handle_client call:
Ok(stream) => {
let html = html.clone();
spawn(proc() {
handle_client(stream, html)
})
}