I am launching multiple threads from incoming tcp connections which need to read from a file or anything that implements the std::io::Read and std::io::Seek trait.
I have the following working implementation (1):
fn upload<R: Read + Seek + Send>(file: &mut R, listen_addr: SocketAddr) -> IoResult<()> {
let listener = TcpListener::bind(listen_addr)?;
let file: Arc<Mutex<&mut R>> = Arc::new(Mutex::new(file));
std::thread::scope(|scope| {
for socket in listener.incoming() {
let socket = socket.unwrap() // error handling omitted ..
let f = file.clone();
scope.spawn(move || {
start_upload_task(socket, f).unwrap();
});
}
}
Ok(())
}
fn start_upload_task<R: Read + Seek>(conn: TcpStream, file: Arc<Mutex<&mut R>>) -> IoResult<()>{
// do something wtih file ..
let mut buf = vec![0u8; 42];
{
let mut file = file.lock().unwrap();
file.seek(SeekFrom::Start(42))?;
file.read_exact(&mut buf)?;
}
Ok(())
}
However, since I want to use wasm_bindgen_futures::spawn_local, the future must be 'static and there is no equivalent to std::thread::scope in rust webassembly that I know of.
What I end up with is the following (2):
fn upload<R: Read + Seek + Send>(file: &mut R, listen_addr: SocketAddr) -> IoResult<()> {
let listener = TcpListener::bind(listen_addr)?;
let file: Arc<Mutex<&mut R>> = Arc::new(Mutex::new(file));
for socket in listener.incoming() {
let socket = socket.unwrap() // error handling omitted ..
let f = file.clone();
std::thread::spawn(move || {
start_upload_task(socket, f).unwrap();
});
}
Ok(())
}
But this gives me the following compile error:
borrowed data escapes outside of associated function
requirement occurs because of the type Mutex<&mut R>, which makes the generic argument &mut R invariant
the struct Mutex<T> is invariant over the parameter T
see https://doc.rust-lang.org/nomicon/subtyping.html for more information about variance rustcE0521
lib.rs(258, 47): file is a reference that is only valid in the associated function body
with the following suggestion:
--> src/lib.rs:273:25
|
258 | fn upload<R: Read + Seek + Send>(&self, file: &mut R, listen_addr: SocketAddr) -> IoResult<()> {
| ---- - let's call the lifetime of this reference `'1`
| |
| `file` is a reference that is only valid in the associated function body
How do I get implementation (2) to work?
Related
let stream = device.build_input_stream(
&config.into(),
move |data, _: &_| {
let b = write_input_data::<f32, f32>(data);
println!("b: {}", b.len());
},
|err| {
eprintln!("an error occurred on stream: {}", err);
},
None
).unwrap();
fn write_input_data<T, U>(input: &[T]) -> Vec<U>
where
T: Sample,
U: Sample + From<T> + hound::Sample + std::fmt::Debug,
{
let mut buffer = Vec::new();
for sample in input.iter() {
buffer.push(U::from(*sample));
}
return buffer;
}
Problem:
I want to access all the data that write_input_data creates. As the closure has a "move" keyword, I'm not able to create any vector in main function and access the data later.
Things I tried:
Removing "move" keyword
Creating a vector in the main function to which I would later push the data(variable called b)
Pass a reference to the write_input_data (useless, move keyword also moves the ownership)
Is this even possible?
let v = Arc::new(Mutex::new(Vec::new()));
let v2 = v.clone();
let stream = device.build_input_stream(
&config.into(),
move |data, _: &_| {
let b = write_input_data::<f32, f32>(data);
println!("b len: {}", b.len());
let mut vec = v2.lock().unwrap();
vec.extend(b);
},
|err| {
eprintln!("an error occurred on stream: {}", err);
},
None
).unwrap();
stream.play();
std::thread::sleep(Duration::from_millis(100));
let mut vec = v.lock().unwrap();
println!("vec len: {}", vec.len());
drop(stream);
}
fn write_input_data<T, U>(input: &[T]) -> Vec<U>
where
T: Sample,
U: Sample + From<T> + hound::Sample,
{
let mut buffer = Vec::new();
for sample in input.iter() {
buffer.push(U::from(*sample));
}
return buffer;
}
To everyone having the same issue in the future, this was the answer. It's important to have two instances of a vector ( v.clone() ) as the variable ownership is moved in the |move| function, but both v and v2 point to the same inner value. Arc and Mutex, although I don't have a perfect understanding of, allow for the variable to be written by both main function & write_input_data.
I'm new to Rust and still reading the Rust book. Below is my program.
use clap::{App, Arg};
type GenericError = Box<dyn std::error::Error + Send + Sync + 'static>;
type GenericResult<T> = Result<T, GenericError>;
fn main() -> GenericResult<()> {
let matches = App::new("test")
.arg(Arg::new("latency")
.takes_value(true))
.get_matches();
let latency_present = matches.is_present("latency");
let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
let latency_pairs: Vec<&str> = latency.split(",").collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
checker.join().unwrap()?;
Ok(())
}
When I run it, it tells me this:
error[E0597]: `matches` does not live long enough
--> src\main.rs:14:19
|
14 | let latency = matches.value_of("latency").unwrap_or("1000:10,300:30");
| ^^^^^^^--------------------
| |
| borrowed value does not live long enough
| argument requires that `matches` is borrowed for `'static`
...
30 | }
| - `matches` dropped here while still borrowed
I don't quite understand the error messages here. But I guess it's because I use latency_pairs in the checker thread and latency_pairs could get dropped while checker is still executing. Is my understanding correct? How to fix the error? I tried for (i, latency_pair) in latency_pairs.clone().iter().enumerate() { in order to pass a cloned value for the thread, but it doesn't help.
latency_pairs holds references into latency which in turn references matches. Thus cloning latency_pairs just clones the references into latency and matches.
Your code would require that latency's type is &'static str but it's actually &'a str where 'a is bound to matches' lifetime.
You can call to_owned() on latency to get an owned value and split the string inside the closure or you can call to_owned() on each of the splits collected in latency_pairs and move that Vec<String> into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
If you need to use latency_pairs outside of the closure, you can clone it before moving it into the closure:
let latency_pairs: Vec<String> = latency.split(",").map(ToOwned::to_owned).collect();
let latency_pairs_ = latency_pairs.clone();
let checker = std::thread::spawn(move || -> GenericResult<()>{
loop {
if latency_present {
for (i, latency_pair) in latency_pairs_.iter().enumerate() {
// let latency_pair: Vec<&str> = latency_pair.split(":").collect();
// let latency = latency_pair[0].parse::<f64>().unwrap();
}
}
}
});
println!("{:?}", latency_pairs);
I've been trying to return a stream, as I've done with tokio::fs::File however I am getting a lifetime error on the BlobClient.
error[E0597]: `blob` does not live long enough
--> src\main.rs:20:27
|
20 | let stream = Box::pin(blob.get().stream(128));
| ^^^^^^^^^^
| |
| borrowed value does not live long enough
| argument requires that `blob` is borrowed for `'static`
...
24 | }
| - `blob` dropped here while still borrowed
I've tried a bunch of different ways of handling the stream but I can't navigate around this lifetime error. I'm sure it might be something simple that I just keep overlooking. Thanks for any assistance.
Here's a repo of what I'm trying to do:
use std::{convert::Infallible, net::SocketAddr};
use azure_core::new_http_client;
use azure_storage::{
blob::prelude::{AsBlobClient, AsContainerClient},
clients::{AsStorageClient, StorageAccountClient},
};
use futures::TryStreamExt;
use hyper::{
service::{make_service_fn, service_fn},
Body, Request, Response, Server,
};
async fn handle(_req: Request<Body>) -> Result<Response<Body>, Infallible> {
let http_client = new_http_client();
let storage_account_client = StorageAccountClient::new_access_key(http_client.clone(), "account", "key");
let storage_client = storage_account_client.as_storage_client();
let blob = storage_client.as_container_client("container").as_blob_client("blob");
let stream = Box::pin(blob.get().stream(128));
let s = stream.and_then(|f| futures::future::ok(f.data));
Ok(Response::new(Body::wrap_stream(s)))
}
#[tokio::main]
async fn main() {
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
let make_service = make_service_fn(|_conn| async { Ok::<_, Infallible>(service_fn(handle)) });
let server = Server::bind(&addr).serve(make_service);
if let Err(e) = server.await {
eprintln!("server error: {}", e);
}
}
The problem is that the stream borrows from blob, but the wrap_stream() function only accepts 'static streams. A workaround is to construct the stream in a new task, and send back the stream items through a channel. The following helper function helps implement this approach:
/// Creates a `'static` stream from a closure returning a (possibly) non-`'static` stream.
///
/// The stream items, closure, and closure argument are still restricted to being `'static`,
/// but the closure can return a non-`'static` stream that borrows from the closure
/// argument.
fn make_static_stream<T, F, U>(
make_stream: F,
mut make_stream_arg: U,
) -> impl Stream<Item = T>
where
T: Send + 'static,
F: FnOnce(&mut U) -> BoxStream<'_, T> + Send + 'static,
U: Send + 'static,
{
let (mut tx, rx) = futures::channel::mpsc::channel(0);
tokio::spawn(async move {
let stream = make_stream(&mut make_stream_arg);
pin_mut!(stream);
while let Some(item) = stream.next().await {
if tx.feed(item).await.is_err() {
// Receiver dropped
break;
}
}
tx.close().await.ok();
});
rx
}
Here's how you would use it in the original code:
// ...
let stream = make_static_stream(
|blob| blob.get().stream(128).map_ok(|x| x.data).boxed(),
blob,
);
Ok(Response::new(Body::wrap_stream(stream)))
}
This question already has answers here:
How to use take_while with futures::Stream?
(2 answers)
Closed 3 years ago.
I want to run an event loop in one thread and handle data from a UDP socket until another thread signals to stop work.
This is a difficult task for me, so I want to start from a simpler task:
one thread starting the event loop and waiting for another thread to signal the end:
use futures::{future, future::Future, stream::Stream, sync::mpsc};
use std::{io, io::BufRead, thread};
fn main() {
let (mut tx, rx) = mpsc::channel::<bool>(1);
let thr = thread::spawn(|| {
let mut runtime = tokio::runtime::current_thread::Runtime::new().unwrap();
runtime.spawn(
future::lazy(|| {
println!("event loop started");
Ok(())
})
.and_then(rx.take_while(|x| *x == true).into_future()),
);
runtime.run()
});
let stdin = io::stdin();
for line in stdin.lock().lines() {
let line = line.unwrap();
println!("{}", line);
if line == "exit" {
tx.try_send(false).unwrap();
break;
}
}
thr.join().unwrap().unwrap();
}
This code doesn't compile:
error[E0277]: the trait bound `bool: futures::future::Future` is not satisfied
--> src/main.rs:14:26
|
14 | .and_then(rx.take_while(|x| *x == true).into_future()),
| ^^^^^^^^^^ the trait `futures::future::Future` is not implemented for `bool`
|
= note: required because of the requirements on the impl of `futures::future::IntoFuture` for `bool`
error[E0599]: no method named `into_future` found for type `futures::stream::take_while::TakeWhile<futures::sync::mpsc::Receiver<bool>, [closure#src/main.rs:14:37: 14:51], bool>` in the current scope
--> src/main.rs:14:53
|
14 | .and_then(rx.take_while(|x| *x == true).into_future()),
| ^^^^^^^^^^^
|
= note: the method `into_future` exists but the following trait bounds were not satisfied:
`futures::stream::take_while::TakeWhile<futures::sync::mpsc::Receiver<bool>, [closure#src/main.rs:14:37: 14:51], bool> : futures::stream::Stream`
`&mut futures::stream::take_while::TakeWhile<futures::sync::mpsc::Receiver<bool>, [closure#src/main.rs:14:37: 14:51], bool> : futures::stream::Stream`
How do I fix the compilation error?
Read and understand the documentation and function signature of methods you attempt to use:
fn take_while<P, R>(self, pred: P) -> TakeWhile<Self, P, R>
where
P: FnMut(&Self::Item) -> R,
R: IntoFuture<Item = bool, Error = Self::Error>,
Self: Sized,
take_while takes a closure that returns some type that must be convertible into a future; a bool is not convertible into a future. The simplest way to do this is via future::ok:
let thr = thread::spawn(|| {
let mut runtime = tokio::runtime::current_thread::Runtime::new().unwrap();
runtime.spawn({
rx.take_while(|&x| future::ok(x))
.for_each(|x| {
println!("{}", x);
future::ok(())
})
});
runtime.run()
});
See also:
The trait bound `(): futures::Future` is not satisfied when using TcpConnectionNew
But my problem also in joining future::lazy and rx.take_while
That's an unrelated problem to what you asked about. Again, we look at the docs, this time for Future::and_then:
fn and_then<F, B>(self, f: F) -> AndThen<Self, B, F>
where
F: FnOnce(Self::Item) -> B,
B: IntoFuture<Error = Self::Error>,
Self: Sized,
Similarly to take_while, it takes a closure and the closure must return something that can be convertible into a future. Your code doesn't provide a closure.
Then look at Stream::into_future. This returns a type that implements Future and returns a tuple. The first item in the tuple is a single value from the stream, the second is the stream itself, to allow getting more values.
To get all the item and error types correct, I've make liberal use of map(drop) and map_err(drop) — you will want to do something better for your data and error handling.
runtime.spawn({
future::lazy(|| {
println!("event loop started");
Ok(())
})
.and_then(|_| {
rx.take_while(|&x| future::ok(x))
.into_future()
.map(drop)
.map_err(drop)
})
.map(drop)
});
Really, you should just use a oneshot channel; it's much simpler:
use futures::{
future::{self, Future},
sync::oneshot,
};
use std::thread;
fn main() {
let (tx, rx) = oneshot::channel();
let thr = thread::spawn(|| {
let mut runtime = tokio::runtime::current_thread::Runtime::new().unwrap();
runtime.spawn({
future::lazy(|| {
println!("event loop started");
Ok(())
})
.and_then(|_| rx.map_err(drop))
});
runtime.run()
});
let lines = ["hello", "goodbye", "exit"];
for &line in &lines {
if line == "exit" {
tx.send(()).unwrap();
break;
}
}
thr.join().unwrap().unwrap();
}
I am writing a Phoenix client library for Rust, taking advantage of the async websocket client from rust-websockets. Right now I am having trouble figuring out how to pass callback functions into the thread that is handling the websocket traffic. I have a simplified struct:
pub struct Socket {
endpoint: String,
connected: Arc<AtomicBool>,
state_change_close: Option<Box<FnMut(String)>>,
}
This struct has a connect function laid out as follows:
pub fn connect(&mut self) -> Result<(), String> {
if self.connected.load(Ordering::Relaxed) {
return Ok(())
}
// Copy endpoint string, otherwise we get an error on thread::spawn
let connection_string = self.endpoint.clone();
let (usr_msg, stdin_ch) = mpsc::channel(0);
let connection_thread = thread::spawn(move || {
// tokio core for running event loop
let mut core = Core::new().unwrap();
let runner = ClientBuilder::new(&connection_string)
.unwrap()
.add_protocol("rust-websocket")
.async_connect_insecure(&core.handle())
.and_then(|(duplex, _)| {
let (sink, stream) = duplex.split();
stream.filter_map(|message| {
println!("Received Message: {:?}", message);
match message {
OwnedMessage::Close(e) => {
// This is the line where I am trying to call the callback
if let Some(ref mut func) = self.state_change_close {
(func)(e.unwrap().reason);
}
Some(OwnedMessage::Close(e))
},
_ => None,
}
})
.select(stdin_ch.map_err(|_| WebSocketError::NoDataAvailable))
.forward(sink)
});
// Start the event loop
core.run(runner).unwrap();
});
self.connected.store(true, Ordering::Relaxed);
return Ok(())
}
When I try to compile this code I get the following error:
error[E0277]: the trait bound `std::ops::FnMut(std::string::String) + 'static: std::marker::Send` is not satisfied
--> src\socket.rs:99:29
|
99 | let connection_thread = thread::spawn(move || {
| ^^^^^^^^^^^^^ the trait `std::marker::Send` is not implemented for `std::ops::FnMut(std::string::String) + 'static`
|
I have tried changing the type of state_change_close to a Mutex<Option<...>> to avoid thread safety issues, but that did not help with this problem. Is what I'm trying to do possible?
After doing some more research I realized that I just had to modify Option<Box<FnMut(String)>> to be Option<Box<FnMut(String) + Send>> and copy that around my code to everywhere that the callback might be set. Learning more about trait objects!