I want to send a hashmap over a channel:
use std::collections::HashMap;
use std::sync::mpsc;
use std::thread;
pub fn frequency(input: &[&str]) {
let handles = vec![];
let (tx, rx) = mpsc::channel();
for txts in input {
let handle = thread::spawn(|| {
tx.send(HashMap::new());
});
handles.push(handle);
}
}
but the compiler won't let me:
error[E0277]: `Sender<HashMap<_, _>>` cannot be shared between threads safely
--> src/lib.rs:9:22
|
9 | let handle = thread::spawn(|| {
| ^^^^^^^^^^^^^ `Sender<HashMap<_, _>>` cannot be shared between threads safely
|
= help: the trait `Sync` is not implemented for `Sender<HashMap<_, _>>`
= note: required because of the requirements on the impl of `Send` for `&Sender<HashMap<_, _>>`
= note: required because it appears within the type `[closure#src/lib.rs:9:36: 11:10]`
note: required by a bound in `spawn`
even though there are Send and Sync implementations for HashMap.
How can I send a hashmap over a channel?
Related
I was designing a type that can be sent between threads or not, depending if it's an Owned type or not.
This code:
use std::cell::RefCell;
use std::rc::Rc;
pub enum M<'a, T> {
Owned(),
Ref(Rc<RefCell<&'a mut [T]>>)
}
fn main() {
let m: M<'static, u8> = M::Owned();
std::thread::spawn(||{
let m = m;
});
}
gives
error[E0277]: `Rc<RefCell<&mut [_]>>` cannot be sent between threads safely
--> src/main.rs:11:5
|
11 | std::thread::spawn(||{
| _____^^^^^^^^^^^^^^^^^^_-
| | |
| | `Rc<RefCell<&mut [_]>>` cannot be sent between threads safely
12 | | let m = m;
13 | | });
| |_____- within this `[closure#src/main.rs:11:24: 13:6]`
I understand that this is not Send because RefCell is !Send. I think this is the only thing preventing M from being Send. Is there a RefCell alternative that is Send?
I understand that this is not Send because RefCell is !Send. I think this is the only thing preventing M from being Send.
Yes, this is correct.
Is there a RefCell alternative that is Send?
Of course, at least two. If you just want the &mut in another thread, you can actually just send it:
#[derive(Debug)]
pub enum M<'a, T> {
Owned(),
Ref(&'a mut [T])
}
fn main() {
let m: M<'static, u8> = M::Owned();
std::thread::spawn(move ||{
println!("{:?}", m);
}).join().unwrap();
}
But if you use the Rc<RefCell<_>> to e.g. make your &mut sharable, you can achieve the same thing in the context of multithreading with Arc<Mutex<_>>, as #Locke already mentioned:
use std::sync::{Arc, Mutex};
#[derive(Debug, Clone)]
pub enum M<'a, T> {
Owned(),
Ref(Arc<Mutex<&'a mut [T]>>)
}
fn main() {
let m: M<'static, u8> = M::Owned();
let m2 = m.clone();
std::thread::spawn(move ||{
println!("{:?}", m2);
}).join().unwrap();
println!("{:?}", m);
}
I'm new to Rust and I'm sorry if I'm using terms incorrect. Maybe my choice of words for question is incorrect.
I was playing with streams and I needed to have some delay between stream elements. So I wrote this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
let mut stream = stream::iter(0..1000).then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
});
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}
I get a lot of compilation errors, but the most important errors are connected with pinning. Here they are:
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>` cannot be unpinned
--> src/main.rs:11:32
|
11 | while let Some(x) = stream.next().await {
| ^^^^ within `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>`
|
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `std::option::Option<impl core::future::future::Future>`
= note: required because it appears within the type `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>` cannot be unpinned
--> src/main.rs:11:25
|
11 | while let Some(x) = stream.next().await {
| ^^^^^^^^^^^^^^^^^^^ within `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:7:64: 10:6 x:_ _]>`
|
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because it appears within the type `std::option::Option<impl core::future::future::Future>`
= note: required because it appears within the type `futures_util::stream::stream::then::_::__Then<'_, futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>`
= note: required because of the requirements on the impl of `core::future::future::Future` for `futures_util::stream::stream::next::Next<'_, futures_util::stream::stream::then::Then<futures_util::stream::iter::Iter<std::ops::Range<{integer}>>, impl core::future::future::Future, [closure#src/main.rs:7:49: 10:6]>>`
If I change my code to this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
let mut stream = stream::iter(0..1000).then(|x| {
futures::future::ready(x + 1)
});
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}
Or to this:
use futures::stream;
use futures::StreamExt;
use tokio::time;
#[tokio::main]
async fn main() {
stream::iter(0..1000)
.then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
})
.for_each(|x| async move { println!("{:?}", x) })
.await;
}
It compiles.
I assume it has something to do with pinning and simultaneous usage of then combinator and while, but I can't wrap my head around it.
I think that the issue boils down to the fact that async blocks are not Unpin. It can be proven with this code:
fn check_unpin<F: Unpin>(_: F) { }
fn main() {
check_unpin(async {});
}
that fails with the rather cryptic message:
error[E0277]: `std::future::from_generator::GenFuture<[static generator#src/main.rs:4:23: 4:25 _]>` cannot be unpinned
--> src/main.rs:4:5
|
1 | fn check_unpin<F: Unpin>(_: F) { }
| ----- required by this bound in `check_unpin`
...
4 | check_unpin(async {});
| ^^^^^^^^^^^ within `impl std::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::from_generator::GenFuture<[static generator#src/main.rs:4:23: 4:25 _]>`
|
= note: required because it appears within the type `impl std::future::Future`
I reckon that GenFuture is the internal type that converts an async block into a impl Future.
Now back to your issue, the combinator then() returns an Unpin value if both the stream and the Future returned by the closure are Unpin (it is not totally clear in the documentation, but I've inferred that from the source code). The stream::iter is Unpin, but when you write |x| async move { x + 1} you are returning an async block that is not Unpin, thus your error.
If you use futures::future::ready(x + 1) it works simply because that Future implements Unpin.
If you use StreamExt::for_each it works because it does not require Self to be Unpin. It is not Unpin itself, but it does not matter because you are sending it up to tokio::main that pins everything internally before polling.
If you want your original code to work you just have to manually pin your stream (playground):
use futures::stream;
use futures::StreamExt;
use tokio::time;
use pin_utils::pin_mut;
#[tokio::main]
async fn main() {
let stream = stream::iter(0..1000).then(|x| async move {
time::delay_for(std::time::Duration::from_millis(500)).await;
x + 1
});
pin_mut!(stream); //<---- here, pinned!
while let Some(x) = stream.next().await {
println!("{:?}", x)
}
}
I'm unable to work out how to resolve the compilation error resulting from adapting a stream using a combinator.
The following Rust Playground demonstrates a fairly minimal example:
use futures::prelude::*;
use futures::StreamExt;
#[derive(Debug)]
pub enum Input {
A,
B(i32),
C(u16),
}
#[derive(Debug)]
enum Output {
Int(i32),
Short(u16),
}
pub struct StreamMaker;
impl StreamMaker {
/// make a stream with a series of inputs
pub fn create(self) -> impl Stream<Item = Input> {
stream::iter(vec![Input::A, Input::C(1u16), Input::B(2)])
}
}
/// consume the creator, and make output messages for a subset
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> {
let mut upstream = creator.create();
upstream.filter_map(|message| async move {
match message {
Input::A => None,
Input::B(v) => Some(Output::Int(v)),
Input::C(v) => Some(Output::Short(v)),
}
})
}
#[tokio::main]
async fn main() -> Result<(), ()> {
let creator = StreamMaker {};
let mut stream = adapt_stream(creator);
while let Some(message) = stream.next().await {
println!("message: {:?}", message)
}
Ok(())
}
Compilation fails with:
error[E0277]: the trait bound `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>: std::marker::Unpin` is not satisfied in `impl core::future::future::Future`
--> src/main.rs:43:38
|
43 | while let Some(message) = stream.next().await {
| ^^^^ within `impl core::future::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>`
|
= help: the following implementations were found:
<std::future::GenFuture<T> as std::marker::Unpin>
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::filter_map::FilterMap<impl futures_core::stream::Stream, impl core::future::future::Future, [closure#src/main.rs:29:25: 35:6]>`
= note: required because it appears within the type `impl futures_core::stream::Stream`
error[E0277]: the trait bound `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>: std::marker::Unpin` is not satisfied in `impl core::future::future::Future`
--> src/main.rs:43:31
|
43 | while let Some(message) = stream.next().await {
| ^^^^^^^^^^^^^^^^^^^ within `impl core::future::future::Future`, the trait `std::marker::Unpin` is not implemented for `std::future::GenFuture<[static generator#src/main.rs:29:46: 35:6 message:Input {}]>`
|
= help: the following implementations were found:
<std::future::GenFuture<T> as std::marker::Unpin>
= note: required because it appears within the type `impl core::future::future::Future`
= note: required because of the requirements on the impl of `std::marker::Unpin` for `futures_util::stream::stream::filter_map::FilterMap<impl futures_core::stream::Stream, impl core::future::future::Future, [closure#src/main.rs:29:25: 35:6]>`
= note: required because it appears within the type `impl futures_core::stream::Stream`
= note: required because of the requirements on the impl of `core::future::future::Future` for `futures_util::stream::stream::next::Next<'_, impl futures_core::stream::Stream>`
I can put a pin_mut!(stream); in the main, but I'd like to be able to push that upstream.
If you do not want the consumer of your stream to have to pin it themselves, you need to return a stream that implements the Unpin trait, meaning that it is safe to move around in memory even after is has been pinned.
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> + Unpin {
// add Unpin trait --^
Adding this, your compiler should complain that the return value doesn't implement Unpin. This is because async move { ... } blocks don't implement Unpin, since they may be self-referential (e.g. contain references to variables they own). The most general way to work around this is to pin the stream to the heap with a Pin<Box<_>>, using the Box::pin constructor:
pub fn adapt_stream(creator: StreamMaker) -> impl Stream<Item = Output> + Unpin {
let mut upstream = creator.create();
Box::pin(upstream.filter_map(|message| async move {
// ^-- pin stream to heap
match message {
Input::A => None,
Input::B(v) => Some(Output::Int(v)),
Input::C(v) => Some(Output::Short(v)),
}
}))
}
Since we're now returning a Pin<Box<_>> pointer to the stream, that pointer can be safely moved around in memory while the inner stream is kept at the same location.
Full playground example
Here's a simple application that duplicates 2 times on stdout the contents of stdin:
use std::{
io,
io::{stdin, stdout, Read, Write},
num::NonZeroUsize,
};
fn dup_input(
input: &mut Box<dyn Read>,
output: &mut Box<dyn Write>,
count: NonZeroUsize,
) -> io::Result<()> {
let mut buf = Vec::new();
input.read_to_end(&mut buf)?;
for _idx in 0..count.get() {
output.write_all(&buf)?;
}
Ok(())
}
fn main() {
let mut input: Box<dyn Read> = Box::new(stdin());
let mut output: Box<dyn Write> = Box::new(stdout());
dup_input(&mut input, &mut output, NonZeroUsize::new(2).unwrap())
.expect("Failed to duplicate input");
}
This part works fine. I want to put a unit test on top of this and this is where the problem lies. The closest I've got to build is with the following attempt:
#[cfg(test)]
mod tests {
use super::*;
use std::{any::Any, io::Cursor};
#[test]
fn test() {
let mut input: Box<dyn Read> = Box::new(Cursor::new([b't', b'e', b's', b't', b'\n']));
let mut output: Box<dyn Write + Any> = Box::new(Vec::<u8>::new());
assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
assert_eq!(output.downcast::<Vec<u8>>().unwrap().len(), 15);
}
}
but rust 1.41.0 doesn't agree:
$ cargo test
Compiling unbox-example v0.1.0 (/XXX/unbox-example)
error[E0225]: only auto traits can be used as additional traits in a trait object
--> src/main.rs:39:41
|
39 | let mut output: Box<dyn Write + Any> = Box::new(Vec::<u8>::new());
| ----- ^^^
| | |
| | additional non-auto trait
| | trait alias used in trait object type (additional use)
| first non-auto trait
| trait alias used in trait object type (first use)
error[E0308]: mismatched types
--> src/main.rs:41:39
|
41 | assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
| ^^^^^^^^^^^ expected trait `std::io::Write`, found a different trait `std::io::Write`
|
= note: expected mutable reference `&mut std::boxed::Box<(dyn std::io::Write + 'static)>`
found mutable reference `&mut std::boxed::Box<(dyn std::io::Write + 'static)>`
error[E0599]: no method named `downcast` found for type `std::boxed::Box<(dyn std::io::Write + 'static)>` in the current scope
--> src/main.rs:43:27
|
43 | assert_eq!(output.downcast::<Vec<u8>>().unwrap().len(), 15);
| ^^^^^^^^ method not found in `std::boxed::Box<(dyn std::io::Write + 'static)>`
error: aborting due to 3 previous errors
Some errors have detailed explanations: E0225, E0308, E0599.
For more information about an error, try `rustc --explain E0225`.
error: could not compile `unbox-example`.
To learn more, run the command again with --verbose.
Is there a way to change the unit test without changing the main code? Note: I could have used generics while implementing dup_input and make the problem significantly easier to solve but this code is part of a broader application and I have to use Read/Write trait objects.
Passing a mutable reference to a Box to dup_input is unnecessarily complicated. You can simply pass a mutable reference to the trait object.
use std::{
io,
io::{stdin, stdout, Read, Write},
num::NonZeroUsize,
};
fn dup_input(
input: &mut dyn Read,
output: &mut dyn Write,
count: NonZeroUsize,
) -> io::Result<()> {
let mut buf = Vec::new();
input.read_to_end(&mut buf)?;
for _idx in 0..count.get() {
output.write_all(&buf)?;
}
Ok(())
}
fn main() {
let mut input = stdin();
let mut output = stdout();
dup_input(&mut input, &mut output, NonZeroUsize::new(2).unwrap())
.expect("Failed to duplicate input");
}
With this version, the test can be written like this:
#[cfg(test)]
mod tests {
use super::*;
use std::io::Cursor;
#[test]
fn test() {
let mut input = Cursor::new([b't', b'e', b's', b't', b'\n']);
let mut output = Vec::<u8>::new();
assert!(dup_input(&mut input, &mut output, NonZeroUsize::new(3).unwrap()).is_ok());
assert_eq!(output.len(), 15);
}
}
We don't need to use Any at all here: output is simply a Vec<u8>.
I'd like to make the following code compile:
struct Provider {}
impl Provider {
fn get_string<'a>(&'a self) -> &'a str { "this is a string" }
}
fn main() {
let provider = Provider{};
let mut vec: Vec<&str> = Vec::new();
// PROBLEM: how do I say that this reference s here
// needs to live as long as vec?
let fun = |s: &str| {
vec.push(s);
};
fun(provider.get_string());
}
Playground link
This is the compile error that I get:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:9:22
|
9 | let mut vec: Vec<&str> = Vec::new();
| ^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the block at 11:24...
--> src/main.rs:11:25
|
11| let fun = |s: &str| {
| ^
note: ...so that reference does not outlive borrowed content
--> src/main.rs:12:18
|
12| vec.push(s);
| ^
note: but, the lifetime must be valid for the block suffix following statement 2 at 13:6...
--> src/main.rs:13:7
|
13| };
| ^
note: ...so that variable is valid at time of its declaration
--> src/main.rs:11:9
|
11| let fun = |s: &str| {
| ^^^
Your code works just fine if remove all the lifetime annotations and let the compiler inference do its job:
struct Provider;
impl Provider {
fn get_string(&self) -> &str { "this is a string" }
}
fn main() {
let provider = Provider;
let mut vec = Vec::new();
let mut fun = |s| {
vec.push(s);
};
fun(provider.get_string());
}
In short, there's no way to explicitly refer to the lifetime of a local variable, only function arguments. The compiler knows how to do it, though.
If you truly needed it, you could create a function to allow annotating the lifetimes:
fn thing<'a>(provider: &'a Provider) -> Vec<&'a str> {
let mut vec: Vec<&'a str> = Vec::new();
{
let mut fun = |s: &'a str| vec.push(s);
fun(provider.get_string());
} // End mutable borrow of `vec`
vec
}
fn main() {
let provider = Provider;
thing(&provider);
}
why did the original annotations stop things from working?
Specifically, it's this bit:
let fun = |s: &str| {
vec.push(s);
};
This declares a new lifetime on the closure. Using a made-up syntax (you can't declare lifetimes on closure arguments), it would be equivalent to:
let fun = <'a> |s: &'a str| {
vec.push(s);
};
Which is why the compiler has the error:
the lifetime cannot outlive the anonymous lifetime #1 defined on [the closure's block]
There's no connection between that generated lifetime and that of the Provider. Leaving it out allows the compiler to insert the desired but unnamable lifetime.
Here's a version which compiles:
use std::marker::PhantomData;
struct Provider<'a> {
_dummy: PhantomData<&'a ()>,
}
impl<'a> Provider<'a> {
fn get_string(&self) -> &'a str {
"this is a string"
}
}
fn f<'b>() {
let provider = Provider { _dummy: PhantomData };
let mut vec: Vec<&str> = Vec::new();
// PROBLEM: how do I say that this reference s here
// needs to live as long as vec?
let mut fun = |s: &'b str| { vec.push(s); };
fun(provider.get_string());
}
fn main() {
f()
}
Playground link
I made the following changes:
Add a lifetime to Provider (I added a PhantomData, but I guess your provider already owns some data it'll provide).
Update the get_string method to show that it returns something with the provider's lifetime, not the input lifetime (ie based on the Provider's lifetime parameter).
Add a new lifetime parameter 'b to the function (which I renamed to f(), since main() can't have one), which I use to name the lifetime of the closure parameter.
The last one is slightly confusing, as apparently merely adding a name to a lifetime (without apparently adding any constraints) has made it work.
I think (but I'd love some documentation for this) that this is because of lifetime elision. A closure is really a hidden struct with a fn call(&self, s: &str) (in this case) method. According to the lifetime elision rules, the s parameter gains the same lifetime as &self, which is the closure itself. In this case, the closure is declared after vec, so the lifetime is too short. The explicit lifetime means that it is decoupled from the closure's own lifetime.