I am trying to build a library defining a generic datasource which can pull data from various sources, synchronously and asynchronously.
When building the async piece, I ran into the following compilation issue which I don't understand how to address:
Here is my simplified code (playground link)
extern crate futures; // futures = "0.1.24"
extern crate tokio; // tokio = "0.1.8"
extern crate serde_json;
use futures::Future;
use serde_json::Value;
use std::collections::HashMap;
trait DataSource {
type Data;
fn read_async(&self, Option<HashMap<String, Value>>) -> Box<futures::Future<Item=Option<Self::Data>, Error=String>>
where Self::Data: 'static + Send;
}
struct DataSourceImpl;
impl DataSource for DataSourceImpl {
type Data = Vec<String>;
fn read_async(&self, _params: Option<HashMap<String, Value>>) -> Box<futures::Future<Item=Option<Self::Data>, Error=String>>
where Self::Data: 'static + Send
{
Box::new(futures::future::ok(Some(vec!["some data".to_string()])))
}
}
fn main() {
let datasource = DataSourceImpl{};
let params = HashMap::new();
tokio::run(datasource.read_async(Some(params))
.map(|content| {
println!("Content read = {:?}", &content);
()
})
.map_err(|err| {
println!("Error {}", &err);
()
})
);
}
I got the following compilation error:
error[E0277]: `dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>` cannot be sent between threads safely
--> src/main.rs:45:13
|
45 | runtime.spawn(future);
| ^^^^^ `dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>` cannot be sent between threads safely
|
= help: the trait `std::marker::Send` is not implemented for `dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>`
= note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>>`
= note: required because it appears within the type `std::boxed::Box<dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>>`
= note: required because it appears within the type `futures::Map<std::boxed::Box<dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>>, [closure#src/main.rs:34:14: 37:10]>`
= note: required because it appears within the type `futures::MapErr<futures::Map<std::boxed::Box<dyn futures::Future<Item=std::option::Option<std::vec::Vec<std::string::String>>, Error=std::string::String>>, [closure#src/main.rs:34:14: 37:10]>, [closure#src/main.rs:38:18: 41:10]>`
Yet when looking into the standard library, I found the following implementations:
impl<T: ?Sized> Send for Box<T> where T: Send
impl<T> Send for Option<T> where T: Send
impl<T> Send for Vec<T> where T: Send
impl Send for String
impl Send for [failure::]Error
What am I missing?
If I get rid of the trait and replace the Box<Future<...>> by impl Future<...> then it works (playground link for the new code); yet I don't understand what is wrong with the trait & Box implementation...
extern crate failure;
extern crate futures; // futures = "0.1.24"
extern crate tokio; // tokio = "0.1.8"
extern crate serde_json;
use futures::Future;
use serde_json::Value;
use std::collections::HashMap;
fn read_async(_params: Option<HashMap<String, Value>>) -> impl futures::Future<Item=Option<Vec<String>>, Error=failure::Error> {
futures::future::ok(Some(vec!["some data".to_string()]))
}
fn main() {
let params = HashMap::new();
let future = read_async(Some(params))
.map(|content| {
println!("Content read = {:?}", &content);
()
})
.map_err(|err| {
println!("Error {}", &err);
()
});
tokio::run(future);
}
Looks like I just needed to change the function signature to
fn read_async(
&self,
_: Option<HashMap<String, Value>>,
) -> Box<Future<Item = Option<Self::Data>, Error = String> + Send> {
// ^^^^^^^
Indeed, Box<T> should be Send when T is Send, but I have to spell it out because Future doesn't derive it manually/automatically.
Thanks to Tobz for pointing this out.
Related
My intention is receiving events through WebSockets and use them on the closures of main.
This works when the messages are pure text (String), but the idea is deserializing that text into some structs.
In this example I've added only Data, Error and Event, but in other cases, it could be different, so I've used generics to do that, but I'm a little lost. The compiler has suggested several things that I've tried, but I don't know how to "force" that the message is casted into a specific type (Data in this example, but EventManager could be used on other parts, so it should be generic).
I've attached this code, that tries to show my idea, although it doesn't compile:
events.rs:
use actix::*;
use actix_web::ws::{Client, Message, ProtocolError};
use futures::Future;
use serde::de;
use serde_json::from_str;
struct MyActor<T> {
manager: EventManager<T>,
}
impl<T: 'static> Actor for MyActor<T> {
type Context = Context<Self>;
}
impl<T: 'static> StreamHandler<Message, ProtocolError> for MyActor<T> {
fn handle(&mut self, msg: Message, _ctx: &mut Context<Self>) {
match msg {
Message::Text(text) => {
debug!("Received {}", text);
for idx in 0..self.manager.events.len() {
let data =
from_str(&text).expect(&format!("Error when deserializing {:?}", text));
(self.manager.events[idx].handler)(data)
}
}
_ => panic!(),
}
}
}
pub struct Event<T> {
handler: Box<Fn(T) + 'static>,
}
pub struct EventManager<T> {
events: Vec<Event<T>>,
}
impl<T: 'static> EventManager<T>
where
T: serde::Deserialize<'static>,
{
pub fn new() -> Self {
Self { events: vec![] }
}
pub fn capture<F>(&mut self, function: F)
where
F: for<'h> Fn(T) + 'static,
{
let event = Event {
handler: Box::new(function),
};
self.events.push(event);
}
pub fn run(self) {
let runner = System::new("example");
debug!("run");
Arbiter::spawn(
Client::new("example")
.connect()
.map(|(reader, _writer)| {
MyActor::create(|ctx| {
MyActor::add_stream(reader, ctx);
MyActor { manager: self }
});
})
.map_err(|err| {}),
);
runner.run();
}
}
main.rs:
#[macro_use]
extern crate log;
extern crate actix;
extern crate actix_web;
extern crate env_logger;
extern crate futures;
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;
pub mod events;
use actix::*;
use serde::de;
use serde::de::{Deserialize, Deserializer};
use events::EventManager;
#[derive(Debug, Message, Serialize, Deserialize)]
#[serde(untagged)]
pub enum Data {
Error(Error),
Event(Event),
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Error {
message: String,
code: String,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Event {
name: String,
content: String,
}
fn main() {
env_logger::init();
let mut client = EventManager::<Data>new();
client.capture(|data| debug!("event: {:?}", data));
client.run();
}
All the code could be see in https://github.com/foochi/how-deserialize-within-actix
There are some fixes to get it compile.
The trick to get it compile is to use Higher-Rank Trait Bounds (HTRB) trait bounds instead of declaring 'static lifetimes.
Follow the compiler suggestion and bind the T: serde::Deserialize<'_> trait:
impl<T> StreamHandler<Message, ProtocolError> for MyActor<T>
where
for<'de> T: serde::Deserialize<'de> + 'static,
Then change also the Deserialize<'static> trait bound associated to theEventManager impl with a HTRB trait bound to get it compatible with the requirement of StreamHandler implementation:
impl<T: 'static> EventManager<T>
where
for<'de> T: serde::Deserialize<'de>,
Finally, if you correct the line the create the client with the right sintax:
let mut client: EventManager<Data> = EventManager::new();
the example code should compile.
Note: For capture the use of a Higher Trait Bound for declaring the Fn requirement is redundant, do simply:
pub fn capture<F>(&mut self, function: F)
where
F: Fn(T) + 'static,
I have the following code based on the actix-web Database Integration sample.
extern crate actix;
extern crate actix_web;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
extern crate r2d2;
extern crate r2d2_mysql;
use actix::prelude::*;
use actix_web::{middleware::Logger, server, App, FutureResponse, HttpRequest, HttpResponse};
mod dbservices {
use actix::prelude::*;
use actix_web::dev::Handler;
use model::DataDictionary;
use r2d2::Pool;
use r2d2_mysql::*;
use std::io;
pub struct MysqlConnection {
db_pool: mysql::PooledConn,
}
impl Actor for MysqlConnection {
type Context = SyncContext<Self>;
}
impl MysqlConnection {
pub fn new(db_url: &str) -> MysqlConnection {
unimplemented!();
}
}
pub struct GetDD;
impl Message for GetDD {
type Result = io::Result<Vec<DataDictionary>>;
}
impl Handler<GetDD> for MysqlConnection {
type Result = io::Result<Vec<DataDictionary>>;
fn handle(&mut self, msg: GetDD, _: &mut Self::Context) -> Self::Result {
unimplemented!();
}
}
}
mod model {
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DataDictionary {
id: i32,
name: String,
}
}
struct State {
db: Addr<MysqlConnection>,
}
fn get_dd(req: HttpRequest<State>) -> FutureResponse<HttpResponse> {
req.clone()
.state()
.db
.send(GetDD)
.from_err()
.and_then(|result| Ok.json(result))
.responder()
}
fn main() {
std::env::set_var("RUST_LOG", "actix_web=debug,info");
const db_url: str = "mysql://connstring";
let addr = SyncArbiter::start(3, move || dbservices::MysqlConnection::new(db_url));
server::new(|| {
App::new()
.middleware(Logger::default())
.middleware(Logger::new("%a %{User-Agent}i"))
.prefix("/api")
.scope("/dd", |dp_scope| {
dp_scope.resource("/", |r| r.h(dbservices::GetDD()))
})
}).bind("127.0.0.1:8088")
.unwrap()
.run();
}
I'm getting the following error when compiling. I'm not sure what I'm doing wrong:
error[E0220]: associated type `Context` not found for `Self`
--> src/main.rs:43:50
|
43 | fn handle(&mut self, msg: GetDD, _: &mut Self::Context) -> Self::Result {
| ^^^^^^^^^^^^^ associated type `Context` not found
Here are my dependencies from Cargo.toml:
[dependencies]
actix-web = "0.6.14"
actix = "0.6.1"
chrono = { version = "0.4.2", features = ["serde"] }
serde = "1.0.60"
serde_derive = "1.0.60"
serde_json = "1.0.17"
log = "0.4.1"
env_logger ="0.5.10"
futures = "0.2.1"
r2d2 = "*"
[dependencies.r2d2_mysql]
git = "https://github.com/outersky/r2d2-mysql"
version="*"
Creating a minimal MCVE almost always makes the problem easier to see:
extern crate actix;
extern crate actix_web;
use actix::prelude::*;
use actix_web::dev::Handler;
use std::io;
pub struct MysqlConnection;
impl Actor for MysqlConnection {
type Context = SyncContext<Self>;
}
pub struct DummyMessage;
impl Message for DummyMessage {
type Result = io::Result<String>;
}
impl Handler<DummyMessage> for MysqlConnection {
type Result = io::Result<String>;
fn handle(&mut self, _: DummyMessage, _: &mut Self::Context) -> Self::Result {
unimplemented!();
}
}
fn main() {}
error[E0220]: associated type `Context` not found for `Self`
--> src/main.rs:22:53
|
22 | fn handle(&mut self, _: DummyMessage, _: &mut Self::Context) -> Self::Result {
| ^^^^^^^^^^^^^ associated type `Context` not found
Problem 1 — multiple versions
actix-web = "0.6.14"
actix = "0.6.1"
This brings in two different versions of actix:
$ cargo tree -d
actix v0.5.8
└── actix-web v0.6.14
└── repro v0.1.0 (file:///private/tmp/repro)
actix v0.6.1
└── repro v0.1.0 (file:///private/tmp/repro)
Pick a single one, actix 0.5.8.
See also:
Why is a trait not implemented for a type that clearly has it implemented?
Problem 2 — wrong trait
You bring in actix_web::dev::Handler:
pub trait Handler<S>: 'static {
type Result: Responder;
fn handle(&mut self, req: HttpRequest<S>) -> Self::Result;
}
You are implementing actix::Handler:
pub trait Handler<M>
where
Self: Actor,
M: Message,
{
type Result: MessageResponse<Self, M>;
fn handle(&mut self, msg: M, ctx: &mut Self::Context) -> Self::Result;
}
actix::Handler has Actor as its supertrait, which means that it can access the associated type Context. actix_web::dev::Handler does not have the supertrait, so it doesn't know about Self::Context.
Pick the appropriate trait and implement it correctly.
You can replace Self::Context with SyncContext<Self>
This is actually strange since there is only single associated type Context but for some reason Rust wants you to specify specifically associated type: <Self as Actor>::Context which should not be required since you have single Context type
Are there any other errors that prevent your struct from compiling?
I'm trying to use a struct created in main() and pass it on to a function that returns a boxed Future. However, I run into lifetime and borrowing issues and can't seem to resolve this cleanly.
Here is my struct and functions:
extern crate futures; // 0.1.21
extern crate tokio_core; // 0.1.17
use futures::{future::ok, Future};
pub struct SomeStruct {
some_val: u32,
}
impl SomeStruct {
pub fn do_something(&self, value: u32) -> u32 {
// Do some work
return self.some_val + value;
}
}
fn main() {
let core = tokio_core::reactor::Core::new().unwrap();
let my_struct = SomeStruct { some_val: 10 };
let future = get_future(&my_struct);
core.run(future);
let future2 = get_future(&my_struct);
core.run(future2);
}
fn get_future(some_struct: &SomeStruct) -> Box<Future<Item = u32, Error = ()>> {
let fut = ok(20).and_then(|val| {
let result = some_struct.do_something(val);
ok(result)
});
Box::new(fut)
}
On compiling, the following error occurs:
error[E0621]: explicit lifetime required in the type of `some_struct`
--> src/main.rs:33:5
|
28 | fn get_future(some_struct: &SomeStruct) -> Box<Future<Item = u32, Error = ()>> {
| ----------- consider changing the type of `some_struct` to `&'static SomeStruct`
...
33 | Box::new(fut)
| ^^^^^^^^^^^^^ lifetime `'static` required
I suppose the error occurs because SomeStruct is used in the Future and might be used outside of main()s scope, hence the compiler asks me to change the lifetime to 'static. Here is what I tried so far (unsuccessfully):
Changing the lifetime to 'static as suggested by the compiler, which creates borrowing issues in main().
Moving val by adding ok(20).and_then(move |val| { as suggested by the compiler, which creates issues in the second invocation of get_future().
Use the lazy_static crate to explicitly initialize SomeStruct as static (as suggested here), however I run into macro errors when trying that.
The whole example can be found here. I have omitted some details to create an minimal example. The issues occur using tokio-core and futures = "0.1". Migrating to version "0.2" is not an option unfortunately, due to a dependency of another library.
Returning a boxed trait object has a 'static bound by default. Do as the compiler suggests and provide an explicit lifetime, but not 'static:
fn get_future<'a>(some_struct: &'a SomeStruct) -> Box<dyn Future<Item = u32, Error = ()> + 'a> {
let fut = future::ok(20).and_then(move |val| {
let result = some_struct.do_something(val);
future::ok(result)
});
Box::new(fut)
}
You also have to use move to transfer ownership of some_struct to the closure and change core to be mutable. You should also handle potential errors resulting from core.run.
For the example provided, you could also return impl Future:
fn get_future<'a>(some_struct: &'a SomeStruct) -> impl Future<Item = u32, Error = ()> +'a {
future::ok(20).and_then(move |val| {
let result = some_struct.do_something(val);
future::ok(result)
})
}
See also:
How do I return a boxed closure from a method that has a reference to the struct?
Why is adding a lifetime to a trait with the plus operator (Iterator<Item = &Foo> + 'a) needed?
What is the correct way to return an Iterator (or any other trait)?
I'm using futures, tokio, hyper, and serde_json to request and deserialize some data that I need to hold until my next request. My initial thought was to make a struct containing the hyper::Chunk and the deserialized data that borrows from the Chunk, but couldn't get the lifetimes right. I tried using the rental crate, but I can't get this to work either. Perhaps I'm using the 'buffer lifetime before declaring the buffer Vec, but maybe I've messed something else up:
#[rental]
pub struct ChunkJson<T: serde::de::Deserialize<'buffer>> {
buffer: Vec<u8>,
json: T
}
Is there some way to make the lifetimes right or should I just use DeserializeOwned and give up on zero-copy?
For more context, the following code works (periodically deserializing JSON from two URLs, retaining the results so we can do something with them both). I'd like to change my X and Y types to use Cow<'a, str> for their fields, changing from DeserializeOwned to Deserialize<'a>. For this to work, I need to store the slice that has been deserialized for each, but I don't know how to do this. I'm looking for examples that use Serde's zero-copy deserialization and retain the result, or some idea for restructuring my code that would work.
#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate serde_json;
extern crate futures;
extern crate tokio_core;
extern crate tokio_periodic;
extern crate hyper;
use std::collections::HashMap;
use std::error::Error;
use futures::future;
use futures::Future;
use futures::stream::Stream;
use hyper::Client;
fn stream_json<'a, T: serde::de::DeserializeOwned + Send + 'a>
(handle: &tokio_core::reactor::Handle,
url: String,
period: u64)
-> Box<Stream<Item = T, Error = Box<Error>> + 'a> {
let client = Client::new(handle);
let timer = tokio_periodic::PeriodicTimer::new(handle).unwrap();
timer
.reset(::std::time::Duration::new(period, 0))
.unwrap();
Box::new(futures::Stream::zip(timer.from_err::<Box<Error>>(), futures::stream::unfold( (), move |_| {
let uri = url.parse::<hyper::Uri>().unwrap();
let get = client.get(uri).from_err::<Box<Error>>().and_then(|res| {
res.body().concat().from_err::<Box<Error>>().and_then(|chunks| {
let p: Result<T, Box<Error>> = serde_json::from_slice::<T>(chunks.as_ref()).map_err(|e| Box::new(e) as Box<Error>);
match p {
Ok(json) => future::ok((json, ())),
Err(err) => future::err(err)
}
})
});
Some(get)
})).map(|x| { x.1 }))
}
#[derive(Serialize, Deserialize, Debug)]
pub struct X {
foo: String,
}
#[derive(Serialize, Deserialize, Debug)]
pub struct Y {
bar: String,
}
fn main() {
let mut core = tokio_core::reactor::Core::new().unwrap();
let handle = core.handle();
let x_stream = stream_json::<HashMap<String, X>>(&handle, "http://localhost/X".to_string(), 2);
let y_stream = stream_json::<HashMap<String, Y>>(&handle, "http://localhost/Y".to_string(), 5);
let mut xy_stream = x_stream.merge(y_stream);
let mut last_x = HashMap::new();
let mut last_y = HashMap::new();
loop {
match core.run(futures::Stream::into_future(xy_stream)) {
Ok((Some(item), stream)) => {
match item {
futures::stream::MergedItem::First(x) => last_x = x,
futures::stream::MergedItem::Second(y) => last_y = y,
futures::stream::MergedItem::Both(x, y) => {
last_x = x;
last_y = y;
}
}
println!("\nx = {:?}", &last_x);
println!("y = {:?}", &last_y);
// Do more stuff with &last_x and &last_y
xy_stream = stream;
}
Ok((None, stream)) => xy_stream = stream,
Err(_) => {
panic!("error");
}
}
}
}
When trying to solve a complicated programming problem, it's very useful to remove as much as you can. Take your code and remove what you can until the problem goes away. Tweak your code a bit and keep removing until you can't any more. Then, turn the problem around and build from the smallest piece and work back to the error. Doing both of these will show you where the problem lies.
First, let's make sure we deserialize correctly:
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
use std::borrow::Cow;
#[derive(Debug, Deserialize)]
pub struct Example<'a> {
#[serde(borrow)]
name: Cow<'a, str>,
key: bool,
}
impl<'a> Example<'a> {
fn info(&self) {
println!("{:?}", self);
match self.name {
Cow::Borrowed(_) => println!("Is borrowed"),
Cow::Owned(_) => println!("Is owned"),
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let decoded: Example = serde_json::from_slice(&data).expect("Couldn't deserialize");
decoded.info();
}
Here, I forgot to add the #[serde(borrow)] attribute, so I'm glad I did this test!
Next, we can introduce the rental crate:
#[macro_use]
extern crate rental;
rental! {
mod holding {
use super::*;
#[rental]
pub struct VecHolder {
data: Vec<u8>,
parsed: Example<'data>,
}
}
}
fn main() {
let data: Vec<_> = br#"{"key": true, "name": "alice"}"#.to_vec();
let holder = holding::VecHolder::try_new(data, |data| {
serde_json::from_slice(data)
});
let holder = match holder {
Ok(holder) => holder,
Err(_) => panic!("Unable to construct rental"),
};
holder.rent(|example| example.info());
// Make sure we can move the data and it's still valid
let holder2 = { holder };
holder2.rent(|example| example.info());
}
Next we try to create a rental of Chunk:
#[rental]
pub struct ChunkHolder {
data: Chunk,
parsed: Example<'data>,
}
Unfortunately, this fails:
--> src/main.rs:29:1
|
29 | rental! {
| ^
|
= help: message: Field `data` must have an angle-bracketed type parameter or be `String`.
Oops! Checking the docs for rental, we can add #[target_ty_hack="[u8]"] to the data field. This leads to:
error[E0277]: the trait bound `hyper::Chunk: rental::__rental_prelude::StableDeref` is not satisfied
--> src/main.rs:29:1
|
29 | rental! {
| ^ the trait `rental::__rental_prelude::StableDeref` is not implemented for `hyper::Chunk`
|
= note: required by `rental::__rental_prelude::static_assert_stable_deref`
That's annoying; since we can't implement that trait for Chunk, we just need to box Chunk, proving that it has a stable address:
#[rental]
pub struct ChunkHolder {
data: Box<Chunk>,
parsed: Example<'data>,
}
I also looked to see if there is a way to get a Vec<u8> back out of Chunk, but it doesn't appear to exist. That would have been another solution with less allocation and indirection.
At this point, "all" that's left is to integrate this back into the futures code. It's a lot of work for anyone but you to recreate that, but I don't foresee any obvious problems in doing so.
I'm trying to create a handler for Iron requests:
extern crate iron;
extern crate mount;
use iron::{Iron, Request, Response, IronResult, status};
use mount::Mount;
use iron::middleware::Handler;
struct Server {
message: String
}
impl Server {
pub fn start(&self){
let mut mount = Mount::new();
mount.mount("/", &self);
Iron::new(mount).http("0.0.0.0:3000").unwrap();
}
}
impl Handler for Server {
fn handle(&self, _req: &mut Request) -> IronResult<Response>{
Ok(Response::with((status::Ok, self.message)))
}
}
fn main() {
Server{message: "test".to_string()}.start();
}
but compiler response is:
error[E0277]: the trait bound `for<'r, 'r, 'r> Server: std::ops::Fn<(&'r mut iron::Request<'r, 'r>,)>` is not satisfied
--> src/main.rs:15:15
|
15 | mount.mount("/", &self);
| ^^^^^ trait `for<'r, 'r, 'r> Server: std::ops::Fn<(&'r mut iron::Request<'r, 'r>,)>` not satisfied
|
= note: required because of the requirements on the impl of `std::ops::FnOnce<(&mut iron::Request<'_, '_>,)>` for `&Server`
= note: required because of the requirements on the impl of `iron::Handler` for `&&Server`
I was unable to understand what Rust is saying to me.
Here's a reproduction of your issue; can you spot the problem?
trait Foo {}
struct Bar;
impl Foo for Bar {}
impl Bar {
fn thing(&self) {
requires_bar(self);
}
}
fn requires_bar<F>(foo: F) where F: Foo {}
fn main() {}
Give up?
You've implemented the trait for your struct:
impl Handler for Server
But are then trying to use a reference to a reference to your struct, which does not implement the trait:
pub fn start(&self) {
// ...
mount.mount("/", &self);
// ...
}
So that cannot work. You need to restructure your code or implement the trait for a reference to your struct.