I've been learning rust, and now I'm beginning to write my own learning examples.
I've been trying to create a simple wrapper (producer/consumer) for rabbitMq (The library project is https://github.com/rabbitmq/rabbitmq-stream-rust-client).
The issue is that I keep getting the error that Stream is not found.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: StreamDoesNotExist { stream: "loan_request" }
My code for the main is the following:
mod consumer;
use tokio;
use consumer::{Consumer, BaseCallback};
#[tokio::main]
async fn main() {
let loan_request = BaseCallback {data: "{loan_id: 1}".to_string()};
let consumer_response = Consumer::set_event_callback("loan_request", loan_request).await;
}
My code to create a consumer, and a queue is the following:
use rabbitmq_stream_client::{Environment};
use serde_json;
use serde::{Serialize, Deserialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct BaseCallback {
pub data: String
}
pub struct Consumer {
environment: Environment,
callback: BaseCallback
}
impl Consumer {
async fn new_connection() -> Environment {
let environment_instance = Environment::builder()
.host("localhost")
.port(5552)
.build()
.await.unwrap();
environment_instance
}
pub async fn set_event_callback(event_name: &str, callback: BaseCallback) {
let connection = Self::new_connection().await;
let consumer = connection.consumer();
consumer
.build(event_name)
.await
.unwrap();
}
}
Any advice on what I'm doing wrong is greatly appreciated.
To anyone wondering, one first needs to create the queue on RabbitMq, for that use the client like this
let environment = Environment::builder()
.host("localhost")
.port(5552)
.build()
.await?;
environment
.stream_creator()
.create("loan_request")
.await
Then we can implement the consumer as above to consume, and handle events in the "loan_request" queue.
Related
I want to write a FFI wrapper for sn_api library, which contains async functions. It will be used in single-threaded non-async code written in Red.
I found, that the easy way is to use Runtime::new().unwrap().block_on(...) in every exported function, although it involves a lot of creating new Tokio runtimes and seem to be too heavy to be run on every call:
use std::os::raw::c_char;
use std::ffi::{CString, CStr};
use sn_api::{BootstrapConfig, Safe};
use tokio::runtime::Runtime;
#[no_mangle]
pub extern "C" _safe_connect(ptr: *const Safe, bootstrap_contact: *const c_char) {
assert!(!ptr.is_null());
let _safe = unsafe {
&*ptr
};
let bootstrap_contact = unsafe {
CStr::from_ptr(bootstrap_contact)
}
let mut bootstrap_contacts = BootstrapConfig::default();
bootstrap_contacts.insert(bootstrap_contact.parse().expect("Invalid bootstrap address"));
// how to reuse the Runtime in other functions?
Runtime::new().unwrap().block_on(_safe.connect(None, None, Some(bootstrap_contacts)));
}
Is it possible to run all async functions on a common Runtime? I imagine it would require creating some singleton / global, but my library is compiled with crate-type = ["cdylib"], which seems not a good place for globals. What would be the best approach?
I've decided for an approach, where I create a Tokio Runtime, and then pass it to every FFI function call containing async code:
use std::os::raw::c_char;
use std::ffi::{CString, CStr};
use sn_api::{BootstrapConfig, Safe};
use tokio::runtime::Runtime;
#[no_mangle]
pub extern "C" fn init_runtime() -> *mut Runtime {
Box::into_raw(Box::new(Runtime::new().unwrap()))
}
#[no_mangle]
pub extern "C" _safe_connect(rt_ptr: *mut Runtime, safe_ptr: *mut Safe, bootstrap_contact: *const c_char) {
assert!(!safe_ptr.is_null());
assert!(!rt_ptr.is_null());
let bootstrap_contact = unsafe {
CStr::from_ptr(bootstrap_contact)
}
let mut bootstrap_contacts = BootstrapConfig::default();
bootstrap_contacts.insert(bootstrap_contact.parse().expect("Invalid bootstrap address"));
unsafe {
let _safe = &mut *safe_ptr;
let rt = &mut *rt_ptr;
rt.block_on(_safe.connect(None, None, Some(bootstrap_contacts))).unwrap();
}
}
I faced the same issue. Here is my cut: export-tokio-to-lib.
plugin.rs:
use async_ffi::{FfiFuture, FutureExt};
use tokio::runtime::Handle;
/// # Safety
#[no_mangle]
pub unsafe extern "C" fn test(arg: f32, handle: *const Handle) -> FfiFuture<safer_ffi::String> {
let handle = &*handle;
async move {
let _enter = handle.enter();
tokio::time::sleep(std::time::Duration::from_secs_f32(arg)).await;
format!("slept {arg} secs").into()
}
.into_ffi()
}
Try this.
From this:
#[tokio::main]
async fn main() {
println!("hello");
}
Transformed into:
fn main() {
let mut rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async {
println!("hello");
})
}
Reference: https://tokio.rs/tokio/tutorial/hello-tokio#async-main-function
when running code like this:
use futures::executor;
...
pub fn store_temporary_password(email: &str, password: &str) -> Result<(), Box<dyn Error>> {
let client = DynamoDbClient::new(Region::ApSoutheast2);
...
let future = client.put_item(input);
executor::block_on(future)?; <- crashes here
Ok(())
}
I get the error:
thread '<unnamed>' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime
My main has the tokio annotation as it should:
#[tokio::main]
async fn main() {
...
My cargo.toml looks like:
[dependencies]
...
futures = { version="0", features=["executor"] }
tokio = "1"
My cargo.lock shows that i only have 1 version of both futures and tokio ("1.2.0" and "0.3.12" respectively).
This exhausts the explanations I found elsewhere for this problem. Any ideas? Thanks.
You have to enter the tokio runtime context before calling block_on:
let handle = tokio::runtime::Handle::current();
handle.enter();
executor::block_on(future)?;
Note that your code is violating the rule that async functions should never spend a long time without reaching a .await. Ideally, store_temporary_password should be marked as async to avoid blocking the current thread:
pub async fn store_temporary_password(email: &str, password: &str) -> Result<(), Box<dyn Error>> {
...
let future = client.put_item(input);
future.await?;
Ok(())
}
If that is not an option, you should wrap any calls to store_temporary_password in tokio::spawn_blocking to run the blocking operation on a separate threadpool.
I am trying to implement tokio::sync::RwLock because my app should re-load a configuration file if/when it changes. This involves re-assigning a Vec variable with a new one, but there could be many readers at any one time.
There is a struct called Manifest that represents a "list" of binaries available for a piece of hardware that I designed. The list is a JSON file which is parsed at startup and whenever it changes.
use notify::{watcher, RecursiveMode, Watcher};
use semver::Version;
use serde::{Deserialize, Serialize};
use std::sync::mpsc::channel;
use std::time::Duration;
use std::{fs::File, sync::Arc, thread};
use tokio::sync::RwLock;
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Binary {
version: Version,
hardware: Version,
requires: Version,
file: String,
}
pub struct Manifest {
path: String,
manifest_changed: bool,
binaries: Arc<RwLock<Vec<Binary>>>,
}
impl Manifest {
pub fn new(path: String) -> Manifest {
let bins = Arc::new(RwLock::new(Vec::new()));
thread::spawn(move || async move {
let bins = Arc::clone(&bins);
let (tx, rx) = channel();
let mut watcher = watcher(tx, Duration::from_secs(10)).unwrap();
watcher.watch(path, RecursiveMode::NonRecursive).unwrap();
loop {
match rx.recv() {
Ok(event) => {
let file = File::open(path).unwrap();
let mut bins = bins.write().await;
*bins = serde_json::from_reader(file).unwrap();
}
Err(e) => println!("watch error: {:?}", e),
}
}
});
Manifest {
path: path,
manifest_changed: true,
binaries: bins,
}
}
pub async fn GetAvailableBinaries(self, hwv: Version, fwv: Version) -> Vec<Binary> {
self.binaries.read().await.to_vec();
}
}
My issue is that the compiler complains:
future cannot be sent between threads safely
future created by async block is not `Send`
help: the trait `std::marker::Sync` is not implemented for `std::sync::mpsc::Receiver<notify::DebouncedEvent>`rustc
manifest.rs(28, 9): future created by async block is not `Send`
manifest.rs(35, 23): has type `&std::sync::mpsc::Receiver<notify::DebouncedEvent>` which is not `Send`
manifest.rs(38, 40): await occurs here, with `rx` maybe used later
manifest.rs(44, 13): `rx` is later dropped here
mod.rs(617, 8): required by this bound in `std::thread::spawn`
I'm not sure I understand why it matters if the channel Receiver implements Send because it's being created in the closure/thread. Further that that reasoning, I really don't know what to look for.
I have a future which wraps a TCP stream in a Framed using the LinesCodec.
When I try to wrap this in a test, I get the future blocking around 20% of the time, but because I have nothing listening on the socket I'm trying to connect to, I expect to always get the error:
thread 'tokio-runtime-worker-0' panicked at 'error: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }', src/lib.rs:35:24 note: Run with 'RUST_BACKTRACE=1' for a backtrace.
This is the test code I have used:
#[macro_use(try_ready)]
extern crate futures; // 0.1.24
extern crate tokio; // 0.1.8
use std::io;
use std::net::SocketAddr;
use tokio::codec::{Framed, LinesCodec};
use tokio::net::TcpStream;
use tokio::prelude::*;
struct MyFuture {
addr: SocketAddr,
}
impl Future for MyFuture {
type Item = Framed<TcpStream, LinesCodec>;
type Error = io::Error;
fn poll(&mut self) -> Result<Async<Framed<TcpStream, LinesCodec>>, io::Error> {
let strm = try_ready!(TcpStream::connect(&self.addr).poll());
Ok(Async::Ready(Framed::new(strm, LinesCodec::new())))
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::Shutdown;
#[test]
fn connect() {
let addr: SocketAddr = "127.0.0.1:4222".parse().unwrap();
let fut = MyFuture { addr: addr }
.and_then(|f| {
println!("connected");
let cn = f.get_ref();
cn.shutdown(Shutdown::Both)
}).map_err(|e| panic!("error: {:?}", e));
tokio::run(fut)
}
}
playground
I have seen patterns in other languages where the test binary itself offers a mechanism to return results asynchronously, but haven't found a good way of using a similar mechanism in Rust.
A simple way to test async code may be to use a dedicated runtime for each test: start it, wait for future completion and shutdown the runtime at the end of the test.
#[test]
fn my_case() {
// setup future f
// ...
tokio::run(f);
}
I don't know if there are consolidated patterns already in the Rust ecosystem; see this discussion about the evolution of testing support for future based code.
Why your code does not work as expected
When you invoke poll(), the future is queried to check if a value is available.
If a value is not available, an interest is registered so that poll() will be invoked again when something happens that can resolve the future.
When your MyFuture::poll() is invoked:
TcpStream::connect creates a new future TcpStreamNew
TcpStreamNew::poll is invoked immediately only once on the future's creation at step 1.
The future goes out of scope, so the next time you invoke MyFuture::poll you never resolve the previously created futures.
You have registered an interest for a future that, if not resolved the first time you poll it, you never ask back again (poll) for a resolved value or for an error.
The reason of the "nondeterministic" behavior is because the first poll sometimes resolve immediately with a ConnectionRefused error and sometimes it waits forever for a future connection event or a failure that it is never retrieved.
Look at mio::sys::unix::tcp::TcpStream used by Tokio:
impl TcpStream {
pub fn connect(stream: net::TcpStream, addr: &SocketAddr) -> io::Result<TcpStream> {
set_nonblock(stream.as_raw_fd())?;
match stream.connect(addr) {
Ok(..) => {}
Err(ref e) if e.raw_os_error() == Some(libc::EINPROGRESS) => {}
Err(e) => return Err(e),
}
Ok(TcpStream {
inner: stream,
})
}
When you connect on a non-blocking socket, the system call may connect/fail immediately or return EINPROGRESS, in this last case a poll must be triggered for retrieving the value of the error.
The issue is not with the test but with the implementation.
This working test case based on yours has no custom future implementation and only calls TcpStream::connect(). It works as you expect it to.
extern crate futures;
extern crate tokio;
#[cfg(test)]
mod tests {
use super::*;
use std::net::Shutdown;
use std::net::SocketAddr;
use tokio::net::TcpStream;
use tokio::prelude::*;
#[test]
fn connect() {
let addr: SocketAddr = "127.0.0.1:4222".parse().unwrap();
let fut = TcpStream::connect(&addr)
.and_then(|f| {
println!("connected");
f.shutdown(Shutdown::Both)
}).map_err(|e| panic!("error: {:?}", e));
tokio::run(fut)
}
}
playground
You are connecting to the same endpoint over and over again in your poll() method. That's not how a future works. The poll() method will be called repeatedly, with the expectation that at some point it will return either Ok(Async::Ready(..)) or Err(..).
If you initiate a new TCP connection every time poll() is called, it will be unlikely to complete in time.
Here is a modified example that does what you expect:
#[macro_use(try_ready)]
extern crate futures;
extern crate tokio;
use std::io;
use std::net::SocketAddr;
use tokio::codec::{Framed, LinesCodec};
use tokio::net::{ConnectFuture, TcpStream};
use tokio::prelude::*;
struct MyFuture {
tcp: ConnectFuture,
}
impl MyFuture {
fn new(addr: SocketAddr) -> MyFuture {
MyFuture {
tcp: TcpStream::connect(&addr),
}
}
}
impl Future for MyFuture {
type Item = Framed<TcpStream, LinesCodec>;
type Error = io::Error;
fn poll(&mut self) -> Result<Async<Framed<TcpStream, LinesCodec>>, io::Error> {
let strm = try_ready!(self.tcp.poll());
Ok(Async::Ready(Framed::new(strm, LinesCodec::new())))
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::Shutdown;
#[test]
fn connect() {
let addr: SocketAddr = "127.0.0.1:4222".parse().unwrap();
let fut = MyFuture::new(addr)
.and_then(|f| {
println!("connected");
let cn = f.get_ref();
cn.shutdown(Shutdown::Both)
}).map_err(|e| panic!("error: {:?}", e));
tokio::run(fut)
}
}
I'm not certain what you intend your future to do, though; I can't comment if it's the right approach.
to some degree, you can drop in tokio's test library to make this easier; it supports async/await in unit tests.
#[tokio::test]
async fn my_future_test() {
let addr: SocketAddr = "127.0.0.1:4222".parse().unwrap();
match MyFuture { addr }.poll().await {
Ok(f) => assert!("something good")
Err(e) => assert!("something bad")
}
}
https://docs.rs/tokio/0.3.3/tokio/attr.test.html
I am using iron. Most of time like 99.* % all is good. But sometimes I get error like Error was: ErrorImpl { code: EofWhileParsingString/List/Object, line: 1, column: 8186 } or InvalidUnicodeCodePoint. I am printing request in log and when i try that request every thing goes well. I also have server written in Golang receiving same request and they never have parsing or json to MyStruct conversion problem.Please note Code would not compile as it is, missing imports, error::from and structure definition. Can not provide reproducible request logs as it only happens when serving lots on concurrent request but if single request is taken it works fine.
I have tried serde_json::from_reader, bodyparser crate and all have same issue.
extern crate serde;
extern crate serde_json;
extern crate iron;
use self::iron;
use self::iron::prelude::*;
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct MyStruct {
}
struct ResponseTime;
impl typemap::Key for ResponseTime {
type Value = u64;
}
#[derive(Debug)]
struct RequestBody;
impl typemap::Key for RequestBody {
type Value = RefCell<Vec<u8>>;
}
impl BeforeMiddleware for ResponseTime {
fn before(&self, req: &mut Request) -> IronResult<()> {
req.extensions.insert::<RequestBody>(RefCell::new(Vec::new()));
req.extensions.insert::<ResponseTime>(precise_time_ns());
Ok(())
}
}
impl AfterMiddleware for ResponseTime {
fn after(&self, req: &mut Request, res: Response) -> IronResult<Response> {
Ok(res)
}
fn catch(&self, req : &mut Request, err : IronError) -> IronResult<Response> {
let ref byte_req = *req.extensions.get::<RequestBody>()
.unwrap()
.borrow();
//just to make sure uft8 is not causing some issue.
let payload = unsafe {
str::from_utf8_unchecked(&byte_req)
};
//but when i send request body all comes good
error!("Error {} for Body {}", err, payload);
Err(err)
}
}
fn iron_handler(req : &mut Request) -> Result<Response, CustomError>{
let mut buffer = req.extensions.get::<server::RequestBody>()
.unwrap()
.borrow_mut();
req.body.read_to_end(&mut buffer)?;
// not seeing InvalidUnicodeCodePoint after this.
let payload = String::from_utf8_lossy(&buffer);
//some request throw error
let my_struct_obj : MyStruct = serde_json::from_str(&payload)?;
Ok(Response::with((iron::status::Ok, "Final Response")))
}
Need help to figure out how to identify problem. Intent of posting here is to see if someone had same issue or can see obvious problem with this. Appreciate everyone'e time do not expect to build and run with examples as can not provide them because of privacy.