I'm trying to implement rabbitmq send/listen functionality in Rust and I have the following code:
struct RabbitMQ {
connection: Connection,
}
impl RabbitMQ {
fn connect() -> Self {
RabbitMQ {
connection: the created connection
}
}
}
impl MessageBroker for RabbitMQ {
async fn publish(&self, topic: &Topic) -> Result<PublisherConfirm, Error> {
let channel = self.connection.create_channel().await.unwrap();
RabbitMQ::create_exchange(&channel, &topic.exchange).await;
let payload = topic.message.as_bytes();
let res = channel.basic_publish(
topic.exchange.name.as_str(),
topic.queue.routing_key.as_str(),
topic.exchange.publish_options,
payload.to_vec(),
BasicProperties::default(),
);
res.await
}
}
So far so good!
Now I want to publish many messages in a for loop without waiting for the confirmation from the server, the problem is that when I spawn tokio async task I need to move my broker value and this makes it invalid for the next iteration of the loop:
let broker = RabbitMQ::connect(&connection_details).await;
for x in 1..10 {
tokio::spawn(async move {
let confirm = broker.publish(&my_topic).await.unwrap();
}).await.unwrap();
}
The above code won't compile with the following error:
error[E0382]: use of moved value: `broker`
--> src/main.rs:47:33
|
21 | let broker = RabbitMQ::connect(&connection_details).await;
| ------ move occurs because `broker` has type `message_broker::RabbitMQ`, which >does not implement the `Copy` trait
...
47 | tokio::spawn(async move {
| _________________________________^
48 | | let confirm = &broker.publish(&enable_cdn).await.unwrap();
| | ------ use occurs due to use in generator
49 | | }).await.unwrap();
| |_________^ value moved here, in previous iteration of loop
I can't implement the Copy trait as Connection isn't primitive and it seems that I can't use reference "&" to the broker.
My question is how can I accomplish this without writing n publish calls?
You're using an async move block, which means any name which is used in the block is moved into the future, regardless of the operations being performed. So writing
&broker.publish
inside the block makes no difference: first broker is moved, and the future (when polled with .await) takes an internal reference to it. So what you need to do is borrow outside the block then move that borrow inside:
let broker = RabbitMQ::connect(&connection_details).await;
for x in 1..10 {
let broker = &broker;
tokio::spawn(async move {
let confirm = broker.publish(&enable_cdn).await.unwrap();
}).await.unwrap();
}
but I think that's not going to work either: tokio::spawn is not scoped, so even though you're await-ing it, the compiler has no idea that it will not outlive broker. As far as it's concerned a tokio task can live as long as it wants. This means you're now probably going to get a lifetime error (the compiler will assume the borrow can outlive the enclosing function, and thus its origin).
An easy solution to that would be to put the Connection behind an Arc or something.
Alternatively, restructure your system to work better with the requirements of rabbitmq: no idea which you're using but amiquip states that connections are thread-safe, and channels while not thread-safe can be sent to other threads.
So rather than publish-ing to an implicit connection, in each iteration of the loop create a channel and move that into the task in order to actually perform the publication.
Also,
Now I want to publish many messages in a for loop without waiting for the confirmation from the server
aren't you still doing that since you're awaiting the result of tokio::spawn?
Related
I implemented the tonic helloworld tutorial. I then tried to change the client code so that I could send multiple requests before awaiting any.
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let num_requests = 10;
let mut client = GreeterClient::connect("http://[::1]:50051").await?;
let mut responses = Vec::with_capacity(num_requests);
for _ in 0..num_requests {
let request = tonic::Request::new(HelloRequest {
name: "Tonic".into(),
});
responses.push(client.say_hello(request));
}
for resp in responses {
assert!(resp.await.is_ok());
}
Ok(())
}
This results in a compilation error:
error[E0499]: cannot borrow `client` as mutable more than once at a time
--> src/client.rs:19:24
|
19 | responses.push(client.say_hello(request));
| ^^^^^^ mutable borrow starts here in previous iteration of loop
Does that mean 'client.say_hello()' returns a type which still references client, and therefore I can't make another call to 'say_hello', which itself requires '&mut self'? Is there a way to continue to make requests before calling to 'await'?
From the Tonic documentation:
Sending a request on a channel requires a &mut self and thus can only send one request in flight. This is intentional and is required to follow the Service contract from the tower library which this channel implementation is built on top of.
...
To work around this and to ease the use of the channel, Channel provides a Clone implementation that is cheap. This is because at the very top level the channel is backed by a tower_buffer::Buffer which runs the connection in a background task and provides a mpsc channel interface. Due to this cloning the Channel type is cheap and encouraged.
Therefore, you can clone the client for each concurrent request you make. This eliminates the possibility of a single client being mutably borrowed more than once at any given time, so the borrow checker is appeased.
let num_requests = 10;
let client = GreeterClient::connect("http://[::1]:50051").await?;
let mut responses = Vec::with_capacity(num_requests);
for _ in 0..num_requests {
let mut client = client.clone();
let request = tonic::Request::new(HelloRequest {
name: "Tonic".into(),
});
responses.push(tokio::spawn(async move {
client.say_hello(request).await
}));
}
for resp in responses {
let resp = resp.await;
assert!(resp.is_ok());
assert!(resp.unwrap().is_ok());
}
I'm trying to implement asynchronous db access using tokio-postgres crate. Here is What I tried:
use tokio_postgres::{Client, NoTls, Error};
pub struct Database{
client: Mutex<Client>
}
impl Database {
pub async fn some_db_operation(&self, /* args */) -> Result<(), Error> {
let connection = &mut self.client.lock().expect("Mutex was poisoned");
let transaction = &mut connection.transaction().await?;
//executing some queries
Ok(())
}
}
The problem is that I want to access the database as a part of incoming http-request handling using warp and therefore everything should be Send. I got the following error:
--> src/db.rs:27:32
|
26 | let connection = &mut self.client.lock().expect("Mutex was poisoned");
| ----------------------------------------------- has type `std::sync::MutexGuard<'_, tokio_postgres::client::Client>` which is not `Send`
27 | let transaction = &mut connection.transaction().await?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ await occurs here, with `self.client.lock().expect("Mutex was poisoned")` maybe used later
Is there a workaround to make it Send?
There was no such a problem with a synchronous client.
MutexGuard (the result of calling lock() on a Mutex) is not Send. The relevant part in the docs is the manual implementation of !Send ("not Send").
I would strongly suggest to look into some sort of connection pooling, perhaps with deadpool_postgres which is built around tokio-postgres. Using a single client behind a mutex will likely tank your async performance anyway.
I'm working on a simple Rust program that reads and parses network packets. For reading the network packets I'm using the pnet libary.
Because the parsing may take some time I'm using two separate threads for reading and parsing the packets.
My idea now was to pass the read packages from the first thread to the second thread via message passing (using mpsc::channel()).
Here's a simplified version of my code which I wrote based on the example given in the pnet doc:
extern crate pnet;
use std::sync::mpsc;
use std::thread;
use pnet::datalink;
use pnet::datalink::Channel::Ethernet;
fn main() {
let (sender, receiver) = mpsc::channel();
thread::spawn(move || {
for packet in receiver.recv() {
println!("{:?}", packet)
}
});
let interface = datalink::interfaces().into_iter()
.find(|interface| interface.name == "enp5s0")
.unwrap();
let (_, mut package_receiver) =
match datalink::channel(&interface, Default::default()) {
Ok(Ethernet(tx, rx)) => (tx, rx),
_ => panic!()
};
loop {
match package_receiver.next() {
Ok(packet) => {
// sender.send("foo"); // this works fine
sender.send(packet);
}
_ => panic!()
}
}
}
This works fine for sending primitive types or Strings over the channel, but not for the network packets. When I try to send a packet to the parser thread via the channel I get the following compiler error:
error[E0597]: `*package_receiver` does not live long enough
--> src/main.rs:28:15
|
28 | match package_receiver.next() {
| ^^^^^^^^^^^^^^^^ borrowed value does not live long enough
...
36 | }
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
I'm pretty new to Rust and would really appreciate the help!
packet is &[u8] type, with some lifetime 'a that is also same as that of the reference taken to package_receiver in the next() call. The next() definition with lifetimes will look like this:
fn next(&'a mut self) -> Result<&'a [u8]>
You send the &[u8] to a thread. But the thread can outlive the references you send to it, leading to dangling references. As a result the compiler complains that they need to have 'static lifetime."foo" works because it is &'static str.
One way would be to take the ownership of the data and then send it as a value to another thread.
Ok(packet) => {
// sender.send("foo"); // this works fine
sender.send(packet.to_owned());
}
You can also have a look at using scoped threads with crossbeam
The package_receiver.next() call is defined as:
pub trait DataLinkReceiver: Send {
fn next(&mut self) -> Result<&[u8]>;
}
And the mspc::Sender defines send as:
pub fn send(&self, t: T) -> Result<(), SendError<T>>
So package_receiver.next() returns a result containing a reference to a slice of bytes, &[u8]. So when you then call sender.send(packet); that's saying you want to send the reference to the other thread. However, the match scope for package_receiver.next() doesn't promise that the reference lives longer than the end of the scope. So the other thread can't be guaranteed that the reference is still valid at the time it's accessing that data.
The str works because it's a static lifetime string. That memory will always be valid to read no matter what thread reads it.
If you change your call to:
sender.send(Vec::from(packet))
This creates a Vec variable, copies the packet slice into the new memory, then passes ownership of that variable to the other thread. This guarantees that the other receiving thread will always have clear access to that data. Because the ownership is clearly passed, the code in the receiving thread is known to the compiler as where the lifetime of received Vec variables ends.
There will also be some miscellaneous errors about using the result of the .send() which could be taken care of with something like this:
if sender.send(Vec::from(packet)).is_err() {
println!("Send error");
}
The other answers here do a great job of explaining why this happens, and it seems EthernetPacket (which derives the packet macro) is simply a wrapper around a reference of bytes. To make a "new" EthernetPacket variable from the reference of bytes, one could do the following:
let packet = EthernetPacket::owned(packet.to_owned()).unwrap();
sender.send(packet).unwrap();
This will clone the packet bytes and make send them through the channel.
Documentation for EthernetPacket::owned can be found here.
I thought the whole purpose of a channel was to share data between threads. I have this code, based on this example:
let tx_thread = tx.clone();
let ctx = self;
thread::spawn(|| {
...
let result = ctx.method()
tx_thread.send((String::from(result), someOtherString)).unwrap();
})
Where tx is a mpsc::Sender<(String, String)>
error[E0277]: the trait bound `std::sync::mpsc::Sender<(std::string::String, std::string::String)>: std::marker::Sync` is not satisfied
--> src/my_module/my_file.rs:137:9
|
137 | thread::spawn(|| {
| ^^^^^^^^^^^^^
|
= note: `std::sync::mpsc::Sender<(std::string::String, std::string::String)>` cannot be shared between threads safely
= note: required because of the requirements on the impl of `std::marker::Send` for `&std::sync::mpsc::Sender<(std::string::String, std::string::String)>`
= note: required because it appears within the type `[closure#src/my_module/my_file.rs:137:23: 153:10 res:&&str, ctx:&&my_module::my_submodule::Reader, tx_thread:&std::sync::mpsc::Sender<(std::string::String, std::string::String)>]`
= note: required by `std::thread::spawn`
I'm confused where I went wrong. Unless I'm looking in the wrong place and my issue is actually my use of let ctx = self;?
Sender cannot be shared between threads, but it can be sent!
It implements the trait Send but not Sync (Sync: Safe to access shared reference to Sender across threads).
The design of channels intends that you .clone() the sender and pass it as a value to a thread (for each thread you have). You are missing the move keyword on the thread's closure, which instructs the closure to capture variables by taking ownership of them.
If you must share a single channel endpoint between several threads, it must be wrapped in a mutex. Mutex<Sender<T>> is Sync + Send where T: Send.
Interesting implementation note: The channel starts out for use as a stream where it has a single producer. The internal data structures are upgraded to a multi-producer implementation the first time a sender is cloned.
You may use std::sync::mpsc::SyncSender from the standard library. The diffrenece is that it implements the Sync trait but it will may block if there is no space in the internal buffer while sending a message.
For more information:
std::sync::mpsc::channel
std::sync::mpsc::sync_channel
I have trouble compiling this program:
use std::env;
use std::sync::mpsc;
use std::thread;
use std::time::Duration;
fn main() {
let args: Vec<_> = env::args().skip(1).collect();
let (tx, rx) = mpsc::channel();
for arg in &args {
let t = tx.clone();
thread::spawn(move || {
thread::sleep(Duration::from_millis(50));
let _new_arg = arg.to_string() + "foo";
t.send(arg);
});
}
for _ in &args {
println!("{}", rx.recv().unwrap());
}
}
I read all arguments from the command line and emulate doing some work on each argument in the thread. Then I print out the results of this work, which I do using a channel.
error[E0597]: `args` does not live long enough
--> src/main.rs:11:17
|
11 | for arg in &args {
| ^^^^ does not live long enough
...
24 | }
| - borrowed value only lives until here
|
= note: borrowed value must be valid for the static lifetime...
If I understood well.. the lifetime of args must be static (i.e. the entire time of program execution), while it only lives within the scope of main function (?). I don't understand the reason behind this, and how I could fix it.
The problem lies in spawning a background thread. When you call thread::spawn you effectively have to pass ownership of any resource used in it to the thread, as it might run indefinitely, which means that its lifetime must be 'static.
There are two options to resolve that: the simplest one would be to pass ownership. Your code here
let new_arg = arg.to_string() + "foo";
t.send(arg);
looks like you actually wanted to send new_arg, in which case you could just create the owned result of arg.to_string() before spawning the thread, thus eliminating the need to pass the reference arg.
Another slightly more involved idea, that might be useful at some point though, are scoped threads as implemented in crossbeam for example. These are bound to an explicit scope, where you spawn them and are joined together at the end. This looks somewhat like this:
crossbeam::scope(|scope| {
scope.spawn(|| {
println!("Hello from a scoped thread!");
});
});
Have a look at the docs for further details.