I am trying to make this code snippet run concurrently instead of sequentially since the number of peers can be a large value. I am using async_std 1.4 and rust 1.41
pub struct Peer {
pub peer_id: String,
pub tcp_stream: Arc<TcpStream>,
pub public_key: [u8; 32],
}
async fn send_to_all_peers(message: Protocol, peers: &HashMap<String,Peer>) -> Result<()> {
for peer in peers.values() {
let mut stream = &*peer.tcp_stream;
stream.write_all(&bincode::serialize(&message)?).await?;
}
Ok(())
}
I've tried to use the futures::future::join_all method without any luck since wrapping future I created and used within async_std::task::spawn requires a static lifetime. Here is what I tried:
async fn send_to_all_peers(message: Protocol, peers: &HashMap<String,Peer>) {
let handles = peers.values().into_iter().map(|peer| {
task::spawn(
async {
let mut stream = &*peer.tcp_stream;
if let Err(err) = stream
.write_all(&bincode::serialize(&message).unwrap())
.await
{
error!("Error when writing to tcp_stream: {}", err);
}
}
)
});
futures::future::join_all(handles).await;
}
I'm sure there is some method I am missing, thanks for any help!
Since you are trying to send message concurrently, each task has to have its own copy of the message:
use async_std::{task, net::TcpStream};
use futures::{future, io::AsyncWriteExt};
use serde::Serialize;
use std::{
collections::HashMap,
error::Error,
sync::Arc,
};
pub struct Peer {
pub peer_id: String,
pub tcp_stream: Arc<TcpStream>,
pub public_key: [u8; 32],
}
#[derive(Serialize)]
struct Protocol;
async fn send_to_all_peers(
message: Protocol,
peers: &HashMap<String, Peer>)
-> Result<(), Box<dyn Error>>
{
let msg = bincode::serialize(&message)?;
let handles = peers.values()
.map(|peer| {
let msg = msg.clone();
let socket = peer.tcp_stream.clone();
task::spawn(async move {
let mut socket = &*socket;
socket.write_all(&msg).await
})
});
future::try_join_all(handles).await?;
Ok(())
}
Have you tried something like
let handles = peers.values().into_iter().map(|peer| {
let mut stream = &*peer.tcp_stream;
stream.write_all(&bincode::serialize(&message).unwrap())
}
let results = futures::future::join_all(handles).await
?
Notice how the .map closure doesn’t await, but straight up returns a future, which is then passed to join_all, and then awaited.
Related
I am using actix-web to run a webserver and want to be able to mutate state through websocket messages.
My current way of using websockets is through implementing the handle method from actix::StreamHandler. However this limits my ability of passing data to it. How can I access the data (actix_web::web::Data) in my handle method?
The only way I can think of solving this issue is to somehow overwrite the function signature of handle, however that doesn't seem possible
Hers is some important code snippets, we have app_name and nonces in app_data:
// main.rs
let nonces = Arc::new(Mutex::new(nonces::Nonces::new()));
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(app_data::AppData {
app_name: String::from("Actix Web"),
nonces: Arc::clone(&nonces),
}))
...
// app_data.rs
pub struct AppData {
pub app_name: String,
pub nonces: Arc<Mutex<nonces::Nonces>>,
}
// ws.rs
struct Ws {
app_data: web::Data<app_data::AppData>,
}
impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for Ws {
fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) {
let app_name = &self.app_data.app_name;
let mut nonces = self.app_data.nonces.lock().unwrap();
println!(">>> {app_name}");
println!(">>> {:?}", nonces.nonces); // I have a nonces data in nonces
...
}
}
async fn index(
req: HttpRequest,
stream: web::Payload,
app_data: web::Data<app_data::AppData>,
) -> Result<HttpResponse, Error> {
ws::start(Ws { app_data: app_data.clone() }, &req, stream)
}
I want to send Events between the game client and server and I already got it working, but I do not know how to do it with bevy.
I am dependent to use tokios async TcpStream, because I have to be able to split the stream into a OwnedWriteHalf and OwnedReadhalf using stream.into_split().
My first idea was to just spawn a thread that handles the connection and then send the received events to a queue using mpsc::channel
Then I include this queue into a bevy resource using app.insert_resource(Queue) and pull events from it in the game loop.
the Queue:
use tokio::sync::mpsc;
pub enum Instruction {
Push(GameEvent),
Pull(mpsc::Sender<Option<GameEvent>>),
}
#[derive(Clone, Debug)]
pub struct Queue {
sender: mpsc::Sender<Instruction>,
}
impl Queue {
pub fn init() -> Self {
let (tx, rx) = mpsc::channel(1024);
init(rx);
Self{sender: tx}
}
pub async fn send(&self, event: GameEvent) {
self.sender.send(Instruction::Push(event)).await.unwrap();
}
pub async fn pull(&self) -> Option<GameEvent> {
println!("new pull");
let (tx, mut rx) = mpsc::channel(1);
self.sender.send(Instruction::Pull(tx)).await.unwrap();
rx.recv().await.unwrap()
}
}
fn init(mut rx: mpsc::Receiver<Instruction>) {
tokio::spawn(async move {
let mut queue: Vec<GameEvent> = Vec::new();
loop {
match rx.recv().await.unwrap() {
Instruction::Push(ev) => {
queue.push(ev);
}
Instruction::Pull(sender) => {
sender.send(queue.pop()).await.unwrap();
}
}
}
});
}
But because all this has to be async I have block the pull() function in the sync game loop.
I do this using the futures-lite crate:
fn event_pull(
communication: Res<Communication>
) {
let ev = future::block_on(communication.event_queue.pull());
println!("got event: {:?}", ev);
}
And this works fine, BUT after around 5 seconds the whole program just halts and does not receive any more events.
It seems like that future::block_on() does block indefinitely.
Having the main function, in which bevy::prelude::App gets built and run, to be the async tokio::main function might also be a problem here.
It would probably be best to wrap the async TcpStream initialisation and tokio::sync::mpsc::Sender and thus also Queue.pull into synchronous functions, but I do not know how to do this.
Can anyone help?
How to reproduce
The repo can be found here
Just compile both server and client and then run both in the same order.
I got it to work by just replacing every tokio::sync::mpsc with crossbeam::channel, which might be a problem, as it does block
and manually initializing the tokio runtime.
so the init code looks like this:
pub struct Communicator {
pub event_bridge: bridge::Bridge,
pub event_queue: event_queue::Queue,
_runtime: Runtime,
}
impl Communicator {
pub fn init(ip: &str) -> Self {
let rt = tokio::runtime::Builder::new_multi_thread()
.enable_io()
.build()
.unwrap();
let (bridge, queue, game_rx) = rt.block_on(async move {
let socket = TcpStream::connect(ip).await.unwrap();
let (read, write) = socket.into_split();
let reader = TcpReader::new(read);
let writer = TcpWriter::new(write);
let (bridge, tcp_rx, game_rx) = bridge::Bridge::init();
reader::init(bridge.clone(), reader);
writer::init(tcp_rx, writer);
let event_queue = event_queue::Queue::init();
return (bridge, event_queue, game_rx);
});
// game of game_rx events to queue for game loop
let eq_clone = queue.clone();
rt.spawn(async move {
loop {
let event = game_rx.recv().unwrap();
eq_clone.send(event);
}
});
Self {
event_bridge: bridge,
event_queue: queue,
_runtime: rt,
}
}
}
And main.rs looks like this:
fn main() {
let communicator = communication::Communicator::init("0.0.0.0:8000");
communicator.event_bridge.push_tcp(TcpEvent::Register{name: String::from("luca")});
App::new()
.insert_resource(communicator)
.add_system(event_pull)
.add_plugins(DefaultPlugins)
.run();
}
fn event_pull(
communication: Res<communication::Communicator>
) {
let ev = communication.event_queue.pull();
if let Some(ev) = ev {
println!("ev");
}
}
Perhaps there might be a better solution.
I'm new to Rust and I'm trying to create a Server struct which listen to an address and starts a TCP socket connection. The problem is that I want to store the client connection inside a hash map so I can use it later..
I tried writing this:
use std::collections::HashMap;
use std::net::TcpListener;
use std::net::TcpStream;
use std::sync::{Arc, RwLock};
use std::thread;
#[derive(Clone, Debug)]
pub struct Server {
id: Arc<RwLock<u32>>,
connections: Arc<RwLock<HashMap<u32, TcpStream>>>,
url: String,
}
impl Server {
pub fn new(url: String) -> Server {
let server = Server {
id: Arc::new(RwLock::new(0)),
connections: Arc::new(RwLock::new(HashMap::new())),
url,
};
server
}
pub fn start(&self) {
thread::spawn(move || {
let mut listener =
TcpListener::bind(self.clone().url).expect("Could not start the server");
println!("Server started succesfully");
for stream in listener.incoming() {
match stream {
Ok(stream) => self.on_client_connect(stream),
Err(error) => eprintln!("Error when tried to use stream"),
}
}
});
}
fn on_client_connect(&mut self, stream: TcpStream) {
let id = self.id.read().unwrap();
self.connections.read().unwrap().insert(id, stream);
let id = self.id.write().unwrap();
*id += 1;
}
}
But of course this doesn't work.. There are 2 things that I don't understand, the first is how to pass the stream into my function and then store in my connections hash map so I can use it later and how to use my id inside my on_client_connect function.
You need to clone outside of thread::spawn and move the cloned instance in thread scope.
Also, on_client_connect do not need &mut self because the fields id and connections are already protected inside RwLock.
use std::net::TcpListener;
use std::net::TcpStream;
use std::collections::HashMap;
use std::sync::{Arc, RwLock};
use std::thread;
#[derive(Clone, Debug)]
pub struct Server {
id: Arc<RwLock<u32>>,
connections: Arc<RwLock<HashMap<u32, TcpStream>>>,
url: String,
}
impl Server {
pub fn new(url: String) -> Server {
let server = Server {
id: Arc::new(RwLock::new(0)),
connections: Arc::new(RwLock::new(HashMap::new())),
url,
};
server
}
pub fn start(&self) {
let me = self.clone(); // Clone it outside
thread::spawn(move || {
let mut listener = TcpListener::bind(&me.url).expect("Could not start the server");
println!("Server started succesfully");
for stream in listener.incoming() {
match stream {
Ok(stream) => me.on_client_connect(stream),
Err(error) => eprintln!("Error when tried to use stream"),
}
}
});
}
fn on_client_connect(&self, stream: TcpStream) { // `&mut self` not needed as the id, connection are inside the lock
let mut id = self.id.write().unwrap();
self.connections.write().unwrap().insert(*id, stream);
*id += 1;
}
}
playground
There's quite a few issues that need minor fixes in this code.
The first one I ran into was the usage of self in the thread::spawn closure.
The thread::spawn needs its argument to have static lifetime, but we've no guarantee that the Server object lives that long.
I solved it by cloning the Server object and moving that into the closure. This is OK as all its data is already behind Arcs.
The next problem was self.connections.read().unwrap().insert(*id, stream); needs to get a write lock, not a read.
Finally id+=1 needs to dereference id.
Once these were fixed, it seems that storing the TcpStream is not an issue. (At least using nightly). I'd thought I'd need to box the TcpStream - but it seems OK as is.
You can see that it compiles in the playground
use std::collections::HashMap;
use std::net::TcpListener;
use std::net::TcpStream;
use std::sync::{Arc, RwLock};
use std::thread;
#[derive(Clone, Debug)]
pub struct Server {
id: Arc<RwLock<u32>>,
connections: Arc<RwLock<HashMap<u32, TcpStream>>>,
url: String,
}
impl Server {
pub fn new(url: String) -> Server {
let server = Server {
id: Arc::new(RwLock::new(0)),
connections: Arc::new(RwLock::new(HashMap::new())),
url,
};
server
}
pub fn start(&self) {
let mut self_clone = self.clone();
thread::spawn(move || {
let mut listener =
TcpListener::bind(&self_clone.url).expect("Could not start the server");
println!("Server started succesfully");
for stream in listener.incoming() {
match stream {
Ok(stream) => self_clone.on_client_connect(stream),
Err(error) => eprintln!("Error when tried to use stream"),
}
}
});
}
fn on_client_connect(&mut self, stream: TcpStream) {
let id = self.id.read().unwrap();
self.connections.write().unwrap().insert(*id, stream);
let mut id = self.id.write().unwrap();
*id += 1;
}
}
I am writing the code to bootstrap and connect to a 2G/3G network using a SIM800L modem. This modem is interfaced with a single serial channel, which I've muxed outside of this project into 4 channels (data, text interface, control interface, status messages).
In order to bootstrap this, I need to run a series of sequential commands. This sequence changes based on the output of the modem (is the SIM locked? What kind of info does the SIM need to be unlocked? What kind of APN are we getting on? What kind of network selection do we want?). I initially thought that this would be a perfect application for futures as each individual operation can be very costly in terms of time spent idling (AT+COPS, one of the command, takes up to 10s to return).
I'm on to something like this, which, while it compiles and seems to execute commands sequentially, the third operation comes out empty. My question is twofold: why do the commands run not pop up in the result of the last future, and is there a more robust way of doing something like this?
#![feature(conservative_impl_trait)]
extern crate futures;
extern crate tokio_core;
use std::sync::{Arc, Mutex};
use futures::{future, Future};
use tokio_core::reactor::Core;
use futures::sync::oneshot;
use std::thread;
use std::io;
use std::time::Duration;
pub struct Channel {
operations: Arc<Mutex<Vec<String>>>,
}
impl Channel {
pub fn ops(&mut self) -> Box<Future<Item = Vec<String>, Error = io::Error>> {
println!("{:?}", self.operations);
let ops = Arc::clone(&self.operations);
let ops = ops.lock().unwrap();
future::ok::<Vec<String>, io::Error>(ops.to_vec()).boxed()
}
pub fn run(&mut self, command: &str) -> Box<Future<Item = Vec<String>, Error = io::Error>> {
let (tx, rx) = oneshot::channel::<Vec<String>>();
let ops = Arc::clone(&self.operations);
let str_cmd = String::from(command);
thread::spawn(move || {
thread::sleep(Duration::new(0, 10000));
let mut ops = ops.lock().unwrap();
ops.push(str_cmd.clone());
println!("Pushing op: {}", str_cmd.clone());
tx.send(vec!["OK".to_string()])
});
rx.map_err(|_| io::Error::new(io::ErrorKind::NotFound, "Test"))
.boxed()
}
}
pub struct Channels {
inner_object: Arc<Mutex<Channel>>,
}
impl Channels {
pub fn one(&self, cmd: &str) -> Box<Future<Item = Vec<String>, Error = io::Error>> {
let v = Arc::clone(&self.inner_object);
let mut v = v.lock().unwrap();
v.run(&cmd)
}
pub fn ops(&self) -> Box<Future<Item = Vec<String>, Error = io::Error>> {
let v = Arc::clone(&self.inner_object);
let mut v = v.lock().unwrap();
v.ops()
}
pub fn run_command(&self) -> Box<Future<Item = (), Error = io::Error>> {
let a = self.one("AT+CMEE=2");
let b = self.one("AT+CREG=0");
let c = self.ops();
Box::new(a.and_then(|result_1| {
assert_eq!(result_1, vec![String::from("OK")]);
b.and_then(|result_2| {
assert_eq!(result_2, vec![String::from("OK")]);
c.map(move |ops| {
assert_eq!(
ops.as_slice(),
["AT+CMEE=2".to_string(), "AT+CREG=0".to_string()]
);
})
})
}))
}
}
fn main() {
let mut core = Core::new().expect("Core should be created");
let channels = Channels {
inner_object: Arc::new(Mutex::new(Channel {
operations: Arc::new(Mutex::new(vec![])),
})),
};
let result = core.run(channels.run_command()).expect("Should've worked");
println!("{:?}", result);
}
playground
why do the commands run not pop up in the result of the last future
Because you haven't sequenced the operations to occur in that way:
let a = self.one("AT+CMEE=2");
let b = self.one("AT+CREG=0");
let c = self.ops();
This immediately builds:
a, b — promises that sleep a while before they respond
c — a promise that gets the ops in the vector
At the point in time that c is created, the sleeps have yet to terminate, so there have been no operations performed, so the vector will be empty.
Future::and_then is intended to be used to define sequential operations. This is complicated in your case as you want to use self in the body of the and_then closure. You can clone the Arc<Channel> and use that instead.
You'll note that I've made a number of simplifications:
Returning a String instead of Vec<String>
Removing unused mut qualifiers and a Mutex
Returning the operations Vec directly.
extern crate futures;
extern crate tokio_core;
use std::sync::{Arc, Mutex};
use futures::Future;
use tokio_core::reactor::Core;
use futures::sync::oneshot;
use std::thread;
use std::io;
use std::time::Duration;
pub struct Channel {
operations: Arc<Mutex<Vec<String>>>,
}
impl Channel {
fn ops(&self) -> Vec<String> {
self.operations.lock().unwrap().clone()
}
fn command(&self, command: &str) -> Box<Future<Item = String, Error = io::Error>> {
let (tx, rx) = oneshot::channel();
let ops = Arc::clone(&self.operations);
let str_cmd = String::from(command);
thread::spawn(move || {
thread::sleep(Duration::new(0, 10000));
println!("Pushing op: {}", str_cmd);
ops.lock().unwrap().push(str_cmd);
tx.send("OK".to_string())
});
Box::new(rx.map_err(|_| io::Error::new(io::ErrorKind::NotFound, "Test")))
}
}
struct Channels {
data: Arc<Channel>,
}
impl Channels {
fn run_command(&self) -> Box<Future<Item = (), Error = io::Error>> {
let d2 = Arc::clone(&self.data);
let d3 = Arc::clone(&self.data);
Box::new(
self.data
.command("AT+CMEE=2")
.and_then(move |cmee_answer| {
assert_eq!(cmee_answer, "OK"); // This should be checked in `command` and be a specific Error
d2.command("AT+CREG=0")
})
.map(move |creg_answer| {
assert_eq!(creg_answer, "OK"); // This should be checked in `command` and be a specific Error
let ops = d3.ops();
assert_eq!(ops, ["AT+CMEE=2", "AT+CREG=0"])
}),
)
}
}
fn main() {
let mut core = Core::new().expect("Core should be created");
let channels = Channels {
data: Arc::new(Channel {
operations: Arc::new(Mutex::new(vec![])),
}),
};
let result = core.run(channels.run_command()).expect("Should've worked");
println!("{:?}", result);
}
However, this isn't the type of code I usually see with futures. Instead of taking &self, many futures take self. Let's see how that would look:
extern crate futures;
extern crate tokio_core;
use std::sync::{Arc, Mutex};
use futures::Future;
use tokio_core::reactor::Core;
use futures::sync::oneshot;
use std::thread;
use std::io;
use std::time::Duration;
#[derive(Clone)]
pub struct Channel {
operations: Arc<Mutex<Vec<String>>>,
}
impl Channel {
fn ops(&self) -> Arc<Mutex<Vec<String>>> {
Arc::clone(&self.operations)
}
fn command(self, command: &str) -> Box<Future<Item = (Self, String), Error = io::Error>> {
let (tx, rx) = oneshot::channel();
let str_cmd = String::from(command);
thread::spawn(move || {
thread::sleep(Duration::new(0, 10000));
println!("Pushing op: {}", str_cmd);
self.operations.lock().unwrap().push(str_cmd);
tx.send((self, "OK".to_string()))
});
Box::new(rx.map_err(|_| io::Error::new(io::ErrorKind::NotFound, "Test")))
}
}
struct Channels {
data: Channel,
}
impl Channels {
fn run_command(self) -> Box<Future<Item = (), Error = io::Error>> {
Box::new(
self.data
.clone()
.command("AT+CMEE=2")
.and_then(|(channel, cmee_answer)| {
assert_eq!(cmee_answer, "OK");
channel.command("AT+CREG=0")
})
.map(|(channel, creg_answer)| {
assert_eq!(creg_answer, "OK");
let ops = channel.ops();
let ops = ops.lock().unwrap();
assert_eq!(*ops, ["AT+CMEE=2", "AT+CREG=0"]);
}),
)
}
}
fn main() {
let mut core = Core::new().expect("Core should be created");
let channels = Channels {
data: Channel {
operations: Arc::new(Mutex::new(vec![])),
},
};
let result = core.run(channels.run_command()).expect("Should've worked");
println!("{:?}", result);
}
I'm going to have multiple functions that all need access to one main socket.
Would it better to:
Pass this socket to each function that needs access to it
Have a globally accessible socket
Can someone provide an example of the best way to do this?
I come from a Python/Nim background where things like this are easily done.
Edit:
How can I pass a socket as an arg to a function being called within a thread.
Ex.
fn main() {
let mut s = BufferedStream::new((TcpStream::connect(server).unwrap()));
let thread = Thread::spawn(move || {
func1(s, arg1, arg2);
});
while true {
func2(s, arg1);
}
}
Answer for updated question
We can use TcpStream::try_clone:
use std::io::Read;
use std::net::{TcpStream, Shutdown};
use std::thread;
fn main() {
let mut stream = TcpStream::connect("127.0.0.1:34254").unwrap();
let stream2 = stream.try_clone().unwrap();
let _t = thread::spawn(move || {
// close this stream after one second
thread::sleep_ms(1000);
stream2.shutdown(Shutdown::Read).unwrap();
});
// wait for some data, will get canceled after one second
let mut buf = [0];
stream.read(&mut buf).unwrap();
}
Original answer
It's usually (let's say 99.9% of the time) a bad idea to have any global mutable state, if you can help it. Just do as you said: pass the socket to the functions that need it.
use std::io::{self, Write};
use std::net::TcpStream;
fn send_name(stream: &mut TcpStream) -> io::Result<()> {
stream.write(&[42])?;
Ok(())
}
fn send_number(stream: &mut TcpStream) -> io::Result<()> {
stream.write(&[1, 2, 3])?;
Ok(())
}
fn main() {
let mut stream = TcpStream::connect("127.0.0.1:31337").unwrap();
let r = send_name(&mut stream).and_then(|_| send_number(&mut stream));
match r {
Ok(..) => println!("Yay, sent!"),
Err(e) => println!("Boom! {}", e),
}
}
You could also pass the TcpStream to a struct that manages it, and thus gives you a place to put similar methods.
use std::io::{self, Write};
use std::net::TcpStream;
struct GameService {
stream: TcpStream,
}
impl GameService {
fn send_name(&mut self) -> io::Result<()> {
self.stream.write(&[42])?;
Ok(())
}
fn send_number(&mut self) -> io::Result<()> {
self.stream.write(&[1, 2, 3])?;
Ok(())
}
}
fn main() {
let stream = TcpStream::connect("127.0.0.1:31337").unwrap();
let mut service = GameService { stream: stream };
let r = service.send_name().and_then(|_| service.send_number());
match r {
Ok(..) => println!("Yay, sent!"),
Err(e) => println!("Boom! {}", e),
}
}
None of this is really Rust-specific, these are generally-applicable programming practices.