This question already has answers here:
What is unwrap in Rust, and what is it used for?
(2 answers)
Closed 5 months ago.
I am writing socks5 proxy server. The program is running asynchronously and I am trying to use tokio::select, but the program terminates due to this error when I want to get the size of the received data:
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 54, kind: ConnectionReset, message: "Connection reset by peer" }'
The function where the error occurs:
async fn exchange_loop(mut client: TcpStream, address: SocketAddr, cmd: u8) {
let mut remote = TcpStream::connect(address).await.unwrap();
let mut buffer_client: [u8; 4096] = [0; 4096];
let mut buffer_remote: [u8; 4096] = [0; 4096];
loop {
tokio::select! {
size = client.read(&mut buffer_client) => {
let size = size.unwrap();
remote.write(buffer_client.as_ref()).await.unwrap();
println!("Send from client {} => {} {} KB", client.peer_addr().unwrap(), remote.peer_addr().unwrap(), size as f32 / 1024.);
if size <= 0 {
break;
};
buffer_client = [0; 4096];
}
size = remote.read(&mut buffer_remote) => {
let size = size.unwrap();
client.write(buffer_remote.as_ref()).await.unwrap();
println!("Send from remote {} => {} {} KB", address, client.peer_addr().unwrap(), size as f32 / 1024.);
if size <= 0 {
break;
};
buffer_remote = [0; 4096];
}
}
}
println!("End connection to {}", address);
}
You need to handle errors, as specified in the previous comments. You need to be able to recover from whatever the client does on his side, interrupting the connection for example.
An example how to recover:
if let Ok(actual_size) = size {
if let Ok(_) = remote.write(buffer_client.as_ref()).await {
println!("Send from client {} => {} {} KB", client.peer_addr().unwrap(), remote.peer_addr().unwrap(), size as f32 / 1024.);
if size <= 0 {
break;
};
buffer_client = [0; 4096];
}
}
I put Ok(_) as I do not know what you expect from the 2nd unwrap.
Related
Good morning people! I'm starting in the Rust world and would like to ask for help.
In the code below I am simulating sending to a client that has disconnected, the program breaks when I try to send a message to a client that is no longer connected. how to treat so that the program does not break?
the match command doesn't even work
Error: Os { code: 10054, kind: ConnectionReset, message: "Forced cancellation
from an existing connection by the remote host." }
use tokio::net::UdpSocket;
use std::io;
#[tokio::main]
async fn main() -> io::Result<()> {
let sock = UdpSocket::bind("0.0.0.0:8080").await?;
let mut buf = [0; 1024];
loop {
let (len, addr) = sock.recv_from(&mut buf).await?;
println!("{:?} bytes received from {:?}", len, addr);
let len = sock.send_to(&buf[..len], addr).await?;
println!("{:?} bytes sent", len);
match sock.send_to(&buf, "127.0.0.1:23451").await
{
Result::Ok(_len) => {
println!("{:?} bytes sent", _len);
}
Result::Err(_) => {
println!("a error")
}
}
}
}
If I don't pass a valid address, I can get the error with match, but passing a valid address from a user that disconnected from the error and I can't get the match
Solution tks Guys!
use std::io;
use tokio::net::UdpSocket;
#[tokio::main]
async fn main() -> io::Result<()> {
let sock = UdpSocket::bind("0.0.0.0:8080").await?;
let mut buf = [0; 1024];
loop {
match sock.recv_from(&mut buf).await {
Result::Ok(data) => {
let (len, addr) = data;
println!("{:?} bytes received from {:?}", len, addr);
let len = sock.send_to(&buf[..len], addr).await?;
println!("{:?} bytes sent", len);
}
Result::Err(e) => {
println!("{:?}", e);
}
}
// SEND MESAGE TO USER DISCONECTED
match sock.send_to(&buf, "127.0.0.1:23451").await {
Result::Ok(_len) => {
println!("{:?} bytes sent", _len);
}
Result::Err(_) => {
println!("a error 2")
}
}
}
}
According to this Reddit post, it is possible the error is detected during recv_from that follows the send_to, where error is not handled.
Damnit, it was something simple, as always. So, the send operation obviously causes the error, but it was not emitted until a subsequent call to recv_from(). Thank you!
https://www.reddit.com/r/rust/comments/a2hh6n/cannot_handle_os_error_while_using_udpsocket/
I use tokio::net::TcpStream to connect a small TCP server, I write a few bytes and expect to read the response from the server.
When I do that with the nc command, it works perfectly
[denis#docker-1 ~]$ echo "get" | nc 10.0.0.11 9090
[37e64dd7-91db-4c13-9f89-f1c87467ffb3][processed]
and the server logs show
Incoming peer instructions.
Waiting for peer instructions...
Reading bytes...
Got a few bytes [4]
Got a few bytes [[103, 101, 116, 10, 0, ...]]
Reading bytes...
Got a few bytes [0]
Got a few bytes [[0, 0, 0, 0, 0, 0,...]]
Writing some data back from peer : [37e64dd7-91db-4c13-9f89-f1c87467ffb3]
But from my Rust client, I can write the bytes but as soon as I want to read the data from the server, everything is locked (even the write action)
use std::collections::HashMap;
use std::ops::DerefMut;
use tokio::io;
use tokio::net::{TcpListener, TcpStream};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use uuid::Uuid;
use std::sync::RwLock;
use lazy_static::*;
#[tokio::main]
async fn main() {
let data = "set".to_string();
let mut stream = TcpStream::connect("10.0.0.11:9090").await.unwrap();
let ( mut read, mut write) = tokio::io::split(stream);
let u2 = data.as_bytes();
write.write_all(u2).await.unwrap();
let mut msg : [u8;1024] = [0;1024];
let _response_size = read.read(&mut msg).await.unwrap();
println!("GOT = {:?}", msg);
}
When looking at the server logs (see below), it reads the 3 bytes sent by the client, but then it is not able to read further, waiting to detect there is 0 byte left to read.
Incoming peer instructions.
Waiting for peer instructions...
Reading bytes...
Got a few bytes [3]
Got a few bytes [[115, 101, 116, 0, 0, ...]]
Reading bytes...
Here is the server code
use std::collections::HashMap;
use std::ops::DerefMut;
use tokio::io;
use tokio::net::{TcpListener, TcpStream};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use uuid::Uuid;
use std::sync::RwLock;
use lazy_static::*;
struct DataPool {
data : [u8;1024],
size : usize,
}
async fn whirl_socket( socket : &mut TcpStream ) -> Vec<DataPool> {
let mut pool: Vec<DataPool> = vec![];
let mut buf = [0; 1024];
// In a loop, read data from the socket until finished
loop {
println!("Reading bytes...");
buf = [0; 1024];
let n = match socket.read(&mut buf).await {
Ok(n) => n,
Err(e) => {
eprintln!("failed to read from socket; err = {:?}", e);
break;
}
};
println!("Got a few bytes [{}]", n);
println!("Got a few bytes [{:?}]", &buf);
pool.push(DataPool {
data: buf,
size: n,
});
if n == 0 {
break;
}
}
pool
}
async fn launch_server_listener() -> io::Result<()> {
println!("Listen to 9090...");
let listener = TcpListener::bind("10.0.0.11:9090").await?;
loop {
println!("Waiting for peer instructions...");
let (mut socket, _) = listener.accept().await?;
println!("Incoming peer instructions.");
tokio::spawn(async move {
let mut pool= whirl_socket(&mut socket).await;
let my_uuid = Uuid::new_v4();
// Write the data back
println!("Writing some data back from peer : [{}]", my_uuid);
let s = format!( "[{}][processed]\n", my_uuid.to_string());
let u = s.as_bytes();
if let Err(e) = socket.write_all(u).await {
eprintln!("failed to write to socket; err = {:?}", e);
return;
}
});
}
}
async fn start_servers() -> Result<(), Box<dyn std::error::Error>> {
let _r = tokio::join!(launch_server_listener());
Ok(())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
start_servers().await?;
Ok(())
}
A read of 0 bytes means the read stream has closed. So in your client code you need to close the write stream. You can do this with .shutdown() from the AsyncWriteExt trait:
write.write_all(u2).await.unwrap();
write.shutdown().await.unwrap();
Using serial-rs it's possible to open a Bluetooth connection between my Mac and Arduino (HC-05). But if I want to open multiple Bluetooth connections at the same time, only the most recent connection stays open.
I am not completely sure how Qt handles this, but it's possible to read/write to multiple devices same time using QSerialPort.
Is this a serial-rs unimplemented feature, or does Qt do something like switching connections (to have only one opened in time) so it looks like multiple connections are handled?
extern crate serial;
#[macro_use]
extern crate text_io;
use std::process::Command;
use std::io;
use std::time::Duration;
use std::sync::mpsc::{Sender, Receiver};
use std::sync::mpsc;
use std::thread;
use std::io::prelude::*;
use serial::prelude::*;
use std::sync::Arc;
use std::sync::Mutex;
fn print_available_ports() {
let status = Command::new("ls")
.arg("/dev/")
.status()
.unwrap_or_else(|e| panic!("Failed to execute process: {}", e));
}
fn print_available_commands() {
println!("Available commands:");
println!(" print_ports - prints available serial ports");
println!(" connect - make an asynchronous connection to port");
println!(" start - signal ports to start collecting data");
println!(" stop - opposite to start");
println!(" monitor - print info of current state of reading");
println!(" help - prints this info.");
println!(" exit");
}
fn connect_to_port(portname: &String,
rate: usize,
tx: Sender<String>,
port_state: Arc<Mutex<bool>>)
-> io::Result<()> {
let mut port = serial::open(portname.trim()).unwrap();
try!(port.reconfigure(&|settings| {
try!(settings.set_baud_rate(serial::BaudRate::from_speed(rate)));
settings.set_char_size(serial::Bits8);
settings.set_parity(serial::ParityNone);
settings.set_stop_bits(serial::Stop1);
settings.set_flow_control(serial::FlowNone);
Ok(())
}));
try!(port.set_timeout(Duration::from_millis(10000)));
println!("Serial port to {} opened successfully.", portname);
println!("Waiting for the start..");
while *(port_state.lock().unwrap()) != true {
}
println!("Port named {} started reading.", portname);
let mut ans_number: usize = 0;
let mut answer = String::new();
let mut bytes_received: usize = 0;
let mut buf = vec![0;128];
loop {
match port.read(&mut buf[..]) {
Ok(n) => {
bytes_received += n;
}
Err(_) => {
println!("Error in reading from {}", portname);
bytes_received = bytes_received;
}
}
if bytes_received > 10000 {
answer = String::new();
answer = format!("#{} Port {} received 10000 bytes of data",
ans_number,
portname);
tx.send(answer);
bytes_received = 0;
ans_number += 1;
}
if *(port_state.lock().unwrap()) == false {
println!("Port named {} got signal to stop. Abort.", portname);
break;
}
}
Ok(())
}
fn main() {
print_available_commands();
let mut reading_active = Arc::new(Mutex::new(false));
let (dtx, drx): (Sender<String>, Receiver<String>) = mpsc::channel();
let mut ports = vec![];
let mut input = String::new();
loop {
input = String::new();
match io::stdin().read_line(&mut input) {
Ok(n) => println!("Command received: {}", input.trim()),
Err(error) => println!("error: {}", error),
}
match input.trim() {
"connect" => {
let portname: String;
let baudrate: usize;
println!("Enter port name:");
portname = read!();
println!("Enter baudrate:");
baudrate = read!();
let thread_state = reading_active.clone();
let thread_tx = dtx.clone();
ports.push(thread::Builder::new().name(portname.clone()).spawn(move || {
connect_to_port(&portname, baudrate, thread_tx, thread_state);
}));
}
"start" => {
*(reading_active.lock().unwrap()) = true;
}
"stop" => {
*(reading_active.lock().unwrap()) = false;
}
"help" => print_available_commands(),
"print_ports" => print_available_ports(),
"exit" => {
println!("Closing used ports..");
}
"monitor" => {
loop {
println!("{:?}", drx.recv());
}
}
_ => println!("Unsupported command."),
}
}
}
I'm trying to port this Python script that sends and receives input to a helper process to Rust:
import subprocess
data = chr(0x3f) * 1024 * 4096
child = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
output, _ = child.communicate(data)
assert output == data
My attempt worked fine until the input buffer exceeded 64k because presumably the OS's pipe buffer filled up before the input was written.
use std::io::Write;
const DATA: [u8; 1024 * 4096] = [0x3f; 1024 * 4096];
fn main() {
let mut child = std::process::Command::new("cat")
.stdout(std::process::Stdio::piped())
.stdin(std::process::Stdio::piped())
.spawn()
.unwrap();
match child.stdin {
Some(ref mut stdin) => {
match stdin.write_all(&DATA[..]) {
Ok(_size) => {}
Err(err) => panic!(err),
}
}
None => unreachable!(),
}
let res = child.wait_with_output();
assert_eq!(res.unwrap().stdout.len(), DATA.len())
}
Is there a subprocess.communicate equivalent in Rust? Maybe a select equivalent? Can mio be used to solve this problem? Also, there seems to be no way to close stdin.
The goal here is to make a high performance system, so I want to avoid spawning a thread per task.
Well it wasn't a small amount of code to get this done, and I needed a combination of mio and nix, because mio wouldn't set AsRawFd items to be nonblocking when they were pipes, so this had to be done first.
Here's the result
extern crate mio;
extern crate bytes;
use mio::*;
use std::io;
use mio::unix::{PipeReader, PipeWriter};
use std::process::{Command, Stdio};
use std::os::unix::io::AsRawFd;
use nix::fcntl::FcntlArg::F_SETFL;
use nix::fcntl::{fcntl, O_NONBLOCK};
extern crate nix;
struct SubprocessClient {
stdin: PipeWriter,
stdout: PipeReader,
output : Vec<u8>,
input : Vec<u8>,
input_offset : usize,
buf : [u8; 65536],
}
// Sends a message and expects to receive the same exact message, one at a time
impl SubprocessClient {
fn new(stdin: PipeWriter, stdout : PipeReader, data : &[u8]) -> SubprocessClient {
SubprocessClient {
stdin: stdin,
stdout: stdout,
output : Vec::<u8>::new(),
buf : [0; 65536],
input : data.to_vec(),
input_offset : 0,
}
}
fn readable(&mut self, _event_loop: &mut EventLoop<SubprocessClient>) -> io::Result<()> {
println!("client socket readable");
match self.stdout.try_read(&mut self.buf[..]) {
Ok(None) => {
println!("CLIENT : spurious read wakeup");
}
Ok(Some(r)) => {
println!("CLIENT : We read {} bytes!", r);
self.output.extend(&self.buf[0..r]);
}
Err(e) => {
return Err(e);
}
};
return Ok(());
}
fn writable(&mut self, event_loop: &mut EventLoop<SubprocessClient>) -> io::Result<()> {
println!("client socket writable");
match self.stdin.try_write(&(&self.input)[self.input_offset..]) {
Ok(None) => {
println!("client flushing buf; WOULDBLOCK");
}
Ok(Some(r)) => {
println!("CLIENT : we wrote {} bytes!", r);
self.input_offset += r;
}
Err(e) => println!("not implemented; client err={:?}", e)
}
if self.input_offset == self.input.len() {
event_loop.shutdown();
}
return Ok(());
}
}
impl Handler for SubprocessClient {
type Timeout = usize;
type Message = ();
fn ready(&mut self, event_loop: &mut EventLoop<SubprocessClient>, token: Token,
events: EventSet) {
println!("ready {:?} {:?}", token, events);
if events.is_readable() {
let _x = self.readable(event_loop);
}
if events.is_writable() {
let _y = self.writable(event_loop);
}
}
}
pub fn from_nix_error(err: ::nix::Error) -> io::Error {
io::Error::from_raw_os_error(err.errno() as i32)
}
fn set_nonblock(s: &AsRawFd) -> io::Result<()> {
fcntl(s.as_raw_fd(), F_SETFL(O_NONBLOCK)).map_err(from_nix_error)
.map(|_| ())
}
const TEST_DATA : [u8; 1024 * 4096] = [40; 1024 * 4096];
pub fn echo_server() {
let mut event_loop = EventLoop::<SubprocessClient>::new().unwrap();
let process =
Command::new("cat")
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn().unwrap();
let raw_stdin_fd;
match process.stdin {
None => unreachable!(),
Some(ref item) => {
let err = set_nonblock(item);
match err {
Ok(()) => {},
Err(e) => panic!(e),
}
raw_stdin_fd = item.as_raw_fd();
},
}
let raw_stdout_fd;
match process.stdout {
None => unreachable!(),
Some(ref item) => {
let err = set_nonblock(item);
match err {
Ok(()) => {},
Err(e) => panic!(e),
}
raw_stdout_fd = item.as_raw_fd();},
}
//println!("listen for connections {:?} {:?}", , process.stdout.unwrap().as_raw_fd());
let mut subprocess = SubprocessClient::new(PipeWriter::from(Io::from_raw_fd(raw_stdin_fd)),
PipeReader::from(Io::from_raw_fd(raw_stdout_fd)),
&TEST_DATA[..]);
let stdout_token : Token = Token(0);
let stdin_token : Token = Token(1);
event_loop.register(&subprocess.stdout, stdout_token, EventSet::readable(),
PollOpt::level()).unwrap();
// Connect to the server
event_loop.register(&subprocess.stdin, stdin_token, EventSet::writable(),
PollOpt::level()).unwrap();
// Start the event loop
event_loop.run(&mut subprocess).unwrap();
let res = process.wait_with_output();
match res {
Err(e) => {panic!(e);},
Ok(output) => {
subprocess.output.extend(&output.stdout);
println!("Final output was {:}\n", output.stdout.len());
},
}
println!("{:?}\n", subprocess.output.len());
}
fn main() {
echo_server();
}
Basically the only way to close stdin was to call process.wait_with_output since the Stdin has no close primitive
Once this happened, the remaining input could extend the output data vector.
There's now a crate that does this
https://crates.io/crates/subprocess-communicate
In this particular example, you know that the input and output amounts are equivalent, so you don't need threads at all. You can just write a bit and then read a bit:
use std::io::{self, Cursor, Read, Write};
static DATA: [u8; 1024 * 4096] = [0x3f; 1024 * 4096];
const TRANSFER_LIMIT: u64 = 32 * 1024;
fn main() {
let mut child = std::process::Command::new("cat")
.stdout(std::process::Stdio::piped())
.stdin(std::process::Stdio::piped())
.spawn()
.expect("Could not start child");
let mut input = Cursor::new(&DATA[..]);
let mut output = Cursor::new(Vec::new());
match (child.stdin.as_mut(), child.stdout.as_mut()) {
(Some(stdin), Some(stdout)) => {
while input.position() < input.get_ref().len() as u64 {
io::copy(&mut input.by_ref().take(TRANSFER_LIMIT), stdin).expect("Could not copy input");
io::copy(&mut stdout.take(TRANSFER_LIMIT), &mut output).expect("Could not copy output");
}
},
_ => panic!("child process input and output were not opened"),
}
child.wait().expect("Could not join child");
let res = output.into_inner();
assert_eq!(res.len(), DATA.len());
assert_eq!(&*res, &DATA[..]);
}
If you didn't have that specific restriction, you will need to use select from the libc crate, which requires file descriptors for the pipes so will probably restrict your code to running on Linux / OS X.
You could also start threads, one for each pipe (and reuse the parent thread for one of the pipes), but you've already ruled that out.
I have a Rust application on on OSX firing up a large amount of threads as can be seen in the code below, however, after looking at how many max threads my version of OSX is allowed to create via the sysctl kern.num_taskthreads command, I can see that it is kern.num_taskthreads: 2048 which explains why I can't spin up over 2048 threads.
How do I go about getting past this hard limit?
let threads = 300000;
let requests = 1;
for _x in 0..threads {
println!("{}", _x);
let request_clone = request.clone();
let handle = thread::spawn(move || {
for _y in 0..requests {
request_clone.lock().unwrap().push((request::Request::new(request::Request::create_request())));
}
});
child_threads.push(handle);
}
Before starting, I'd encourage you to read about the C10K problem. When you get into this scale, there's a lot more things you need to keep in mind.
That being said, I'd suggest looking at mio...
a lightweight IO library for Rust with a focus on adding as little overhead as possible over the OS abstractions.
Specifically, mio provides an event loop, which allows you to handle a large number of connections without spawning threads. Unfortunately, I don't know of a HTTP library that currently supports mio. You could create one and be a hero to the Rust community!
Not sure how helpful this will be, but I was trying to create a small pool of threads that will create connections and then send them over to an event loop via a channel for reading.
I'm sure this code is probably pretty bad, but here it is anyways for examples. It uses the Hyper library, like you mentioned.
extern crate hyper;
use std::io::Read;
use std::thread;
use std::thread::{JoinHandle};
use std::sync::{Arc, Mutex};
use std::sync::mpsc::channel;
use hyper::Client;
use hyper::client::Response;
use hyper::header::Connection;
const TARGET: i32 = 100;
const THREADS: i32 = 10;
struct ResponseWithString {
index: i32,
response: Response,
data: Vec<u8>,
complete: bool
}
fn main() {
// Create a client.
let url: &'static str = "http://www.gooogle.com/";
let mut threads = Vec::<JoinHandle<()>>::with_capacity((TARGET * 2) as usize);
let conn_count = Arc::new(Mutex::new(0));
let (tx, rx) = channel::<ResponseWithString>();
for _ in 0..THREADS {
// Move var references into thread context
let conn_count = conn_count.clone();
let tx = tx.clone();
let t = thread::spawn(move || {
loop {
let idx: i32;
{
// Lock, increment, and release
let mut count = conn_count.lock().unwrap();
*count += 1;
idx = *count;
}
if idx > TARGET {
break;
}
let mut client = Client::new();
// Creating an outgoing request.
println!("Creating connection {}...", idx);
let res = client.get(url) // Get URL...
.header(Connection::close()) // Set headers...
.send().unwrap(); // Fire!
println!("Pushing response {}...", idx);
tx.send(ResponseWithString {
index: idx,
response: res,
data: Vec::<u8>::with_capacity(1024),
complete: false
}).unwrap();
}
});
threads.push(t);
}
let mut responses = Vec::<ResponseWithString>::with_capacity(TARGET as usize);
let mut buf: [u8; 1024] = [0; 1024];
let mut completed_count = 0;
loop {
if completed_count >= TARGET {
break; // No more work!
}
match rx.try_recv() {
Ok(r) => {
println!("Incoming response! {}", r.index);
responses.push(r)
},
_ => { }
}
for r in &mut responses {
if r.complete {
continue;
}
// Read the Response.
let res = &mut r.response;
let data = &mut r.data;
let idx = &r.index;
match res.read(&mut buf) {
Ok(i) => {
if i == 0 {
println!("No more data! {}", idx);
r.complete = true;
completed_count += 1;
}
else {
println!("Got data! {} => {}", idx, i);
for x in 0..i {
data.push(buf[x]);
}
}
}
Err(e) => {
panic!("Oh no! {} {}", idx, e);
}
}
}
}
}