I am trying to write integration tests for the official multithreaded web server tutorial.
In integrational_tests.rs, I have something like this:
fn connect_and_send_request() {
let stream = TcpStream::connect("127.0.0.1:7879").unwrap();
let mut writer = BufWriter::new(stream);
let s = b"GET / HTTP/1.1\r\n\r\n";
writer.write(s).unwrap();
writer.flush().unwrap();
}
fn test_main_route() {
let server_address = [SocketAddr::from(([127, 0, 0, 1], 7879))];
let listener: TcpListener = TcpListener::bind(&server_address[..]).unwrap();
let pool = ThreadPool::new(4);
connect_and_send_request();
for stream in listener.incoming() {
let mut stream = stream.unwrap();
pool.execute(|| {
let main_mock = mock("GET", "/").create();
handle_connection(stream);
println!("Main mock {}", main_mock);
main_mock.assert();
});
}
fn handle_connection(mut stream: TcpStream) {
let mut response = [0; 512];
stream.read(&mut response).unwrap();
stream.flush().unwrap();
println!("Request: {}", String::from_utf8_lossy(&response[..]));
}
}
In my "Request" println! I see my request string, but main_mock.assert() based on Mockito fails:
thread <unnamed> panicked at assertion failed: (left == right)
expect GET / ... but recived 0.
Related
I'm making a simple chat using Tokio for learning. I've made a server with TcpListener that works fine when I use telnets conexions. For example, if I open 2 terminals with telnet and send data, the server receives the conexion and send the data to the address that is not the sender.
The problem I have is with the TcpStream. For some reason (I think is some type of blocking problem) the stream is not sending the data until I stop the app.
I'd tried multiple configurations, but I think that the simple requiered that is close to "work" is the next one.
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let mut stream = TcpStream::connect("127.0.0.1:8080").await?;
println!("INFO: Socket Address: {:?}", stream.local_addr().unwrap());
let (read, mut writer) = stream.split();
let (sender, receiver) = channel();
thread::spawn(move || loop {
let mut line_buf = String::new();
stdin().read_line(&mut line_buf);
let line = line_buf.trim_end().to_string();
if line.len() > 0 {
sender.send(line_buf.trim_end().to_string()).unwrap();
}
});
loop {
let line = receiver.recv().unwrap();
println!("DEBUG User Input {line}");
writer.write_all(line.as_bytes()).await?;
}
P.D: this code doesn't have anything for receive the messages sended by the server. For manage that, I had something like:
tokio::spawn(async move {
let (read, mut writer) = socket.split();
let mut reader = BufReader::new(read);
let mut line = String::new();
loop {
tokio::select! {
result = reader.read_line(&mut line) => {
if result.unwrap() == 0 {
break;
}
print!("DEBUG: Message Received: {line}");
tx.send((line.clone(), addr)).unwrap();
line.clear();
}
result = rx.recv() => {
let (msg, other_addr) = result.unwrap();
println!("INFO: Message Address: {addr}\n\n");
if addr != other_addr {
writer.write_all(msg.as_bytes()).await.unwrap();
}
}
}
}
});
I ahve a really simple tcp client/server setup that I found somewhere online.
use std::net::{Shutdown,TcpListener, TcpStream};
use std::thread;
use std::io::{Read,Write,Error};
fn handle_client(mut stream: TcpStream)-> Result<(), Error> {
println!("incoming connection from: {}", stream.peer_addr()?);
let mut buf = [0;512];
loop {
let bytes_read = stream.read(&mut buf)?;
if bytes_read == 0 {return Ok(())}
let tmp = format!("{}", String::from_utf8_lossy(&buf).trim());
eprintln!("getting {}",tmp);
stream.write(&vec![1,2,3,4])?;
}
}
fn main() {
let listener = TcpListener::bind("0.0.0.0:8888").expect("Could not bind");
let mut i = 0;
for stream in listener.incoming() {
match stream {
Err(e)=> {eprintln!("failed: {}", e)}
Ok(stream) => {
thread::spawn(move || {
handle_client(stream).unwrap_or_else(|error| eprintln!("{:?}", error));
});
}
}
}
}
and the client looks like so:
use std::net::TcpStream;
use std::str;
use std::io::{self,BufRead,BufReader,Write};
fn main() {
let mut stream = TcpStream::connect("0.0.0.0:8888").expect("could not connect");
loop {
let mut input = String::new();
let mut buffer : Vec<u8> = Vec::new();
io::stdin().read_line(&mut input).expect("failed to read stdin");
stream.write(input.as_bytes()).expect("Failed to write to server");
let mut reader = BufReader::new(&stream);
reader.read_until(b'\n', &mut buffer).expect("Could not read into buffer");
println!("{}", str::from_utf8(&buffer).expect("msg: &str"))
}
println!("Hello, world!");
}
(actually vs code tells me that the code inside the loop here is unreachable, which is false)
But this basically just lets the client send a message to the server, that the server sends back.
I would actually like to send vectors back and forth, which actually seems like a simpler task.
So, I change the line in the server that writes to the stream to this:
stream.write(&vec![1,2,3,4])?;
and now, in the client that prints the message it gets from the serve to:
println!("{:?}", &buffer)
But when I do this, nothing happens on the clientside, nothing is printed, and it seems like the loop is just stuck somewhere.
I gugess it has something to do with this line:
reader.read_until(b'\n', &mut buffer).expect("Could not read into buffer");
and I read some other way of reading in. I tried using a function from tcpstream called "read_vectored", but I can't use it at all for the stream object I have.
for downloading with reqwest and tokio and progress I am using the code below
pub async fn download_file(client: &Client, url: &str, path: &str) -> Result<(), String> {
// Reqwest setup
let res = client
.get(url)
.send()
.await
.or(Err(format!("Failed to GET from '{}'", &url)))?;
let total_size = res
.content_length()
.ok_or(format!("Failed to get content length from '{}'", &url))?;
// Indicatif setup
let pb = ProgressBar::new(total_size);
pb.set_style(ProgressStyle::default_bar()
.template("{msg}\n{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {bytes}/{total_bytes} ({bytes_per_sec}, {eta})")
.progress_chars("#>-"));
pb.set_message(format!("Downloading {}", url));
// download chunks
let mut file = File::create(path).or(Err(format!("Failed to create file '{}'", path)))?;
let mut downloaded: u64 = 0;
let mut stream = res.bytes_stream();
while let Some(item) = stream.next().await {
let chunk = item.or(Err(format!("Error while downloading file")))?;
file.write(&chunk)
.or(Err(format!("Error while writing to file")))?;
let new = min(downloaded + (chunk.len() as u64), total_size);
downloaded = new;
pb.set_position(new);
}
pb.finish_with_message(format!("Downloaded {} to {}", url, path));
return Ok(());
}
from the while loop I can set progress and see progressbar like examples here https://github.com/mitsuhiko/indicatif
now I am trying to find make progressbar from upload, but could not find the way to monitor reqwest client, code below is my upload function
pub async fn upload_file(client: &Client, url: &str, path: &str) -> Result<(), String> {
let f = File::open(path).expect("Unable to open file");
let total_size = f.metadata().unwrap().len();
// Indicatif setup
let pb = ProgressBar::new(total_size);
pb.set_style(ProgressStyle::default_bar()
.template("{msg}\n{spinner:.green} [{elapsed_precise}] [{wide_bar:.cyan/blue}] {bytes}/{total_bytes} ({bytes_per_sec}, {eta})")
.progress_chars("#>-"));
pb.set_message(format!("Posting {}", url));
let file = tokio::fs::File::open(path).await.unwrap();
let stream = FramedRead::new(file, BytesCodec::new());
let res=client
.post(url)
.body(Body::wrap_stream(stream))
.send()
.await;
pb.finish_with_message(format!("Uploaded {} to {}", url, path));
return Ok(());
}
upload works but no progressbar with percent or any indicator. there should have been status monitor, like below
.post(url)
.body(Body::wrap_stream(stream))
.send()
.monitorStatus(|stat|{
pb.set_position(stat);
}).....
you can see working code here https://github.com/ozkanpakdil/rust-examples/blob/5f4965f2b086d07c8294352182639dc75232bb30/download_upload/src/download_file.rs#L43 just uncomment those tests and run cargo test
My question is, how to monitor reqwest client for upload and making a progressbar from it ?
You can create an async_stream and yield chunks of the input to upload:
let file = tokio::fs::File::open(&input).await.unwrap();
let total_size = file.metadata().await.unwrap().len();
let input_ = input.to_string();
let output_ = output.to_string();
let mut reader_stream = ReaderStream::new(file);
let mut uploaded = HTTPSHandler::get_already_uploaded(output).await;
bar.set_length(total_size);
let async_stream = async_stream::stream! {
while let Some(chunk) = reader_stream.next().await {
if let Ok(chunk) = &chunk {
let new = min(uploaded + (chunk.len() as u64), total_size);
uploaded = new;
bar.set_position(new);
if(uploaded >= total_size){
bar.finish_upload(&input_, &output_);
}
}
yield chunk;
}
};
Then, just wrap the stream when building the Body:
let _ = reqwest::Client::new()
.put(output)
.header("content-type", "application/octet-stream")
.header("Range", "bytes=".to_owned() + &uploaded.to_string() + "-")
.header(
reqwest::header::USER_AGENT,
reqwest::header::HeaderValue::from_static(CLIENT_ID),
)
.body(reqwest::Body::wrap_stream(async_stream))
.send()
.await
.unwrap();
Btw, have a look at the implementation of aim, I've faced similar problems there!
I'm trying to upload a file to aws in rust, for that I'm using the s3 rust client by rusoto_s3, I managed to get the multipart upload code working when these parts are sent from a single thread, however, that is not what I want, I want to upload big files and I want to be able to send these parts in multiple threads, for that, I did a little bit of googling and I came across rayon.
For info the way multipart upload works is as follows:
Initiate the multipart -> aws will return an ID
Use this ID to send the different parts, pass the file chunk, and the part number -> aws will return an Etag
Once you sent all the parts, send a complete upload request with all the completed parts as an array contains the Etag and the part number.
I'm new to rust, coming from C++ and Java background, here is my code:
#[tokio::test]
async fn if_multipart_then_upload_multiparts_dicom() {
let now = Instant::now();
dotenv().ok();
let local_filename = "./files/test_big.DCM";
let destination_filename = "24_time_test.dcm";
let mut file = std::fs::File::open(local_filename).unwrap();
const CHUNK_SIZE: usize = 7_000_000;
let mut buffer = Vec::with_capacity(CHUNK_SIZE);
let client = super::get_client().await;
let create_multipart_request = CreateMultipartUploadRequest {
bucket: client.bucket_name.to_owned(),
key: destination_filename.to_owned(),
..Default::default()
};
// Start the multipart upload and note the upload_id generated
let response = client
.s3
.create_multipart_upload(create_multipart_request)
.await
.expect("Couldn't create multipart upload");
let upload_id = response.upload_id.unwrap();
// Create upload parts
let create_upload_part = |body: Vec<u8>, part_number: i64| -> UploadPartRequest {
UploadPartRequest {
body: Some(body.into()),
bucket: client.bucket_name.to_owned(),
key: destination_filename.to_owned(),
upload_id: upload_id.to_owned(),
part_number: part_number,
..Default::default()
}
};
let completed_parts = Arc::new(Mutex::new(vec![]));
rayon::scope(|scope| {
let mut part_number = 1;
loop {
let maximum_bytes_to_read = CHUNK_SIZE - buffer.len();
println!("maximum_bytes_to_read: {}", maximum_bytes_to_read);
file.by_ref()
.take(maximum_bytes_to_read as u64)
.read_to_end(&mut buffer)
.unwrap();
println!("length: {}", buffer.len());
println!("part_number: {}", part_number);
if buffer.len() == 0 {
// The file has ended.
break;
}
let next_buffer = Vec::with_capacity(CHUNK_SIZE);
let data_to_send = buffer;
let completed_parts_cloned = completed_parts.clone();
scope.spawn(move |_| {
let part = create_upload_part(data_to_send.to_vec(), part_number);
{
let part_number = part.part_number;
let client = executor::block_on(super::get_client());
let response = executor::block_on(client.s3.upload_part(part));
completed_parts_cloned.lock().unwrap().push(CompletedPart {
e_tag: response
.expect("Couldn't complete multipart upload")
.e_tag
.clone(),
part_number: Some(part_number),
});
}
});
buffer = next_buffer;
part_number = part_number + 1;
}
});
let completed_upload = CompletedMultipartUpload {
parts: Some(completed_parts.lock().unwrap().to_vec()),
};
let complete_req = CompleteMultipartUploadRequest {
bucket: client.bucket_name.to_owned(),
key: destination_filename.to_owned(),
upload_id: upload_id.to_owned(),
multipart_upload: Some(completed_upload),
..Default::default()
};
client
.s3
.complete_multipart_upload(complete_req)
.await
.expect("Couldn't complete multipart upload");
println!(
"time taken: {}, with chunk:: {}",
now.elapsed().as_secs(),
CHUNK_SIZE
);
}
here is a sample of the output and error I'm getting:
maximum_bytes_to_read: 7000000
length: 7000000
part_number: 1
maximum_bytes_to_read: 7000000
length: 7000000
part_number: 2
maximum_bytes_to_read: 7000000
thread '<unnamed>' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', C:\Users\DNDT\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.2.0\src\runtime\blocking\pool.rs:85:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'there is no reactor running, must be called from the context of a Tokio 1.x runtime', C:\Users\DNDT\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-1.2.0\src\runtime\blocking\pool.rs:85:33
length: 7000000
I googled this error but I did not have a clear understanding on what actually its:
there is no reactor running, must be called from the context of Tokio runtime”
Here is what I found:
another question with the same error
and another question
Which seems its some compatibility issue because s3 might be using some version of tokio that is not compatible with the version of tokio I have.
Here are some relevant dependencies:
tokio = { version = "1", features = ["full"] }
tokio-compat-02 = "0.1.2"
rusoto_s3 = "0.46.0"
rusoto_core = "0.46.0"
rusoto_credential = "0.46.0"
rayon = "1.5.0"
I think the main issue comes on actually wanting to run async code in a rayon thread. I tried changing my async code to blocking code using executor::block_on, I also spend some time trying to make the compiler happy, I have multiple threads they all want to write to let completed_parts = Arc::new(Mutex::new(vec![])); so I did some cloning here to make the complier happy.
Also if my used craes matter, here are they:
extern crate dotenv;
extern crate tokio;
use bytes::Bytes;
use dotenv::dotenv;
use futures::executor;
use futures::*;
use rusoto_core::credential::{EnvironmentProvider, ProvideAwsCredentials};
use rusoto_s3::util::{PreSignedRequest, PreSignedRequestOption};
use rusoto_s3::PutObjectRequest;
use rusoto_s3::StreamingBody;
use rusoto_s3::{
CompleteMultipartUploadRequest, CompletedMultipartUpload, CompletedPart,
CreateMultipartUploadRequest, UploadPartRequest, S3,
};
use std::io::Read;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use std::time::Instant;
use tokio::fs;
New to rust, so there a lot of moving pieces to get this one right!
Thanks #Jmb for the discussion, I got rid of the threads and I spawn a tokio task as follows:
create a vector to hold or the futures so we could wait for them:
let mut multiple_parts_futures = Vec::new();
spawn the async task:
loop { // loop file chuncks
...
let send_part_task_future = tokio::task::spawn(async move {
// Upload part
...
}
and then later wait for all futures:
let _results = futures::future::join_all(multiple_parts_futures).await;
worth mentioning, the completed parts need to be sorted:
let mut completed_parts_vector = completed_parts.lock().unwrap().to_vec();
completed_parts_vector.sort_by_key(|part| part.part_number);
The whole code is:
#[tokio::test]
async fn if_multipart_then_upload_multiparts_dicom() {
let now = Instant::now();
dotenv().ok();
let local_filename = "./files/test_big.DCM";
let destination_filename = generate_unique_name();
let destination_filename_clone = destination_filename.clone();
let mut file = std::fs::File::open(local_filename).unwrap();
const CHUNK_SIZE: usize = 6_000_000;
let mut buffer = Vec::with_capacity(CHUNK_SIZE);
let client = super::get_client().await;
let create_multipart_request = CreateMultipartUploadRequest {
bucket: client.bucket_name.to_owned(),
key: destination_filename.to_owned(),
..Default::default()
};
// Start the multipart upload and note the upload_id generated
let response = client
.s3
.create_multipart_upload(create_multipart_request)
.await
.expect("Couldn't create multipart upload");
let upload_id = response.upload_id.unwrap();
let upload_id_clone = upload_id.clone();
// Create upload parts
let create_upload_part = move |body: Vec<u8>, part_number: i64| -> UploadPartRequest {
UploadPartRequest {
body: Some(body.into()),
bucket: client.bucket_name.to_owned(),
key: destination_filename_clone.to_owned(),
upload_id: upload_id_clone.to_owned(),
part_number: part_number,
..Default::default()
}
};
let create_upload_part_arc = Arc::new(create_upload_part);
let completed_parts = Arc::new(Mutex::new(vec![]));
let mut part_number = 1;
let mut multiple_parts_futures = Vec::new();
loop {
let maximum_bytes_to_read = CHUNK_SIZE - buffer.len();
println!("maximum_bytes_to_read: {}", maximum_bytes_to_read);
file.by_ref()
.take(maximum_bytes_to_read as u64)
.read_to_end(&mut buffer)
.unwrap();
println!("length: {}", buffer.len());
println!("part_number: {}", part_number);
if buffer.len() == 0 {
// The file has ended.
break;
}
let next_buffer = Vec::with_capacity(CHUNK_SIZE);
let data_to_send = buffer;
let completed_parts_cloned = completed_parts.clone();
let create_upload_part_arc_cloned = create_upload_part_arc.clone();
let send_part_task_future = tokio::task::spawn(async move {
let part = create_upload_part_arc_cloned(data_to_send.to_vec(), part_number);
{
let part_number = part.part_number;
let client = super::get_client().await;
let response = client.s3.upload_part(part).await;
completed_parts_cloned.lock().unwrap().push(CompletedPart {
e_tag: response
.expect("Couldn't complete multipart upload")
.e_tag
.clone(),
part_number: Some(part_number),
});
}
});
multiple_parts_futures.push(send_part_task_future);
buffer = next_buffer;
part_number = part_number + 1;
}
let client = super::get_client().await;
println!("waiting for futures");
let _results = futures::future::join_all(multiple_parts_futures).await;
let mut completed_parts_vector = completed_parts.lock().unwrap().to_vec();
completed_parts_vector.sort_by_key(|part| part.part_number);
println!("futures done");
let completed_upload = CompletedMultipartUpload {
parts: Some(completed_parts_vector),
};
let complete_req = CompleteMultipartUploadRequest {
bucket: client.bucket_name.to_owned(),
key: destination_filename.to_owned(),
upload_id: upload_id.to_owned(),
multipart_upload: Some(completed_upload),
..Default::default()
};
client
.s3
.complete_multipart_upload(complete_req)
.await
.expect("Couldn't complete multipart upload");
println!(
"time taken: {}, with chunk:: {}",
now.elapsed().as_secs(),
CHUNK_SIZE
);
}
I have a Rust application on on OSX firing up a large amount of threads as can be seen in the code below, however, after looking at how many max threads my version of OSX is allowed to create via the sysctl kern.num_taskthreads command, I can see that it is kern.num_taskthreads: 2048 which explains why I can't spin up over 2048 threads.
How do I go about getting past this hard limit?
let threads = 300000;
let requests = 1;
for _x in 0..threads {
println!("{}", _x);
let request_clone = request.clone();
let handle = thread::spawn(move || {
for _y in 0..requests {
request_clone.lock().unwrap().push((request::Request::new(request::Request::create_request())));
}
});
child_threads.push(handle);
}
Before starting, I'd encourage you to read about the C10K problem. When you get into this scale, there's a lot more things you need to keep in mind.
That being said, I'd suggest looking at mio...
a lightweight IO library for Rust with a focus on adding as little overhead as possible over the OS abstractions.
Specifically, mio provides an event loop, which allows you to handle a large number of connections without spawning threads. Unfortunately, I don't know of a HTTP library that currently supports mio. You could create one and be a hero to the Rust community!
Not sure how helpful this will be, but I was trying to create a small pool of threads that will create connections and then send them over to an event loop via a channel for reading.
I'm sure this code is probably pretty bad, but here it is anyways for examples. It uses the Hyper library, like you mentioned.
extern crate hyper;
use std::io::Read;
use std::thread;
use std::thread::{JoinHandle};
use std::sync::{Arc, Mutex};
use std::sync::mpsc::channel;
use hyper::Client;
use hyper::client::Response;
use hyper::header::Connection;
const TARGET: i32 = 100;
const THREADS: i32 = 10;
struct ResponseWithString {
index: i32,
response: Response,
data: Vec<u8>,
complete: bool
}
fn main() {
// Create a client.
let url: &'static str = "http://www.gooogle.com/";
let mut threads = Vec::<JoinHandle<()>>::with_capacity((TARGET * 2) as usize);
let conn_count = Arc::new(Mutex::new(0));
let (tx, rx) = channel::<ResponseWithString>();
for _ in 0..THREADS {
// Move var references into thread context
let conn_count = conn_count.clone();
let tx = tx.clone();
let t = thread::spawn(move || {
loop {
let idx: i32;
{
// Lock, increment, and release
let mut count = conn_count.lock().unwrap();
*count += 1;
idx = *count;
}
if idx > TARGET {
break;
}
let mut client = Client::new();
// Creating an outgoing request.
println!("Creating connection {}...", idx);
let res = client.get(url) // Get URL...
.header(Connection::close()) // Set headers...
.send().unwrap(); // Fire!
println!("Pushing response {}...", idx);
tx.send(ResponseWithString {
index: idx,
response: res,
data: Vec::<u8>::with_capacity(1024),
complete: false
}).unwrap();
}
});
threads.push(t);
}
let mut responses = Vec::<ResponseWithString>::with_capacity(TARGET as usize);
let mut buf: [u8; 1024] = [0; 1024];
let mut completed_count = 0;
loop {
if completed_count >= TARGET {
break; // No more work!
}
match rx.try_recv() {
Ok(r) => {
println!("Incoming response! {}", r.index);
responses.push(r)
},
_ => { }
}
for r in &mut responses {
if r.complete {
continue;
}
// Read the Response.
let res = &mut r.response;
let data = &mut r.data;
let idx = &r.index;
match res.read(&mut buf) {
Ok(i) => {
if i == 0 {
println!("No more data! {}", idx);
r.complete = true;
completed_count += 1;
}
else {
println!("Got data! {} => {}", idx, i);
for x in 0..i {
data.push(buf[x]);
}
}
}
Err(e) => {
panic!("Oh no! {} {}", idx, e);
}
}
}
}
}