What signature can I use to download files using Axum and Tokio? - rust

I'm using axum and this code (found here) to download files:
use axum::{
body::StreamBody,
http::{header, StatusCode},
response::{Headers, IntoResponse},
routing::get,
Router,
};
use std::net::SocketAddr;
use tokio_util::io::ReaderStream;
#[tokio::main]
async fn main() {
let app = Router::new().route("/", get(handler));
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
axum::Server::bind(&addr)
.serve(app.into_make_service())
.await
.unwrap();
}
async fn handler() -> impl IntoResponse {
// `File` implements `AsyncRead`
let file = match tokio::fs::File::open("Cargo.toml").await {
Ok(file) => file,
Err(err) => return Err((StatusCode::NOT_FOUND, format!("File not found: {}", err))),
};
// convert the `AsyncRead` into a `Stream`
let stream = ReaderStream::new(file);
// convert the `Stream` into an `axum::body::HttpBody`
let body = StreamBody::new(stream);
let headers = Headers([
(header::CONTENT_TYPE, "text/toml; charset=utf-8"),
]);
Ok((headers, body))
}
Everything works. But I cannot find a way to move the below code in a separate function:
let file = match tokio::fs::File::open("Cargo.toml").await {
Ok(file) => file,
Err(err) => return Err((StatusCode::NOT_FOUND, format!("File not found: {}", err))),
};
I would like to use both tokio::fs::File and https://crates.io/crates/rust-s3 methods in this function.
So I need a "common type" which appear to be AsyncRead, I think.
What should be the signature of the function?
I tried with:
use tokio::io::AsyncRead;
pub struct Player {
db: Arc<DB>
}
impl Handler {
pub async fn player_pdf(
&self,
id: &str,
) -> Result<&(dyn AsyncRead)> {
//...use id here...
let file = &tokio::fs::File::open("player.pdf").await?;
Ok(file)
}
}
but I get the error:
error[E0308]: mismatched types
|
55 | Ok(file)
| -- ^^^^
| | |
| | expected reference, found struct `tokio::fs::File`
| | help: consider borrowing here: `&file`
| arguments to this enum variant are incorrect
|
= note: expected reference `&dyn tokio::io::AsyncRead`
found struct `tokio::fs::File`
I tried with: let file = &tokio::fs::File::open("player.pdf").await?; and I got:
error[E0515]: cannot return value referencing temporary value
|
43 | let file = &tokio::fs::File::open(...
| --------------------------- temporary value created here
...
55 | Ok(file)
| ^^^^^^^^ returns a value referencing data owned by the current function
What can I use?

Returning a generic "boxed" value might be the solution here:
impl Handler {
pub async fn player_pdf(
&self,
id: &str,
) -> Result<Box<dyn AsyncRead>> {
//...use id here...
Ok(Box::new(tokio::fs::File::open("player.pdf").await?))
}
}
Where now there's no dangling reference, it's encapsulated and fully owned.

Related

Share state between actix-web server and async closure

I want to periodically fetch data (using asynchronous reqwest), which is then served at an http endpoint using actix-web as a server.
(I have a data source that has a fixed format, that I want to have read by a service that require a different format, so I need to transform the data.)
I've tried to combine actix concepts with the thread sharing state example from the Rust book, but I don't understand the error or how to solve it.
This is the code minified as much as I was able:
use actix_web::{get, http, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use tokio::time::{sleep, Duration};
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Mutex<AppState>>) -> impl Responder {
let state = data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.to_owned())
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
let status_string = get_state().await.unwrap();
let app_data = Arc::new(Mutex::new(web::Data::new(AppState {
status: status_string,
})));
let app_data1 = Arc::clone(&app_data);
actix_web::rt::spawn(async move {
loop {
println!("I get executed every 2-ish seconds!");
sleep(Duration::from_millis(2000)).await;
let res = get_state().await;
let mut app_data = app_data1.lock().unwrap();
// Edit 2: this line is not accepted by the compiler
// Edit 2: *app_data.status = res.unwrap();
// Edit 2: but this line is accepted
*app_data = web::Data::new(AppState { status: res });
}
});
let app_data2 = Arc::clone(&app_data);
// Edit 2: but I get an error here now
HttpServer::new(move || App::new().app_data(app_data2).service(index))
.bind(("127.0.0.1", 9090))?
.run()
.await
}
async fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let client = reqwest::Client::new().get("http://ipecho.net/plain".to_string());
let status = client.send().await?.text().await?;
println!("got status: {status}");
Ok(status)
}
But I get the following error:
error[E0308]: mismatched types
--> src/main.rs:33:32
|
33 | *app_data.status = res.unwrap();
| ---------------- ^^^^^^^^^^^^ expected `str`, found struct `String`
| |
| expected due to the type of this binding
error[E0277]: the size for values of type `str` cannot be known at compilation time
--> src/main.rs:33:13
|
33 | *app_data.status = res.unwrap();
| ^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `std::marker::Sized` is not implemented for `str`
= note: the left-hand-side of an assignment must have a statically known size
Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
Why do I suddenly get a str? Is there an easy fix or is my approach to solving this wrong?
Edit: Maybe removing the * is the right way to go, as Peter Hall suggests, but that gives me the following error instead:
error[E0594]: cannot assign to data in an `Arc`
--> src/main.rs:33:13
|
33 | app_data.status = res.unwrap();
| ^^^^^^^^^^^^^^^ cannot assign
|
= help: trait `DerefMut` is required to modify through a dereference, but it is not implemented for `Arc<AppState>`
error[E0507]: cannot move out of `app_data2`, a captured variable in an `Fn` closure
--> src/main.rs:38:49
|
37 | let app_data2 = Arc::clone(&app_data);
| --------- captured outer variable
38 | HttpServer::new(move || App::new().app_data(app_data2).service(index))
| ------- ^^^^^^^^^ move occurs because `app_data2` has type `Arc<std::sync::Mutex<Data<AppState>>>`, which does not implement the `Copy` trait
| |
| captured by this `Fn` closure
Some errors have detailed explanations: E0507, E0594.
For more information about an error, try `rustc --explain E0507`.
Edit 2: I now get the following error (code changes commented with 'Edit 2' above):
error[E0507]: cannot move out of `app_data2`, a captured variable in an `Fn` closure
--> src/main.rs:46:49
|
45 | let app_data2 = app_data.clone();
| --------- captured outer variable
46 | HttpServer::new(move || App::new().app_data(app_data2).service(index))
| ------- ^^^^^^^^^ move occurs because `app_data2` has type `Arc<Mutex<Data<AppState>>>`, which does not implement the `Copy` trait
| |
| captured by this `Fn` closure
For more information about this error, try `rustc --explain E0507`.
My Cargo.toml dependencies:
[dependencies]
actix-web = "4.2.1"
reqwest = "0.11.12"
tokio = "1.21.2"
async solution
I had my types mixed up a bit, having the app state as Arc<Mutex<T>> seemed to be the way to go, maybe it would be better with Arc<RwLock<T>>.
use actix_web::{get, http, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use tokio::time::{sleep, Duration};
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Arc<Mutex<AppState>>>) -> impl Responder {
let state = data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.to_owned())
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
let status_string = get_state().await.unwrap();
let app_data = Arc::new(Mutex::new(AppState {
status: status_string,
}));
let app_data1 = app_data.clone();
actix_web::rt::spawn(async move {
loop {
println!("I get executed every 2-ish seconds!");
sleep(Duration::from_millis(2000)).await;
let res = get_state().await.unwrap();
let mut app_data = app_data1.lock().unwrap();
*app_data = AppState { status: res };
}
});
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(app_data.clone()))
.service(index)
})
.bind(("127.0.0.1", 9090))?
.run()
.await
}
async fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let client = reqwest::Client::new().get("http://ipecho.net/plain".to_string());
let status = client.send().await?.text().await?;
println!("got status: {status}");
Ok(status)
}
async/sync solution
Instead of doing the async get with reqwest I have a solution with the synchronous crate minreq (that I found after a lot of searching). I also chose to not use the #[actix_web::main] macro, and instead start the runtime explicitly at the end of my main function.
use actix_web::{get, http, rt, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Arc<Mutex<AppState>>>) -> impl Responder {
let state = &data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.clone())
}
fn main() -> std::io::Result<()> {
let status_string = get_state().unwrap();
let app_data = Arc::new(Mutex::new(AppState {
status: status_string,
}));
let app_data1 = Arc::clone(&app_data);
thread::spawn(move || loop {
thread::sleep(Duration::from_millis(2000));
let res = get_state().unwrap();
let mut app_data = app_data1.lock().unwrap();
*app_data = AppState { status: res };
});
rt::System::new().block_on(
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(app_data.clone()))
.service(index)
})
.bind(("127.0.0.1", 9090))?
.run(),
)
}
fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let resp = minreq::get("http://ipecho.net/plain").send().unwrap();
let state = resp.as_str().unwrap();
Ok(state.to_string())
}

Rust "this parameter and the return type are declared with different lifetimes"

I'm using the smol library from Rust. None of the other answers to this question helped.
The smol's Executor::spawn() is declared like so:
pub fn spawn<T: Send + 'a>(&self, future: impl Future<Output = T> + Send + 'a) -> Task<T> {
Now I have a function and want to call spawn recursively like so:
async fn start(executor: &Executor<'_>) {
let server_task = executor.spawn(async {
executor.spawn(async { println!("hello"); }).await;
});
}
But I'm getting this error:
9 | async fn start(executor: &Executor<'_>) {
| ------------ -
| |
| this parameter and the return type are declared with different lifetimes...
...
18 | let server_task = executor.spawn(async {
| ^^^^^ ...but data from `executor` is returned here
How can I resolve this error? I'm very confused.
use {
smol::{block_on, Executor},
std::sync::Arc,
};
// --
fn main() {
let ex = Arc::new(Executor::new());
block_on(ex.run(start(ex.clone())));
}
async fn start(executor: Arc<Executor<'_>>) {
let ex2 = executor.clone();
let server_task = executor.spawn(async move {
let t = ex2.spawn(async {
println!("hello");
});
t.await;
});
server_task.await;
}

mismatched String type in Rust

I'm currently trying to learn Rust, and when I was trying to follow the official book tutorial I have encountered an error that was not described in the book.
use std::env;
use std::fs;
fn main() {
let args: Vec<String> = env::args().collect();
let config = Config::new(&args);
println!("Searching for {}", config.query);
println!("In file {}", config.filename);
println!("{:?}", args);
let contents = fs::read_to_string(config.filename)
.expect("Something went wrong reading the file");
println!("With text:\n{}", contents);
}
struct Config {
query: String,
filename: String,
}
impl Config {
fn new(args: &[String]) -> Config {
let query = &args[1].clone();
let filename = &args[2].clone();
Config { query, filename }
}
}
but I kept getting the following error from the compiler:
error[E0308]: mismatched types
--> src/main.rs:25:18
|
25 | Config { query, filename }
| ^^^^^
| |
| expected struct `std::string::String`, found `&str`
| help: try using a conversion method: `query: query.to_string()`
error[E0308]: mismatched types
--> src/main.rs:25:25
|
25 | Config { query, filename }
| ^^^^^^^^
| |
| expected struct `std::string::String`, found `&std::string::String`
| help: try using a conversion method: `filename: filename.to_string()`
error: aborting due to 2 previous errors
I have tried doing something like:
let query = &args[1].clone().to_string();
let filename = &args[2].clone().to_owned().to_string();
but it brought the same error. Finally, I have managed to solve this issue by applying the following changes:
struct Config<'a> {
query: &'a str,
filename: &'a str,
}
impl Config <'_> {
fn new(args: &[String]) -> Config {
let query = &*args[1];
let filename = &*args[2];
Config { query, filename }
}
}
My understanding is that new() could not return a struct of owned strings because values query and filename were assigned referenced values and for some reason, neither of the functions to_string(), to_owned() and clone() were not converting a referenced Sting into an owned one.
Does anyone know why I had to use referenced sices with lifetimes instead of strings?
Also, is there a way to use String type just like in the tutorial(https://doc.rust-lang.org/book/ch12-03-improving-error-handling-and-modularity.html)?
the query and file name type is &String, the clone method use &self, You should do like this
impl Config {
fn new(args: &[String]) -> Config {
let query = args[1].clone();
let filename = args[2].clone();
Config { query, filename }
}
}

How to write an asynchronous recursive walkdir function with an asynchronous callback

I'm trying to write an async function that will traverse the filesystem tree, recursively, and calls an asynchronous callback for each file found.
This is for a learning effort, I have no real use case.
Here is what I have so far:
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0, features = ["unstable"]
use futures::{
executor::block_on,
future::{BoxFuture, FutureExt},
}; // 0.3.4
use std::{marker::Sync, pin::Pin};
fn main() {
fn walkdir<F>(path: String, cb: &'static F) -> BoxFuture<'static, ()>
where
F: Fn(&DirEntry) -> BoxFuture<()> + Sync + Send,
{
async move {
let mut entries = fs::read_dir(&path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path().to_str().unwrap().to_string();
if entry.path().is_file().await {
cb(&entry).await
} else {
walkdir(path, cb).await
}
}
}
.boxed()
}
let foo = async {
walkdir(".".to_string(), &|entry: &DirEntry| async {
async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
})
.await
};
block_on(foo);
}
I get this far by some sort of trial and error, but now I'm stuck on async closure callback with this error
warning: unused import: `path::*`
--> src/main.rs:3:5
|
3 | path::*,
| ^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `pin::Pin`
--> src/main.rs:10:25
|
10 | use std::{marker::Sync, pin::Pin};
| ^^^^^^^^
error[E0308]: mismatched types
--> src/main.rs:33:54
|
33 | walkdir(".".to_string(), &|entry: &DirEntry| async {
| ______________________________________________________^
34 | | async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
35 | | })
| |_________^ expected struct `std::pin::Pin`, found opaque type
|
= note: expected struct `std::pin::Pin<std::boxed::Box<dyn core::future::future::Future<Output = ()> + std::marker::Send>>`
found opaque type `impl core::future::future::Future`
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0
use futures::{future::{Future, FutureExt, LocalBoxFuture}, executor}; // 0.3.4
fn main() {
async fn walkdir<R>(path: impl AsRef<Path>, mut cb: impl FnMut(DirEntry) -> R)
where
R: Future<Output = ()>,
{
fn walkdir_inner<'a, R>(path: &'a Path, cb: &'a mut dyn FnMut(DirEntry) -> R) -> LocalBoxFuture<'a, ()>
where
R: Future<Output = ()>,
{
async move {
let mut entries = fs::read_dir(path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path();
if path.is_file().await {
cb(entry).await
} else {
walkdir_inner(&path, cb).await
}
}
}.boxed_local()
}
walkdir_inner(path.as_ref(), &mut cb).await
}
executor::block_on({
walkdir(".", |entry| async move {
async_std::println!(">> {}", entry.path().display()).await
})
});
}
Notable changes:
Take in AsRef<Path> instead of a String and a generic closure instead of a trait object reference
Change the closure type to be FnMut as it's more permissive
The closure returns any type that is a future.
There's an inner implementation function that hides the ugly API required for recursive async functions.
The callback takes the DirEntry by value instead of by reference.
See also:
How to asynchronously explore a directory and its sub-directories?
How to using async fn callback in rust

How do I use a value after calling Option::map?

I am trying to:
Get an Option<&str> from somewhere, and build a PathBuf from it.
If None, print some message, and return.
If the path is not a directory, print a message saying the path is not a directory, and return.
If everything is good, continue the program.
use std::path::PathBuf;
fn it_works() {
let path_str = Some("/tmp/abc");
let path = path_str.map(|s| PathBuf::from(s));
if !path.map_or(false, |p| p.is_dir()) {
match path {
Some(p) => println!("The folder {:?} is not a directory!", p),
None => println!("The repository folder is not set!"),
}
return;
}
}
The pattern matching in the above snippet doesn't work because the value has been moved in the map_or combinator:
error[E0382]: use of moved value
--> src/lib.rs:8:18
|
5 | let path = path_str.map(|s| PathBuf::from(s));
| ---- move occurs because `path` has type `std::option::Option<std::path::PathBuf>`, which does not implement the `Copy` trait
6 | if !path.map_or(false, |p| p.is_dir()) {
| ---- value moved here
7 | match path {
8 | Some(p) => println!("The folder {:?} is not a directory!", p),
| ^ value used here after move
I can do something like this, but it doesn't feel very "idiomatic" because of the unwrap and multiple if clauses:
let path_str = Some("/tmp/abc");
let path = path_str.map(|s| PathBuf::from(s));
if path.is_none() {
println!("The repository folder is not set!");
return;
}
let p = path.unwrap();
if !p.is_dir() {
println!("The folder {:?} is not a directory!", p);
}
Could there be a better to solve this?
If the closure in .map(...) (or any similar functions on an Option) doesn't need ownership of the value in the option (i.e. it only needs a reference to the value), you can always use option.as_ref() or option.as_mut() to turn an &Option<T> or &mut Option<T> into an Option<&T> or Option<&mut T>. Then calling .map() will not take ownership because references are copyable, so it's just copied into the provided closure.
With this in mind, your code would be modified to be this:
fn it_works() {
let path_str = Some("/tmp/abc");
let path = path_str.map(|s| PathBuf::from(s));
if !path.as_ref().map_or(false, |p| p.is_dir()) {
// ^^^^^^^^^ using .as_ref() here
// ^^^ now p is a '&PathBuf' instead of 'PathBuf'
match path {
// ^^^^ we didn't take ownership so compiler doesn't complain here
Some(p) => println!("The folder {:?} is not a directory!", p),
None => println!("The repository folder is not set!"),
}
return;
}
}
PathBuf implements FromStr trait, so you can use it, in combination with powerful pattern matching.
fn it_works() {
use std::path::*;
use std::str::FromStr;
let path_str: Option<&str> = Some("/tmp/abc");
match path_str.map(PathBuf::from_str) {
Some(Ok(p)) => if !p.is_dir() {},
Some(Err(e)) => {}
None => {}
};
}

Resources