Share state between actix-web server and async closure - rust

I want to periodically fetch data (using asynchronous reqwest), which is then served at an http endpoint using actix-web as a server.
(I have a data source that has a fixed format, that I want to have read by a service that require a different format, so I need to transform the data.)
I've tried to combine actix concepts with the thread sharing state example from the Rust book, but I don't understand the error or how to solve it.
This is the code minified as much as I was able:
use actix_web::{get, http, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use tokio::time::{sleep, Duration};
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Mutex<AppState>>) -> impl Responder {
let state = data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.to_owned())
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
let status_string = get_state().await.unwrap();
let app_data = Arc::new(Mutex::new(web::Data::new(AppState {
status: status_string,
})));
let app_data1 = Arc::clone(&app_data);
actix_web::rt::spawn(async move {
loop {
println!("I get executed every 2-ish seconds!");
sleep(Duration::from_millis(2000)).await;
let res = get_state().await;
let mut app_data = app_data1.lock().unwrap();
// Edit 2: this line is not accepted by the compiler
// Edit 2: *app_data.status = res.unwrap();
// Edit 2: but this line is accepted
*app_data = web::Data::new(AppState { status: res });
}
});
let app_data2 = Arc::clone(&app_data);
// Edit 2: but I get an error here now
HttpServer::new(move || App::new().app_data(app_data2).service(index))
.bind(("127.0.0.1", 9090))?
.run()
.await
}
async fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let client = reqwest::Client::new().get("http://ipecho.net/plain".to_string());
let status = client.send().await?.text().await?;
println!("got status: {status}");
Ok(status)
}
But I get the following error:
error[E0308]: mismatched types
--> src/main.rs:33:32
|
33 | *app_data.status = res.unwrap();
| ---------------- ^^^^^^^^^^^^ expected `str`, found struct `String`
| |
| expected due to the type of this binding
error[E0277]: the size for values of type `str` cannot be known at compilation time
--> src/main.rs:33:13
|
33 | *app_data.status = res.unwrap();
| ^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
|
= help: the trait `std::marker::Sized` is not implemented for `str`
= note: the left-hand-side of an assignment must have a statically known size
Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
Why do I suddenly get a str? Is there an easy fix or is my approach to solving this wrong?
Edit: Maybe removing the * is the right way to go, as Peter Hall suggests, but that gives me the following error instead:
error[E0594]: cannot assign to data in an `Arc`
--> src/main.rs:33:13
|
33 | app_data.status = res.unwrap();
| ^^^^^^^^^^^^^^^ cannot assign
|
= help: trait `DerefMut` is required to modify through a dereference, but it is not implemented for `Arc<AppState>`
error[E0507]: cannot move out of `app_data2`, a captured variable in an `Fn` closure
--> src/main.rs:38:49
|
37 | let app_data2 = Arc::clone(&app_data);
| --------- captured outer variable
38 | HttpServer::new(move || App::new().app_data(app_data2).service(index))
| ------- ^^^^^^^^^ move occurs because `app_data2` has type `Arc<std::sync::Mutex<Data<AppState>>>`, which does not implement the `Copy` trait
| |
| captured by this `Fn` closure
Some errors have detailed explanations: E0507, E0594.
For more information about an error, try `rustc --explain E0507`.
Edit 2: I now get the following error (code changes commented with 'Edit 2' above):
error[E0507]: cannot move out of `app_data2`, a captured variable in an `Fn` closure
--> src/main.rs:46:49
|
45 | let app_data2 = app_data.clone();
| --------- captured outer variable
46 | HttpServer::new(move || App::new().app_data(app_data2).service(index))
| ------- ^^^^^^^^^ move occurs because `app_data2` has type `Arc<Mutex<Data<AppState>>>`, which does not implement the `Copy` trait
| |
| captured by this `Fn` closure
For more information about this error, try `rustc --explain E0507`.
My Cargo.toml dependencies:
[dependencies]
actix-web = "4.2.1"
reqwest = "0.11.12"
tokio = "1.21.2"

async solution
I had my types mixed up a bit, having the app state as Arc<Mutex<T>> seemed to be the way to go, maybe it would be better with Arc<RwLock<T>>.
use actix_web::{get, http, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use tokio::time::{sleep, Duration};
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Arc<Mutex<AppState>>>) -> impl Responder {
let state = data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.to_owned())
}
#[actix_web::main]
async fn main() -> std::io::Result<()> {
let status_string = get_state().await.unwrap();
let app_data = Arc::new(Mutex::new(AppState {
status: status_string,
}));
let app_data1 = app_data.clone();
actix_web::rt::spawn(async move {
loop {
println!("I get executed every 2-ish seconds!");
sleep(Duration::from_millis(2000)).await;
let res = get_state().await.unwrap();
let mut app_data = app_data1.lock().unwrap();
*app_data = AppState { status: res };
}
});
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(app_data.clone()))
.service(index)
})
.bind(("127.0.0.1", 9090))?
.run()
.await
}
async fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let client = reqwest::Client::new().get("http://ipecho.net/plain".to_string());
let status = client.send().await?.text().await?;
println!("got status: {status}");
Ok(status)
}

async/sync solution
Instead of doing the async get with reqwest I have a solution with the synchronous crate minreq (that I found after a lot of searching). I also chose to not use the #[actix_web::main] macro, and instead start the runtime explicitly at the end of my main function.
use actix_web::{get, http, rt, web, App, HttpResponse, HttpServer, Responder};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct AppState {
status: String,
}
#[get("/")]
async fn index(data: web::Data<Arc<Mutex<AppState>>>) -> impl Responder {
let state = &data.lock().unwrap();
HttpResponse::Ok()
.insert_header(http::header::ContentType::plaintext())
.body(state.status.clone())
}
fn main() -> std::io::Result<()> {
let status_string = get_state().unwrap();
let app_data = Arc::new(Mutex::new(AppState {
status: status_string,
}));
let app_data1 = Arc::clone(&app_data);
thread::spawn(move || loop {
thread::sleep(Duration::from_millis(2000));
let res = get_state().unwrap();
let mut app_data = app_data1.lock().unwrap();
*app_data = AppState { status: res };
});
rt::System::new().block_on(
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(app_data.clone()))
.service(index)
})
.bind(("127.0.0.1", 9090))?
.run(),
)
}
fn get_state() -> Result<String, Box<dyn std::error::Error>> {
let resp = minreq::get("http://ipecho.net/plain").send().unwrap();
let state = resp.as_str().unwrap();
Ok(state.to_string())
}

Related

What signature can I use to download files using Axum and Tokio?

I'm using axum and this code (found here) to download files:
use axum::{
body::StreamBody,
http::{header, StatusCode},
response::{Headers, IntoResponse},
routing::get,
Router,
};
use std::net::SocketAddr;
use tokio_util::io::ReaderStream;
#[tokio::main]
async fn main() {
let app = Router::new().route("/", get(handler));
let addr = SocketAddr::from(([127, 0, 0, 1], 3000));
axum::Server::bind(&addr)
.serve(app.into_make_service())
.await
.unwrap();
}
async fn handler() -> impl IntoResponse {
// `File` implements `AsyncRead`
let file = match tokio::fs::File::open("Cargo.toml").await {
Ok(file) => file,
Err(err) => return Err((StatusCode::NOT_FOUND, format!("File not found: {}", err))),
};
// convert the `AsyncRead` into a `Stream`
let stream = ReaderStream::new(file);
// convert the `Stream` into an `axum::body::HttpBody`
let body = StreamBody::new(stream);
let headers = Headers([
(header::CONTENT_TYPE, "text/toml; charset=utf-8"),
]);
Ok((headers, body))
}
Everything works. But I cannot find a way to move the below code in a separate function:
let file = match tokio::fs::File::open("Cargo.toml").await {
Ok(file) => file,
Err(err) => return Err((StatusCode::NOT_FOUND, format!("File not found: {}", err))),
};
I would like to use both tokio::fs::File and https://crates.io/crates/rust-s3 methods in this function.
So I need a "common type" which appear to be AsyncRead, I think.
What should be the signature of the function?
I tried with:
use tokio::io::AsyncRead;
pub struct Player {
db: Arc<DB>
}
impl Handler {
pub async fn player_pdf(
&self,
id: &str,
) -> Result<&(dyn AsyncRead)> {
//...use id here...
let file = &tokio::fs::File::open("player.pdf").await?;
Ok(file)
}
}
but I get the error:
error[E0308]: mismatched types
|
55 | Ok(file)
| -- ^^^^
| | |
| | expected reference, found struct `tokio::fs::File`
| | help: consider borrowing here: `&file`
| arguments to this enum variant are incorrect
|
= note: expected reference `&dyn tokio::io::AsyncRead`
found struct `tokio::fs::File`
I tried with: let file = &tokio::fs::File::open("player.pdf").await?; and I got:
error[E0515]: cannot return value referencing temporary value
|
43 | let file = &tokio::fs::File::open(...
| --------------------------- temporary value created here
...
55 | Ok(file)
| ^^^^^^^^ returns a value referencing data owned by the current function
What can I use?
Returning a generic "boxed" value might be the solution here:
impl Handler {
pub async fn player_pdf(
&self,
id: &str,
) -> Result<Box<dyn AsyncRead>> {
//...use id here...
Ok(Box::new(tokio::fs::File::open("player.pdf").await?))
}
}
Where now there's no dangling reference, it's encapsulated and fully owned.

Error in Async closure: Lifetime may not live long enough; returning this value requires that `'1` must outlive `'2`

I'm trying to use a Mutex<Sender> inside an async closure but I'm not entirely sure why I'm getting the error below:
error: lifetime may not live long enough
--> src/main.rs:120:41
|
120 | io.add_method(MARK_ITSELF, move |_| async {
| ________________________________--------_^
| | | |
| | | return type of closure `impl std::future::Future<Output = Result<jsonrpc::serde_json::Value, jsonrpc_http_server::jsonrpc_core::Error>>` contains a lifetime `'2`
| | lifetime `'1` represents this closure's body
121 | | trace!("Mark itself!");
122 | | let tx = sender.lock().map_err(to_internal)?;
123 | | tx.send(Action::MarkItself)
124 | | .map_err(to_internal)
125 | | .map(|_| Value::Bool(true))
126 | | });
| |_____^ returning this value requires that `'1` must outlive `'2`
|
= note: closure implements `Fn`, so references to captured variables can't escape the closure
My main function looks like this:
use jsonrpc::serde_json::Value;
use jsonrpc::Error as ClientError;
use jsonrpc::{
serde_json::value::RawValue,
simple_http::{self, SimpleHttpTransport},
Client,
};
use jsonrpc_http_server::jsonrpc_core::{Error as ServerError, IoHandler};
use jsonrpc_http_server::ServerBuilder;
use log::{debug, error, trace};
use serde::Deserialize;
use std::rc::Rc;
use std::sync::mpsc::Receiver;
use std::sync::Mutex;
use std::thread;
use std::{
env, fmt,
net::SocketAddr,
sync::mpsc::{channel, Sender},
};
const START_ROLL_CALL: &str = "start_roll_call";
const MARK_ITSELF: &str = "mark_itself";
fn create_client(url: &str, user: &str, pass: &str) -> Result<Client, simple_http::Error> {
let t = SimpleHttpTransport::builder()
.url(url)?
.auth(user, Some(pass))
.build();
Ok(Client::with_transport(t))
}
fn spawn_worker() -> Result<Sender<Action>, failure::Error> {
let (tx, rx): (Sender<Action>, Receiver<Action>) = channel();
let next: SocketAddr = env::var("NEXT")?.parse()?;
thread::spawn(move || {
let remote = Remote::new(next).unwrap();
let mut in_roll_call = false;
for action in rx.iter() {
match action {
Action::StartRollCall => {
if !in_roll_call {
if remote.start_roll_call().is_ok() {
debug!("ON");
in_roll_call = true;
}
} else {
if remote.mark_itself().is_ok() {
debug!("OFF");
in_roll_call = false;
}
}
}
Action::MarkItself => {
if in_roll_call {
if remote.mark_itself().is_ok() {
debug!("OFF");
in_roll_call = false;
}
} else {
debug!("SKIP");
}
}
}
}
});
Ok(tx)
}
enum Action {
StartRollCall,
MarkItself,
}
struct Remote {
client: Client,
}
impl Remote {
fn new(addr: SocketAddr) -> Result<Self, simple_http::Error> {
let url = format!("http://{}", addr);
let client = create_client(&url, "", "")?;
Ok(Self { client })
}
fn call_method<T>(&self, method: &str, params: &[Box<RawValue>]) -> Result<T, ClientError>
where
T: for<'de> Deserialize<'de>,
{
let request = self.client.build_request(method, params);
self.client
.send_request(request)
.and_then(|res| res.result::<T>())
}
fn start_roll_call(&self) -> Result<bool, ClientError> {
self.call_method(START_ROLL_CALL, &[])
}
fn mark_itself(&self) -> Result<bool, ClientError> {
self.call_method(MARK_ITSELF, &[])
}
}
fn main() -> Result<(), failure::Error> {
env_logger::init();
let tx = spawn_worker()?;
let addr: SocketAddr = env::var("ADDRESS")?.parse()?;
let mut io = IoHandler::default();
let sender = Mutex::new(tx.clone());
io.add_method(START_ROLL_CALL, move |_| async move {
trace!("Starting roll call!");
let tx = sender.lock().map_err(to_internal)?;
tx.send(Action::StartRollCall)
.map_err(to_internal)
.map(|_| Value::Bool(true))
});
let sender = Mutex::new(tx.clone());
io.add_method(MARK_ITSELF, move |_| async {
trace!("Mark itself!");
let tx = sender.lock().map_err(to_internal)?;
tx.send(Action::MarkItself)
.map_err(to_internal)
.map(|_| Value::Bool(true))
});
let server = ServerBuilder::new(io).start_http(&addr)?;
Ok(server.wait())
}
fn to_internal<E: fmt::Display>(err: E) -> ServerError {
error!("Error: {}", err);
ServerError::internal_error()
}
The main idea is to pass the mspc sender to the closure so that the method can send the Action(An enum). Is there something I'm doing wrong?
The problem is add_method requires that
your closure is static, and
the Future returned from your closure is static.
Try this
let sender = Arc::new(Mutex::new(tx.clone()));
io.add_method(START_ROLL_CALL, move |_| {
let cloned_sender = sender.clone();
async move {
trace!("Starting roll call!");
let tx = cloned_sender.lock().map_err(to_internal)?;
tx.send(Action::StartRollCall)
.map_err(to_internal)
.map(|_| Value::Bool(true))
}
});
Side note: you are using sync Mutex in an async environment. This undermines the benefits of async. Consider using async Mutex such as Tokio Mutex or async-lock Mutex.

Warp: single route works, multiple with .or() do not

Hoping someone can help me understand why running warp with a single route like this compiles fine:
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// GET /stats
let stats = warp::get()
.and(warp::path("stats"))
.map(|| {
let mut sys = System::new_all();
sys.refresh_all();
let local = LocalSystem::from_sys(&sys);
warp::reply::json(&local);
});
// GET /
let index = warp::get()
.and(warp::path::end())
.map(|| warp::reply::json(&last_ten_logs()));
warp::serve(index).run(([127, 0, 0, 1], 4000)).await;
Ok(())
}
But changing the warp::serve() line to serve two routes like in the examples in the repo causes a compilation error:
error[E0277]: the trait bound `(): Reply` is not satisfied
--> src/main.rs:140:17
|
140 | warp::serve(stats.or(index)).run(([127, 0, 0, 1], 4000)).await;
| ----------- ^^^^^^^^^^^^^^^ the trait `Reply` is not implemented for `()`
| |
| required by a bound introduced by this call
|
= note: required because of the requirements on the impl of `Reply` for `((),)`
= note: 2 redundant requirements hidden
= note: required because of the requirements on the impl of `Reply` for `(warp::generic::Either<((),), (Json,)>,)`
I don't understand what the compiler is asking me to change.
The error is explicit:
the trait Reply is not implemented for ()
The problem is that your stats endpoint do not return anything, just remove the last ; so it gets returned as the last expresion in the closure:
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// GET /stats
let stats = warp::get()
.and(warp::path("stats"))
.map(|| {
let mut sys = System::new_all();
sys.refresh_all();
let local = LocalSystem::from_sys(&sys);
warp::reply::json(&local)
});
...
}

What's the best way of handling errors in main method of actix-web?

Is there a way to handle errors inside main method in actix-web?
I have the following code, as you can see there's three places in that function that can panic: the reference to the DATABASE_URL environment variable, the creation of the connection pool and the initialization of the template engine (tera).
#[actix_web::main]
async fn main() -> std::io::Result<()> {
dotenv().ok();
let database_url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set.");
let manager = ConnectionManager::<PgConnection>::new(database_url);
let pool = Pool::builder().build(manager).expect("Failed to create connection pool.");
HttpServer::new(move || {
let tera = Tera::new("templates/**/*").unwrap();
App::new()
.data(pool.clone())
.data(tera)
.route("/", web::get().to(index))
})
.bind("127.0.0.1:8000")?
.run()
.await
}
So, I tried to change the return type on that function and use a previously defined Enum in order to handle those errors:
async fn main() -> Result<HttpServer, ServerError> {
dotenv().ok();
let database_url = std::env::var("DATABASE_URL")?;
let manager = ConnectionManager::<PgConnection>::new(database_url);
let pool = Pool::builder().build(manager)?;
HttpServer::new(move || {
let tera = Tera::new("templates/**/*")?;
App::new()
.data(pool.clone())
.data(tera)
.route("/", web::get().to(index))
})
.bind("127.0.0.1:8000")?
.run()
.await
}
But the compiler shows this message:
[Running 'cargo run']
error[E0107]: missing generics for struct `HttpServer`
|
| async fn main() -> Result<HttpServer, ServerError> {
| ^^^^^^^^^^ expected 4 type arguments
|
note: struct defined here, with 4 type parameters: `F`, `I`, `S`, `B`
|
| pub struct HttpServer<F, I, S, B>
| ^^^^^^^^^^ - - - -
help: use angle brackets to add missing type arguments
|
| async fn main() -> Result<HttpServer<F, I, S, B>, ServerError> {
| ^^^^^^^^^^^^
error: aborting due to previous error
For more information about this error, try `rustc --explain E0107`.
error: could not compile
What's the best course of action that I can follow here? Should I handle this kind of errors within the main method?

How to write an asynchronous recursive walkdir function with an asynchronous callback

I'm trying to write an async function that will traverse the filesystem tree, recursively, and calls an asynchronous callback for each file found.
This is for a learning effort, I have no real use case.
Here is what I have so far:
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0, features = ["unstable"]
use futures::{
executor::block_on,
future::{BoxFuture, FutureExt},
}; // 0.3.4
use std::{marker::Sync, pin::Pin};
fn main() {
fn walkdir<F>(path: String, cb: &'static F) -> BoxFuture<'static, ()>
where
F: Fn(&DirEntry) -> BoxFuture<()> + Sync + Send,
{
async move {
let mut entries = fs::read_dir(&path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path().to_str().unwrap().to_string();
if entry.path().is_file().await {
cb(&entry).await
} else {
walkdir(path, cb).await
}
}
}
.boxed()
}
let foo = async {
walkdir(".".to_string(), &|entry: &DirEntry| async {
async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
})
.await
};
block_on(foo);
}
I get this far by some sort of trial and error, but now I'm stuck on async closure callback with this error
warning: unused import: `path::*`
--> src/main.rs:3:5
|
3 | path::*,
| ^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `pin::Pin`
--> src/main.rs:10:25
|
10 | use std::{marker::Sync, pin::Pin};
| ^^^^^^^^
error[E0308]: mismatched types
--> src/main.rs:33:54
|
33 | walkdir(".".to_string(), &|entry: &DirEntry| async {
| ______________________________________________________^
34 | | async_std::println!(">> {}\n", &entry.path().to_str().unwrap()).await
35 | | })
| |_________^ expected struct `std::pin::Pin`, found opaque type
|
= note: expected struct `std::pin::Pin<std::boxed::Box<dyn core::future::future::Future<Output = ()> + std::marker::Send>>`
found opaque type `impl core::future::future::Future`
use async_std::{
fs::{self, *},
path::*,
prelude::*,
}; // 1.5.0
use futures::{future::{Future, FutureExt, LocalBoxFuture}, executor}; // 0.3.4
fn main() {
async fn walkdir<R>(path: impl AsRef<Path>, mut cb: impl FnMut(DirEntry) -> R)
where
R: Future<Output = ()>,
{
fn walkdir_inner<'a, R>(path: &'a Path, cb: &'a mut dyn FnMut(DirEntry) -> R) -> LocalBoxFuture<'a, ()>
where
R: Future<Output = ()>,
{
async move {
let mut entries = fs::read_dir(path).await.unwrap();
while let Some(path) = entries.next().await {
let entry = path.unwrap();
let path = entry.path();
if path.is_file().await {
cb(entry).await
} else {
walkdir_inner(&path, cb).await
}
}
}.boxed_local()
}
walkdir_inner(path.as_ref(), &mut cb).await
}
executor::block_on({
walkdir(".", |entry| async move {
async_std::println!(">> {}", entry.path().display()).await
})
});
}
Notable changes:
Take in AsRef<Path> instead of a String and a generic closure instead of a trait object reference
Change the closure type to be FnMut as it's more permissive
The closure returns any type that is a future.
There's an inner implementation function that hides the ugly API required for recursive async functions.
The callback takes the DirEntry by value instead of by reference.
See also:
How to asynchronously explore a directory and its sub-directories?
How to using async fn callback in rust

Resources