I am using Reqwest to call some APIs. I want to have a general function to parse the response as below:
async fn response_to_result(response: &Response) -> anyhow::Result<()> {
let status = response.status().as_u16();
let response_message = response.text().await?; // Move here because of response.text()
return if status == 200 {
Ok(())
} else {
Err(anyhow::anyhow!(response_message))
};
}
This is the error I got:
move occurs because `*response` has type `reqwest::Response`, which does not implement the `Copy` trait
The move occurs by calling response.text() (method definition: pub async fn text(self) -> crate::Result<String>). Normally with params we can pass by reference, but with the method I don't have any idea. Does anyone have a solution for this?
Normally with params, we can pass with reference
That's true as long as the method doesn't consume the value. text() does, since it assembles all the chunks together for you. Here's its signature:
pub async fn text(self) -> Result<String>
Note the self rather than &self. After running text(), the whole stream is consumed, a String is constructed, and it's returned to you to manage. Response does not have an internal buffer that stores all this data. It's gone once the stream is consumed; keeping track of it is the job of the caller. Future calls to text() (or bytes()) couldn't work. So text() destroys the entire Response. You can't continue to use it after calling text().
So you'll need to pass the actual value, not a borrow of it. Remove the & on Response.
If you want a (mutable) borrowing version of this, you'll need to assemble the chunks yourself with chunk(). But you probably don't want to do this. You probably just want to pass Response (without the &) and let response_to_result() consume the value like text().
Related
I have a list of functions with variable arguments, and I want to randomly pick one of them, in runtime, and call it, on a loop. I'm looking to enhance the performance of my solution.
I have a function that calculates the arguments based on some randomness, and then (should) return a function pointer, which I could then call.
pub async fn choose_random_endpoint(
&self,
rng: ThreadRng,
endpoint_type: EndpointType,
) -> impl Future<Output = Result<std::string::String, MyError>> {
match endpoint_type {
EndpointType::Type1 => {
let endpoint_arguments = self.choose_endpoint_arguments(rng);
let endpoint = endpoint1(&self.arg1, &self.arg2, &endpoint_arguments.arg3);
endpoint
}
EndpointType::Type2 => {
let endpoint_arguments = self.choose_endpoint_arguments(rng);
let endpoint = endpoint2(
&self.arg1,
&self.arg2,
&endpoint_arguments.arg3,
rng.clone(),
);
endpoint
}
EndpointType::Type3 => {
let endpoint_arguments = self.choose_endpoint_arguments(rng);
let endpoint = endpoint3(
&self.arg1,
&self.arg2,
&endpoint_arguments.arg3,
rng.clone(),
);
endpoint
}
}
}
The error I obtain is
expected opaque type `impl Future<Output = Result<std::string::String, MyError>>` (opaque type at <src/calls/type1.rs:14:6>)
found opaque type `impl Future<Output = Result<std::string::String, MyError>>` (opaque type at <src/type2.rs:19:6>)
. The compiler advises me to await the endpoints, and this solves the issue, but is there a performance overhead to this?
Outer function:
Aassume there is a loop calling this function:
pub async fn make_call(arg1: &str, arg2: &str) -> Result<String> {
let mut rng = rand::thread_rng();
let random_endpoint_type = choose_random_endpoint_type(&mut rng);
let random_endpoint = choose_random_endpoint(&rng, random_endpoint_type);
// call the endpoint
Ok(response)
}
Now, I want to call make_call every X seconds, but I don't want my main thread to block during the endpoint calls, as those are expensive. I suppose the right way to approach this is spawning a new thread per X seconds of interval, that call make_call?
Also, performance-wise: having so many clones on the rng seems quite expensive. Is there a more performant way to do this?
The error you get is sort of unrelated to async. It's the same one you get when you try to return two different iterators from a function. Your function as written doesn't even need to be async. I'm going to remove async from it when it's not needed, but if you need async (like for implementing an async-trait) then you can add it back and it'll probably work the same.
I've reduced your code into a simpler example that has the same issue (playground):
async fn a() -> &'static str {
"a"
}
async fn b() -> &'static str {
"b"
}
fn a_or_b() -> impl Future<Output = &'static str> {
if rand::random() {
a()
} else {
b()
}
}
What you're trying to write
When you want to return a trait, but the specific type that implements that trait isn't known at compile time, you can return a trait object. Futures need to be Unpin to be awaited, so this uses a pinned box (playground).
fn a_or_b() -> Pin<Box<dyn Future<Output = &'static str>>> {
if rand::random() {
Box::pin(a())
} else {
Box::pin(b())
}
}
You may need the type to be something like Pin<Box<dyn Future<Output = &'static str> + Send + Sync + 'static>> depending on the context.
What you should write
I think the only reason you'd do the above is if you want to generate the future with some kind of async rng, then do something else, and then run the generated future after that. Otherwise there's no need to have nested futures; just await the inner futures when you call them (playground).
async fn a_or_b() -> &'static str {
if rand::random() {
a().await
} else {
b().await
}
}
This is conceptually equivalent to the Pin<Box> method, just without having to allocate a Box. Instead, you have an opaque type that implements Future itself.
Blocking
The blocking behavior of these is only slightly different. Pin<Box> will block on non-async things when you call it, while the async one will block on non-async things where you await it. This is probably mostly the random generation.
The blocking behavior of the endpoint is the same and depends on what happens inside there. It'll block or not block wherever you await either way.
If you want to have multiple make_call calls happening at the same time, you'll need to do that outside the function anyway. Using the tokio runtime, it would look something like this:
use tokio::task;
use futures::future::join_all;
let tasks: Vec<_> = (0..100).map(|_| task::spawn(make_call())).collect();
let results = join_all(tasks).await;
This also lets you do other stuff while the futures are running, in between collect(); and let results.
If something inside your function blocks, you'd want to spawn it with task::spawn_blocking (and then await that handle) so that the await call in make_call doesn't get blocked.
RNG
If your runtime is multithreaded, the ThreadRng will be an issue. You could create a type that implements Rng + Send with from_entropy, and pass that into your functions. Or you can call thread_rng or even just rand::random where you need it. This makes a new rng per thread, but will reuse them on later calls since it's a thread-local static. On the other hand, if you don't need as much randomness, you can go with a Rng + Send type from the beginning.
If your runtime isn't multithreaded, you should be able to pass &mut ThreadRng all the way through, assuming the borrow checker is smart enough. You won't be able to pass it into an async function and then spawn it, though, so you'd have to create a new one inside that function.
I have a function like this
fn get_html(address: &str) -> String {
let mut response = reqwest::blocking::get(
address,
);
response = response.unwrap_or_else(|_e| {String::from("")});
response = response.text().unwrap_or_else(|_e| {String::from("")});
return response
}
Where I'm checking for html content. I would like to return an empty String if any kind of an error occurs somewhere in this function.
I'm not sure how to deal with this because unwrap_or_else expecting Result not String.
The reqwest::blocking::get() function is returning a Result<Response>.
To obtain the html, you have to unwrap this result and call .text() method.
That will return a Result<String>, that you have to unwrap again.
In your code you assign a String::from("") when you unwrap the Result<Response>, and that is not right, because you have a Response when it is Ok and a String when it is an Err.
Instead you should match the result and return the String from the function:
fn get_html(address: &str) -> String {
let mut response = reqwest::blocking::get(
address,
);
match response {
Ok(response) => response.text().unwrap_or_else(|_e| String::from("")),
Err(_) => String::from(""),
}
}
In this code, you use unwrap_or_else() just on the .text() result.
While if you have an error on the response itself, you return a String from the function.
An idiomatic way to solve your issue would be to refactor slightly your code: because code in get_html could fail, it's natural that its signature reflects this, so instead of returning a String, it could return an Option<String>, leaving the caller decide on what to do in this case:
fn get_html(address: &str) -> Option<String> {
reqwest::blocking::get(address)
.ok()?
.text()
.ok()
}
See the playground.
This makes the code much more straightforward. However, you may really want the signature of the function to be -> String, not -> Option<String>. In this case, there are two solutions.
The first solution would be to use the experimental try block. I mention this solution, not because it's currently the most adequate, but because it will be one day (most likely).
fn get_html(address: &str) -> String {
let result: Option<_> = try {
reqwest::blocking::get(address)
.ok()?
.text()
.ok()?
};
result.unwrap_or_default()
}
See the playground.
Note that, as is, Rust is not able to figure out types on its own for the try block, so we have to help it, which makes this more verbose. This aspect will probably improve over time, when try blocks are stabilized.
Also note that, since the Default::default constructor of String produces an empty string, we can directly use .unwrap_or_default() instead of .unwrap_or(String::new()) or .unwrap_or_else(|| String::new()) (since an empty string is not allocated, the first option is also acceptable).
The second solution would simply be to add an other function with the wanted signature, and make it use the first function
fn get_html_failable(address: &str) -> Option<String> {
reqwest::blocking::get(address)
.ok()?
.text()
.ok()
}
fn get_html(address: &str) -> String {
get_html_failable(address).unwrap_or_default()
}
This may seem unconvenient, but lots of libraries in Rust have internal error propagation using types that represent failure (such as Result or Option), because it's really convenient to write such functions and to compose them (both reqwest::blocking::get and .text() return a Result, and see how easy it is to use them in a function that returns an Option), and have wrapper function that expose the wanted signature (most libraries will still expose fail types, but if you are writing an application it's not necessarily the best choice).
A variable of both preceding workarounds is to "simulate" a try block in stable Rust, without relying on an additional toplevel function.
fn get_html(address: &str) -> String {
(|| reqwest::blocking::get(address).ok()?.text().ok())()
.unwrap_or_default()
}
See the playground.
Note that, in this case, countrary to the try version, Rust is able to figure out the right types on its own.
This is probably the least readable solution, so I wouldn't recommend it (use try blocks instead). But it works.
I am writing logic to fetch certs from keycloak api and when I compile the program I am getting below error.
Ok(keys) returns a value referencing data owned by the current function
async fn get_jwt_keys() -> Result<&'static KeycloakCertRespKeys, Error> {
let request_url = format!("http://x.x.x.x:5000/auth/realms/test/protocol/openid-connect/certs");
println!("{}", request_url);
let response = reqwest::get(&request_url).await?;
// println!("{:?}", response);
let certs: KeycloakCertResp = response.json().await?;
let keys: &KeycloakCertRespKeys = certs.keys.first().unwrap();
println!("{:?}", keys.kid);
Ok(keys)
}
The compiler is right. If you look when the variable keys is created, its type is a reference. The function returns Ok(that reference).
The reference indeed references data available only to that function, that gets dealocated when that variable gets out of scope.
To solve the problem, you may call to_owned method and return the actual object instead of a reference, or clone if the type you have, KeycloakCertRespKeys, does not implement ToOwned trait. Note that the return type might change (remove &'static).
As mentioned in one of the previous comments, the compiler forces you to specify a static lifetime to make sure that the data is available throughout the whole time while the program executes.
I have a hyper server set up more or less exactly as the third example here: https://docs.rs/hyper/0.14.16/hyper/server/index.html . My version of the handle function calls some other async functions, and everything works fine, until I try to encode some URL query params into a string in one of those async functions. My project stops compiling when I include the four lines involving the Serializer in one of the functions called by handle:
async fn broken_func(&self, param: &str) -> Result<String, Infallible> {
// ...
let mut s = url::form_urlencoded::Serializer::new(String::new());
// the presence or absence of these two lines has no effect on the bug, but
// they demonstrate how I'm trying to use the Serializer
s.append_pair("key", "value");
println!("{:?}", s.finish());
drop(s); // <-- thought this might help, but it doesn't
// ...
Ok(query_string)
}
The error I get is
generator cannot be sent between threads safely
the trait `std::marker::Sync` is not implemented for `dyn for<'r> std::ops::Fn(&'r str) -> std::borrow::Cow<'_, [u8]>`
I have no idea what this has to do with form_urlencoded::Serializer. However, I am aware that Serializer is both !Send and !Sync, but in this case I'm only using it within a single function so I don't think that should make a difference? If I remove those four lines above, it goes back to compiling.
So instead, to serialize some key/value pairs into URL query parameters, I have to use the following, which kind of seems ridiculous -- not just because this is needlessly complex for something so simple, but also because url::Url::parse_with_params uses form_urlencoded::Serializer under the hood.
let query_string = url::Url::parse_with_params(
"http://example.com",
&[("key", "value")]
)
.unwrap()
.query()
.map(|s| s.to_owned())
.unwrap();
Any idea why trying to explicitly use Serializer inside an async function causes things to break?
Caesar hit the nail on the head. The trick of putting s in a scope fixed it.
I'm working on an app that optionally uses a GUI to display video data that's roughly structured like this:
fn main() {
let (window_tx, window_rx) =
MainContext::channel::<MyStruct>(PRIORITY_DEFAULT);
let some_thread = thread::spawn(move || -> () {
// send data to window_tx
});
let application =
gtk::Application::new(Some("com.my.app"), Default::default());
application.connect_activate(move |app: >k::Application| {
build_ui(app, window_rx);
});
application.run();
some_thread.join().unwrap();
}
fn build_ui(application: >k::Application, window_rx: Receiver<MyStruct>) {
window_rx.attach( ... );
}
The gtk rust library requires a Fn callback passed to application.connect_activate on startup, so I can't use a FnOnce or FnMut closure to move the glib::Receiver in the callback. The compiler throws this error:
error[E0507]: cannot move out of `window_rx`, a captured variable in an `Fn` closure
I've tried to avoid the move by wrapping window_rx in a Rc, ie:
let r = Rc::new(RefCell::new(window_rx));
application.connect_activate(move |app: >k::Application| {
build_ui(app, Rc::clone(&r));
});
But upon dereferencing the Rc in my build_ui function, I get this error:
error[E0507]: cannot move out of an `Rc`
The fallback I've used thus far is to just move the channel creation and thread creation into my build_ui function, but because the GUI is not required, I was hoping to avoid using GTK and the callback entirely if GUI is not used. Is there some way I can either safely move window_rx within a closure or otherwise dereference it in the callback without causing an error?
When you need to move a value out from code that, by the type system but not in practice, could be called more than once, the simple tool to reach for is Option. Wrapping the value in an Option allows it to be swapped with an Option::None.
When you need something to be mutable even though you're inside a Fn, you need interior mutability; in this case, Cell will do. Here's a complete compilable program that approximates your situation:
use std::cell::Cell;
// Placeholders to let it compile
use std::sync::mpsc;
fn wants_fn_callback<F>(_f: F) where F: Fn() + 'static {}
struct MyStruct;
fn main() {
let (_, window_rx) = mpsc::channel::<MyStruct>();
let window_rx: Cell<Option<mpsc::Receiver<MyStruct>>> = Cell::new(Some(window_rx));
wants_fn_callback(move || {
let _: mpsc::Receiver<MyStruct> = window_rx.take().expect("oops, called twice");
});
}
Cell::take() removes the Option<Receiver> from the Cell, leaving None in its place. The expect then removes the Option wrapper (and handles the possibility of the function being called twice by panicking in that case).
Applied to your original problem, this would be:
let window_rx: Option<Receiver<MyStruct>> = Cell::new(Some(window_rx));
application.connect_activate(move |app: >k::Application| {
build_ui(app, window_rx.take().expect("oops, called twice"));
});
However, be careful: if the library requires a Fn closure, there might be some condition under which the function could be called more than once, in which case you should be prepared to do something appropriate in that circumstance. If there isn't such a condition, then the library's API should be improved to take a FnOnce instead.