I'm trying to iterate over logs from a docker container by using the bollard crate.
Here's my code:
use std::default::Default;
use bollard::container::LogsOptions;
use bollard::Docker;
fn main() {
let docker = Docker::connect_with_http_defaults().unwrap();
let options = Some(LogsOptions::<String>{
stdout: true,
..Default::default()
});
let data = docker.logs("2f6c52410d", options);
// ...
}
docker.logs() returns impl Stream<Item = Result<LogOutput, Error>>. I'd like to iterate over the results, but I have no idea how to do that. I've managed to find an example that uses try_collect::<Vec<LogOutput>>() from the future_utils crate, but I'd like to iterate over the results in a while loop instead of collecting the results in a vector. I know that I can iterate over a vector, but performing tasks in a loop will be better for my use case.
I've tried to call poll_next() method for the stream, but it requires a mysterious Context object which I don't understand. The poll_next() method was unavailable until I've used pin_mut!() macro on the stream.
How do I iterate over stream? What should I read to understand what's going on here? I know that the streams are related to Futures, but calling await or next() doesn't work here.
You typically bring in your library of choice's StreamExt trait, and then do something like
while let Some(foo) = stream.next().await {
// ...
}
Related
I am developing a CLI program for rendering template files using the new MiniJinja library by mitsuhiko.
The program is here: https://github.com/benwilber/temple.
I would like to be able to extend the program by allowing the user to load custom Lua scripts for things like custom filters, functions, and tests. However, I am running into Rust lifetime errors that I've not been able to solve.
Basically, I would like to be able to register a Lua function as a custom filter function. But it's showing an error when compiling. Here is the code:
https://github.com/benwilber/temple/compare/0.3.1..lua
Error:
https://gist.github.com/c649a0b240cf299d3dbbe018c24cbcdc
How can I call a Lua function from the MiniJinja add_filter function? I would prefer to try to do this in the regular/safe way. But I'm open to unsafe alternatives if required.
Thanks!
Edit: Posted the same on Reddit and users.rust-lang.org
Lua uses state that is not safe to use from more than one thread.
A consequence of this is that LuaFunction is neither Sync or Send.
This is being enforced by this part of the error message:
help: within `LuaFunction<'_>`, the trait `Sync` is not implemented for `*mut rlua::ffi::lua_State`
In contrast a minijinja::Filter must implement Send + Sync + 'static.
(See https://docs.rs/minijinja/0.5.0/minijinja/filters/trait.Filter.html)
This means we can't share LuaFunctions (or even LuaContext) between calls to the Filters.
One option is to not pass your lua state into the closures, and instead create a new lua state every call, something like this.
env.add_filter(
"concat2",
|_env: &Environment, s1: String, s2: String|
-> anyhow::Result<String, minijinja::Error> {
lua.context(|lua_ctx| {
lua_ctx.load(include_str!("temple.lua")).exec().unwrap();
let globals = lua_ctx.globals();
let temple: rlua::Table = globals.get("temple").unwrap();
let filters: rlua::Table = temple.get("_filters").unwrap();
let concat2: rlua::Function = filters.get("concat2").unwrap();
let res: String = concat2.call::<_, String>((s1, s2)).unwrap();
Ok(res)
}
}
);
This is likely to have relatively high overhead.
Another option is to create your rlua state in one thread and communicate with it via pipes. This would look more like this:
pub fn test() {
let mut env = minijinja::Environment::new();
let (to_lua_tx, to_lua_rx) = channel::<(String,String,SyncSender<String>)>();
thread::spawn(move|| {
let lua = rlua::Lua::new();
lua.context(move |lua_ctx| {
lua_ctx.load("some_code").exec().unwrap();
let globals = lua_ctx.globals();
let temple: rlua::Table = globals.get("temple").unwrap();
let filters: rlua::Table = temple.get("_filters").unwrap();
let concat2: rlua::Function = filters.get("concat2").unwrap();
while let Ok((s1,s2, channel)) = to_lua_rx.recv() {
let res: String = concat2.call::<_, String>((s1, s2)).unwrap();
channel.send(res).unwrap()
}
})
});
let to_lua_tx = Mutex::new(to_lua_tx);
env.add_filter(
"concat2",
move |_env: &minijinja::Environment,
s1: String,
s2: String|
-> anyhow::Result<String, minijinja::Error> {
let (tx,rx) = sync_channel::<String>(0);
to_lua_tx.lock().unwrap().send((s1,s2,tx)).unwrap();
let res = rx.recv().unwrap();
Ok(res)
}
);
}
It would even be possible to start multiple lua states this way, but would require a bit more plumbing.
DISCLAIMER: This code is all untested - however, it builds with a stubbed version of minijinja and rlua in the playground. You probably want better error handling and might need some additional code to handle cleanly shutting down all the threads.
When trying to return tiberius::QueryResult I am unable to do so because it references data owned. How do I return stream if this is now allowed?
pub async fn sql_conn(str_query: &str) -> std::result::Result<tiberius::QueryResult<'_>, tiberius::error::Error>{
let mut config = Config::new();
config.host("host");
config.port(1433);
config.authentication(AuthMethod::sql_server("usr", "pw"));
config.trust_cert();
let tcp = TcpStream::connect(config.get_addr()).await?;
tcp.set_nodelay(true)?;
let mut client = Client::connect(config, tcp.compat_write()).await?;
let stream = client.query(
str_query
, &[]).await?;
Ok(stream)
}
Error:
cannot return value referencing local variable `client`
returns a value referencing data owned by the current function
The reason this isn't working is because your query result object references your client and depends on resources that it uses. Most likely, that's because your query result is streaming and the client owns the connection required for that streaming to occur.
Rust won't let you return the query result because it needs the client and the client, as a local variable, is destroyed when the function returns, since it goes out of scope. If Rust let you return the query result, it would likely reference the closed client, and your program would either fail or segfault. This is a common problem in many languages that don't provide garbage collection, and Rust is specifically designed not to allow you to make this mistake.
There are a couple of options here. First, you can create a function which creates the SQL connection and returns a client, then use the client and the query results it returns in the function where you want the data. That way, both the client and the query results will have the right lifetimes.
You could also try to create a struct which instantiates and holds your client and then use that to make the query. For example (untested):
struct Connection<'a> {
client: tiberius::Client<'a>
}
impl<'a> Connection<'a> {
fn query(&mut self, query: &str) -> Result<tiberius::QueryResult<'a>, tiberius::error::Error> {
client.query(str_query, &[]).await
}
}
This is essentially the same as the first situation, just with a different structure.
The third option is to both instantiate the client and totally consume the results in the same function, and then return some structure (like a Vec) with the results. This means that you will have to consume the entirety of the data, which you may not want to do for efficiency reasons, but it does solve the lifetime issue, and depending on your scenario, may be a valid option.
mod loginfo{
use std::io::Error;
use chrono::prelude::*;
use std::io::prelude::*;
use std::fs::OpenOptions;
const LOG_SYS :&'static str = "log.txt";
const LOG_ERR :&'static str = "log_error.txt";
pub fn set_log_error(info: String)->Result<(), String>{
let mut handler = OpenOptions::new().append(true)
.open(LOG_ERR);
if handler.is_err(){
create_file(LOG_ERR.to_owned()).unwrap();
set_log_error(info).unwrap();
}
if let Err(_errno) = handler.write_fmt(
format_args!("{:?}\t{:?} ->[Last OS error({:?})]\n",
Utc::now().to_rfc2822().to_string(), info,
Error::last_os_error()) ){
panic!(
"\nCannot write info log error\t Info\t:{:?}\n",
Error::last_os_error());
}
Ok(())
}
pub fn set_log(info: String)->Result<(), String>{
let mut handler = OpenOptions::new().append(true)
.open(LOG_SYS);
if handler.is_err(){
set_log_error("Cannot write info log".to_owned())
.unwrap();
}
if let Err(_errno) = handler.write_fmt(
format_args!("{:?}\t{:?}\n",
Utc::now().to_rfc2822().to_string(), info)){
set_log_error("Cannot write data log file".to_owned())
.unwrap();
}
Ok(())
}
pub fn create_file(filename : String)->Result<(), String>{
let handler = OpenOptions::new().write(true)
.create(true).open(filename);
if handler.is_err(){
panic!(
"\nCannot create log file\t Info\t:{:?}\n",
Error::last_os_error());
}
Ok(())
}
}
When compiling, I get the following errors, "error[E0599]: no method named write_fmt found for enum std::result::Result<std::fs::File, std::io::Error> in the current scope --> src/loginfo.rs:19:38`"
but despite using the right imports, I still get the same errors. Is this due to a bad implementation of the module?
Thank you in advance for your answers and remarks?
+1 #Masklinn Ok I think I understand it would be easier to just write
pub fn foo_write_log( info: String){
let mut handler = OpenOptions::new().append(true)
.create(true).open(LOG_SYS).expect("Cannot create log");
handler.write_fmt(
format_args!("{:?}\t{:?} ->[Last OS error({:?})]\n",
Utc::now().to_rfc2822().to_string(), info,
Error::last_os_error())).unwrap();
}
but despite using the right imports, I still get the same errors. Is this due to a bad implementation of the module?
Kind-of? If you look at the type specified in the error, handler is a Result<File, Error>. And while io::Write is implemented on File, it's not implemented on Result.
The problem is that while you're checking whether handler.is_err() you never get the file out of it, nor do you ever return in the error case. Normally you'd use something like match or if let or one of the higher-order methods (e.g. Result::map, Result::and_then) in order to handle or propagate the various cases.
And to be honest the entire thing is rather odd and awkward e.g. your functions can fail but they panic instead (you never actually return an Err); if you're going to try and create a file when opening it for writing fails, why not just do that directly[0]; you are manually calling write_fmt and format_args why not just write!; write_fmt already returns an io::Error why do you discard it then ask for it again via Error::last_os_error; etc...
It's also a bit strange to hand-roll your own logger thing when the rust ecosystem already has a bunch of them though you do you; and the naming is also somewhat awkward e.g. I'd expect something called set_X to actually set the X, so to me set_log would be a way to set the file being logged to.
[0] .create(true).append(true) should open the file in append mode if it exists and create it otherwise; not to mention your version has a concurrency issue: if the open-for-append fails you create the file in write mode, but someone else could have created the file -- with content -- between the two calls, in which case you're going to partially overwrite the file
I'm trying to understand how to implement polling multiple futures with different types. For context, I'm calling an API that will return something like:
[{"type": "source_a", "id": 123}, {"type": "source_b", "id": 234}, ...]
Each type in the API response requires a call to another API, with each API returning different data types. The code I've written works something like this:
async fn get_data(sources: Vec<Source>) -> Data {
let mut data = Default::default();
for source in sources {
if source.kind == "source_a" {
let source_data = get_source_a(source).await;
process_source_a(source_data, &mut data);
} else if source.kind == "source_b" {
...
}
}
data
}
This won't run concurrently, it will simply fetch sources one at a time and process them. How can I rewrite this so that each source is fetched concurrently and then processed once data is available? Speaking Rustily, I think what I want is to execute a closure that mutably borrows data when the future is ready. Should I be looking at something like an Arc<RefCell<Data>>?
To process the futures in parallel, you need to await something like join_all, which will run them concurrently and return when they are all done. For this to work, you have to resolve two issues:
join_all requires futures of the same type, so you need to box them or otherwise unify them.
data needs to be accessed by multiple async blocks, so it needs to be protected by Arc and Mutex.
The first issue can be solved simply by spawning the async fns as tasks, which has the added advantage of potentially running them in parallel (in addition to them being run concurrently). The example below uses tokio::spawn, but it would be almost exactly the same for async_std. Since there is no reproducible example, I can't test the code, but it could look like this:
async fn get_data(sources: Vec<Source>) -> Data {
let data = Arc::new(Mutex::new(Data::default()));
let mut tasks = vec![];
for source in sources {
if source.kind == "source_a" {
let data = Arc::clone(&data);
tasks.push(tokio::task::spawn(async move {
let source_data = get_source_a(source).await;
process_source_a(source_data, &mut data.lock().unwrap());
}));
} else if source.kind == "source_b" {
// ...
}
}
// Wait for all sources to finish and propagate the panic if any.
// With async_std this wouldn't require the `for_each()`.
futures::future::join_all(tasks)
.await
.for_each(|x| x.unwrap());
// As all tasks are done, there should be no references to `data` at
// this point, so we can extract it out of the Arc<Mutex<_>> wrapping.
data.try_unwrap().unwrap().into_inner()
}
I'm trying to write a simple game that runs in the browser, and I'm having a hard time modeling a game loop given the combination of restrictions imposed by the browser, rust, and wasm-bindgen.
A typical game loop in the browser follows this general pattern:
function mainLoop() {
update();
draw();
requestAnimationFrame(mainLoop);
}
If I were to model this exact pattern in rust/wasm-bindgen, it would look like this:
let main_loop = Closure::wrap(Box::new(move || {
update();
draw();
window.request_animation_frame(main_loop.as_ref().unchecked_ref()); // Not legal
}) as Box<FnMut()>);
Unlike javascript, I'm unable to reference main_loop from within itself, so this doesn't work.
An alternative approach that someone suggested is to follow the pattern illustrated in the game of life example. At a high-level, it involves exporting a type that contains the game state and includes public tick() and render() functions that can be called from within a javascript game loop. This doesn't work for me because my gamestate requires lifetime parameters, since it effectively just wraps a specs World and Dispatcher struct, the latter of which has lifetime parameters. Ultimately, this means that I can't export it using #[wasm_bindgen].
I'm having a hard time finding ways to work around these restrictions, and am looking for suggestions.
The easiest way to model this is likely to leave invocations of requestAnimationFrame to JS and instead just implement the update/draw logic in Rust.
In Rust, however, what you can also do is to exploit the fact that a closure which doesn't actually capture any variables is zero-size, meaning that Closure<T> of that closure won't allocate memory and you can safely forget it. For example something like this should work:
#[wasm_bindgen]
pub fn main_loop() {
update();
draw();
let window = ...;
let closure = Closure::wrap(Box::new(|| main_loop()) as Box<Fn()>);
window.request_animation_frame(closure.as_ref().unchecked_ref());
closure.forget(); // not actually leaking memory
}
If your state has lifetimes inside of it, that is unfortunately incompatible with returning back to JS because when you return all the way back to the JS event loop then all WebAssembly stack frames have been popped, meaning that any lifetime is invalidated. This means that your game state persisted across iterations of the main_loop will need to be 'static
I'm a Rust novice, but here's how I addressed the same issue.
You can eliminate the problematic window.request_animation_frame recursion and implement an FPS cap at the same time by invoking window.request_animation_frame from a window.set_interval callback which checks a Rc<RefCell<bool>> or something to see if there's an animation frame request still pending. I'm not sure if the inactive tab behavior will be any different in practice.
I put the bool into my application state since I'm using an Rc<RefCell<...>> to that anyway for other event handling. I haven't checked that this below compiles as is, but here's the relevant parts of how I'm doing this:
pub struct MyGame {
...
should_request_render: bool, // Don't request another render until the previous runs, init to false since we'll fire the first one immediately.
}
...
let window = web_sys::window().expect("should have a window in this context");
let application_reference = Rc::new(RefCell::new(MyGame::new()));
let request_animation_frame = { // request_animation_frame is not forgotten! Its ownership is moved into the timer callback.
let application_reference = application_reference.clone();
let request_animation_frame_callback = Closure::wrap(Box::new(move || {
let mut application = application_reference.borrow_mut();
application.should_request_render = true;
application.handle_animation_frame(); // handle_animation_frame being your main loop.
}) as Box<FnMut()>);
let window = window.clone();
move || {
window
.request_animation_frame(
request_animation_frame_callback.as_ref().unchecked_ref(),
)
.unwrap();
}
};
request_animation_frame(); // fire the first request immediately
let timer_closure = Closure::wrap(
Box::new(move || { // move both request_animation_frame and application_reference here.
let mut application = application_reference.borrow_mut();
if application.should_request_render {
application.should_request_render = false;
request_animation_frame();
}
}) as Box<FnMut()>
);
window.set_interval_with_callback_and_timeout_and_arguments_0(
timer_closure.as_ref().unchecked_ref(),
25, // minimum ms per frame
)?;
timer_closure.forget(); // this leaks it, you could store it somewhere or whatever, depends if it's guaranteed to live as long as the page
You can store the result of set_interval and the timer_closure in Options in your game state so that your game can clean itself up if needed for some reason (maybe? I haven't tried this, and it would seem to cause a free of self?). The circular reference won't erase itself unless broken (you're then storing Rcs to the application inside the application effectively). It should also enable you to change the max fps while running, by stopping the interval and creating another using the same closure.