I have the following function which is a tauri command. The purpose of this function is to detect whether the file in a path is a valid video file
This code panics when given a path to a file which is not a valid video. (Specifically i'm giving it a .rs file)
use std::{panic, thread};
use libvmaf_rs::video::Video;
#[tauri::command]
pub async fn validate_video(path: String) -> Result<(), String> {
let did_panic = thread::spawn(|| match Video::new(path, 640, 480) {
Ok(_) => Ok(()),
Err(e) => Err(e.frames().into_iter().map(|f| format!("{f:?}")).fold(
String::new(),
|mut prev, next| {
prev.push_str(&next);
prev.push_str("\n");
prev
},
)),
});
match did_panic.join() {
Ok(e) => e,
Err(e) => Err("Panicked while trying to load video".to_string()),
}
}
As you can see, I attempted to handle this by wrapping the entire handler in a thread and checking if the thread panicked, but unfortunately this still crashes my entire program when this function panics. I have no control of the underlying library. I've also tried panic::catch_unwind with the same results
Here's the console output. This is coming from libswscale which is linked to the ffmpeg-next crate
[sunrast # 0x7f1e44003040] this is not sunras encoded data
[image2 # 0x7f1e44000900] Could not find codec parameters for stream 0 (Video: sunrast, none): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Assertion desc failed at libswscale/swscale_internal.h:725
It seems that there's some sort of assertation happening in libswscale which is panic-ing before it gets to rust. There's no way to recover from this!
Related
I'm looking to build a basic HTTP(S) server using rust hyper, with the purpose of throughput measurement. Essentially it has two functions
on GET request, send an infinite (or arbitrary large) stream of bytes.
on POST, discard all incoming bytes, then send a short acknowledgement.
I have this working for HTTP using std::net, but would like to add it to hyper to be able to measure HTTP and HTTPS. Being fairly new to rust, I am wondering how to add it to the hyper HTTPS example server - that is, how I can get the response builder to expose a stream (io::stream?) I can write a static buffer of random bytes to, without building the entire response body in memory.
Essentially, I would like to go
loop {
match stream.write(rand_bytes) {
Ok(_) => {},
Err(_) => break,
}
}
here
async fn echo(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
let mut response = Response::new(Body::empty());
match (req.method(), req.uri().path()) {
// Help route.
(&Method::GET, "/") => {
*response.body_mut() = Body::from("I want to be a stream.\n");
}
...
I see that I could wrap a futures stream using wrap_stream, so maybe my question is how to define a stream iterator that I can use in wrap_stream which returns the same bytes over and over again.
Thanks to the comment from Caesar above, the following snippet worked for me:
let infstream: futures::stream::Iter<std::iter::Repeat<Result<String, String>>> = stream::iter(std::iter::repeat(Ok(rand_string)));
*response.body_mut() = Body::wrap_stream(infstream);
This does not implement any termination criteria, but it could easily be modified to return a fixed number of bytes.
I am writing a multi-threaded concurrent Kafka producer using Rust and Tokio. The project has 2 modes, an interactive mode that runs in an infinite loop and a file mode which takes a file as an argument and then reads the file and sends these messages to Kafka via multiple threads. Interactive mode works fine! but file mode has issues.
To achieve this, I had initially started with Rayon, but then switched to a more flexible runtime; tokio. Now, I am able to parallelize the task of sending data over a specified number of threads within tokio, however, it seems that runtime is getting dropped before all messages are produced. Here is my code:
pub fn worker(brokers: String, f: File, t: usize, topic: Arc<String>) {
let reader = BufReader::new(f);
let mut rt = runtime::Builder::new()
.threaded_scheduler()
.core_threads(t)
.build()
.unwrap();
let producers: Arc<Vec<Mutex<BaseProducer>>> = Arc::new(
(0..t)
.map(|_| get_producer(&brokers))
.collect::<Vec<Mutex<BaseProducer>>>(),
);
let acounter = atomic::AtomicUsize::new(0);
let _results: Vec<_> = reader
.lines()
.map(|line| line.unwrap())
.map(move |line| {
let prods = producers.clone();
let tp = topic.clone();
let cnt = acounter.swap(
(acounter.load(atomic::Ordering::SeqCst) + 1) % t,
atomic::Ordering::SeqCst,
);
rt.block_on(async move {
match prods[cnt]
.lock()
.unwrap()
.send(BaseRecord::to(&(*tp)).payload(&line).key(""))
{
Ok(_) => (),
Err(e) => eprintln!("{:?}", e),
};
})
})
.collect();
}
fn get_producer(brokers: &String) -> Mutex<BaseProducer> {
Mutex::new(
BaseProducer::from_config(
ClientConfig::new()
.set("bootstrap.servers", &brokers)
.set("message.timeout.ms", "5000"),
)
.expect("Producer creation error"),
)
}
As a high-level walkthrough: I create mutable producers equal to the number of threads specified and every task within this thread will use one of these producers. The file is read line by line sequentially and every line is moved into the closure that produces it as a message to Kafka.
The code works fine, for the most part, but there are issues related to the runtime exiting without completing all tasks, even when I am using the block_on function in the runtime. Which is supposed to block until the future is complete (Async block in my case here).
I believe the issue is that the issue is with runtime getting dropped without all the threading within Tokio exiting successfully.
I tried reading a file with this approach habing 100,000 records, on a single thread, I was able to produce 28,000 records. On 2 threads, close to 46,000 records. And while utilising all 8 cores of my CPU, I was getting 99,000-100,000 messages indeterministically.
I have checked several answers on SO, but none help in my case. I also read through the documentation of tokio::runtime::Runtime here and tried to use spawn and then use futures::future::join, but that didn't work either.
Any help is appreciated!
My rust project uses Command to execute a process.
Sometimes (low frequency) when I run this code the call to status.code() returns None. I am usually using Mac OS Catalina Beta 1, rustc 1.36.0 - but it happens in Travis too (will have to go and find logs of OS/rustc there).
I was treating this as an error but "randomly" it would cause local and travis builds to fail, so now I'm ignoring it - but it would be nice to understand what's causing it.
In failure cases, re-running immediately will cause it to succeed.
let output = Command::new(&command)
.args(command_args)
.stdin(Stdio::inherit())
.stdout(Stdio::inherit())
.stderr(Stdio::piped())
.output()
.chain_err(|| "Error while attempting to spawn command to compile and run flow")?;
match output.status.code() {
Some(0) => Ok("Flow ran to completion".to_string()),
Some(code) => {
error!(
"Process STDERR:\n{}",
String::from_utf8_lossy(&output.stderr)
);
bail!("Exited with status code: {}", code)
}
None => Ok("No return code - ignoring".to_string()),
}
My question is not why this could happen (I know that the docs say "terminated by signal") but why it is happening, as no-one AFAIK is sending a signal to it, I seriously doubt any OOM or other such issues.
Read the manual:
On Unix, this will return None if the process was terminated by a signal; std::os::unix provides an extension trait for extracting the signal and other details from the ExitStatus.
use std::os::unix::process::ExitStatusExt;
use std::process::Command;
fn main() {
let mut child = Command::new("sleep")
.args(&["10"])
.spawn()
.expect("failed to spawn child");
child.kill().expect("failed to kill on child");
let status = child.wait().expect("failed to wait on child");
match status.code() {
None => {
println!("{:?}", status.signal());
()
}
_ => (),
}
}
You could use from_c_int() to have a pretty print of the signal type.
Calling write_all on a file returns an error with the description: os error. Debug printing the error outputs: Err(Error { repr: Os(9) })
What does the error mean?
You didn't include any code, so I had to make wild guesses about what you are doing. Here's one piece of code that reproduces your error:
use std::fs;
use std::io::Write;
fn main() {
let mut f = fs::File::open("/").unwrap();
// f.write_all(b"hello").unwrap();
// Error { repr: Os(9) }
match f.write_all(b"hello") {
Ok(..) => {},
Err(e) => println!("{}", e),
}
// Bad file descriptor (os error 9)
}
If you use the Display ({}) format instead of Debug ({:?}), you will see an error message that is nicer than just the error code. Note that unwrap will use the Debug formatter, so you have to use match in this case.
You could also look up the error code in the kernel source. You don't indicate if you are running Windows (unlikely), OS X or Linux, so I guessed Linux.
There are lots of SO questions that then explain what the code can mean, but I'm sure you know how to search through those, now that you have a handle on the problem.
This question refers to Rust as of October 2014.
If you are using Rust 1.0 or above, you best look elsewhere for a solution.
I have a long running Rust process that generates log values, which I'm running using Process.
It looks at though I might be able to periodically "check on" the running process using set_timeout() and wait() and do something kind of high level loop like:
let mut child = match Command::new("thing").arg("...").spawn() {
Ok(child) => child,
Err(e) => fail!("failed to execute child: {}", e),
};
loop {
child.set_timeout(Some(100));
match child.wait() {
// ??? Something goes here
}
}
The things I'm not 100% on are; how do I tell the difference between a timeout error and a process-return error from wait(), and how to a use the PipeStream to "read as much as you can without blocking from the stream" every interval to push out.
Is this the best approach? Should I start a task to monitor stdout and stderr instead?
For distinguishing the errors from the process from the timeout, you have to manage the returns from wait, an example here:
fn run() {
let mut child = match Command::new("sleep").arg("1").spawn() {
Ok(child) => child,
Err(e) => fail!("failed to execute child: {}", e),
};
loop {
child.set_timeout(Some(1000));
match child.wait() {
// Here assume any error is timeout, you can filter from IoErrorKind
Err(..) => println!("Timeout"),
Ok(ExitStatus(0)) => {
println!("Finished without errors");
return;
}
Ok(ExitStatus(a)) => {
println!("Finished with error number: {}", a);
return;
}
Ok(ExitSignal(a)) => {
println!("Terminated by signal number: {}", a);
return;
}
}
}
}
About using streams, check with wait_with_output, or implement something similar with channels and threads : http://doc.rust-lang.org/src/std/home/rustbuild/src/rust-buildbot/slave/nightly-linux/build/src/libstd/io/process.rs.html#601
Hope it helped
Have a look in cargo:
https://docs.rs/cargo-util/0.1.1/cargo_util/struct.ProcessBuilder.html#method.exec_with_streaming
The only downside is that cargo-util seems to need openssl even with default-features=false...
But you can at least see how it and read2 are done.