Trouble with buffer types in mio - rust

I'd like to write an asynchronous server in Rust using mio and I have trouble with the buffer types. I've tried different buffer types and can't get it to work. My current code is:
extern crate mio;
extern crate bytes;
use std::io;
use std::io::{Error, ErrorKind};
use std::net::SocketAddr;
use std::str::FromStr;
use std::io::Cursor;
use self::mio::PollOpt;
use self::mio::EventLoop;
use self::mio::EventSet;
use self::mio::Token;
use self::mio::Handler;
use self::mio::io::TryRead;
use self::mio::io::TryWrite;
//use self::mio::buf::ByteBuf;
//use self::mio::buf::Buf;
use self::mio::tcp::*;
use self::bytes::buf::Buf;
use self::bytes::buf::byte::ByteBuf;
struct EventHandler;
impl Handler for EventHandler {
type Timeout = ();
type Message = ();
fn ready(&mut self, event_loop: &mut EventLoop<EventHandler>, token: Token, events: EventSet) {
}
}
pub struct Connection {
sock: TcpStream,
send_queue: Vec<ByteBuf>,
}
impl Connection {
pub fn writable(&mut self, event_loop: &mut EventLoop<EventHandler>) -> Result<(), String> {
while !self.send_queue.is_empty() {
if !self.send_queue.first().unwrap().has_remaining() {
self.send_queue.pop();
}
let buf = self.send_queue.first_mut().unwrap();
match self.sock.try_write_buf(&mut buf) {
Ok(None) => {
return Ok(());
}
Ok(Some(n)) => {
continue;
}
Err(e) => {
return Err(format!("{}", e));
}
}
}
Ok(())
}
}
fn main() {
println!("Hello, world!");
}
The Cargo.toml contains the following dependencies:
mio = "*"
bytes = "*"
which currently translates to bytes 0.2.11 and mio 0.4.3 in Cargo.lock.
The error I am getting is this:
main.rs:45:29: 45:52 error: the trait `bytes::buf::Buf` is not implemented
for the type `&mut bytes::buf::byte::ByteBuf` [E0277]
main.rs:45 match self.sock.try_write_buf(&mut buf) {
I'd want to be able to write a Vec<u8> into the socket and handle the case when the buffer is only partially written. How can I accomplish that?
I don't need explanation about the code that properly handles the return values, this question is about the buffer type. I have no idea which buffer type I have to use.

The problem is this:
let buf = self.send_queue.first_mut().unwrap();
match self.sock.try_write_buf(&mut buf) {
You pass in an &mut &mut ByteBuf to try_write_buf because buf is already an &mut ByteBuf. Just drop the extra &mut:
let buf = self.send_queue.first_mut().unwrap();
match self.sock.try_write_buf(buf) {

Related

How to convert a Bytes Iterator into a Stream in Rust

I'm trying to figure out build a feature which requires reading the contents of a file into a futures::stream::BoxStream but I'm having a tough time figuring out what I need to do.
I have figured out how to read a file byte by byte via Bytes which implements an iterator.
use std::fs::File;
use std::io::prelude::*;
use std::io::{BufReader, Bytes};
// TODO: Convert this to a async Stream
fn async_read() -> Box<dyn Iterator<Item = Result<u8, std::io::Error>>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
let iter = reader.bytes().into_iter();
Box::new(iter)
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
for b in async_read().into_iter() {
println!("{:?}", b);
}
}
However, I've been struggling a bunch trying to figure out how I can turn this Box<dyn Iterator<Item = Result<u8, std::io::Error>>> into an Stream.
I would have thought something like this would work:
use futures::stream;
use std::fs::File;
use std::io::prelude::*;
use std::io::{BufReader, Bytes};
// TODO: Convert this to a async Stream
fn async_read() -> stream::BoxStream<'static, dyn Iterator<Item = Result<u8, std::io::Error>>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
let iter = reader.bytes().into_iter();
std::pin::Pin::new(Box::new(stream::iter(iter)))
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
while let Some(b) = async_read().poll() {
println!("{:?}", b);
}
}
But I keep getting a ton of compiler errors, I've tried other permutations but generally getting no where.
One of the compiler errors:
std::pin::Pin::new
``` --> src/main.rs:14:24
|
14 | std::pin::Pin::new(Box::new(stream::iter(iter)))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected trait object `dyn std::iter::Iterator`, found enum `std::result::Result`
Anyone have any advice?
I'm pretty new to Rust, and specifically Streams/lower level stuff so I apologize if I got anything wrong, feel free to correct me.
For some additional background, I'm trying to do this so you can CTRL-C out of a command in nushell
I think you are overcomplicating it a bit, you can just return impl Stream from async_read, there is no need to box or pin (same goes for the original Iterator-based version). Then you need to set up an async runtime in order to poll the stream (in this example I just use the runtime provided by futures::executor::block_on). Then you can call futures::stream::StreamExt::next() on the stream to get a future representing the next item.
Here is one way to do this:
use futures::prelude::*;
use std::{
fs::File,
io::{prelude::*, BufReader},
};
fn async_read() -> impl Stream<Item = Result<u8, std::io::Error>> {
let f = File::open("/dev/random").expect("Could not open file");
let reader = BufReader::new(f);
stream::iter(reader.bytes())
}
async fn async_main() {
while let Some(b) = async_read().next().await {
println!("{:?}", b);
}
}
fn main() {
ctrlc::set_handler(move || {
println!("received Ctrl+C!");
std::process::exit(0);
})
.expect("Error setting Ctrl-C handler");
futures::executor::block_on(async_main());
}

How to add special NotReady logic to tokio-io?

I'm trying to make a Stream that would wait until a specific character is in buffer. I know there's read_until() on BufRead but I actually need a custom solution, as this is a stepping stone to implement waiting until a specific string in in buffer (or, for example, a regexp match happens).
In my project where I first encountered the problem, problem was that future processing just hanged when I get a Ready(_) from inner future and return NotReady from my function. I discovered I shouldn't do that per docs (last paragraph). However, what I didn't get, is what's the actual alternative that is promised in that paragraph. I read all the published documentation on the Tokio site and it doesn't make sense for me at the moment.
So following is my current code. Unfortunately I couldn't make it simpler and smaller as it's already broken. Current result is this:
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
Err(Custom { kind: Other, error: Error(Shutdown) })
<ad infinum>
Expected result is getting some Ok(Ready(_)) out of it, while printing W and W', and waiting for specific character in buffer.
extern crate futures;
extern crate tokio_core;
extern crate tokio_io;
extern crate tokio_io_timeout;
extern crate tokio_process;
use futures::stream::poll_fn;
use futures::{Async, Poll, Stream};
use tokio_core::reactor::Core;
use tokio_io::AsyncRead;
use tokio_io_timeout::TimeoutReader;
use tokio_process::CommandExt;
use std::process::{Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct Process {
child: tokio_process::Child,
stdout: Arc<Mutex<tokio_io_timeout::TimeoutReader<tokio_process::ChildStdout>>>,
}
impl Process {
fn new(
command: &str,
reader_timeout: Option<Duration>,
core: &tokio_core::reactor::Core,
) -> Self {
let mut cmd = Command::new(command);
let cat = cmd.stdout(Stdio::piped());
let mut child = cat.spawn_async(&core.handle()).unwrap();
let stdout = child.stdout().take().unwrap();
let mut timeout_reader = TimeoutReader::new(stdout);
timeout_reader.set_timeout(reader_timeout);
let timeout_reader = Arc::new(Mutex::new(timeout_reader));
Self {
child,
stdout: timeout_reader,
}
}
}
fn work() -> Result<(), ()> {
let window = Arc::new(Mutex::new(Vec::new()));
let mut core = Core::new().unwrap();
let process = Process::new("cat", Some(Duration::from_secs(20)), &core);
let mark = Arc::new(Mutex::new(b'c'));
let read_until_stream = poll_fn({
let window = window.clone();
let timeout_reader = process.stdout.clone();
move || -> Poll<Option<u8>, std::io::Error> {
let mut buf = [0; 8];
let poll;
{
let mut timeout_reader = timeout_reader.lock().unwrap();
poll = timeout_reader.poll_read(&mut buf);
}
match poll {
Ok(Async::Ready(0)) => Ok(Async::Ready(None)),
Ok(Async::Ready(x)) => {
{
let mut window = window.lock().unwrap();
println!("W: {:?}", *window);
println!("buf: {:?}", &buf[0..x]);
window.extend(buf[0..x].into_iter().map(|x| *x));
println!("W': {:?}", *window);
if let Some(_) = window.iter().find(|c| **c == *mark.lock().unwrap()) {
Ok(Async::Ready(Some(1)))
} else {
Ok(Async::NotReady)
}
}
}
Ok(Async::NotReady) => Ok(Async::NotReady),
Err(e) => Err(e),
}
}
});
let _stream_thread = thread::spawn(move || {
for o in read_until_stream.wait() {
println!("{:?}", o);
}
});
match core.run(process.child) {
Ok(_) => {}
Err(e) => {
println!("Child error: {:?}", e);
}
}
Ok(())
}
fn main() {
work().unwrap();
}
This is complete example project.
If you need more data you need to call poll_read again until you either find what you were looking for or poll_read returns NotReady.
You might want to avoid looping in one task for too long, so you can build yourself a yield_task function to call instead if poll_read didn't return NotReady; it makes sure your task gets called again ASAP after other pending tasks were run.
To use it just run return yield_task();.
fn yield_inner() {
use futures::task;
task::current().notify();
}
#[inline(always)]
pub fn yield_task<T, E>() -> Poll<T, E> {
yield_inner();
Ok(Async::NotReady)
}
Also see futures-rs#354: Handle long-running, always-ready futures fairly #354.
With the new async/await API futures::task::current is gone; instead you'll need a std::task::Context reference, which is provided as parameter to the new std::future::Future::poll trait method.
If you're already manually implementing the std::future::Future trait you can simply insert:
context.waker().wake_by_ref();
return std::task::Poll::Pending;
Or build yourself a Future-implementing type that yields exactly once:
pub struct Yield {
ready: bool,
}
impl core::future::Future for Yield {
type Output = ();
fn poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::Context<'_>) -> core::task::Poll<Self::Output> {
let this = self.get_mut();
if this.ready {
core::task::Poll::Ready(())
} else {
cx.waker().wake_by_ref();
this.ready = true; // ready next round
core::task::Poll::Pending
}
}
}
pub fn yield_task() -> Yield {
Yield { ready: false }
}
And then use it in async code like this:
yield_task().await;

Rust "if let" not working?

I am wrapping libxml2 in Rust as an exercise in learning the Rust FFI, and I have come across something strange. I am new to Rust, but I believe the following should work.
In main.rs I have:
mod xml;
fn main() {
if let doc = xml::parse_file("filename") {
doc.simple_function();
}
}
And xml.rs is:
extern create libc;
use libc::{c_void, c_char, c_int, c_ushort};
use std::ffi::CString;
// There are other struct definitions I've implemented that xmlDoc
// depends on, but I'm not going to list them because I believe
// they're irrelevant
#[allow(non_snake_case)]
#[allow(non_camel_case_types)]
#[repr(C)]
struct xmlDoc {
// xmlDoc structure defined by the libxml2 API
}
pub struct Doc {
ptr: *mut xmlDoc
}
impl Doc {
pub fn simple_function(&self) {
if self.ptr.is_null() {
println!("ptr doesn't point to anything");
} else {
println!("ptr is not null");
}
}
}
#[allow(non_snake_case)]
#[link(name = "xml2")]
extern {
fn xmlParseFile(filename: *const c_char) -> *mut xmlDoc;
}
pub fn parse_file(filename: &str) -> Option<Doc> {
unsafe {
let result;
match CString::new(filename) {
Ok(f) => { result = xmlParseFile(f.as_ptr()); },
Err(_) => { return None; }
}
if result.is_null() {
return None;
}
Some(Doc { ptr: result })
}
}
I'm wrapping the C struct, xmlDoc in a nice Rust-friendly struct, Doc, to have a clear delineation between the safe (Rust) and unsafe (C) data types and functions.
This all works for the most part except when I compile, I get an error in main.rs:
src/main.rs:38:13: 38:28 error: no method named 'simple_function' found
for type 'std::option::Option<xml::Doc>' in the current scope
src/main.rs:38 doc.simple_function();
^~~~~~~~~~~~~~~
error: aborting due to previous error`
It seems convinced that doc is an Option<xml::Doc> even though I'm using the if let form that should unwrap the Option type. Is there something I'm doing incorrectly?
match xml::parse_file("filename") {
Some(doc) => doc.simple_function(),
None => {}
}
The above works fine, but I'd like to use the if let feature of Rust if I'm able.
You need to pass the actual pattern to if let (unlike languages like Swift which special case if let for Option types):
if let Some(doc) = xml::parse_file("filename") {
doc.simple_function();
}

A client for HTTP server push (streaming) in Rust?

For the lack of a better example, let's say I want to write a simple client with Rust that could establish a connection and receive data from Twitter's HTTP Streaming API. Is this possible yet? I've been keeping an eye on Iron and Nickel which seem like good frameworks, but I don't think they have this feature yet?
The http client hyper supports reading responses incrementally (as anything that implements rust's Reader trait), but I wasn't able to find anything to parse the response incrementally, or that implements twitter's particular protocol (to end objecs with \r\n).
That said, I was able to implement a quick'n'dirty proof of concept:
EDIT: See and play with it on github.
use rustc_serialize::json::Json;
use std::str;
pub trait JsonObjectStreamer {
fn json_objects(&mut self) -> JsonObjects<Self>;
}
impl<T: Buffer> JsonObjectStreamer for T {
fn json_objects(&mut self) -> JsonObjects<T> {
JsonObjects { reader: self }
}
}
pub struct JsonObjects<'a, B> where B: 'a {
reader: &'a mut B
}
impl<'a, B> Iterator for JsonObjects<'a, B> where B: Buffer + 'a {
type Item = Json;
fn next(&mut self) -> Option<Json> {
let mut line_bytes = match self.reader.read_until(b'\r') {
Ok(bytes) => bytes,
Err(_) => return None,
};
if line_bytes.last() == Some(&b'\r') {
// drop the \r
line_bytes.pop();
// skip the \n
match self.reader.read_char() {
Ok(_) => (),
Err(_) => return None,
}
}
let line = match str::from_utf8(&line_bytes) {
Ok(line) => line,
Err(_) => return None
};
Json::from_str(line).ok()
}
}
Usage: (assuming you have dropped it on a src/json_streamer.rs file on your project)
#![feature(io)]
extern crate hyper;
extern crate "rustc-serialize" as rustc_serialize;
mod json_streamer;
use hyper::Client;
use std::old_io::BufferedReader;
use json_streamer::JsonObjectStreamer;
fn main() {
let mut client = Client::new();
let res = client.get("http://localhost:4567/").send().unwrap();
for obj in BufferedReader::new(res).json_objects() {
println!("object arrived: {}", obj);
}
}
I've used this tiny sinatra app to test it:
require 'sinatra'
require 'json'
class Stream
def each
hash = { index: 0 }
loop do
hash[:index] += 1
yield hash.to_json + "\r\n"
sleep 0.5
end
end
end
get '/' do
Stream.new
end

"Registering" trait implementations + factory method for trait objects

Say we want to have objects implementations switched at runtime, we'd do something like this:
pub trait Methods {
fn func(&self);
}
pub struct Methods_0;
impl Methods for Methods_0 {
fn func(&self) {
println!("foo");
}
}
pub struct Methods_1;
impl Methods for Methods_1 {
fn func(&self) {
println!("bar");
}
}
pub struct Object<'a> { //'
methods: &'a (Methods + 'a),
}
fn main() {
let methods: [&Methods; 2] = [&Methods_0, &Methods_1];
let mut obj = Object { methods: methods[0] };
obj.methods.func();
obj.methods = methods[1];
obj.methods.func();
}
Now, what if there are hundreds of such implementations? E.g. imagine implementations of cards for collectible card game where every card does something completely different and is hard to generalize; or imagine implementations for opcodes for a huge state machine. Sure you can argue that a different design pattern can be used -- but that's not the point of this question...
Wonder if there is any way for these Impl structs to somehow "register" themselves so they can be looked up later by a factory method? I would be happy to end up with a magical macro or even a plugin to accomplish that.
Say, in D you can use templates to register the implementations -- and if you can't for some reason, you can always inspect modules at compile-time and generate new code via mixins; there are also user-defined attributes that can help in this. In Python, you would normally use a metaclass so that every time a new child class is created, a ref to it is stored in the metaclass's registry which allows you to look up implementations by name or parameter; this can also be done via decorators if implementations are simple functions.
Ideally, in the example above you would be able to create Object as
Object::new(0)
where the value 0 is only known at runtime and it would magically return you an Object { methods: &Methods_0 }, and the body of new() would not have the implementations hard-coded like so "methods: [&Methods; 2] = [&Methods_0, &Methods_1]", instead it should be somehow inferred automatically.
So, this is probably extremely buggy, but it works as a proof of concept.
It is possible to use Cargo's code generation support to make the introspection at compile-time, by parsing (not exactly parsing in this case, but you get the idea) the present implementations, and generating the boilerplate necessary to make Object::new() work.
The code is pretty convoluted and has no error handling whatsoever, but works.
Tested on rustc 1.0.0-dev (2c0535421 2015-02-05 15:22:48 +0000)
(See on github)
src/main.rs:
pub mod implementations;
mod generated_glue {
include!(concat!(env!("OUT_DIR"), "/generated_glue.rs"));
}
use generated_glue::Object;
pub trait Methods {
fn func(&self);
}
pub struct Methods_2;
impl Methods for Methods_2 {
fn func(&self) {
println!("baz");
}
}
fn main() {
Object::new(2).func();
}
src/implementations.rs:
use super::Methods;
pub struct Methods_0;
impl Methods for Methods_0 {
fn func(&self) {
println!("foo");
}
}
pub struct Methods_1;
impl Methods for Methods_1 {
fn func(&self) {
println!("bar");
}
}
build.rs:
#![feature(core, unicode, path, io, env)]
use std::env;
use std::old_io::{fs, File, BufferedReader};
use std::collections::HashMap;
fn main() {
let target_dir = Path::new(env::var_string("OUT_DIR").unwrap());
let mut target_file = File::create(&target_dir.join("generated_glue.rs")).unwrap();
let source_code_path = Path::new(file!()).join_many(&["..", "src/"]);
let source_files = fs::readdir(&source_code_path).unwrap().into_iter()
.filter(|path| {
match path.str_components().last() {
Some(Some(filename)) => filename.split('.').last() == Some("rs"),
_ => false
}
});
let mut implementations = HashMap::new();
for source_file_path in source_files {
let relative_path = source_file_path.path_relative_from(&source_code_path).unwrap();
let source_file_name = relative_path.as_str().unwrap();
implementations.insert(source_file_name.to_string(), vec![]);
let mut file_implementations = &mut implementations[*source_file_name];
let mut source_file = BufferedReader::new(File::open(&source_file_path).unwrap());
for line in source_file.lines() {
let line_str = match line {
Ok(line_str) => line_str,
Err(_) => break,
};
if line_str.starts_with("impl Methods for Methods_") {
const PREFIX_LEN: usize = 25;
let number_len = line_str[PREFIX_LEN..].chars().take_while(|chr| {
chr.is_digit(10)
}).count();
let number: i32 = line_str[PREFIX_LEN..(PREFIX_LEN + number_len)].parse().unwrap();
file_implementations.push(number);
}
}
}
writeln!(&mut target_file, "use super::Methods;").unwrap();
for (source_file_name, impls) in &implementations {
let module_name = match source_file_name.split('.').next() {
Some("main") => "super",
Some(name) => name,
None => panic!(),
};
for impl_number in impls {
writeln!(&mut target_file, "use {}::Methods_{};", module_name, impl_number).unwrap();
}
}
let all_impls = implementations.values().flat_map(|impls| impls.iter());
writeln!(&mut target_file, "
pub struct Object;
impl Object {{
pub fn new(impl_number: i32) -> Box<Methods + 'static> {{
match impl_number {{
").unwrap();
for impl_number in all_impls {
writeln!(&mut target_file,
" {} => Box::new(Methods_{}),", impl_number, impl_number).unwrap();
}
writeln!(&mut target_file, "
_ => panic!(\"Unknown impl number: {{}}\", impl_number),
}}
}}
}}").unwrap();
}
The generated code:
use super::Methods;
use super::Methods_2;
use implementations::Methods_0;
use implementations::Methods_1;
pub struct Object;
impl Object {
pub fn new(impl_number: i32) -> Box<Methods + 'static> {
match impl_number {
2 => Box::new(Methods_2),
0 => Box::new(Methods_0),
1 => Box::new(Methods_1),
_ => panic!("Unknown impl number: {}", impl_number),
}
}
}

Resources