In Python it is possible to write
from io import StringIO
with StringIO("some text...") as stream:
for line in stream:
# work with the data
process_line(line)
Is there a way I can do the same thing, treat some string as a file object, and apply Read trait to it?
Yes, you can use std::io::Cursor:
use std::io::{Read, Cursor};
fn use_read_trait(s: String, buff: &mut [u8]) -> usize {
let mut c = Cursor::new(s);
c.read(buff).unwrap()
}
Related
I want to write some code that can read bytes from:
stdin
files
a string
TCP
and maybe others. What is the best way to do this in Rust?
I thought the std::io::Read trait was the way to go, but it seems to be lacking implementations for string at least (I just needed this particular one for testing - maybe I can use something else)?
You may find it help to use the impl<'_> Read for &' [u8] trait for reading bytes from a string. As the type indicates, you'll have to first convert your nice string into a slice of bytes. Here is a short, dumb example.
use std::fs::File;
use std::io::{Error, Read, BufReader};
fn whoo<T: Read>(mut readable: T) {
let mut buffer = [0; 10];
readable.read(&mut buffer).expect("panic");
println!("{:?}", buffer);
}
fn main() -> Result<(), Error> {
whoo("hello there".as_bytes());
whoo("".as_bytes());
let dict = File::open("/usr/share/dict/words")?;
let reader = BufReader::new(dict);
whoo(reader);
Ok(())
}
I have a bytes::Bytes (in this case its the body of a request in actix-web) and another function that expects a string slice argument: foo: &str. What is the proper way to convert the bytes::Bytes to &str so that no copy is made? I've tried &body.into() but I get:
the trait `std::convert::From<bytes::bytes::Bytes>` is not implemented for `str`
Here are the basic function signatures:
pub fn parse_body(data: &str) -> Option<&str> {
// Do stuff
// ....
Ok("xyz")
}
fn consume_data(req: HttpRequest<AppState>, body: bytes::Bytes) -> HttpResponse {
let foo = parse_body(&body);
// Do stuff
HttpResponse::Ok().into()
}
Bytes dereferences to [u8], so you can use any existing mechanism to convert &[u8] to a string.
use bytes::Bytes; // 0.4.10
use std::str;
fn example(b: &Bytes) -> Result<&str, str::Utf8Error> {
str::from_utf8(b)
}
See also:
How do I convert a Vector of bytes (u8) to a string
I've tried &body.into()
From and Into are only for infallible conversions. Not all arbitrary chunks of data are valid UTF-8.
I've poked the serde-yaml and yaml-rust crates a bit, but I haven't seen any examples.
serde-yaml's documentation has the following 4 functions:
from_reader — Deserialize an instance of type T from an IO stream of YAML.
from_slice — Deserialize an instance of type T from bytes of YAML text.
from_str — Deserialize an instance of type T from a string of YAML text.
from_value — Interpret a serde_yaml::Value as an instance of type T.
Using from_reader as an example:
use serde_yaml; // 0.8.7
fn main() -> Result<(), Box<dyn std::error::Error>> {
let f = std::fs::File::open("something.yaml")?;
let d: String = serde_yaml::from_reader(f)?;
println!("Read YAML string: {}", d);
Ok(())
}
something.yaml:
"I am YAML"
You can deserialize into the looser-typed Value if you don't know your format (String in this example), but be sure to read the Serde guide for full details of how to do type-directed serialization and deserialization instead.
See also:
How do I parse a JSON File?
Deserializing TOML into vector of enum with values
In general, using any Serde format is pretty much the same as all the rest.
This example uses the yaml_rust crate
use std::fs::File;
use std::io::prelude::*;
use yaml_rust::yaml::{Hash, Yaml};
use yaml_rust::YamlLoader;
fn main() {
println!("Hello, Yaml");
let file = "./etc/my_yaml_file.yaml";
load_file(file);
}
fn load_file(file: &str) {
let mut file = File::open(file).expect("Unable to open file");
let mut contents = String::new();
file.read_to_string(&mut contents)
.expect("Unable to read file");
let docs = YamlLoader::load_from_str(&contents).unwrap();
// iterate / process doc[s] ..
}
The answer from Shepmaster is great if you want to do it properly. Here's a complete example to get started with.
data['foo']['bar'].as_str() returns an Option<str>.
fn example() -> Result<String> {
let f = std::fs::File::open("something.yaml")?;
let data: serde_yaml::Value = serde_yaml::from_reader(f)?;
data["foo"]["bar"]
.as_str()
.map(|s| s.to_string())
.ok_or(anyhow!("Could not find key foo.bar in something.yaml"))
}
How does one stream data from a reader to a write in Rust?
My end goal is actually to write out some gzipped data in a streaming fashion. It seems like what I am missing is a function to iterate over data from a reader and write it out to a file.
This task would be easy to accomplish with read_to_string, etc. But my requirement is to stream the data to keep memory usage down. I have not been able to find a simple way to do this that doesn't make lots of buffer allocations.
use std::io;
use std::io::prelude::*;
use std::io::{BufReader};
use std::fs::File;
use flate2::read::{GzEncoder};
use flate2::{Compression};
pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<()> {
let file = File::create(file)?;
let gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
read_write(gz, file)
}
pub fn read_write<R: BufRead, W: Write>(mut r: R, mut w: W) -> io::Result<()> {
// ?
}
Your read_write function sounds exactly like io::copy. So this would be
pub fn gzipped<R: Read>(file: String, stream: R) -> io::Result<u64> {
let mut file = File::create(file)?;
let mut gz = BufReader::new(GzEncoder::new(stream, Compression::Default));
io::copy(&mut gz, &mut file)
}
The only difference is that io::copy takes mutable references, and returns Result<u64>.
I need a completely in-memory object that I can give to BufReader and BufWriter. Something like Python's StringIO. I want to write to and read from such an object using methods ordinarily used with Files.
Is there a way to do this using the standard library?
In fact there is a way: Cursor<T>!
(please also read Shepmaster's answer on why often it's even easier)
In the documentation you can see that there are the following impls:
impl<T> Seek for Cursor<T> where T: AsRef<[u8]>
impl<T> Read for Cursor<T> where T: AsRef<[u8]>
impl Write for Cursor<Vec<u8>>
impl<T> AsRef<[T]> for Vec<T>
From this you can see that you can use the type Cursor<Vec<u8>> just as an ordinary file, because Read, Write and Seek are implemented for that type!
Little example (Playground):
use std::io::{Cursor, Read, Seek, SeekFrom, Write};
// Create fake "file"
let mut c = Cursor::new(Vec::new());
// Write into the "file" and seek to the beginning
c.write_all(&[1, 2, 3, 4, 5]).unwrap();
c.seek(SeekFrom::Start(0)).unwrap();
// Read the "file's" contents into a vector
let mut out = Vec::new();
c.read_to_end(&mut out).unwrap();
println!("{:?}", out);
For a more useful example, check the documentation linked above.
You don't need a Cursor most of the time.
object that I can give to BufReader and BufWriter
BufReader requires a value that implements Read:
impl<R: Read> BufReader<R> {
pub fn new(inner: R) -> BufReader<R>
}
BufWriter requires a value that implements Write:
impl<W: Write> BufWriter<W> {
pub fn new(inner: W) -> BufWriter<W> {}
}
If you view the implementors of Read you will find impl<'a> Read for &'a [u8].
If you view the implementors of Write, you will find impl Write for Vec<u8>.
use std::io::{Read, Write};
fn main() {
// Create fake "file"
let mut file = Vec::new();
// Write into the "file"
file.write_all(&[1, 2, 3, 4, 5]).unwrap();
// Read the "file's" contents into a new vector
let mut out = Vec::new();
let mut c = file.as_slice();
c.read_to_end(&mut out).unwrap();
println!("{:?}", out);
}
Writing to a Vec will always append to the end. We also take a slice to the Vec that we can update. Each read of c will advance the slice further and further until it is empty.
The main differences from Cursor:
Cannot seek the data, so you cannot easily re-read data
Cannot write to anywhere but the end
If you want to use BufReader with an in-memory String, you can use the as_bytes() method:
use std::io::BufRead;
use std::io::BufReader;
use std::io::Read;
fn read_buff<R: Read>(mut buffer: BufReader<R>) {
let mut data = String::new();
let _ = buffer.read_line(&mut data);
println!("read_buff got {}", data);
}
fn main() {
read_buff(BufReader::new("Potato!".as_bytes()));
}
This prints read_buff got Potato!. There is no need to use a cursor for this case.
To use an in-memory String with BufWriter, you can use the as_mut_vec method. Unfortunately it is unsafe and I have not found any other way. I don't like the Cursor approach since it consumes the vector and I have not found a way yet to use the Cursor together with BufWriter.
use std::io::BufWriter;
use std::io::Write;
pub fn write_something<W: Write>(mut buf: BufWriter<W>) {
buf.write("potato".as_bytes());
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::{BufWriter};
#[test]
fn testing_bufwriter_and_string() {
let mut s = String::new();
write_something(unsafe { BufWriter::new(s.as_mut_vec()) });
assert_eq!("potato", &s);
}
}