How do I insert a dynamic byte string into a vector? - rust

I need to create packet to send to the server. For this purpose I use vector with byteorder crate. When I try to append string, Rust compiler tells I use unsafe function and give me an error.
use byteorder::{LittleEndian, WriteBytesExt};
fn main () {
let login = "test";
let packet_length = 30 + (login.len() as i16);
let mut packet = Vec::new();
packet.write_u8(0x00);
packet.write_i16::<LittleEndian>(packet_length);
packet.append(&mut Vec::from(String::from("game name ").as_bytes_mut()));
// ... rest code
}
The error is:
packet.append(&mut Vec::from(String::from("game name ").as_bytes_mut()));
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
This is playground to reproduce: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=381c6d14660d47beaece15d068b3dc6a
What is the correct way to insert some string as bytes into vector ?

The unsafe function called was as_bytes_mut(). This creates a mutable reference with exclusive access to the bytes representing the string, allowing you to modify them. You do not really need a mutable reference in this case, as_bytes() would have sufficed.
However, there is a more idiomatic way. Vec<u8> also functions as a writer (it implements std::io::Write), so you can use one of its methods, or even the write! macro, to write encoded text on it.
use std::io::Write;
use byteorder::{LittleEndian, WriteBytesExt};
fn main () -> Result<(), std::io::Error> {
let login = "test";
let packet_length = 30 + (login.len() as i16);
let mut packet = Vec::new();
packet.write_u8(0x00)?;
packet.write_i16::<LittleEndian>(packet_length)?;
let game_name = String::from("game name");
write!(packet, "{} ", game_name)?;
Ok(())
}
Playground
See also:
Use write! macro with a string instead of a string literal
What's the de-facto way of reading and writing files in Rust 1.x?

You can use .extend() on the Vec and pass in the bytes representation of the String:
use byteorder::{LittleEndian, WriteBytesExt};
fn main() {
let login = "test";
let packet_length = 30 + (login.len() as i16);
let mut packet = Vec::new();
packet.write_u8(0x00);
packet.write_i16::<LittleEndian>(packet_length);
let string = String::from("game name ");
packet.extend(string.as_bytes());
}
Playground

Related

Writing a Vec<String> to files using std::fs::write

I'm writing a program that handles a vector which is combination of numbers and letters (hence Vec<String>). I sort it with the .sort() method and am now trying to write it to a file.
Where strvec is my sorted vector that I'm trying to write using std::fs::write;
println!("Save results to file?");
let to_save: String = read!();
match to_save.as_str() {
"y" => {
println!("Enter filename");
let filename: String = read!();
let pwd = current_dir().into();
write("/home/user/dl/results", strvec);
Rust tells me "the trait AsRef<[u8]> is not implemented for Vec<String>". I've also tried using &strvec.
How do I avoid this/fix it?
When it comes to writing objects to the file you might want to consider serialization. Most common library for this in Rust is serde, however in this example where you want to store vector of Strings and if you don't need anything human readable in file (but it comes with small size :P), you can also use bincode:
use std::fs;
use bincode;
fn main() {
let v = vec![String::from("aaa"), String::from("bbb")];
let encoded_v = bincode::serialize(&v).expect("Could not encode vector");
fs::write("file", encoded_v).expect("Could not write file");
let read_v = fs::read("file").expect("Could not read file");
let decoded_v: Vec<String> = bincode::deserialize(&read_v).expect("Could not decode vector");
println!("{:?}", decoded_v);
}
Remember to add bincode = "1.3.3" under dependencies in Cargo.toml
#EDIT:
Actually you can easily save String to the file so simple join() should do:
use std::fs;
fn main() {
let v = vec![
String::from("aaa"),
String::from("bbb"),
String::from("ccc")];
fs::write("file", v.join("\n")).expect("");
}
Rust can't write anything besides a &[u8] to a file. There are too many different ways which data can be interpreted before it gets flattened, so you need to handle all of that ahead of time. For a Vec<String>, it's pretty simple, and you can just use concat to squish everything down to a single String, which can be interpreted as a &[u8] because of its AsRef<u8> trait impl.
Another option would be to use join, in case you wanted to add some sort of delimiter between your strings, like a space, comma, or something.
fn main() {
let strvec = vec![
"hello".to_string(),
"world".to_string(),
];
// "helloworld"
std::fs::write("/tmp/example", strvec.concat()).expect("failed to write to file");
// "hello world"
std::fs::write("/tmp/example", strvec.join(" ")).expect("failed to write to file");
}
You can't get a &[u8] from a Vec<String> without copying since a slice must refer to a contiguous sequence of items. Each String will have its own allocation on the heap somewhere, so while each individual String can be converted to a &[u8], you can't convert the whole vector to a single &[u8].
While you can .collect() the vector into a single String and then get a &[u8] from that, this does some unnecessary copying. Consider instead just iterating the Strings and writing each one to the file. With this helper, it's no more complex than using std::fs::write():
use std::path::Path;
use std::fs::File;
use std::io::Write;
fn write_each(
path: impl AsRef<Path>,
items: impl IntoIterator<Item=impl AsRef<[u8]>>,
) -> std::io::Result<()> {
let mut file = File::create(path)?;
for i in items {
file.write_all(i.as_ref())?;
}
// Surface any I/O errors that could otherwise be swallowed when
// the file is closed implicitly by being dropped.
file.sync_all()
}
The bound impl IntoIterator<Item=impl AsRef<[u8]>> is satisfied by both Vec<String> and by &Vec<String>, so you can call this as either write_each("path/to/output", strvec) (to consume the vector) or write_each("path/to/output", &strvec) (if you need to hold on to the vector for later).

How to `read` a number of bytes into a `Vec`? [duplicate]

I have a struct with a BufStream<T> where T: Read+Write.
The BufStream can be a TcpStream and I'd like to read n bytes from it.
Not a fixed amount of bytes in a predefined buffer, but I have a string/stream which indicates the number of bytes to read next.
Is there a nice way to do that?
Since Rust 1.6, Read::read_exact can be used to do this. If bytes_to_read is the number of bytes you need to read, possibly determined at runtime, and reader is the stream to read from:
let mut buf = vec![0u8; bytes_to_read];
reader.read_exact(&mut buf)?;
The part that wasn't clear to me from the read_exact documentation was that the target buffer can be a dynamically-allocated Vec.
Thanks to the Rust Gitter community for pointing me to this solution.
It sounds like you want Read::take and Read::read_to_end.
This will allow you to read data into a &mut Vec<u8>, which is useful when you want to reuse an existing buffer or don't have an appropriately sized slice already. This allows you to avoid initializing the data with dummy values before overwriting them with the newly-read information:
use std::{
io::{prelude::*, BufReader},
str,
};
fn read_n<R>(reader: R, bytes_to_read: u64) -> Vec<u8>
where
R: Read,
{
let mut buf = vec![];
let mut chunk = reader.take(bytes_to_read);
// Do appropriate error handling for your situation
// Maybe it's OK if you didn't read enough bytes?
let n = chunk.read_to_end(&mut buf).expect("Didn't read enough");
assert_eq!(bytes_to_read as usize, n);
buf
}
fn main() {
let input_data = b"hello world";
let mut reader = BufReader::new(&input_data[..]);
let first = read_n(&mut reader, 5);
let _ = read_n(&mut reader, 1);
let second = read_n(&mut reader, 5);
println!(
"{:?}, {:?}",
str::from_utf8(&first),
str::from_utf8(&second)
);
}
If you are worried that Read::take consumes the reader by reference, note that take comes from Read and Read is implemented for any mutable reference to a type that implements Read. You can also use Read::by_ref to create this mutable reference.
See also:
Whats the idiomatic way to reference BufReader/BufWriter when passing it between functions?
Why does Iterator::take_while take ownership of the iterator?

How do I specify the calling convention for a C callback that uses &mut [u8] and u32? [duplicate]

Can I somehow get an array from std::ptr::read?
I'd like to do something close to:
let mut v: Vec<u8> = ...
let view = &some_struct as *const _ as *const u8;
v.write(&std::ptr::read<[u8, ..30]>(view));
Which is not valid in this form (can't use the array signature).
If you want to obtain a slice from a raw pointer, use std::slice::from_raw_parts():
let slice = unsafe { std::slice::from_raw_parts(some_pointer, count_of_items) };
If you want to obtain a mutable slice from a raw pointer, use std::slice::from_raw_parts_mut():
let slice = unsafe { std::slice::from_raw_parts_mut(some_pointer, count_of_items) };
Are you sure you want read()? Without special care it will cause disaster on structs with destructors. Also, read() does not read a value of some specified type from a pointer to bytes; it reads exactly one value of the type behind the pointer (e.g. if it is *const u8 then read() will read one byte) and returns it.
If you only want to write byte contents of a structure into a vector, you can obtain a slice from the raw pointer:
use std::mem;
use std::io::Write;
struct SomeStruct {
a: i32,
}
fn main() {
let some_struct = SomeStruct { a: 32 };
let mut v: Vec<u8> = Vec::new();
let view = &some_struct as *const _ as *const u8;
let slice = unsafe { std::slice::from_raw_parts(view, mem::size_of::<SomeStruct>()) };
v.write(slice).expect("Unable to write");
println!("{:?}", v);
}
This makes your code platform-dependent and even compiler-dependent: if you use types of variable size (e.g. isize/usize) in your struct or if you don't use #[repr(C)], the data you wrote into the vector is likely to be read as garbage on another machine (and even #[repr(C)] may not lift this problem sometimes, as far as I remember).

No method named to_str found for type Option<&std::ffi::OsStr>?

Here is my current attempt at iterating through a directory and adding filetype extensions to a map and counting how many files have that extension type.
The call to to_str() fails because
no method named `to_str` found for type `std::option::Option<&std::ffi::OsStr>` in the current scope [E0599]
I tried converting to a string elsewhere, I tried not even evaluating the Option and trying to just insert it as-is into the map, but that didn't work either.
extern crate walkdir;
use std::ffi::OsStr;
use std::path::Path;
use walkdir::WalkDir;
extern crate serde_json;
use serde_json::{Map, Number, Value};
fn main() {
let mut map = Map::new();
let walker = WalkDir::new("/Users/jamescampbell/").into_iter();
for entry in walker {
let entry = entry.unwrap();
let os_str = OsStr::new(entry.file_name());
let path = Path::new(os_str);
let extensioner = path.extension();
let my_new_string: String = match extensioner.to_str() {
None => String::from("crap, os_str failed"),
Some(s) => s,
};
println!("should be foo: {}", my_new_string);
if !map[s] {
map.insert(s.to_string(), Value::Number(Number::from(1u64)));
} else {
map[s] += 1;
}
}
}
A couple of issues.
The first one is that the result of extension() is an Option<T>. Specifically, its a Option<&std::ffi::OsStr>. So you must unwrap it:
That isn't the only issue though, because calling to_str() on an &OsStr will return a Option<&str>, which itself is also not a String.
Leaving out all of the potential ignored issues with the conversions between string types ... this is a minimal recreation of the part that is causing you issues:
use std::path::Path;
use std::ffi::OsStr;
fn main() {
let os_str = OsStr::new("example.txt");
let path = Path::new(os_str);
let extensioner = path.extension();
let my_new_string: String = extensioner.unwrap().to_str().unwrap().into();
println!("Extension: {}", my_new_string);
}
Here it is running on the playground
Basically, we:
unwrap the Option<&OsStr> into an &OsStr.
Call to_str() on the &OsStr
unwrap the resulting Option<&str> into an &str
Call into(), which will convert it into a String (this invokes the From trait implementation of String)

How to get the byte offset between `&str`

I have two &str pointing to the same string, and I need to know the byte offset between them:
fn main() {
let foo = " bar";
assert_eq!(offset(foo, foo.trim()), Some(2));
let bar = "baz\nquz";
let mut lines = bar.lines();
assert_eq!(offset(bar, lines.next().unwrap()), Some(0));
assert_eq!(offset(bar, lines.next().unwrap()), Some(4));
assert_eq!(offset(foo, bar), None); // not a sub-string
let quz = "quz".to_owned();
assert_eq!(offset(bar, &quz), None); // not the same string, could also return `Some(4)`, I don't care
}
This is basically the same as str::find, but since the second slice is a sub-slice of the first, I would have hoped something faster. Also str::find won't work in the lines() case if several lines are identical.
I thought I could just use some pointer arithmetic to do that with something like foo.trim().as_ptr() - foo.as_ptr() but it turns out that Sub is not implemented on raw pointers.
but it turns out that Sub is not implemented on raw pointers.
You can use the offset_from method:
fn main() {
let source = "hello, world";
let a = &source[1..];
let b = &source[5..];
// I copied this unsafe code from Stack Overflow without
// reading the text that told me how to know if this was safe
let diff = unsafe { b.as_ptr().offset_from(a.as_ptr()) };
println!("{diff}");
}
Please be sure to read the documentation for this method as it describes under what circumstances it will not cause undefined behavior.
In older versions of Rust, you can convert the pointer to a usize to do math on it:
fn main() {
let source = "hello, world";
let a = &source[1..];
let b = &source[5..];
let diff = b.as_ptr() as usize - a.as_ptr() as usize;
println!("{diff}");
}
This is of course kind of unsafe, but if you want arithmetic, you can just cast the pointers to usize with as and subtract that.
(Note: it's not so unsafe that the compiler will actually complain.)

Resources