I am new to embedded and Rust programming. I am working on sensor and I am able to get the readings of the sensor.
The problem:
The data is in the itm.txt file which includes headers and a variable-length payload, Data in itm.txt file. When I am getting the readings of the sensor I am getting this.
Can someone please help me to parse this data, I am finding no way in Rust to parse this data.
This is the code I am using to get the file content.
use std::io;
use std::fs::File;
use std::io::{Read, BufReader};
fn decode<R: io::Read>(mut r: R) -> io::Result<Vec<u8>> {
let mut output = vec![];
loop {
let mut len = 0u8;
r.read_exact(std::slice::from_mut(&mut len))?;
let len = 1 << (len - 1);
let mut buf = vec![0; len];
let res = r.read_exact(&mut buf);
if buf == b"\0" {
break;
}
output.extend(buf);
}
Ok(output)
}
fn main() {
//let data = "{ x: 579, y: -197 , z: -485 }\0";
let mut file = File::open("/tmp/itm.txt").expect("No such file is present at this location");
let mut buf_reader = Vec::new();
file.read_to_end(&mut buf_reader).expect("error");
let content = std::str::from_utf8(&buf_reader).unwrap();
println!("raw str: {:?}", content);
println!("raw hex: {:x?}", content.as_bytes());
let decoded = decode(content.as_bytes()).unwrap();
let s = std::str::from_utf8(&decoded).expect("Failed");
println!("decoded str: {:?}", s);
}
The error I am getting now
raw str: "\u{2}Un\u{3}scal\u{3}edMe\u{3}asur\u{3}emen\u{1}t\u{1} \u{2}{ \u{1}x\u{2}: \u{2}31\u{1}3\u{1},\u{1} \n\n"
raw hex: [2, 55, 6e, 3, 73, 63, 61, 6c, 3, 65, 64, 4d, 65, 3, 61, 73, 75, 72, 3, 65, 6d, 65, 6e, 1, 74, 1, 20, 2, 7b, 20, 1, 78, 2, 3a, 20, 2, 33, 31, 1, 33, 1, 2c, 1, 20, a, a]
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }', src/main.rs:51:46
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
This is the link to itm thing.
Related
I'm trying to stream bytes to a tcp server by using io::copy(&mut reader, &mut writer), but it gives me this error: the trait "std::io::Read" is not implemented for "Vec<{integer}>". Here I have a vector of bytes which would be the same as me opening a file and converting it to bytes. I want to write the bytes to the BufWriter. What am I doing wrong?
use std::io;
use std::net::TcpStream;
use std::io::BufWriter;
pub fn connect() {
if let Ok(stream) = TcpStream::connect("localhost:8080") {
println!("Connection established!");
let mut reader = vec![
137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 70, 0, 0, 0, 70,
];
let mut writer = BufWriter::new(&stream);
io::copy(&mut reader, &mut writer).expect("Failed to write to stream");
} else {
println!("Couldn't connect to the server")
}
}
error[E0277]: the trait bound `Vec<{integer}>: std::io::Read` is not satisfied
--> src/lib.rs:12:31
|
12 | io::copy(&mut reader, &mut writer).expect("Failed to write to stream");
| -------- ^^^^^^^^^^^ the trait `std::io::Read` is not implemented for `Vec<{integer}>`
| |
| required by a bound introduced by this call
|
note: required by a bound in `std::io::copy`
the compiler have a little trouble here, Vec doesn't implement Read but &[u8] do, you just have a get a slice from the vec before create a mutable reference:
copy(&mut reader.as_slice(), &mut writer).expect("Failed to write to stream");
See also:
What is the difference between storing a Vec vs a Slice?
What are the differences between Rust's `String` and `str`?
Using .as_slice() like so works for me:
pub fn connect() {
if let Ok(stream) = TcpStream::connect("localhost:8080") {
println!("Connection established!");
let reader = vec![
137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 70, 0, 0, 0, 70,
];
let mut writer = BufWriter::new(&stream);
io::copy(&mut reader.as_slice(), &mut writer).expect("Failed to write to stream");
} else {
println!("Couldn't connect to the server")
}
}
That’s because std::io::Read supports slices.
fn main() {
let arr: [u8;8] = [97, 112, 112, 108, 101];
println!("Len is {}",arr.len());
println!("Elements are {:?}",arr);
}
error[E0308]: mismatched types
--> src/main.rs:2:23
|
2 | let arr: [u8;8] = [97, 112, 112, 108, 101];
| ------ ^^^^^^^^^^^^^^^^^^^^^^^^ expected an array with a fixed size of 8 elements, found one with 5 elements
| |
| expected due to this
Is there any way to pad the remaining elements with 0's? Something like:
let arr: [u8;8] = [97, 112, 112, 108, 101].something();
In addition to the other answers, you can use const generics to write a dedicated method.
fn pad_zeroes<const A: usize, const B: usize>(arr: [u8; A]) -> [u8; B] {
assert!(B >= A); //just for a nicer error message, adding #[track_caller] to the function may also be desirable
let mut b = [0; B];
b[..A].copy_from_slice(&arr);
b
}
Playground
You can use concat_arrays macro for it:
use concat_arrays::concat_arrays;
fn main() {
let arr: [u8; 8] = concat_arrays!([97, 112, 112, 108, 101], [0; 3]);
println!("{:?}", arr);
}
I don't think it's possible to do without external dependencies.
You could start with zeros and set the initial values afterwards. This requires arr to be mut though.
fn main() {
let mut arr: [u8;8] = [0;8];
let init = [97, 112, 112, 108, 101];
arr[..init.len()].copy_from_slice(&init);
println!("Len is {}",arr.len());
println!("Elements are {:?}",arr);
}
Link to Playground
I am trying to use serde together with bincode to de-serialize an arbitrary bitcoin network message. Given that the payload is handled ubiquitously as a byte array, how do I de-serialize it when the length is unknown at compile-time? bincode does by default handle Vec<u8> by assuming it's length is encoded as u64 right before the elements of the vector. However, this assumption does not hold here because the checksum comes after the length of the payload.
I have the following working solution
Cargo.toml
[package]
name = "serde-test"
version = "0.1.0"
edition = "2018"
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.11"
bincode = "1.3.3"
main.rs
use bincode::Options;
use serde::{Deserialize, Deserializer, de::{SeqAccess, Visitor}};
#[derive(Debug)]
struct Message {
// https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure
magic: u32,
command: [u8; 12],
length: u32,
checksum: u32,
payload: Vec<u8>,
}
struct MessageVisitor;
impl<'de> Visitor<'de> for MessageVisitor {
type Value = Message;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("Message")
}
fn visit_seq<V>(self, mut seq: V) -> Result<Self::Value, V::Error> where V: SeqAccess<'de>,
{
let magic = seq.next_element()?.unwrap();
let command = seq.next_element()?.unwrap();
let length: u32 = seq.next_element()?.unwrap();
let checksum = seq.next_element()?.unwrap();
let payload = (0..length).map(|_| seq.next_element::<u8>().unwrap().unwrap()).collect();
// verify payload checksum (omitted for brevity)
Ok(Message {magic, command, length, checksum, payload})
}
}
impl<'de> Deserialize<'de> for Message {
fn deserialize<D>(deserializer: D) -> Result<Message, D::Error> where D: Deserializer<'de>,
{
deserializer.deserialize_tuple(5000, MessageVisitor) // <-- overallocation
}
}
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let msg: Message = bincode::DefaultOptions::new().with_fixint_encoding().deserialize(bytes).unwrap();
println!("{:?}", msg);
}
Output:
Message { magic: 3652501241, command: [118, 101, 114, 115, 105, 111, 110, 0, 0, 0, 0, 0], length: 101, checksum: 3530103391, payload: [114, 17, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 188, 143, 94, 84, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 198, 27, 100, 9, 32, 141, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 203, 0, 113, 192, 32, 141, 18, 128, 53, 203, 201, 121, 83, 248, 15, 47, 83, 97, 116, 111, 115, 104, 105, 58, 48, 46, 57, 46, 51, 47, 207, 5, 5, 0, 1] }
I dislike this solution because of how payload is handled. It requires me to allocate some "large enough" buffer to take into account the dynamic size of the payload, In the code snippet above 5000 is sufficient. I would much rather de-serialize payload as a single element and use deserializer.deserialize_tuple(5, MessageVisitor) instead.
Is there a way to handle this kind of deserialization in a succint manner?
Similar question I could find: Can I deserialize vectors with variable length prefix with Bincode?
Your problem is that the source message is not encoded as bincode, so you are doing weird things to treat non-bincode data as if it was.
Serde is designed for creating serializers and deserializers for general-purpose formats, but your message is in a very specific format that can only be interpreted one way.
A library like nom is much more suitable for this kind of work, but it may be overkill considering how simple the format is and you can just parse it from the bytes directly:
use std::convert::TryInto;
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let (magic_bytes, bytes) = bytes.split_at(4);
let magic = u32::from_le_bytes(magic_bytes.try_into().unwrap());
let (command_bytes, bytes) = bytes.split_at(12);
let command = command_bytes.try_into().unwrap();
let (length_bytes, bytes) = bytes.split_at(4);
let length = u32::from_le_bytes(length_bytes.try_into().unwrap());
let (checksum_bytes, bytes) = bytes.split_at(4);
let checksum = u32::from_le_bytes(checksum_bytes.try_into().unwrap());
let payload = bytes[..length as usize].to_vec();
let msg = Message {
magic,
command,
length,
checksum,
payload,
};
println!("{:?}", msg);
}
There are hundreds of cryptocurrency projects in Rust and there are many crates already written for handling cryptocurrency data structures. These crates are battle-tested and will have much better error-handling (my example above has none). As mentioned in the comments, you can perhaps look at the bitcoin crate.
I'm trying to make a function in Rust that will return a HMAC-SHA256 digest. I've been working from the description at Wikipedia and RFC 2104.
I've been struggling with returning the correct HMAC. I'm using ring for the SHA256 digests but no matter what I try, I can't seem to get the right result. I suspect it might have something to do with .as_ref().to_vec() conversions. Even if that's true, I don't know how to continue from that. Not everything from RFC 2104 is implemented in the following code, but it highlights my issue.
extern crate ring;
use ring::{digest, test};
pub fn hmac(k: Vec<u8>, mut m: Vec<u8>) -> Vec<u8> {
// Initialize ipad and opad as byte vectors with SHA256 blocksize
let ipad = vec![0x5C; 64];
let opad = vec![0x36; 64];
// iround and oround are used to seperate the two steps with XORing
let mut iround = vec![];
let mut oround = vec![];
for count in 0..k.len() {
iround.push(k[count] ^ ipad[count]);
oround.push(k[count] ^ opad[count]);
}
iround.append(&mut m); // m is emptied here
iround = (digest::digest(&digest::SHA256, &iround).as_ref()).to_vec();
oround.append(&mut iround); // iround is emptied here
oround = (digest::digest(&digest::SHA256, &oround).as_ref()).to_vec();
let hashed_mac = oround.to_vec();
return hashed_mac;
}
#[test]
fn test_hmac_digest() {
let k = vec![0x61; 64];
let m = vec![0x62; 64];
let actual = hmac(k, m);
// Expected value taken from: https://www.freeformatter.com/hmac-generator.html#ad-output
let expected = test::from_hex("f6cbb37b326d36f2f27d294ac3bb46a6aac29c1c9936b985576041bfb338ae70").unwrap();
assert_eq!(actual, expected);
}
These are the digests:
Actual = [139, 141, 144, 52, 11, 3, 48, 112, 117, 7, 56, 151, 163, 65, 152, 195, 163, 164, 26, 250, 178, 100, 187, 230, 89, 61, 191, 164, 146, 228, 180, 62]
Expected = [246, 203, 179, 123, 50, 109, 54, 242, 242, 125, 41, 74, 195, 187, 70, 166, 170, 194, 156, 28, 153, 54, 185, 133, 87, 96, 65, 191, 179, 56, 174, 112]
As mentioned in a comment, you have swapped the bytes for the inner and outer padding. Refer back to the Wikipedia page:
o_key_pad = key xor [0x5c * blockSize] //Outer padded key
i_key_pad = key xor [0x36 * blockSize] //Inner padded key
Here's what my take on the function would look like. I believe it has less allocation:
extern crate ring;
use ring::{digest, test};
const BLOCK_SIZE: usize = 64;
pub fn hmac(k: &[u8], m: &[u8]) -> Vec<u8> {
assert_eq!(k.len(), BLOCK_SIZE);
let mut i_key_pad: Vec<_> = k.iter().map(|&k| k ^ 0x36).collect();
let mut o_key_pad: Vec<_> = k.iter().map(|&k| k ^ 0x5C).collect();
i_key_pad.extend_from_slice(m);
let hash = |v| digest::digest(&digest::SHA256, v);
let a = hash(&i_key_pad);
o_key_pad.extend_from_slice(a.as_ref());
hash(&o_key_pad).as_ref().to_vec()
}
#[test]
fn test_hmac_digest() {
let k = [0x61; BLOCK_SIZE];
let m = [0x62; BLOCK_SIZE];
let actual = hmac(&k, &m);
// Expected value taken from: https://www.freeformatter.com/hmac-generator.html#ad-output
let expected = test::from_hex("f6cbb37b326d36f2f27d294ac3bb46a6aac29c1c9936b985576041bfb338ae70").unwrap();
assert_eq!(actual, expected);
}
I have some file content which delimited by pipe | symbol. Named, important.txt.
1|130|80|120|110|E
2|290|420|90|70|B
3|100|220|30|80|C
Then, I use Rust BufReader::split to read its content.
use std::error::Error;
use std::fs::File;
use std::io::BufReader;
use std::io::Prelude::*;
use std::path::Path;
fn main() {
let path = Path::new("important.txt");
let display = path.display();
//Open read-only
let file = match File::open(&path) {
Err(why) => panic!("can't open {}: {}", display,
Error::description(why)),
Ok(file) => file,
}
//Read each line
let reader = BufReader::new(&file);
for vars in reader.split(b'|') {
println!("{:?}\n", vars.unwrap());
}
}
The problem is, vars.unwrap() would return bytes instead of string.
[49]
[49, 51, 48]
[56, 48]
[49, 50, 48]
[49, 49, 48]
[69, 10, 50]
[50, 57, 48]
[52, 50, 48]
[57, 48]
[55, 48]
[66, 10, 51]
[49, 48, 48]
[50, 50, 48]
[51, 48]
[56, 48]
[67, 10]
Do you have any ideas how to parse this delimited file into variable in Rust?
Since your data is line-based, you can use BufRead::lines:
use std::io::{BufReader, BufRead};
fn main() {
let input = r#"1|130|80|120|110|E
2|290|420|90|70|B
3|100|220|30|80|C
"#;
let reader = BufReader::new(input.as_bytes());
for line in reader.lines() {
for value in line.unwrap().split('|') {
println!("{}", value);
}
}
}
This gives you an iterator over Strings for each line in the input. You then use str::split to get the pieces.
Alternatively, you can take the &[u8] you already have and make a string from it with str::from_utf8:
use std::io::{BufReader, BufRead};
use std::str;
fn main() {
let input = r#"1|130|80|120|110|E
2|290|420|90|70|B
3|100|220|30|80|C
"#;
let reader = BufReader::new(input.as_bytes());
for vars in reader.split(b'|') {
let bytes = vars.unwrap();
let s = str::from_utf8(&bytes).unwrap();
println!("{}", s);
}
}
You may also want to look into the csv crate if you are reading structured data like a CSV that just happens to be pipe-delimited.