I have a small tcp client and server, they run on each their device.
The client looks like this:
fn handle_connection(mut stream: TcpStream)-> Result<(), Error> {
for i in 1..1000 {
thread::sleep(Duration::from_millis(2000));
let uplink = encrpyt(&[1;33], &i_c_i_cpy);
stream.write_all(&uplink);(&uplink);
println!("uplinklen {}",uplink.len());
}
fn main() {
match TcpStream::connect("192.168.1.227:8888") {
Ok(mut stream) => {
println!("Successfully connected to server");
handle_connection(stream);
}
Err(e) => {
println!("Failed to connect: {}", e);
}
};
}
The encryption function just returns a vector, that is encrypted and encoded. So It's just a vector.
THe server looks like this:
fn handle_connection(mut stream: TcpStream)-> Result<(), Error> {
loop {
println!("start reading input");
let mut buf = [0;46];
stream.read_exact(&mut buf)?;
let incoming = &buf;
println!("getting {:?}", incoming);
}
fn main() {
// static key material of r (stored in efuse)
let listener = TcpListener::bind("192.168.1.227:8888").unwrap();
for stream in listener.incoming() {
let stream = stream.unwrap();
handle_connection(stream);
}
}
and this is just the client sending a bunch of messages to the server.
THe encryption function always outputs a start of 7,0, but if I run these two for a while, the after about a minute or so, I start to see these inputs at the server:
getting [43, 196, 203, 62, 25, 251, 78, 211, 7, 0, 24, 32, 88, 41, 59, 70, 187, 214, 222, 9, 217, 240, 4, 174, 221, 222, 77, 34, 199, 32, 160, 105, 199, 249, 108, 2, 29, 211, 83, 110, 85, 161, 69, 160, 9, 45]
Where I can see that the message has been moved a bit, and prepended with a part of another message.
So, the server is getting the messages out of order. I actually thought I had this fixed by sending fixed length messages, and always reading into a bufffer of that size.
When looking around a common solution for this type of problem is to use BufReader, but all the bufreader examples I can find send pure string messages, and have messages where the input doesnt not include EOF's, and the vectors I need to send, needs to be able to contain all type of ´u8´'s.
How can I send messages over tcp, without them getting mixed together?
It is worth noting that this problem is especially bad when I run the client on my esp32 wroom
Related
I am trying to convert a sha1 hash info_hash into a string format that can be used in an BitTorrent tracker announce URL. I know how to get the correct url, but I am unsatisfied with the use of "unsafe" code and I do not understand why println!("{}", h) shows ��9��(�l�[�����ۨζ and not %D8%F79%CE%C3%28%95l%CC%5B%BF%1F%86%D9%FD%CF%DB%A8%CE%B6.
use url::Url;
fn main() {
let h: [u8; 20] = [216, 247, 57, 206, 195, 40, 149, 108, 204, 91, 191, 31, 134, 217, 253, 207, 219, 168, 206, 182];
let h: &str = unsafe {
std::str::from_utf8_unchecked(&h)
};
println!("{}", h); // this shows ��9��(�l�[�����ۨζ
let url = Url::parse_with_params("http://bttracker.org:6969/test",
&[
("info_hash", h),
]).unwrap().to_string();
println!("{}", url); // this shows the correct string format of info_hash
// this causes panic (?)
//assert_eq!(h, "%D8%F79%CE%C3%28%95l%CC%5B%BF%1F%86%D9%FD%CF%DB%A8%CE%B6");
}
output:
��9��(�l�[�����ۨζ
http://bttracker.org:6969/test?info_hash=%D8%F79%CE%C3%28%95l%CC%5B%BF%1F%86%D9%FD%CF%DB%A8%CE%B6
playground link
How do I convert the 20-byte array h: [u8; 20] into the string "%D8%F79%CE%C3%28%95l%CC%5B%BF%1F%86%D9%FD%CF%DB%A8%CE%B6" without using unsafe?
There are many ways to encode binary data into a "readable" format. What you see is called "percent encoding" or "URL encoding" wikipedia which writes invalid characters as %XX. The url crate uses the percent-encoding crate under the hood:
use percent_encoding::{NON_ALPHANUMERIC, percent_encode}; // 2.1.0
fn main() {
let h: [u8; 20] = [216, 247, 57, 206, 195, 40, 149, 108, 204, 91, 191, 31, 134, 217, 253, 207, 219, 168, 206, 182];
println!("{}", percent_encode(&h, NON_ALPHANUMERIC));
}
%D8%F79%CE%C3%28%95l%CC%5B%BF%1F%86%D9%FD%CF%DB%A8%CE%B6
I am trying to use serde together with bincode to de-serialize an arbitrary bitcoin network message. Given that the payload is handled ubiquitously as a byte array, how do I de-serialize it when the length is unknown at compile-time? bincode does by default handle Vec<u8> by assuming it's length is encoded as u64 right before the elements of the vector. However, this assumption does not hold here because the checksum comes after the length of the payload.
I have the following working solution
Cargo.toml
[package]
name = "serde-test"
version = "0.1.0"
edition = "2018"
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.11"
bincode = "1.3.3"
main.rs
use bincode::Options;
use serde::{Deserialize, Deserializer, de::{SeqAccess, Visitor}};
#[derive(Debug)]
struct Message {
// https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure
magic: u32,
command: [u8; 12],
length: u32,
checksum: u32,
payload: Vec<u8>,
}
struct MessageVisitor;
impl<'de> Visitor<'de> for MessageVisitor {
type Value = Message;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("Message")
}
fn visit_seq<V>(self, mut seq: V) -> Result<Self::Value, V::Error> where V: SeqAccess<'de>,
{
let magic = seq.next_element()?.unwrap();
let command = seq.next_element()?.unwrap();
let length: u32 = seq.next_element()?.unwrap();
let checksum = seq.next_element()?.unwrap();
let payload = (0..length).map(|_| seq.next_element::<u8>().unwrap().unwrap()).collect();
// verify payload checksum (omitted for brevity)
Ok(Message {magic, command, length, checksum, payload})
}
}
impl<'de> Deserialize<'de> for Message {
fn deserialize<D>(deserializer: D) -> Result<Message, D::Error> where D: Deserializer<'de>,
{
deserializer.deserialize_tuple(5000, MessageVisitor) // <-- overallocation
}
}
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let msg: Message = bincode::DefaultOptions::new().with_fixint_encoding().deserialize(bytes).unwrap();
println!("{:?}", msg);
}
Output:
Message { magic: 3652501241, command: [118, 101, 114, 115, 105, 111, 110, 0, 0, 0, 0, 0], length: 101, checksum: 3530103391, payload: [114, 17, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 188, 143, 94, 84, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 198, 27, 100, 9, 32, 141, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 203, 0, 113, 192, 32, 141, 18, 128, 53, 203, 201, 121, 83, 248, 15, 47, 83, 97, 116, 111, 115, 104, 105, 58, 48, 46, 57, 46, 51, 47, 207, 5, 5, 0, 1] }
I dislike this solution because of how payload is handled. It requires me to allocate some "large enough" buffer to take into account the dynamic size of the payload, In the code snippet above 5000 is sufficient. I would much rather de-serialize payload as a single element and use deserializer.deserialize_tuple(5, MessageVisitor) instead.
Is there a way to handle this kind of deserialization in a succint manner?
Similar question I could find: Can I deserialize vectors with variable length prefix with Bincode?
Your problem is that the source message is not encoded as bincode, so you are doing weird things to treat non-bincode data as if it was.
Serde is designed for creating serializers and deserializers for general-purpose formats, but your message is in a very specific format that can only be interpreted one way.
A library like nom is much more suitable for this kind of work, but it may be overkill considering how simple the format is and you can just parse it from the bytes directly:
use std::convert::TryInto;
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let (magic_bytes, bytes) = bytes.split_at(4);
let magic = u32::from_le_bytes(magic_bytes.try_into().unwrap());
let (command_bytes, bytes) = bytes.split_at(12);
let command = command_bytes.try_into().unwrap();
let (length_bytes, bytes) = bytes.split_at(4);
let length = u32::from_le_bytes(length_bytes.try_into().unwrap());
let (checksum_bytes, bytes) = bytes.split_at(4);
let checksum = u32::from_le_bytes(checksum_bytes.try_into().unwrap());
let payload = bytes[..length as usize].to_vec();
let msg = Message {
magic,
command,
length,
checksum,
payload,
};
println!("{:?}", msg);
}
There are hundreds of cryptocurrency projects in Rust and there are many crates already written for handling cryptocurrency data structures. These crates are battle-tested and will have much better error-handling (my example above has none). As mentioned in the comments, you can perhaps look at the bitcoin crate.
This question already has answers here:
Show u8 slice in hex representation
(5 answers)
Closed 2 years ago.
How to print the hash as string?
use sha3::{Digest, Keccak256};
fn main() {
let vote = "Alice".to_owned();
let mut hasher = Keccak256::new();
hasher.update(vote.as_bytes());
let result = hasher.finalize();
let hashstring = result.to_owned();
println!("{:?}", hashstring);
// [129, 55, 107, 152, 104, 178, 146, 164, 106, 28, 72, 109, 52, 78, 66, 122, 48, 136, 101, 127, 218, 98, 155, 95, 74, 100, 120, 34, 211, 41, 205, 106]
// I need the hex string, instead of numbers
}
sha3 docs
As per Andrew Morton, here is my answer:
use sha3::{Digest, Keccak256};
use hex;
fn main() {
let vote = "Alice".to_owned();
let mut hasher = Keccak256::new();
hasher.update(vote.as_bytes());
let result = hasher.finalize();
println!("{:?}", hex::encode(result));
// "81376b9868b292a46a1c486d344e427a3088657fda629b5f4a647822d329cd6a"
}
You don't need any additional crates. Just format it as either upper- or lower-case hex:
use sha3::{Digest, Keccak256};
fn main() {
let vote = "Alice".to_owned();
let mut hasher = Keccak256::new();
hasher.update(vote.as_bytes());
let result = hasher.finalize();
println!("{:x}", result); // 81376b9868b292a46a1c486d344e427a3088657fda629b5f4a647822d329cd6a
println!("{:X}", result); // 81376B9868B292A46A1C486D344E427A3088657FDA629B5F4A647822D329CD6A
}
I'm trying to make a function in Rust that will return a HMAC-SHA256 digest. I've been working from the description at Wikipedia and RFC 2104.
I've been struggling with returning the correct HMAC. I'm using ring for the SHA256 digests but no matter what I try, I can't seem to get the right result. I suspect it might have something to do with .as_ref().to_vec() conversions. Even if that's true, I don't know how to continue from that. Not everything from RFC 2104 is implemented in the following code, but it highlights my issue.
extern crate ring;
use ring::{digest, test};
pub fn hmac(k: Vec<u8>, mut m: Vec<u8>) -> Vec<u8> {
// Initialize ipad and opad as byte vectors with SHA256 blocksize
let ipad = vec![0x5C; 64];
let opad = vec![0x36; 64];
// iround and oround are used to seperate the two steps with XORing
let mut iround = vec![];
let mut oround = vec![];
for count in 0..k.len() {
iround.push(k[count] ^ ipad[count]);
oround.push(k[count] ^ opad[count]);
}
iround.append(&mut m); // m is emptied here
iround = (digest::digest(&digest::SHA256, &iround).as_ref()).to_vec();
oround.append(&mut iround); // iround is emptied here
oround = (digest::digest(&digest::SHA256, &oround).as_ref()).to_vec();
let hashed_mac = oround.to_vec();
return hashed_mac;
}
#[test]
fn test_hmac_digest() {
let k = vec![0x61; 64];
let m = vec![0x62; 64];
let actual = hmac(k, m);
// Expected value taken from: https://www.freeformatter.com/hmac-generator.html#ad-output
let expected = test::from_hex("f6cbb37b326d36f2f27d294ac3bb46a6aac29c1c9936b985576041bfb338ae70").unwrap();
assert_eq!(actual, expected);
}
These are the digests:
Actual = [139, 141, 144, 52, 11, 3, 48, 112, 117, 7, 56, 151, 163, 65, 152, 195, 163, 164, 26, 250, 178, 100, 187, 230, 89, 61, 191, 164, 146, 228, 180, 62]
Expected = [246, 203, 179, 123, 50, 109, 54, 242, 242, 125, 41, 74, 195, 187, 70, 166, 170, 194, 156, 28, 153, 54, 185, 133, 87, 96, 65, 191, 179, 56, 174, 112]
As mentioned in a comment, you have swapped the bytes for the inner and outer padding. Refer back to the Wikipedia page:
o_key_pad = key xor [0x5c * blockSize] //Outer padded key
i_key_pad = key xor [0x36 * blockSize] //Inner padded key
Here's what my take on the function would look like. I believe it has less allocation:
extern crate ring;
use ring::{digest, test};
const BLOCK_SIZE: usize = 64;
pub fn hmac(k: &[u8], m: &[u8]) -> Vec<u8> {
assert_eq!(k.len(), BLOCK_SIZE);
let mut i_key_pad: Vec<_> = k.iter().map(|&k| k ^ 0x36).collect();
let mut o_key_pad: Vec<_> = k.iter().map(|&k| k ^ 0x5C).collect();
i_key_pad.extend_from_slice(m);
let hash = |v| digest::digest(&digest::SHA256, v);
let a = hash(&i_key_pad);
o_key_pad.extend_from_slice(a.as_ref());
hash(&o_key_pad).as_ref().to_vec()
}
#[test]
fn test_hmac_digest() {
let k = [0x61; BLOCK_SIZE];
let m = [0x62; BLOCK_SIZE];
let actual = hmac(&k, &m);
// Expected value taken from: https://www.freeformatter.com/hmac-generator.html#ad-output
let expected = test::from_hex("f6cbb37b326d36f2f27d294ac3bb46a6aac29c1c9936b985576041bfb338ae70").unwrap();
assert_eq!(actual, expected);
}
Here's my code:
let mut altbuf: Vec<u8> = Vec::new();
// Stuff here...
match stream.read_byte() {
Ok(d) => altbuf.push(d),
Err(e) => { println!("Error: {}", e); doneflag = true; }
}
for x in altbuf.iter() {
println!("{}", x);
}
The code prints u8 bytes which are correct, but I can't figure out for the life of me how to convert a vector of pure u8 bytes into a string? The only other answer to a similar question on stack overflow assumes that you're working with a vector of type &[u8].
If you look at the String documentation, there are a few methods you could use. There's String::from_utf8 that takes a Vec<u8>, and there's also String::from_utf8_lossy which takes a &[u8].
Note that a Vec<T> is more-or-less an owned, resizable wrapper around a [T]. That is, if you have a Vec<u8>, you can turn it into a &[u8], most easily by re-borrowing it (i.e. &*some_vec). You can also call any methods defined on &[T] directly on a Vec<T> (in general, this is true of things that implement the Deref trait).
If your altbuf is a vector of u8 as shown, this should work:
println!("{:?}", altbuf);
Here is an edited piece of code I have that does something similar:
let rebuilt: Vec<u8>;
unsafe {
ret = proc_pidpath(pid, buffer_ptr, buffer_size);
rebuilt = Vec::from_raw_parts(buffer_ptr as *mut u8, ret as usize, buffer_size as usize);
};
println!("Returned a {} byte string", ret);
println!("{:?}", rebuilt);
That rebuilds a vector of u8 values from a buffer filled by a C function called via FFI so the bytes could be anything, maybe not valid UTF-8.
When I run it, the output is:
Returned a 49 byte string
[47, 85, 115, 101, 114, 115, 47, 97, 110,
100, 114, 101, 119, 47, 46, 114, 98, 101, 110, 118, 47, 118, 101, 114,
115, 105, 111, 110, 115, 47, 49, 46, 57, 46, 51, 45, 112, 51, 57, 50,
47, 98, 105, 110, 47, 114, 117, 98, 121]
You could format the numbers printed (in hex, octal, etc) using different format strings inside the {}.
You can get a String from that using String::from_utf8(rebuilt) - which may return an error.
match String::from_utf8(rebuilt) {
Ok(path) => Ok(path),
Err(e) => Err(format!("Invalid UTF-8 sequence: {}", e)),
}
To print bytes as a UTF-8 string, use std::str::from_utf8 when the bytes may be malformed. Use the unsafe std::str::from_utf8_unchecked when the bytes are always valid UTF-8.
println!("{}", std::str::from_utf8(&altbuf).unwrap());
Use the write method from std::io:
use std::{io, io::Write};
fn main() -> io::Result<()> {
io::stdout().write(b"March\n")?;
Ok(())
}
It prints a slice of u8, also known as a bytestring.
io::stdout