When I send DNS query to root server for com name servers over UDP, it's working just fine, but the message is truncated. When I retry over TCP, the Root server does not times out.
use std::net::TcpStream;
use std::io::Write;
fn main() {
let mut stream = TcpStream::connect("199.9.14.201:53")
.unwrap();
stream.write(&[251, 188, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 3, 99, 111, 109, 0, 0, 2, 0, 1])
.expect("ugh");
let mut buf: [u8; 1000] = [1; 1000];
match stream.peek(&mut buf) {
Ok(..) => {
println!("{:?}", buf);
}
Err(e) => {
println!("{:?}", e);
}
}
println!("Hello, world!");
}
From RFC 1035 section 4.2.2:
Messages sent over TCP connections use server port 53 (decimal). The
message is prefixed with a two byte length field which gives the message length, excluding the two byte length field. This length
field allows the low-level processing to assemble a complete message
before beginning to parse it.
I looks like your code does not include the required message length. Thus the first two bytes of your message are treated as length which describe a length much larger than what you has send. Therefore the server is waiting for more data.
Related
I'm trying to stream bytes to a tcp server by using io::copy(&mut reader, &mut writer), but it gives me this error: the trait "std::io::Read" is not implemented for "Vec<{integer}>". Here I have a vector of bytes which would be the same as me opening a file and converting it to bytes. I want to write the bytes to the BufWriter. What am I doing wrong?
use std::io;
use std::net::TcpStream;
use std::io::BufWriter;
pub fn connect() {
if let Ok(stream) = TcpStream::connect("localhost:8080") {
println!("Connection established!");
let mut reader = vec![
137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 70, 0, 0, 0, 70,
];
let mut writer = BufWriter::new(&stream);
io::copy(&mut reader, &mut writer).expect("Failed to write to stream");
} else {
println!("Couldn't connect to the server")
}
}
error[E0277]: the trait bound `Vec<{integer}>: std::io::Read` is not satisfied
--> src/lib.rs:12:31
|
12 | io::copy(&mut reader, &mut writer).expect("Failed to write to stream");
| -------- ^^^^^^^^^^^ the trait `std::io::Read` is not implemented for `Vec<{integer}>`
| |
| required by a bound introduced by this call
|
note: required by a bound in `std::io::copy`
the compiler have a little trouble here, Vec doesn't implement Read but &[u8] do, you just have a get a slice from the vec before create a mutable reference:
copy(&mut reader.as_slice(), &mut writer).expect("Failed to write to stream");
See also:
What is the difference between storing a Vec vs a Slice?
What are the differences between Rust's `String` and `str`?
Using .as_slice() like so works for me:
pub fn connect() {
if let Ok(stream) = TcpStream::connect("localhost:8080") {
println!("Connection established!");
let reader = vec![
137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 70, 0, 0, 0, 70,
];
let mut writer = BufWriter::new(&stream);
io::copy(&mut reader.as_slice(), &mut writer).expect("Failed to write to stream");
} else {
println!("Couldn't connect to the server")
}
}
That’s because std::io::Read supports slices.
I need to implement a long-running program that receives messages via stdin. The protocol defines that messages are in form of length indicator (for simplicity 1 byte integer) and then string of a length represented by length indicator. Messages are NOT separated by any whitespace.
The program is expected to consume all messages from stdin and wait for another messages.
How do I implement such waiting on stdin?
I implemented the iterator in a way that it tries to read from stdin and repeats in case of error. It works, but it is very inefficient.
I would like the iterator to read the message when new data comes.
My implementation is using read_exact:
use std::io::{Read, stdin, Error as IOError, ErrorKind};
pub struct In<R>(R) where R: Read;
pub trait InStream{
fn read_one(&mut self) -> Result<String, IOError>;
}
impl <R>In<R> where R: Read{
pub fn new(stdin: R) -> In<R> {
In(stdin)
}
}
impl <R>InStream for In<R> where R: Read{
/// Read one message from stdin and return it as string
fn read_one(&mut self) -> Result<String, IOError>{
const length_indicator: usize = 1;
let stdin = &mut self.0;
let mut size: [u8;length_indicator] = [0; length_indicator];
stdin.read_exact(&mut size)?;
let size = u8::from_be_bytes(size) as usize;
let mut buffer = vec![0u8; size];
let _bytes_read = stdin.read_exact(&mut buffer);
String::from_utf8(buffer).map_err(|_| IOError::new(ErrorKind::InvalidData, "not utf8"))
}
}
impl <R>Iterator for In<R> where R:Read{
type Item = String;
fn next(&mut self) -> Option<String>{
self.read_one()
.ok()
}
}
fn main(){
let mut in_stream = In::new(stdin());
loop{
match in_stream.next(){
Some(x) => println!("x: {:?}", x),
None => (),
}
}
}
I went trough Read and BufReader documentation, but none method seems to solve my problem as read doc contains following text:
This function does not provide any guarantees about whether it blocks waiting for data, but if an object needs to block for a read and cannot, it will typically signal this via an Err return value.
How do I implement waiting for data on stdin?
===
Edit: minimum use-case that does not block and loops giving UnexpectedEof error instead of waiting for data:
use std::io::{Read, stdin};
fn main(){
let mut stdin = stdin();
let mut stdin_handle = stdin.lock();
loop{
let mut buffer = vec![0u8; 4];
let res = stdin_handle.read_exact(&mut buffer);
println!("res: {:?}", res);
println!("buffer: {:?}", buffer);
}
I run it on OSX by cargo run < in where in is named pipe. I fill the pipe by echo -n "1234" > in.
It waits for the first input and then it loops.
res: Ok(())
buffer: [49, 50, 51, 52]
res: Err(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
buffer: [0, 0, 0, 0]
res: Err(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
buffer: [0, 0, 0, 0]
res: Err(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
buffer: [0, 0, 0, 0]
res: Err(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
buffer: [0, 0, 0, 0]
res: Err(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
...
I would like the program to wait until there is sufficient data to fill the buffer.
As others explained, the docs on Read are written very generally and don't apply to standard input, which is blocking. In other words, your code with the buffering added is fine.
The problem is how you use the pipe. For example, if you run mkfifo foo; cat <foo in one shell, and echo -n bla >foo in another, you'll see that the cat in the first shell will display foo and exit. That closing the last writer of the pipe sends EOF to the reader, rendering your program's stdin useless.
You can work around the issue by starting another program in the background that opens the pipe in write mode and never exits, for example tail -f /dev/null >pipe-filename. Then echo -n bla >foo will be observed by your program, but won't cause its stdin to close. The "holding" of the write end of the pipe could probably also be achieved from Rust as well.
I am trying to use serde together with bincode to de-serialize an arbitrary bitcoin network message. Given that the payload is handled ubiquitously as a byte array, how do I de-serialize it when the length is unknown at compile-time? bincode does by default handle Vec<u8> by assuming it's length is encoded as u64 right before the elements of the vector. However, this assumption does not hold here because the checksum comes after the length of the payload.
I have the following working solution
Cargo.toml
[package]
name = "serde-test"
version = "0.1.0"
edition = "2018"
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_bytes = "0.11"
bincode = "1.3.3"
main.rs
use bincode::Options;
use serde::{Deserialize, Deserializer, de::{SeqAccess, Visitor}};
#[derive(Debug)]
struct Message {
// https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure
magic: u32,
command: [u8; 12],
length: u32,
checksum: u32,
payload: Vec<u8>,
}
struct MessageVisitor;
impl<'de> Visitor<'de> for MessageVisitor {
type Value = Message;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("Message")
}
fn visit_seq<V>(self, mut seq: V) -> Result<Self::Value, V::Error> where V: SeqAccess<'de>,
{
let magic = seq.next_element()?.unwrap();
let command = seq.next_element()?.unwrap();
let length: u32 = seq.next_element()?.unwrap();
let checksum = seq.next_element()?.unwrap();
let payload = (0..length).map(|_| seq.next_element::<u8>().unwrap().unwrap()).collect();
// verify payload checksum (omitted for brevity)
Ok(Message {magic, command, length, checksum, payload})
}
}
impl<'de> Deserialize<'de> for Message {
fn deserialize<D>(deserializer: D) -> Result<Message, D::Error> where D: Deserializer<'de>,
{
deserializer.deserialize_tuple(5000, MessageVisitor) // <-- overallocation
}
}
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let msg: Message = bincode::DefaultOptions::new().with_fixint_encoding().deserialize(bytes).unwrap();
println!("{:?}", msg);
}
Output:
Message { magic: 3652501241, command: [118, 101, 114, 115, 105, 111, 110, 0, 0, 0, 0, 0], length: 101, checksum: 3530103391, payload: [114, 17, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 188, 143, 94, 84, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 198, 27, 100, 9, 32, 141, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 203, 0, 113, 192, 32, 141, 18, 128, 53, 203, 201, 121, 83, 248, 15, 47, 83, 97, 116, 111, 115, 104, 105, 58, 48, 46, 57, 46, 51, 47, 207, 5, 5, 0, 1] }
I dislike this solution because of how payload is handled. It requires me to allocate some "large enough" buffer to take into account the dynamic size of the payload, In the code snippet above 5000 is sufficient. I would much rather de-serialize payload as a single element and use deserializer.deserialize_tuple(5, MessageVisitor) instead.
Is there a way to handle this kind of deserialization in a succint manner?
Similar question I could find: Can I deserialize vectors with variable length prefix with Bincode?
Your problem is that the source message is not encoded as bincode, so you are doing weird things to treat non-bincode data as if it was.
Serde is designed for creating serializers and deserializers for general-purpose formats, but your message is in a very specific format that can only be interpreted one way.
A library like nom is much more suitable for this kind of work, but it may be overkill considering how simple the format is and you can just parse it from the bytes directly:
use std::convert::TryInto;
fn main() {
let bytes = b"\xf9\xbe\xb4\xd9version\x00\x00\x00\x00\x00e\x00\x00\x00_\x1ai\xd2r\x11\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\xbc\x8f^T\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xc6\x1bd\t \x8d\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xcb\x00q\xc0 \x8d\x12\x805\xcb\xc9yS\xf8\x0f/Satoshi:0.9.3/\xcf\x05\x05\x00\x01";
let (magic_bytes, bytes) = bytes.split_at(4);
let magic = u32::from_le_bytes(magic_bytes.try_into().unwrap());
let (command_bytes, bytes) = bytes.split_at(12);
let command = command_bytes.try_into().unwrap();
let (length_bytes, bytes) = bytes.split_at(4);
let length = u32::from_le_bytes(length_bytes.try_into().unwrap());
let (checksum_bytes, bytes) = bytes.split_at(4);
let checksum = u32::from_le_bytes(checksum_bytes.try_into().unwrap());
let payload = bytes[..length as usize].to_vec();
let msg = Message {
magic,
command,
length,
checksum,
payload,
};
println!("{:?}", msg);
}
There are hundreds of cryptocurrency projects in Rust and there are many crates already written for handling cryptocurrency data structures. These crates are battle-tested and will have much better error-handling (my example above has none). As mentioned in the comments, you can perhaps look at the bitcoin crate.
I'm trying to receive a single data packet using tokio:
extern crate tokio;
extern crate tokio_io;
use tokio::net::{TcpListener};
use tokio::prelude::*;
use std::net::SocketAddr;
fn main() {
let addr = "0.0.0.0:8080".parse::<SocketAddr>().unwrap();
let socket = TcpListener::bind(&addr).unwrap();
println!("Listening on: {}", addr);
let done = socket
.incoming()
.map_err(|e| println!("failed to accept socket; error = {:?}", e))
.for_each(move |mut socket| {
let mut bytes = vec![];
bytes.reserve(1024);
let processor = socket.read_buf(&mut bytes).into_future()
.and_then(move |_size| {
println!("bytes: {:?}", bytes);
Ok(())
})
.map_err(|_| ());;
tokio::spawn(processor)
});
tokio::run(done);
}
This code prints an empty packet. How do I change this code to print the received packet with data?
For myself, I almost found the answer. Very helpful Similar question.
struct AsWeGetIt<R>(R);
impl<R> Stream for AsWeGetIt<R>
where
R: AsyncRead,
{
type Item = BytesMut;
type Error = std::io::Error;
fn poll(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
let mut buf = BytesMut::with_capacity(1000);
self.0
.read_buf(&mut buf)
.map(|async| async.map(|_| Some(buf)))
}
}
....
let processor = AsWeGetIt(socket).into_future()
.and_then(|(bytes,_)| {
println!("bytes: {:?}", bytes);
Ok(())
}).map_err(|_| ());
But for a better understanding how to do without a separate structure ...
And why and what is the map using?
If your goal is really to receive one packet, I think you've succeeded!
I've tested the program a few times, and I get a response. I'm testing with:
nc 127.0.0.1 8080 <<< hello
After running that a few times, I get the following output:
Listening on: 0.0.0.0:8080
bytes: [104, 101, 108, 108, 111, 10]
bytes: []
bytes: [104, 101, 108, 108, 111, 10]
bytes: []
bytes: [104, 101, 108, 108, 111, 10]
bytes: []
bytes: [104, 101, 108, 108, 111, 10]
bytes: [104, 101, 108, 108, 111, 10]
As you can see, sometimes we have data ready, and sometimes we don't. I think in your testing you've just been unlucky and only gotten TCP responses before any data has been sent?
I'm about 90% sure that TCP streams can include empty packets, and this is what we're seeing. (if someone has more knowledge here, feel free to edit the answer or comment).
To fix your program you might want to reconsider your goal.
Reading one TCP packet seems rarely helpful. Instead, generally, you want to read some amount of bytes and process the data as it comes. My understanding of TCP is as a stream of bytes, not really a stream of packets. The packets are just a way to get bytes from one place to another, and they could be any length without breaking compatibility. "One packet" is a fairly nebulous concept.
Here's an example reading the first 16 bytes of the stream using the tokio::io::read_exact function:
extern crate tokio;
use tokio::net::TcpListener;
use tokio::prelude::*;
use std::net::SocketAddr;
fn main() {
let addr = "0.0.0.0:8080".parse::<SocketAddr>().unwrap();
let socket = TcpListener::bind(&addr).unwrap();
println!("Listening on: {}", addr);
let done = socket
.incoming()
.map_err(|e| println!("failed to accept socket; error = {:?}", e))
.for_each(move |mut socket| {
// this function deals with bytes a bit differently and will just fill the
// buffer exactly rather than adding onto the end.
let mut bytes = vec![0; 16];
let processor = tokio::io::read_exact(socket, bytes)
.and_then(move |(socket, bytes)| {
println!("bytes: {:?}", bytes);
Ok(())
})
.map_err(|_| ());
tokio::spawn(processor)
});
tokio::run(done);
}
Editor's note: This code example is from a version of Rust prior to 1.0 and is not valid Rust 1.0 code. The concepts discussed in the question are still valid.
I'm experimenting with torrent scraping using Rust. I can see the incoming package in Wireshark, but my recv_from calls always return Error("End of file"). Here's my program:
use std::io::net::ip::{Ipv4Addr, SocketAddr};
use std::io::net::udp::UdpSocket;
use std::rand;
use std::io::MemWriter;
fn main() {
let addr = SocketAddr { ip: Ipv4Addr(0, 0, 0, 0), port: 35000 };
let mut socket = match UdpSocket::bind(addr) {
Ok(s) => s,
Err(e) => panic!("couldn't bind socket: {}", e),
};
let mut buf: Vec<u8> = Vec::with_capacity(1000);
let transaction_id: u32 = rand::random();
let mut req_data = MemWriter::with_capacity(16);
req_data.write_be_u64(0x41727101980).unwrap(); // connection_id, identifies the protocol.
req_data.write_be_u32(0).unwrap(); // action: connect
req_data.write_be_u32(transaction_id).unwrap();
println!("{}", socket.send_to(req_data.get_ref(), ("31.172.63.252", 80)));
match socket.recv_from(buf.as_mut_slice()) {
Ok((amt, src)) => {
println!("Got {} bytes from {}.", amt, src);
},
Err(err) => println!("Can't recv_from: {}", err)
}
}
The output is always:
➜ udp-bug git:(master) ✗ cargo run
Compiling udp-bug v0.0.1 (file:///home/omer/rust/udp-bug)
Running `target/udp-bug`
Ok(())
Can't recv_from: end of file
However, I can see the expected response coming in Wireshark:
20235 3512.148636000 31.172.63.252 192.168.1.4 QUIC 60 CID: 0, Seq: 0
This package has a 16-byte payload, exactly what I expect. What's going wrong?
Editor's note: This code example is from a version of Rust prior to 1.0 and is not valid Rust 1.0 code. The concepts discussed in the answer are still valid.
I think your problem is that you're using Vec::with_capacity() as a mutable slice. Vec::with_capacity() only creates a vector with the specified capacity (naturally), but its length is zero. Consequently, the length of the slice taken from the vector will also be zero:
let v = Vec::with_capacity(128);
println!("{}", v.as_mut_slice().len()); // prints 0
Slices can't grow, so recv_from() has no space to write to and it fails with the error.
You have essentially two options here. First one is to use unsafe set_len() method:
let mut buf: Vec<u8> = Vec::with_capacity(1000);
unsafe { buf.set_len(1000); }
This way the buffer will have the correct length but its contents will likely be just garbage. This is not very important for this use case, however, as long as you only access the correct amount of bytes (using the information returned by recv_from()).
There is a better way, however. You can use stack-allocated fixed-size array:
let mut buf = [0u8, ..1000];
// ...
match socket.recv_from(buf.as_mut_slice()) {
// ...
}
Same thing goes for your req_data: you can use a statically sized array and a BufWriter:
let transaction_id: u32 = rand::random();
let mut req_data_buf = [0u8, ..16];
let mut req_data = BufWriter::new(req_data_buf);
req_data.write_be_u64(0x41727101980).unwrap(); // connection_id, identifies the protocol.
req_data.write_be_u32(0).unwrap(); // action: connect
req_data.write_be_u32(transaction_id).unwrap();
println!("{}", socket.send_to(req_data_buf, ("31.172.63.252", 80)));
This will only work with fixed-size buffers though. If you don't know the size of the buffer, you will still need a Vec.