I'm using libpcap (pcap crate) and I want to reconstruct individual tcp flows from packets.
I have to match those packets to a flow, in a way that works for packets going in both directions (client->server and server-client) with as little overhead as possible.
A flow is identified by its 4-tuple (src-addr, dst-addr, src-port, dst-port).
I was wondering if XOR-ing the ip and port (src and dst) would be good enough to match the tcp flow. Basically something like this:
create a hashmap (flow_key, array-of-packets)
get the next packet from libpcap
extract src/dst ip and port
flow-key = hash( XOR(src-ip, dst-ip), XOR (src-port, dst-port) )
if flow_key is in the hashmap then add the packet to the hashmap value
if not, create a new entry in the hashmap
drop the entry when FIN or RST is observed and process the collected packets
This way, I would not care in which direction the packet is flowing and there is no need to implement more detailed connection tracking, dealing with connection state and etc (I assume).
I did some tests using rust, like the code below which seems to be working as expected but I'm not sure if the idea is valid or not:
use std::collections::hash_map::DefaultHasher;
use std::{net::Ipv4Addr, str::FromStr};
use std::hash::{Hash, Hasher};
#[derive(Debug, PartialEq)]
struct TestS {
src_ip: Ipv4Addr,
dst_ip: Ipv4Addr,
src_port: u16,
dst_port: u16
}
impl Hash for TestS {
fn hash<H: Hasher>(&self, state: &mut H) {
let hashable_ip = u32::from(self.src_ip) ^ u32::from(self.dst_ip);
let hashable_port = self.src_port ^ self.dst_port;
hashable_ip.hash(state);
hashable_port.hash(state);
}
}
fn main() {
let a_packet = TestS{
src_ip: Ipv4Addr::from_str("192.168.0.1").unwrap(),
dst_ip: Ipv4Addr::from_str("127.0.0.1").unwrap(),
src_port: 7879,
dst_port:80
};
let another_packet = TestS{
src_ip: Ipv4Addr::from_str("127.0.0.1").unwrap(),
dst_ip: Ipv4Addr::from_str("192.168.0.1").unwrap(),
src_port: 80,
dst_port: 7879
};
let mut hash_state = DefaultHasher::new();
a_packet.hash(&mut hash_state);
let hash1 = hash_state.finish();
let mut hash_state2 = DefaultHasher::new();
another_packet.hash(&mut hash_state2);
let hash2 = hash_state2.finish();
println!("{} -- {}", hash1, hash2);
//hash1 and hash2 will be equal
}
Note that if you have streams with crossed ports (e.g. a stream between S:1 and C:2 and another stream between S:2 and C:1), then they will be taken as the same flow.
A more robust solution would be to sort the (IP, port) pairs prior to hashing:
let src_ip = u32::from (self.src_ip);
let dst_ip = u32::from (self.dst_ip)
let (ip1, port1) = min ((src_ip, src_port), (dst_ip, dst_port));
let (ip2, port2) = max ((src_ip, src_port), (dst_ip, dst_port));
ip1.hash (state);
ip2.hash (state);
port1.hash (state);
port2.hash (state);
Related
sorry, i'm a complete newbie to Rust. I try to read the temp from the sensor mentioned above on a Raspberry Pi using the code provided on this site: https://github.com/fuchsnj/ds18b20
Actually, i want to call the function
get_temperature
but i have no idea how to declare the parameters, especially delay and one_wire_bus.
I was able to resolve all the 'namespaces' or name bindings (sorry, coming from C++) but got stuck with the parameters. Can someone give me an example how to call and use this function like this:
use ds18b20::{Resolution, Ds18b20};
use embedded_hal::blocking::delay::{DelayUs, DelayMs};
use embedded_hal::digital::v2::{OutputPin, InputPin};
use one_wire_bus::{self, OneWire, OneWireResult};
use core::fmt::Debug;
use std::io::Write;
fn main() {
let mut delay = ?????;
let mut one_wire_bus = ?????;
let mut tx = ?????; //&mut Vec::new();
let temp = get_temperature(delay, tx, one_wire_bus);
...
//do something whit the temp
...
}
This is the implementation of the function from the website
fn get_temperature<P, E>(
delay: &mut (impl DelayUs<u16> + DelayMs<u16>),
tx: &mut impl Write,
one_wire_bus: &mut OneWire<P>,
) -> OneWireResult<(), E>
where
P: OutputPin<Error=E> + InputPin<Error=E>,
E: Debug
{
// initiate a temperature measurement for all connected devices
ds18b20::start_simultaneous_temp_measurement(one_wire_bus, delay)?;
// wait until the measurement is done. This depends on the resolution you specified
// If you don't know the resolution, you can obtain it from reading the sensor data,
// or just wait the longest time, which is the 12-bit resolution (750ms)
Resolution::Bits12.delay_for_measurement_time(delay);
// iterate over all the devices, and report their temperature
let mut search_state = None;
loop {
if let Some((device_address, state)) = one_wire_bus.device_search(search_state.as_ref(), false, delay)? {
search_state = Some(state);
if device_address.family_code() != ds18b20::FAMILY_CODE {
// skip other devices
continue;
}
// You will generally create the sensor once, and save it for later
let sensor = Ds18b20::new(device_address)?;
// contains the read temperature, as well as config info such as the resolution used
let sensor_data = sensor.read_data(one_wire_bus, delay)?;
writeln!(tx, "Device at {:?} is {}°C", device_address, sensor_data.temperature);
} else {
break;
}
}
Ok(())
}
I am writing a program that pings a set of targets 100 times, and stores each RTT value returned from the ping into a vector, thus giving me a set of RTT values for each target. Say I have n targets, I would like all of the pinging to be done concurrently. The rust code looks like this:
let mut sample_rtts_map = HashMap::new();
for addr in targets.to_vec() {
let mut sampleRTTvalues: Vec<f32> = vec![];
//sample_rtts_map.insert(addr, sampleRTTvalues);
thread::spawn(move || {
while sampleRTTvalues.len() < 100 {
let sampleRTT = ping(addr);
sampleRTTvalues.push(sampleRTT);
// thread::sleep(Duration::from_millis(5000));
}
});
}
The hashmap is used to tell which vector of values belongs to which target. The problem is, how do I retrieve the updated sampleRTTvalues from each thread after the thread is done executing? I would like something like:
let (name, sampleRTTvalues) = thread::spawn(...)
The name, being the name of the thread, and sampleRTTvalues being the vector. However, since I'm creating threads in a for loop, each thread is being instantiated the same way, so how I differentiate them?
Is there some better way to do this? I've looked into schedulers, future, etc., but it seems my case can just be done with simple threads.
I go the desired behavior with the following code:
use std::thread;
use std::sync::mpsc;
use std::collections::HashMap;
use rand::Rng;
use std::net::{Ipv4Addr,Ipv6Addr,IpAddr};
const RTT_ONE: IpAddr = IpAddr::V4(Ipv4Addr::new(127,0,0,1));
const RTT_TWO: IpAddr = IpAddr::V6(Ipv6Addr::new(0,0,0,0,0,0,0,1));
const RTT_THREE: IpAddr = IpAddr::V4(Ipv4Addr::new(127,0,1,1));//idk how ip adresses work, forgive if this in invalid but you get the idea
fn ping(address: IpAddr) -> f32 {
rand::thread_rng().gen_range(5.0..107.0)
}
fn main() {
let targets = [RTT_ONE,RTT_TWO,RTT_THREE];
let mut sample_rtts_map: HashMap<IpAddr,Vec<f32>> = HashMap::new();
for addr in targets.into_iter() {
let (sample_values,moved_values) = mpsc::channel();
let mut sampleRTTvalues: Vec<f32> = vec![];
thread::spawn(move || {
while sampleRTTvalues.len() < 100 {
let sampleRTT = ping(addr);
sampleRTTvalues.push(sampleRTT);
//thread::sleep(Duration::from_millis(5000));
}
});
sample_rtts_map.insert(addr,moved_values.recv().unwrap());
}
}
note that the use rand::Rng can be removed when implementing, as it is only so the example works. what this does is pass data from the spawned thread to the main thread, and in the method used it waits until the data is ready before adding it to the hash map. If this is problematic (takes a long time, etc.) then you can use try_recv instead of recv which will add an error / option type that will return a recoverable error if the value is ready when unwrapped, or return the value if it's ready
You can use a std::sync::mpsc channel to collect your data:
use std::collections::HashMap;
use std::sync::mpsc::channel;
use std::thread;
fn ping(_: &str) -> f32 { 0.0 }
fn main() {
let targets = ["a", "b"]; // just for example
let mut sample_rtts_map = HashMap::new();
let (tx, rx) = channel();
for addr in targets {
let tx = tx.clone();
thread::spawn(move || {
for _ in 0..100 {
let sampleRTT = ping(addr);
tx.send((addr, sampleRTT));
}
});
}
drop(tx);
// exit loop when all thread's tx have dropped
while let Ok((addr, sampleRTT)) = rx.recv() {
sample_rtts_map.entry(addr).or_insert(vec![]).push(sampleRTT);
}
println!("sample_rtts_map: {:?}", sample_rtts_map);
}
This will run all pinging threads simultaneously, and collect data in main thread synchronously, so that we can avoid using locks. Do not forget to drop sender in main thread after cloning to all pinging threads, or the main thread will hang forever.
I made a loop for a webserver.
On a windows client I didn't have any problems but on a linux client the server didn't responding to requests.
The problem: I found out that if request_size % buffer_size == 0 then the loop runs once more waiting for more data.
The question: Is there an efficient way of reading data that takes into consideration slow connections, connections that drop packages. (Not just using non_blocking or nodelay.)
let listener = TcpListener::bind("127.0.0.1:80").unwrap();
while let Ok((mut stream, _)) = listener.accept() {
let mut data: Vec<u8> = Vec::new();
let mut buf = [0u8; 32];
while let Ok(size) = stream.read(&mut buf) {
data.extend(buf[..size].iter());
if size != buf.len() { break; }
}
// do something with the data
}
I could increase the buffer size but that wouldn't solve the problem.
First, to detect EOF reliably, you should test the returned size of Read::read against zero and not your buffer size, because if you have a 'slow connections' you might not get enough data to fill the entire buffer at once, causing your loop to quite early with an incomplete message in data.
There are essentially 3 ways to make sure you received the entire message:
Read until EOF
Read a fixed-sized message
Encode some 'content length' and read that many bytes
Notice, that only the last two variants allow your client to eventually send more data over the same stream. Also notice, that these two variants can be implemented comparably easy via Read::read_exact.
Besides notice, if you don't trust your client, it might be helpful to set up TcpStream::set_read_timeout with a reasonably long timeout (e.g. 2 min).
Read until EOF
This is probably the easiest and, according to your title and code, probably the method you are aiming for. However, to generate an EOF, the client must shutdown at least its write channel. So, if your server is stuck in read, I assume you forgot to shutdown your client (tho I have to guess here).
On the server side, if you really want to read until EOF, you don't need a loop yourself, you can simply use the Read::read_to_end utility function. Here is an example for a client & server with the client sending a single message terminated by EOF:
use std::io::Read;
use std::io::Write;
use std::net::TcpListener;
use std::net::TcpStream;
// --- Client code
const SERVER_ADDR: &str = "localhost:1234";
pub fn client() {
let mut socket = TcpStream::connect(SERVER_ADDR).expect("Failed to connect");
// Send a 'single' message, the flushes kinda simulates a very slow connection
for _ in 0..3 {
socket.write(b"Hello").expect("Failed to send");
socket.flush().unwrap();
}
// Instead of shutdow, you can also drop(socket), but than you can't read.
socket.shutdown(std::net::Shutdown::Write).unwrap();
// go reading, or whatever
}
// --- Server code
const SERVER_BIND: &str = "127.0.0.1:1234";
pub fn server() {
let listener = TcpListener::bind(SERVER_BIND).expect("Failed to bind");
while let Ok((stream, _)) = listener.accept() {
let _ = handle_client(stream); // don't care if the client screwed up
}
}
pub fn handle_client(mut socket: TcpStream) -> std::io::Result<()> {
let mut data: Vec<u8> = Vec::new();
// Read all bytes until EOF
socket.read_to_end(&mut data)?;
println!("Data: {:?}", data); // or whatever
Ok(())
}
I am trying to read and write to my serial port on Linux to communicate with a microcontroller and I'm trying to do so in Rust.
My normal pattern when developing in say C++ or Python is to have two threads: one which sends requests out over serial periodically and one which reads bytes out of the buffer and handles them.
In Rust, I'm running into trouble with the borrow checker while using the serial crate. This makes sense to me why this is, but I'm unsure what designing for an asynchronous communication interface would look like in Rust. Here's a snippet of my source:
let mut port = serial::open(&device_path.as_os_str()).unwrap();
let request_temperature: Vec<u8> = vec![0xAA];
thread::spawn(|| {
let mut buffer: Vec<u8> = Vec::new();
loop {
let _bytes_read = port.read(&mut buffer);
// process data
thread::sleep(Duration::from_millis(100));
}
});
loop {
port.write(&request_temperature);
thread::sleep(Duration::from_millis(1000));
}
How can I emulate this functionality where I have two threads holding onto a mutable resource in Rust? I know that since this specific example could be done in a single thread, but I'm thinking for an eventual larger program this would end up being multiple threads.
You can wrap your port in a Arc and a Mutex, then you can write something like:
use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
struct Port;
impl Port {
pub fn read(&mut self, _v: &mut Vec<u8>) {
println!("READING...");
}
pub fn write(&mut self, _v: &Vec<u8>) {
println!("WRITING...");
}
}
pub fn main() {
let mut port = Arc::new(Mutex::new(Port));
let p2 = port.clone();
let handle = thread::spawn(move || {
let mut buffer: Vec<u8> = Vec::new();
for j in 0..100 {
let _bytes_read = p2.lock().unwrap().read(&mut buffer);
thread::sleep(Duration::from_millis(10));
}
});
let request_temperature: Vec<u8> = vec![0xAA];
for i in 0..10 {
port.lock().unwrap().write(&request_temperature);
thread::sleep(Duration::from_millis(100));
}
handle.join();
}
So that this will run on a test machine, I've replaced the serial port with a stub class, reduced the sleeps and replaced the infinite loop with some finite loops.
While this works, you'll probably actually want proper communication between the threads at some stage, at which point you'll want to look at std::sync::mpsc::channel
Editor's note: This code example is from a version of Rust prior to 1.0 and is not valid Rust 1.0 code. The concepts discussed in the question are still valid.
I'm experimenting with torrent scraping using Rust. I can see the incoming package in Wireshark, but my recv_from calls always return Error("End of file"). Here's my program:
use std::io::net::ip::{Ipv4Addr, SocketAddr};
use std::io::net::udp::UdpSocket;
use std::rand;
use std::io::MemWriter;
fn main() {
let addr = SocketAddr { ip: Ipv4Addr(0, 0, 0, 0), port: 35000 };
let mut socket = match UdpSocket::bind(addr) {
Ok(s) => s,
Err(e) => panic!("couldn't bind socket: {}", e),
};
let mut buf: Vec<u8> = Vec::with_capacity(1000);
let transaction_id: u32 = rand::random();
let mut req_data = MemWriter::with_capacity(16);
req_data.write_be_u64(0x41727101980).unwrap(); // connection_id, identifies the protocol.
req_data.write_be_u32(0).unwrap(); // action: connect
req_data.write_be_u32(transaction_id).unwrap();
println!("{}", socket.send_to(req_data.get_ref(), ("31.172.63.252", 80)));
match socket.recv_from(buf.as_mut_slice()) {
Ok((amt, src)) => {
println!("Got {} bytes from {}.", amt, src);
},
Err(err) => println!("Can't recv_from: {}", err)
}
}
The output is always:
➜ udp-bug git:(master) ✗ cargo run
Compiling udp-bug v0.0.1 (file:///home/omer/rust/udp-bug)
Running `target/udp-bug`
Ok(())
Can't recv_from: end of file
However, I can see the expected response coming in Wireshark:
20235 3512.148636000 31.172.63.252 192.168.1.4 QUIC 60 CID: 0, Seq: 0
This package has a 16-byte payload, exactly what I expect. What's going wrong?
Editor's note: This code example is from a version of Rust prior to 1.0 and is not valid Rust 1.0 code. The concepts discussed in the answer are still valid.
I think your problem is that you're using Vec::with_capacity() as a mutable slice. Vec::with_capacity() only creates a vector with the specified capacity (naturally), but its length is zero. Consequently, the length of the slice taken from the vector will also be zero:
let v = Vec::with_capacity(128);
println!("{}", v.as_mut_slice().len()); // prints 0
Slices can't grow, so recv_from() has no space to write to and it fails with the error.
You have essentially two options here. First one is to use unsafe set_len() method:
let mut buf: Vec<u8> = Vec::with_capacity(1000);
unsafe { buf.set_len(1000); }
This way the buffer will have the correct length but its contents will likely be just garbage. This is not very important for this use case, however, as long as you only access the correct amount of bytes (using the information returned by recv_from()).
There is a better way, however. You can use stack-allocated fixed-size array:
let mut buf = [0u8, ..1000];
// ...
match socket.recv_from(buf.as_mut_slice()) {
// ...
}
Same thing goes for your req_data: you can use a statically sized array and a BufWriter:
let transaction_id: u32 = rand::random();
let mut req_data_buf = [0u8, ..16];
let mut req_data = BufWriter::new(req_data_buf);
req_data.write_be_u64(0x41727101980).unwrap(); // connection_id, identifies the protocol.
req_data.write_be_u32(0).unwrap(); // action: connect
req_data.write_be_u32(transaction_id).unwrap();
println!("{}", socket.send_to(req_data_buf, ("31.172.63.252", 80)));
This will only work with fixed-size buffers though. If you don't know the size of the buffer, you will still need a Vec.