I am trying to work out what could be causing a strange issue I have been observing when testing a system.
The system under test recieves HTTP requests via UDP from an external source. For testing I have mocked out this source using Python sending the UDP packets over a local network for testing I send 1, 20, 133 and 1000 messages.
When I run the test on my local windows machine I regularily get a drop out of UDP packets with sometimes even the single message failing to be recieved. however when I create a virtual Linux Box (Centos) using Virtual Box and run the same tests from there I get 100% success rate 100% of the time.
Are there any know issues with sending UDP messages from Windows, any strange buffering differences between Windows and Linux I could adjust for on my machine?
UDP stacks of operating systems usually drops a received UDP packet, if the incoming packet does not fit to receive buffer. The receive buffer may be 'full', if the receiving application does not read it (=socket) fast enough.
Default size of receive buffer of your Windows may be smaller than in the Linux.
For handling bursts of incoming UDP packets better, you could increase size of the receiving buffer:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 65536)
You can also use getsockopt for checking the size.
having researched and investigated it seems that despite my efforts to try and prove otherwise the answer is, it's just Windows. Windows Network Buffering is causing the problem. As fixing this is outside the remit of this question i will close this and if necessary open a new question, thanks for your help.
Related
I've written my own packet sniffer in Linux.
I open a socket with socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) and then process the Ethernet packets - unpacking ARP packets, IP packets (and ICMP / TCP / UDP packets inside those).
This is all working fine so far.
Now I can read packets like this - and I can also inject packets by wrapping up a suitable Ethernet packet and sending it.
But what I'd like is a means to block packets - to consume them, as it were, so that they don't get further delivered into the system.
That is, if a TCP packet is being sent to port 80, then I can see the packet with my packet sniffer and it'll get delivered to the web server in the usual fashion.
But, basically, I'd like it that if I spot something wrong with the packet - not coming from the right MAC address, malformed in some way, or just breaking security policy - that I can just "consume" the packet, and it won't get further delivered onto the web server.
Because I can read packets and write packets - if I can also just block packets as well, then I'll have all I need.
Basically, I don't just want to monitor network traffic, but sometimes have control over it. E.g. "re-route" a packet by consuming the original incoming packet and then writing out a new slightly-altered packet to a different address. Or just plain block packets that shouldn't be being delivered at all.
My application is to be a general "network traffic management" program. Monitors and logs traffic. But also controls it too - blocking packets as a firewall, re-routing packets as a load balancer.
In other words, I've got a packet sniffer - but if it sniffs something that smells bad, then I'd like it to be able to stop that packet. Discard it early, so it's not further delivered anywhere.
(Being able to alter packets on the way through might be handy too - but if I can block, then there's always the possibility to just block the original packet completely, but then write out a new altered packet in its place.)
What you are looking for is libnetfilter_queue. The documentation is still incredibly bad, but the code in this example should get you started.
I used this library to develop a project that queued network packets and replayed them at a later time.
A bit of a tangent, but it was relevant when I was resolving my problem. Blocking raw packets is relatively complicated, so it might make sense to consider doing that at a different layer. In other words, does your cloud provider let you set up firewall rules to drop specific kind of traffic?
In my case it was easier to do, which is why I'm suggesting such a lateral solution.
I'm working on Linux 4.13.x. And I'm considering to send a packet response in the kernel.
Consider an echo TCP or UDP server running in the userland and there is also another node running a TCP or UDP client. Clients are sending requests to the server. I want to send the packet response back to the client without any involvement of server application running at userspace.
Here is my thoughts about this problem:
I started thinking how it is possible and I come across to a solution like netfilter. If I can capture the packets in NF_INET_PRE_ROUTING and then try to swap the source and destination IP addresses of IP header and also swapping the ports in the TCP header, then according to this answers and this presumably modified packet should be forwarded to the originator throughout the routing system.
Actually, I tried this scenario and it seems it is not possible to do so from netfilter hooks, however, I'm not sure of it. I thought that it is not working since it has problem with checksums because I'm manipulating packets so I did another experiment to figure this issue out. I just change the packet data and everything worked well. I think checksums don't have any problem since they will be check at NIC while receiving and also same situation while sending so manipulation in between doesn't make anything wrong. I also activate the IPv4 forwarding at the server host(sysctl.config) still nothing changes.
I don't want to create new packet, I only want to alter this packet and send it back. There is another similar question which is creating another packet. Moreover, I'm just thinking why this scenario is not working? But based on the netfilter's architecture it should work.
Thank you
I am also working on this, actually kernel validate the source ip address after ip_rcv function in NF_HOOK which check the source ip address. So just try below command:-
sudo sysctl -w "net.ipv4.conf.all.rp_filter=0"
after doing this also disable your interface from which you send and receive packet just like below:-
sudo sysctl -w "net.ipv4.conf.enp2s0.rp_filter=0"
I'm an Automation Developer and lately I've taken it upon myself to control an IP Phone on my desk (Cisco 7940).
I have a third party application that can control the IP phone with SCCP (Skinny) packets. Through Wireshark, I see that the application will send 4 unique SCCP packets and then receives a TCP ACK message.
SCCP is not very well known, but it looks like this:
Ethernet( IP( TCP( SCCP( ))))
Using a Python packet builder: Scapy, I've been able to send the same 4 packets to the IP Phone, however I never get the ACK. In my packets, I have correctly set the sequence, port and acknowledge values in the TCP header. The ID field in the IP header is also correct.
The only thing I can imagine wrong is that it takes Python a little more than a full second to send the four packets. Whereas the application takes significantly less time. I've tried raising the priority for the Python shell with no luck.
Does anyone have an idea why I may not be receiving the ACK back?
This website may be helpful in debugging why on your machine you aren't seeing the traffic you expect, and taking steps to modify your environment to produce the desired output.
Normally, the Linux kernel takes care of setting up and sending and
receiving network traffic. It automatically sets appropriate header
values and even knows how to complete a TCP 3 way handshake. Uising
the kernel services in this way is using a "cooked" socket.
Scapy does not use these kernel services. It creates a "raw" socket. The
entire TCP/IP stack of the OS is circumvented. Because of this, Scapy
give us compete control over the traffic. Traffic to and from Scapy
will not be filtered by iptables. Also, we will have to take care of
the TCP 3 way handshake ourselves.
http://www.packetlevel.ch/html/scapy/scapy3way.html
I am trying to generate a series of packets to simulate the TCP 3-way handshake procedure, my first step is to capture the real connecting packets, and try to re-send the same packets from the same machine, but it didn't work at first.
finally I found it out that the packet I captured with tcpdump is not exactly what my computer sent out, the TCP's checksum field is changed and it lead me to thinkk that I can establish a tcp connection even the TCP checksum is incorrect.
so my question is how is the checksum field calculated? is it modified by tcpdump or hardware? why is it changed? Is it a bug of tcpdump? or it's because the calculation is omitted.
the following is the screenshot I captured from my host machine and a virtual machinne, you can see that the same packet captured on differnet machine are all the same except for the TCP checksum.
and the small window is my virtual machine, I used command "ssh 10.82.25.138" from the host to generate these packets
What you are seeing may be the result of checksum offloading. To quote from the wireshark wiki (http://wiki.wireshark.org/CaptureSetup/Offloading):
Most modern operating systems support some form of network offloading,
where some network processing happens on the NIC instead of the CPU.
Normally this is a great thing. It can free up resources on the rest
of the system and let it handle more connections. If you're trying to
capture traffic it can result in false errors and strange or even
missing traffic.
On systems that support checksum offloading, IP, TCP, and UDP
checksums are calculated on the NIC just before they're transmitted on
the wire. In Wireshark these show up as outgoing packets marked black
with red Text and the note [incorrect, should be xxxx (maybe caused by
"TCP checksum offload"?)].
Wireshark captures packets before they are sent to the network
adapter. It won't see the correct checksum because it has not been
calculated yet. Even worse, most OSes don't bother initialize this
data so you're probably seeing little chunks of memory that you
shouldn't.
Although this is for wireshark, the same principle applies. In your host machine, you see the wrong checksum because it just hasn't been filled in yet. It looks right on the guest, because before it's sent out on the "wire" it is filled in. Try disabling checksum offloading on the interface which is handling this traffic, e.g.:
ethtool -K eth0 rx off tx off
if it's eth0.
Little bit confused on this issue :-). I have some code running on Ubuntu 10.04, which is using the recvmsg call to receive Audio UDP packets. This piece is part of a bigger SIP client which I've been testing with a remote system across the net.
Typically I am able to establish a call no problem and audio happily travels from the remote source to my program. However after a random amount of time I'm running into a read error on the port. When this happens I get the errno EHOSTUNREACH. On this error I shutdown my port and kill the connection.
The strange thing about this is that I was under the impression that this would happen in response to an ICMP message. Perhaps due to a momentary network glitch. However after reproducing this issue and doing a full packet capture, all I saw where ICMP pings and responses. I didn't see any of the ICMP error messages that the kernel interprets as EHOSTUNREACH.
My UDP port is pretty basic. I can post the code if its really needed. But this is just a basic SOCK_DGRAM. The socket is able to receive data for upwards of 8 hours sometimes before it hits this error condition.
Any ideas how I can further tackle this issue. I'm trying to understand why I'm receiving this errno with no ICMP message to correlate it with.
Are you using the same socket to send packets out with? Or a different socket sharing the same port? I'm curious if it's possible another socket is actually generating the error.
I remember years ago dealing with ICMP messages causing both sendto and recvfrom to fail
intermittently like you describe. And if I recall, the workaround is to just ignore it and do another recvfrom/recvmsg.
As to why you are getting that error code in the absence of seeing an ICMP message on the wire is beyond me. Are you certain you are dismissing the ICMP messages you observed as "pings and responses" correctly? ICMP messages don't have port numbers, so it's possible that all sockets (or a random one) associated with that remote host return this error code.
I thought there might be an ioctl to disable this behavior, but I couldn't find it.
A similar discussion here.