I'm implementing a setup in which I receive duplicate datagrams in a video-streaming based on UDP. I noticed that in layer 3 (IP) there's a header field named identification in which I can use to detect duplicated frames. The problem is that duplicated frames reach the top of the TCP/IP (Application Layer) stack and consequently create conflict with streaming.
Now my question, ¿How can I discard datagrams, in the receiving station, by detecting duplicates in the identification field until it reaches the application layer, using linux?
Related
I have one doubt.
Using uclinux 2.4.x. In this linux I have my own adapter code for reading the frames from ingress port.
Have added a debug log at exact place from where reading frames word by word and it is sure that I received all number of frames transmitted by peer.
( at Layer2 I am receiving all frames).
From here onward now invoking "netif_rx" to send all received framer to upper layer (i.e Network layer and Transport Layer).
Doubt : I observe that there are some drop of packets at (Transport Layer)UDP protocol.
How did I confirm: Added a count check at Layer 1 and Layer3(UDP Layer) and both counts are not equal.
that means even if we receiving all framer from layer 1 but when it reaching to UDP in between somewhere drop is happening.
So, Could any one suggest that where exactly the problem could be, How to check if memory is full or skb_alloc is not happening for more packets at UDP layer.
Kindly suggest your opinion, It will be of great help and support.
Please let me know if you need more information.
BR
Karn
I am developing a program that sniffs network packets using a raw socket (AF_PACKET, SOCK_RAW) and processes them in some way.
I am not sure whether my program runs fast enough and succeeds to capture all packets on the socket. I am worried that the recieve buffer for this socket occainally gets full (due to traffic bursts) and some packets are dropped.
How do I know if packets were dropped due to lack of space in the
socket's receive buffer?
I have tried running ss -f link -nlp.
This outputs the number of bytes that are currently stored in the revice buffer for that socket, but I can not tell if any packets were dropped.
I am using Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-52-generic x86_64).
Thanks.
I was having a similar problem as you. I knew that tcpdump was able to to generate statistics about packet drops, so I tried to figure out how it did that. By looking at the code of tcpdump, I noticed that it is not generating those statistic by itself, but that it is using the libpcap library to get those statistics. The libpcap is on the other hand getting those statistics by accessing the if_packet.h header and calling the PACKET_STATISTICS socket option (at least I think so, but I'm no C expert).
Therefore, I saw only two solutions to the problem:
I had to interact somehow with the linux header files from my Pyhton script to get the packet statistics, which seemed a bit complicated.
Use the Python version of libpcap which is pypcap to get those information.
Since I had no clue how to do the first thing, I implemented the second option. Here is an example how to get packet statistics using pypcap and how to get the packet data using dpkg:
import pcap
import dpkt
import socket
pc=pcap.pcap(name="eth0", timeout_ms=10000, immediate=True)
def packet_handler(ts,pkt):
#printing packet statistic (packets received, packets dropped, packets dropped by interface
print pc.stats()
#example packet parsing using dpkt
eth=dpkt.ethernet.Ethernet(pkt)
if eth.type != dpkt.ethernet.ETH_TYPE_IP:
return
ip =eth.data
layer4=ip.data
ipsrc=socket.inet_ntoa(ip.src)
ipdst=socket.inet_ntoa(ip.dst)
pc.loop(0,packet_handler)
tpacket_stats structure is defined in linux/packet.h header file
Create variable using the tpacket_stats structre and pass it to getSockOpt with PACKET_STATISTICS SOL_SOCKET options will give packets received and dropped count.
-- some times drop can be due to buffer size
-- so if you want to decrease the drop count check increasing the buffersize using setsockopt function
First off, switch your operating system.
You need a reliable, network oriented operating system. Not some pink fluffy "ease of use" with "security" functionality enabled. NetBSD or Gentoo/ArchLinux (the bare installations, not the GUI kitted ones).
Start a simultaneous tcpdump on a network tap and capture the traffic you're supposed to receive along side of your program and compare the results.
There's no efficient way to check if you've received all the packets you intended to on the receiving end since the packets might be dropped on a lower level than you anticipate.
Also this is a question for Unix # StackOverflow, there's no programming here what I can see, at least there's no code.
The only certain way to verify packet drops is to have a much more beefy sender (perhaps a farm of machines that send packets) to a single client, record every packet sent to your reciever. Have the statistical data analyzed and compared against your senders and see how much you dropped.
The cheaper way is to buy a network tap or even more ad-hoc enable port mirroring in your switch if possible. This enables you to dump as much traffic as possible into a second machine.
This will give you a more accurate result because your application machine will be busy as it is taking care of incoming traffic and processing it.
Further more, this is why network taps are effective because they split the communication up into two channels, the receiving and sending directions of your traffic if you will. This enables you to capture traffic on two separate machines (also using tcpdump, but instead of a mirrored port, you get a more accurate traffic mirroring).
So either use port mirroring
Or you buy one of these:
When capturing network traffic for debugging, there seem to be two common approaches:
Use a raw socket.
Use libpcap.
Performance-wise, is there much difference between these two approaches? libpcap seems a nice compatible way to listen to a real network connection or to replay some canned data, but does that feature set come with a performance hit?
The answer is intended to explain more about the libpcap.
libpcap uses the PF_PACKET to capture packets on an interface. Refer to the following link.
https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt
From the above link
In Linux 2.4/2.6/3.x if PACKET_MMAP is not enabled, the capture process is very
inefficient. It uses very limited buffers and requires one system call to
capture each packet, it requires two if you want to get packet's timestamp
(like libpcap always does).
In the other hand PACKET_MMAP is very efficient. PACKET_MMAP provides a size
configurable circular buffer mapped in user space that can be used to either
send or receive packets. This way reading packets just needs to wait for them,
most of the time there is no need to issue a single system call. Concerning
transmission, multiple packets can be sent through one system call to get the
highest bandwidth. By using a shared buffer between the kernel and the user
also has the benefit of minimizing packet copies.
performance improvement may vary depending on PF_PACKET implementation is used.
From https://www.kernel.org/doc/Documentation/networking/packet_mmap.txt -
It is said that TPACKET_V3 brings the following benefits:
*) ~15 - 20% reduction in CPU-usage
*) ~20% increase in packet capture rate
The downside of using libpcap -
If an application needs to hold the packet then it may need to make
a copy of the incoming packet.
Refer to manpage of pcap_next_ex.
pcap_next_ex() reads the next packet and returns a success/failure indication. If the packet was read without problems, the pointer
pointed to by the pkt_header argument is set to point to the
pcap_pkthdr struct for the packet, and the pointer pointed to by the
pkt_data argument is set to point to the data in the packet. The
struct pcap_pkthdr and the packet data are not to be freed by the
caller, and are not guaranteed to be valid after the next call to
pcap_next_ex(), pcap_next(), pcap_loop(), or pcap_dispatch(); if the
code needs them to remain valid, it must make a copy of them.
Performance penalty if application only interested in incoming
packets.
PF_PACKET works as taps in the kernel i.e. all the incoming and outgoing packets are delivered to PF_SOCKET. Which results in an expensive call to packet_rcv for all the outgoing packets. Since libpcap uses the PF_PACKET, so libpcap can capture all the incoming as well outgoing packets.
if application is only interested in incoming packets then outgoing packets can be discarded by setting pcap_setdirection on the libpcap handle. libpcap internally discards the outgoing packets by checking the flags on the packet metadata.
So in essence, outgoing packets are still seen by the libpcap but only to be discarded later. This is performance penalty for the application which is interested in incoming packets only.
Raw packet works on IP level (OSI layer 3), pcap on data link layer (OSI layer 2). So its less a performance issue and more a question of what you want to capture. If performance is your main issue search for PF_RING etc, that's what current IDS use for capturing.
Edit: raw packets can be either IP level (AF_INET) or data link layer (AF_PACKET), pcap might actually use raw sockets, see Does libpcap use raw sockets underneath them?
We have developed an Application Specific Integrated Circuit for power line communications. The chip has an ethernet interface. If the ASIC receives an ethernet frame containing TCP/IP or ARP payload (ethertypes 0x0800 IPv4, 0x0806 ARP and 0x86DD IPv6), it simply forwards the frame onto the power line and does the same in the other direction. We call such frames data frames.
If the ASIC receives an ethernet frame of a specific ethertype (we use 0x88b5 which is allocated for experimental//public use on local networks), it consumes this frame itself. These frames contain configuration settings for the ASIC. We call these configuration frames.
The chip is connected to an Ethernet LAN on one side and to power line on the other end. So it basically bridges the two network. The ASIC requires throttling of the data frames passing through it. This is due to the fact that the speeds over power line are 100 times less than the 100 Mbps Ethernet and also because the number of data frames that the ASIC can handle per second are limited.
We use raw sockets to form the configuration frames and send it via ethernet to the ASIC. Is there a way in which whenever configuration frame (0x88b5), it is queued in front of all the pending data frames (ethertypes 0x0800, 0x0806, 0x86dd) in the netdev_queue?
Can this be done via some supporting functionality implemented using hacks & hooks in a kernel module?
We came across a similar question (although improperly tagged) here: Setting up priority of packets that are being transmitted over the network
My aim was to find a way to process(drop,accept,forward and etc.) packets that are from Layer 2 ...
I know that "iptables" in *inux allow us to send packet to "NFQUEUE" for further packet processing ....
but it support layer 3 packets ... which means it does not detect packets that are from Layer 2..
although "arptables" detect packets that are destine for Layer 2, I couldn't find a way to send it to "NFQUEUE"
is there any way that can let us choose whether or not we should accept/drop/continue the layer packets?
Only ebtables has a target (-j arpreply) to generate ARP packets as of this date, though you can filter with either ebtables and arptables. NFQUEUE is also usable from ebtables, and in fact, can be quickly extended for arptables by just adding an entry to it, but so far, arptables has been pretty much a nice program, even more so than ebtables.