I am writing a linux program that controls internet traffic. In other words, how much bytes I have used while some amount of time. I use a Pcap4J for java (implementation of libpcap) and I have question about it. What happens if my program hasn't proceeded a package while a new one has arrived.
1. It slows down the download(upload) rate for the whole OS?
2. It skips a new one, and my program will never know that it passed by?
In other words, I've downloaded the 1G of data on my computer. How many bytes my program get: 100% or it may be passed my program by but still got the destination place!
And give me know if it is a bad idea to write a control traffic app using this lib!
Your application loses packets. In your words, they pass by.
However, if your idea is to have a metric of how many packets went in and out of your system in a given time, there are definitely better ways to achieve it.
On Linux you can just do a script that does something like this:
DEVICE=eth0
RX0=$(cat /sys/net/$DEVICE/statistics/rx_bytes)
TX0=$(cat /sys/net/$DEVICE/statistics/tx_bytes)
while : ; do
sleep 5
RX1=$(cat /sys/net/$DEVICE/statistics/rx_bytes)
TX1=$(cat /sys/net/$DEVICE/statistics/tx_bytes)
echo "RX bytes: $(($RX1-$RX0))"
echo "TX bytes: $(($TX1-$TX0))"
RX0=RX1
TX0=TX1
done
You can adjust times or whether is a parameter, I think you'll get the idea.
Related
I encountered this issue while trying to make a ping program in C myself and used wireshark for further digging into the problem: ping which sends echo requests to a destination IP also ads a timestamp field of 8 bytes (TOD timestamp) after the ICMP header in linux. Ping in Windows doesn't add that timestamp but rather I think makes the time calculations locally. Now my question is how do you convert the time from Unix Epoch format (the number of seconds from 1970 which you get with the 'time' function in C) to that TOD format of 8 bytes? I got to this question as, finally, after quite a time of research, my ping.c program sends the ICMP echo request message to the destination, where after a test with 2 hosts I noticed that it manages to arrive, but gets no ping echo reply message back while the native linux ping works properly. I can only imagine 2 possible causes:
I didnt complete well the fields of the ICMP and IP header. To be honest, I myself pretty much doubt this possiblity because wireshark shows the message arrives to the destination and is recognized as an echo request message, but doesn't trigger any echo reply answer. However, if it would to be this, the only thing I can think off is that timestamp which I don`t know how to convert in TOD form to occupy at most 8 bytes.
There might be a firewall at the destination or some other system dependent fact.
https://www.ibm.com/docs/en/zos/2.2.0?topic=problems-using-ping-command
The Ping command does not use the ICMP/ICMPv6 header sequence number field (icmp_seq or icmp6_seq) to correlate requests with ICMP/ICMPv6 Echo Replies. Instead, it uses the ICMP/ICMPv6 header identifier field (icmp_id or icmp6_id) plus an 8-byte TOD time stamp field to correlate requests with replies. The TOD time stamp is the first 8-bytes of data after the ICMP/ICMPv6 header.
Finally, to repeat the initial question:
How do you convert the UNIX Epoch time to the TOD timestamp form which linux ping adds at the end of the ICMP header/begining of data field?
An useful explanation, but I don't think sufficient I found here:
https://towardsdatascience.com/3-tips-to-handle-timestamps-in-c-ad5b36892294
I should probably mention I`m working on Ubuntu 20.04 focalfossa.
I found a related post here. The book "Principles of Operation" is mentioned in the comments. I skimmed through it but it seems to be generally lower level than C so if anyone knows another place/way to answer the question it would be better.
Background
Including the UNIX timestamp of the time of transmission in the first data bytes of the ICMP Echo message is a trick/optimization the original ping by Mike Muuss used to avoid keeping track of it locally. It exploits the following guarantee made by RFC 792's Echo or Echo Reply Message description:
The data received in the echo message must be returned in the echo reply message.
Many (if not all) BSD ping implementations are based on Mike Muuss' original implementation and therefore kept this behavior. On Linux systems, ping is typically provided by iputils, GNU inetutils, or Busybox. All exhibit the same behavior. fping is a notable exception, which stores a mapping from target host and sequence number to timestamp locally.
Implementations typically store the timestamp in the sender's native precision and byte order as opposed to a well-defined precision in conventional network byte order (big endian), that is normally used for data transmitted over the network, as it intended to be only be interpreted by the original sender and others should just treat it as opaque stream of bytes.
Because this is so common however, the Wireshark ICMP dissector (as of v3.6.2) tries to be clever and heuristically decode it nonetheless, if the first 8 data bytes look like a struct timeval timestamp in 32-bit precision for seconds and microseconds in either byte order. Please note that if the sender was actually using big endian 64-bit precision, this will fail and if it was using little endian 64-bit precision, it will truncate the microseconds before the Epochalypse and fail after that.
Obtaining and serializing epoch time
To answer your actual question:
How do you convert the UNIX Epoch time to the TOD timestamp form which linux ping adds at the end of the ICMP header/begining of data field?
Most implementations use the obsolescent gettimeofday(2) instead of the newer clock_gettime(2). The following snippet is taken from iputils:
struct icmphdr *icp /* = pointer to ICMP packet */;
struct timeval tmp_tv;
gettimeofday(&tmp_tv, NULL);
memcpy(icp + 1, &tmp_tv, sizeof(tmp_tv));
memcpy from a temporary variable instead of directly passing the icp + 1 as target to gettimeofday is used to avoid potential improper alignment, effective type and strict aliasing violation issues.
I appreciate the clear answers. I actually managed to solve the problem and make the ping function work and I have to say the problem was certainly not the timestamp because, yes indeed, i was talking about the "Echo or Echo Reply Message". One way of implementing ping is by using the ICMP feature of Echo and Echo reply messages. The fact is when I put this question I was obviously stuck probably because I wasn't clearly differentiating the main aspects of the problem. Thus, I started to examine the packet sent by the native ping on my Ubuntu 20.04 focal_fossa (with Wireshark), hopefully trying to get a better grasp of how to fill the headers of the packet sent by my program (the IP and ICMP headers). This question simply arised from the fact that I noticed that in this version of Ubuntu, ping adds a timestamp (indeed of 32 bits) after the end of ICMP header, basically in the data/payload section. As a matter of fact, I also used Wireshark on Windows10 and saw that there is no timestamp added after the header. So yes, it might be about different versions of the program being used.
What is the main point I want to emphasize is that my final version of ping has nothing to do with any timestamps. So yes, they are not a crucial aspect for ping to work.
I need to build a small size tcpdump for the embedded project that I am working on. Since the memory size of my embedded device is limited, I need to strip all the unwanted functionality in the TCPDUMP while building it. My target is make the tcpdump executable size less that 300KB. After using "strip tcpdump option" and disabling package options in the configure, I have reached 750KB. To achieve this, I want to remove all the protocol decoding capability of tcpdump. I want the tcpdump to have no more that hex dump capability. I have a below initial list of unwanted protocols that has to be removed.
print-802_11.c
print-802_15_4.c
print-ah.c
print-ahcp.c
print-aodv.c
print-aoe.c
print-ap1394.c
print-atalk.c
print-atm.c
print-babel.c
print-bootp.C
print-bt.c
print-calm-fast.c
print-carp.c
print-cdp.c
print-cfm.c
print-chdlc.c
print-cip.c
print-cnfp.c
print-dccp.c
print-decnet.c
print-dtp.c
print-dvmrp.c
print-eap.c
print-egp.c
print-eigrp.c
print-enc.c
print-esp.c
print-fddi.c
print-forces.c
print-ipx.c
print-isakmp.c
print-isoclns.c
print-juniper.c
print-krb.c
print-lane.c
print-m3ua.c
print-sip.c
print-sl.c
print-sll.c
print-sunatm.c
print-zephyr.c
print-usb.c
print-vjc.c
print-vqp.c
print-timed.c
print-tipc.c
print-token.c
I started to remove these from Makefile.in and removing the function calls manually in the source code. But then I realized this approach is not scalable.
Is there a better way to do this ? Someway by using configure options?
I am new to this. So please explain.
Is there a better way to do this ? Someway by using configure options?
No, there are no such configure options. You'll have to do it the non-scalable way.
"I want to remove all the protocol decoding capability of tcpdump. I
want the tcpdump to have no more that hex dump capability. [...] Is
there a better way to do this ?"
I think there is, but with a very different approach.
If all you want from tcpdump is:
the capability of specifying an interface,
put this interface on promiscuous mode or not, or monitor mode if it's a Wi-Fi interface,
apply a capture filter,
and then spit the output in a file or as hex to stdout,
...you'd be better write your own from scratch, using libpcap (which is what tcpdump uses BTW).
This should be no more than 100-400 lines of C code depending on the options you want to have, you'll have a very, very small executable, and no more dependencies than tcpdump which require libpcap anyway. All the complexity is in the dissection, once you remove all that, what you have is basically... a pcap loop.
It's not that hard, and looks to me as far less work than your approach - and also more interesting work.
There's a tutorial to start with (30-60 minutes read):
http://www.tcpdump.org/pcap.html
...at the end of this tutorial, you'll already have the core of your program.
And you can find loads of info (and ask questions) in the related SO tags:
https://stackoverflow.com/tags/libpcap/info
https://stackoverflow.com/tags/pcap/info
...and have about 70 well-written man pages documenting the full pcap API (you'll end up using maybe 10-20 of these).
I am developing a program that sniffs network packets using a raw socket (AF_PACKET, SOCK_RAW) and processes them in some way.
I am not sure whether my program runs fast enough and succeeds to capture all packets on the socket. I am worried that the recieve buffer for this socket occainally gets full (due to traffic bursts) and some packets are dropped.
How do I know if packets were dropped due to lack of space in the
socket's receive buffer?
I have tried running ss -f link -nlp.
This outputs the number of bytes that are currently stored in the revice buffer for that socket, but I can not tell if any packets were dropped.
I am using Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-52-generic x86_64).
Thanks.
I was having a similar problem as you. I knew that tcpdump was able to to generate statistics about packet drops, so I tried to figure out how it did that. By looking at the code of tcpdump, I noticed that it is not generating those statistic by itself, but that it is using the libpcap library to get those statistics. The libpcap is on the other hand getting those statistics by accessing the if_packet.h header and calling the PACKET_STATISTICS socket option (at least I think so, but I'm no C expert).
Therefore, I saw only two solutions to the problem:
I had to interact somehow with the linux header files from my Pyhton script to get the packet statistics, which seemed a bit complicated.
Use the Python version of libpcap which is pypcap to get those information.
Since I had no clue how to do the first thing, I implemented the second option. Here is an example how to get packet statistics using pypcap and how to get the packet data using dpkg:
import pcap
import dpkt
import socket
pc=pcap.pcap(name="eth0", timeout_ms=10000, immediate=True)
def packet_handler(ts,pkt):
#printing packet statistic (packets received, packets dropped, packets dropped by interface
print pc.stats()
#example packet parsing using dpkt
eth=dpkt.ethernet.Ethernet(pkt)
if eth.type != dpkt.ethernet.ETH_TYPE_IP:
return
ip =eth.data
layer4=ip.data
ipsrc=socket.inet_ntoa(ip.src)
ipdst=socket.inet_ntoa(ip.dst)
pc.loop(0,packet_handler)
tpacket_stats structure is defined in linux/packet.h header file
Create variable using the tpacket_stats structre and pass it to getSockOpt with PACKET_STATISTICS SOL_SOCKET options will give packets received and dropped count.
-- some times drop can be due to buffer size
-- so if you want to decrease the drop count check increasing the buffersize using setsockopt function
First off, switch your operating system.
You need a reliable, network oriented operating system. Not some pink fluffy "ease of use" with "security" functionality enabled. NetBSD or Gentoo/ArchLinux (the bare installations, not the GUI kitted ones).
Start a simultaneous tcpdump on a network tap and capture the traffic you're supposed to receive along side of your program and compare the results.
There's no efficient way to check if you've received all the packets you intended to on the receiving end since the packets might be dropped on a lower level than you anticipate.
Also this is a question for Unix # StackOverflow, there's no programming here what I can see, at least there's no code.
The only certain way to verify packet drops is to have a much more beefy sender (perhaps a farm of machines that send packets) to a single client, record every packet sent to your reciever. Have the statistical data analyzed and compared against your senders and see how much you dropped.
The cheaper way is to buy a network tap or even more ad-hoc enable port mirroring in your switch if possible. This enables you to dump as much traffic as possible into a second machine.
This will give you a more accurate result because your application machine will be busy as it is taking care of incoming traffic and processing it.
Further more, this is why network taps are effective because they split the communication up into two channels, the receiving and sending directions of your traffic if you will. This enables you to capture traffic on two separate machines (also using tcpdump, but instead of a mirrored port, you get a more accurate traffic mirroring).
So either use port mirroring
Or you buy one of these:
Is there any way in Linux to measure number of outgoing packets from a machine in a certain amount of time, lets say per second or per minute?
There are quite a few programs that can do this (most of the good ones are not standard in base distrobution). The one I highly recommend is iptraf. Another one is ntop. Other than that, writing a custom shell script to cat the output of ifconfig interface's TX packets and have it loop at a desired interval can also do the trick.
I need to gather some network statistic to test my server application. I've tried many linux tools, but nothing I've found suits my needs.
Basically I want to gather some UDP statistics (bytes/time_interval, packets/time_interval, packets_loss), but regarding only two particular hosts - for example I want to get UDP statistic from traffic going from IP_A:PORT_A to IP_B:PORT_B.
Tools like tcpdump/wireshark can easily dump such traffic but I have problems with getting statistics like temporary speed (too see throughput peeks), and linux system statistics gives me number for all traffic.
It would be better to get text output so it will be possible to parse it.
Anyone has any idea how can I achieve it?
Thanks in advance
Harnen
Here's a tutorial for the libpcap library:
http://www.systhread.net/texts/200805lpcap1.php
To determine packets lost, your program will probably want to work on a pair of logs, and make sure UDP messages on the source are found on the destination. A good method for doing this is to maintain a window of packets equal to the amount of time your timeout is set to, load all the packets into the window, sort them, then search for all the packets in the desired time frame, marking them as found as you go. Once you've exhausted a minute, remove half of that minute from the buffer, and load the next thirty seconds and re-sort.
If you have lots (millions? probably should profile it) of packets, it may be faster to use what's called a Counting Bloom Filter, so you can determine if your packet is "probably" in there very quickly.
If you weren't looking for programming advice, take your question to serverfault.