I would like to know if there is any possibility to live capture network packets and save it to a variable for example in python.
I need some information to get from it and not to save it to a file.
I need to capture http packets and get source addres and its content, which should be a html code, to extract only text from it and then do the rest of the job on that information.
There is no reason to save every packet to a file because whole process would be more slower. I was looking for quite a long time for any tool to do this but no success.
Please, if you know any tool that could help me to do this, write about it.
I'm sure you have seen this link about a library to process PCAP files. Now the question is how to acquire in real time without storing in a file.
Probably easiest is to use a fifo
$ mkfifo /tmp/tcpdump.fifo
Now you can capture and feed data into the named fifo
$ sudo tcpdump -s0 -i eth0 -f /tmp/tcpdump.fifo tcp port 80
And in your python program you can open '/tmp/tcpdump.fifo' as the input file as per the instructions in the link.
Alternatively you can try opening '/dev/stdin' in your program and reading the data from there; you could then pipe the PCAP data straight into stdin using the shell and skipping the intermediate named fifo.
$ sudo tcpdump -s0 -i eth0 -f - tcp port 80 | ./youprogram.py
Related
I have socat creating two serial ports: ttyclient and ttyserver, ttyclient will be called by an application and I have a script listening to ttyserver. The sequence of events I need is
My socat creates the two via /usr/bin/socat -d -d PTY,link=./ttyserver,raw,echo=0 PTY,link=./ttyclient,raw,echo=0
My script writes a few bytes to ttyserver (to be read by ttyclient)
At some later point in time the application runs, reads the bytes from ttyclient and goes to work.
The problem I am having is the bytes are only readable if something is read during step 2. If I run minicom -D ./ttyclient before step 2 I see the bytes being transmitted as expected, but afterwards shows nothing. The data appears to be discarded.
Is this expected behavior? I'm unsure of how to keep the data around so when something reads it is presented with those bytes. Alternatively I'd happy if I had some way to know ttyclient has been opened and send data on that event.
The issue is that I was using Minicom to read ttyclient. For whatever reason Minicom won't display cached data like that... If I just use cat I get my desired behavior.
I've ecountered a problem using a serial GPS/GNSS device on a Raspberry Pi. The device in question is a u-blox GNSS receiver symlinked to /dev/gps.
I try to achieve logging the output data from this device and simultaneously sending correction data to it.
To be more specific, I use RTKLIBs (http://www.rtklib.com/) str2str tool for sending NTRIP/RTCM correction data to the GNSS receiver in order to get better position estimations using DGNSS/RTK.
The receiver's output data will be logged by a python script which is based on the GPS deamon (gpsd).
However, I guess the main issue is related to the serial port control.
When I run the writing process (str2str) first and afterwards any reading process (my python script/gpsd frontends (e.g. cgps) /cat) at the same time, the reading process will output data for a few seconds and freeze then. It doesn't matter which tool I use for reading the data.
I found this question: https://superuser.com/questions/488908/sharing-a-serial-port-between-two-processes. Therefore I made sure that the processes got rw access to the device and even tried running them as superuser. Furthermore I stumbled upon socat and virtual serial ports, but didn't find any use for it. (Virtual Serial Port for Linux)
Is there any way to read data from a serial port with one process and write to it with another? The only solution I know of right now might be to rewrite the read and write process in python using pySerial. This would allow to only have one process accessing the serial device, but would mean plenty of work.
Finally I found a soultion using a construction somehow similar to this: https://serverfault.com/questions/453032/socat-to-share-a-serial-link-between-multiple-processes
A first socat instance (A) gets GNSS correction data from a TCP connection, which is piped to socat B. Socat B manages the connection to the serial device and pipes output data to another socat instance C, which allows other processes such as gpsd to connect and get the receiver's output from TCP port.
In total, this looks like:
socat -d -d -d -u -lpA TCP4:127.0.0.1:10030 - 2>>log.txt |
socat -d -d -d -t3 -lpB - /dev/gps,raw 2>>log.txt|
socat -d -d -d -u -lpC - TCP4-LISTEN:10031,forever,reuseaddr,fork 2>>log.txt
With only one process managing the serial connection, it doesn't block anymore.
This is my first time using raw sockets (yes, I need to use them as I must modify a field inside a network header) and all the documentation or tutorials I read describe a solution to sniff packets but that is not exactly what I need. I need to create a script which intercepts the packet, process it and sends it further to the destination, i.e. the packets should not reach the destination unless my script decides to.
In order to learn, I created a small prototype which detects pings and just prints "PING". I would expect ping not to work as I intercept the packets and I don't include the logic to send them to its destination. However ping is working (again, it seems as it is just sniffing/mirroring packets). My goal is that the ping packets are "trapped" in my script and I don't know how to do that. This is what I do in my current python script (I avoid writing how I do the decode for simplicity)
sock = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(0x0003))
sock.bind((eth0, 0))
packet = sock.recvfrom(65565)
decode_eth(packet)
decode_ip(packet)
if (ipheader.ip_proto == 1):
print("\nPING")
Can somebody explain how can I achieve my goal or point me to the right documentation?
Your description seems to be different from what your title suggest. My understanding is that you want to receive, modify and possibly drop incoming network packets. And this is to be done on Linux. In that case I suggest you use a netfilter prerouting hook, which will make things a lot simpler (and likely more stable). Netfilter is well documented, a nice overview including information related to your requirements can be seen here. The important function to use is nf_register_hook(), read the answer to this question to get an idea of how to set things up.
I suppose that your Linux box is configured as a router (not a bridge). The packet will pass through your Linux because you have enabled IP Forwarding. So there are two solution:
Solution 1:
Disable IP Forwarding and then receive the packet from one interface and do the appropriate task (forwarding to another interface or dropping it).
Solution 2:
Use NetFilterQueue.
Install it on your Linux box (Debian/Ubuntu in my example):
apt-get install build-essential python-dev libnetfilter-queue-dev
Use iptables to send packets coming from input interface (eth0 in my example):
iptables -I INPUT -i eth0 -j NFQUEUE --queue-num 1
Run this script to handle packets forwarded to the Queue No.1 :
from netfilterqueue import NetfilterQueue
def print_and_accept(pkt):
print pkt
pkt.accept()
nfqueue = NetfilterQueue()
nfqueue.bind(1, print_and_accept)
try:
nfqueue.run()
except KeyboardInterrupt:
print
Note that pkt.drop() cause dropping the packet. Also you should accept/drop every packet.
I'm working on sending large data files between two Linux computers via a 10 Gigabit Ethernet cable and netcat with a UDP transfer, but seem to be having issues.
After running several tests, I've come to the conclusion that netcat is the issue. I've tested the UDP transfer using [UDT][1], [Tsunami-UDP]2, and a Python UDT transfer as well, and all of which have not had any packet loss issues.
On the server side, we've been doing:
cat "bigfile.txt" | pv | nc -u IP PORT
then on the client side, we've been doing:
nc -u -l PORT > "outputFile.txt"
A few things that we've noticed:
On one of the computers, regardless of whether it's the client or server, it just "hangs". That is to say, even once the transfer is complete, Linux doesn't kill the process and move to the next line in the terminal.
If we run pipe view on the receiving side as well, the incoming data rate is significantly lower than what the sending side thinks it's sending.
Running Wireshark doesn't show any packet loss.
Running the system performance monitor in Linux shows that the incoming data rate (for the receiving side) is the same as the outgoing data rate from the sending side. This is in contrast to what pipe view thinks (see #2)
We're not sure where the issue is with netcat, and if there is a way around it. Any help/insights would be greatly appreciated.
Also, for what it's worth, using netcat with a TCP transfer works fine. And, I do understand that UDP isn't known for reliability, and that packet loss should be expected, but it's the protocol we must use.
Thanks
It could well be that the sending instance is sending the data too fast for the receiving instance. Note that this can occur even if you see no drops on the receiving NIC (as you seem to be saying), because the loss can occur at OS level instead. Your OS could have its UDP buffers overflowing. Run this command:
watch -d "cat /proc/net/snmp | grep -w Udp"
To see if your RcvbufErrors field is non-zero and/or growing while your file transfer is going on.
This answer (How to send only one UDP packet with netcat?) says that nc sends one packet per line. Assuming that's true, this could lead to a significantly higher number of packets than your other transfer mechanisms. Presumably, as #Smeeheey suggested, you're running out of receive buffers on the receiving end.
To cause your sending end to exit, you can add -q 1 to the command line (exit 1 second after seeing end of file).
But there's no way that the the receiving end nc can know when the transfer is complete. This is why these other mechanisms are "protocols" -- they have mechanisms built into them to communicate the bounds of a file. Raw UDP has no concept of end of file.
Tuning the Linux networking stack is a bit complicated, as there are many components to tune to figure out where data is being dropped.
If possible/feasible, I'd recommend that you start by monitoring packet drops throughout the entire network stack. Once you've done that, you can determine where exactly packets are being dropped and then adjust tuning parameters as needed. There are a lot of different files to measure with lots of different fields. I wrote a detailed blog post about monitoring and tuning each component of the Linux networking stack from top to bottom. It's a bit difficult to summarize all the information there, but take a look, I think it can help guide you.
Shell Scripting: How to calculate the number of bytes uploaded to a remote node?
1) I have used IPTRAF command it can capture all the data but when I try to run the script from a local system using vxargs(python), it doesn't work.
2) I have also tried to use IFTOP command but it doesn't save the output to a file.
3) Also /proc/net/dev doesn't work because its captures some extra packets.
Does tcpdump command capture only the number of bytes/packets of the data load transmitted or does the data transmitted will have some headers attached to it?
I would like to capture only data packets from my running network intensive application.
I don't think this can be done trivially. See http://mutrics.iitis.pl/tracedump for a program that seems to accomplish this. However it has not been updated since 2012 and only works on 32 bit arch.
A crude solution could be to run the application through strace and then summarize the return values of the relevant syscalls like write and sendto.
Maybe not what you want, but if you run the process with specified user, let's say appuser, you can add a firewall rule which will count the traffic for you.
example:
iptables -I OUTPUT -m owner --uid-owner appuser -j ACCEPT
check how much packet and bytes sent out by the user:
iptables -vL OUTPUT |sed -n '2p;/appuser/p'