I have an application that sends HTTP GET requests and receives answers from some web server.
Is it possible to accomplish following: read the incoming data on on wire on port 80 using Linux commands (no code) and forward it to multicast group or just any upd port. Tried to do it with nc, no success.
I am aware it can be easily done on application level, I am trying to avoid unnecessary latency on bringing the data up to application level and then putting it down to wire. Time is very crucial here.
Thanks all.
You can use tcpflow to capture the actual data in a TCP stream. For example I can start:
sudo tcpflow -C -v -i eth0 "tcp src port 80"
Then for example if I run curl google.com in another window, then tcpflow dumps the TCP stream.
You can then pipe this output to socat to send it to a UDP socket.
UDP listener:
socat -u UDP-RECV:5555 STDOUT
Pipe the tcpflow output to this UPD listener:
sudo tcpflow -C -v -i eth0 "tcp src port 80" | socat STDIN UDP-SENDTO:localhost:5555
Bear in mind that TCP and UDP are quite different protocols, and thus have quite different use-cases. TCP is stream-oriented, whereas UDP is datagram-oriented. Trying to stuff a stream over a datagram-oriented socket will at best be unreliable.
Also while you may be able to get the above to work, I'm not sure how much performance gain you'll actually see. No matter what happens, TCP stream reassembly must happen and the kernel will generally do this very efficiently. Unlikely that tcpflow will do this any better than the kernel.
Related
I've been using /sys/class/net/eno1/statistics/rx_bytes and tx_bytes to gather stats on my network interface. The trouble is, that network has a device (a Silicon Dust HDHOMERUN HDTV tuner) which constantly streams UDP packets at a very high rate that I don't want to monitor. I'd like to remove that traffic from the monitor - perhaps by only looking at TCP packets.
Is there any way to separate out the TCP and UDP stats?
netstat -st gives some info but it's somewhat cryptic - just how big is a 'segment'? The MTU? The man page is silent on that.
$ netstat -st | grep 'segments received'
25449056 segments received
1683 bad segments received
$ netstat -st | grep 'segments sent out'
37860139 segments sent out
Based on this answer from serverfault. If you are using iptables you can add a rule to each of the INPUT and OUTPUT chains which will count every packet which carries TCP in the payload. It is possible that you will need to invoke every iptables command with sudo.
Create the rules:
# Match all TCP-carrying packets incoming on 'eno1' iface
iptables -I INPUT -i eno1 -p tcp
# Match all TCP-carrying packets outgoing through 'eno1' iface
iptables -I OUTPUT -o eno1 -p tcp
Afterwards, you can use iptables -nvxL INPUT or OUTPUT to be presented with the number of bytes processed by the rule:
Chain INPUT (policy ACCEPT 9387 packets, 7868103 bytes)
pkts bytes target prot opt in out source destination
10582 9874623 tcp -- eno1 * 0.0.0.0/0 0.0.0.0/0
In case you already have other rules defined it might be handy to create a separate chain entirely. This is also described in the answer i referenced, though you also want the -i and -o options in the in/out chains respectively. These allow you to filter on a single interface (use -i for INPUT and -o for OUTPUT).
iptables -N count_in # create custom chain named 'count_in'
iptables -A count_in -j RETURN # append RETURN action to chain 'count_in'
iptables -I INPUT -j count_in # insert chain at the top of chain INPUT
iptables -I count_in 1 -i eno1 -p tcp # insert rule that matches all tcp packets on eno1
# and has no action (does nothing)
iptables -nvxL count_in # list chain 'count_in' rules
I am not sure whether the "bytes" counter includes the IP header, or just the TCP segment bytes but it is still probably the closest metric to what you want to measure (TCP-only rx/tx bytes).
Additionally keep in mind that oftentimes rules defined with iptables are not actually saved and will get deleted on a system reboot. To enable them persistently on every reboot you may use the iptables-save and iptables-restore commands. To learn their usage you should probably look in your Linux distro's documentation as well as iptables manual.
Finally, AFAIK iptables is considered legacy by now and it is being slowly replaced by nftables. I myself still have iptables installed in my system by default. If you want to switch/are already using nftables, then you need to translate above commands to the syntax supported by the nft command. There is a utility called iptables-translate available which may help with this. It's purpose is to translate old iptables commands to equivalent nft commands. I mention this mostly for the sake of completeness, you should be just fine using iptables for your particular task if you have it installed.
You can use iptraf-ng.
Install with:
sudo apt install iptraf-ng
This will give you statistics per protocol (IPv4/IPv6/TCP/UDP/ICMP/...) on a specific interface:
sudo iptraf-ng -d eth0
You can also use this to have details per ports:
sudo iptraf-ng -s eth0
On a Linux-based system I need data incoming on a TCP port to be automatically redirected to other 50 local ports without going through user-space's send/recv. Each port needs to receive a copy of all incoming traffic. All ports are local to the same machine.
I've discarded the splice syscall due to the limit of one endpoint being a file. I guess that iptables is the right tool for this purpose, but I can't figure out the right syntax for this purpose. It should be something similar to:
iptables -t nat -A PREROUTING -p tcp --dport <in_port> -j REDIRECT --to-ports <out_port1>-<out_port50>
I wonder e.g. if the option -m multiport is needed.
I am not able to find an answer to a simple thing I will try to achive:
once a tcp connection is established to my linux server, let's say ssh / tcp 22 or x11 / tcp 6000 display -> how do I close this connection without killing the process (sshd / x11 display server).
I saw also some suggestoin to use iptables, but it does not work for me, the connection is still visible in netstat -an.
would be good if someone can point me to the right direction.
what I tried so far
tcpkill: kills the process, not good for me
iptables: does not close the established connection, but prevent further connections.
Thanks in adavnce
DJ
Ok, I found at least one solution (killcx) which is working. Maybe we will be able to find an easier solution.
Also, i saw the comment from "zb" - thanks - which might also work, but I was not able to find a working syntax, since this tool seems to be really useful but complex.
So here is an example how to work with the 1. solution which is working for me:
netstat -anp | grep 22
output: tcp 0 0 192.168.0.82:22 192.168.0.77:33597 VERBUNDEN 25258/0
iptables -A INPUT -j DROP -s 192.168.0.77 (to prevent reconnect)
perl killcx.pl 192.168.0.77:33597 (to kill the tcp connection)
killcx can be found here: http://killcx.sourceforge.net/
it "steals" the connection from the foreign host (192.168.0.77) and close it. So that solution is working fine, but to complex to setup quickly if you are under stress. Here are the required packages:
apt-get install libnetpacket-perl libnet-pcap-perl libnet-rawip-perl
wget http://killcx.sourceforge.net/killcx.txt -O killcx.pl
however, would be good to have an easier solution.
tcpkill wont work, since it will only kill any new connection, it doesnt kill existing ESTABLISHED connections
heres how you remove an Established TCP connection
find the PID of the process and the IP of the client connecting,
lets say you are on serverA and someone is connecting from serverB
root#A> netstat -tulpan | grep ssh | grep serverB
should see something like,
tcp 0 0 <serverA IP>:<port> <serverB>:<port> ESTABLISHED 221955/sshd
use lsof utility to get the File Descriptor of this connection using the parent PID
root#A> lsof -np 221995 | grep serverB IP
should see something like this
sshd 221955 <user> 17u IPv4 2857516568 0t0 TCP <serverA IP>:<port>-><serverB IP>:<port> (ESTABLISHED)
get the File Descriptor number (4th column) = 17u
use GDB to shut down this connection, w/out killing sshd
root#A> gdb -p 211955 --batch -ex 'call shutdown(17u, 2)'
should see something similar,
0x00007f0b138c0b40 in __read_nocancel () from /usr/lib64/libc.so.6
$1 = 0
[Inferior 1 (process 211955) detached]
that TCP connection should now be closed
I have been using wire-shark to analyse the packets of socket programs, Now i want to see the traffic of other hosts traffic, as i found that i need to use monitor mode that is only supported in Linux platform, so i tried but i couldn't capture any packets that is transferred in my network, listing as 0 packets captured.
Scenario:
I'm having a network consisting of 50+ hosts (all are powered by windows Except mine), my IP address is 192.168.1.10, when i initiate a communication between any 192.168.1.xx it showing the captured traffic.
But my requirement is to monitor the traffic of 192.168.1.21 b/w 192.168.1.22 from my host i,e. from 192.168.1.10.
1: is it possible to capture the traffic as i mentioned?
2: If it is possible then is wire-shark is right tool for it (or should i have to use differnt one)?
3: if it is not possible, then why?
Just adapt this a bit with your own filters and ips : (on local host)
ssh -l root <REMOTE HOST> tshark -w - not tcp port 22 | wireshark -k -i -
or using bash :
wireshark -k -i <(ssh -l root <REMOTE HOST> tshark -w - not tcp port 22)
You can use tcpdump instead of tshark if needed :
ssh -l root <REMOTE HOST> tcpdump -U -s0 -w - -i eth0 'port 22' |
wireshark -k -i -
You are connected to a switch which is "switching" traffic. It bases the traffic you see on your mac address. It will NOT send you traffic that is not destined to your mac address. If you want to monitor all the traffic you need to configure your switch to use a "port mirror" and plug your sniffer into that port. There is no software that you can install on your machine that will circumvent the way network switching works.
http://en.wikipedia.org/wiki/Port_mirroring
In a Linux specific way, how can I get the list of all open UDP ports and all TCP ports currently accepting connections in some interface?
The most common way is to use netstat console utility with the following flags:
netstat -plan
where:
-p : Show the PID and name of the program to which each socket belongs;
-l : Show only listening sockets;
-a : Show both listening and non-listening sockets;
-n : Show numerical addresses instead of trying to determine symbolic host, port or user names.
For additional output options and flags please check man pages man netstat. Based on your particular needs, only TCP or UDP (for example) protocol connections can be examined:
netstat -4 --tcp --udp --all
Alternatively, lsof -i might be helpful.
Most likely you are interested in the following information (special /proc filesystem):
/proc - Mount point for the proc filesystem, which gives access to kernel status information via the following files:
/proc/net/dev - device information
/proc/net/raw - raw socket information
/proc/net/tcp - TCP socket information
/proc/net/udp - UDP socket information
/proc/net/igmp - IGMP multicast information
/proc/net/unix - Unix domain socket information
/proc/net/ipx - IPX socket information
/proc/net/ax25 - AX25 socket information
/proc/net/appletalk - DDP (appletalk) socket information
/proc/net/nr - NET/ROM socket information
/proc/net/route - IP routing information
/proc/net/ax25_route - AX25 routing information
/proc/net/ipx_route - IPX routing information
/proc/net/nr_nodes - NET/ROM nodelist
/proc/net/nr_neigh - NET/ROM neighbours
/proc/net/ip_masquerade - masqueraded connections
/proc/net/snmp - statistics
List all listening processes and what port they are listening too. Running without sudo will list only user processes.
sudo ss -tlpn
explanation:
-t tcp
-l listening
-p show process
-n numeric, don't replace port numbers with their "popular" name.