How to parse netstat command to get the send-q number from the line - linux

I have this output from netstat -naputeo:
tcp 0 0 :::44500 :::* LISTEN 2000 773788772 18117/java off (0.00/0/0)
tcp 0 0 :::22 :::* LISTEN 0 9419 4186/sshd off (0.00/0/0)
tcp 0 0 ::ffff:127.0.0.1:61666 ::ffff:127.0.0.1:43940 ESTABLISHED 2000 788032760 18122/java off (0.00/0/0)
tcp 0 0 ::ffff:192.168.1.202:56510 ::ffff:192.168.1.202:3000 ESTABLISHED 0 791652028 6804/java_ndsagent keepalive (7185.05/0/0)
tcp 0 0 ::ffff:192.168.1.202:56509 ::ffff:192.168.1.202:3000 TIME_WAIT 0 0 - timewait (41.13/0/0)
tcp 0 0 ::ffff:192.168.1.202:56508 ::ffff:192.168.1.202:3000 TIME_WAIT 0 0 - timewait (21.13/0/0)
tcp 0 4656 ::ffff:192.168.1.202:22 ::ffff:84.208.36.125:48507 ESTABLISHED 0 791474860 24141/1 on (0.19/0/0)
tcp 0 0 ::ffff:127.0.0.1:61616 ::ffff:127.0.0.1:45121 ESTABLISHED 2000 788032761 18117/java off (0.00/0/0)
tcp 0 0 ::ffff:192.168.1.202:3000 ::ffff:192.168.1.202:56510 ESTABLISHED 0 791651217 8044/rmiregistry off (0.00/0/0)
The Send-Q is the 3rd field, here the offender is port 22 and 4656KB.
The problem is that i need to output that specific line and that number/port/process to an output file [only if it is above 4000, that will be sent to my inbox and alert me.
I have seen similar answers but I can't extract the line using those suggestions. I don't know what process will be filling the Q but I know the ports. It's not just the 22 it could be more at any giving time.
I tried:
netstat -naputeo | awk '$3 == 0 && $4 ~ /[^0-9]22$/'
But that gives me the wrong line. [that is the :::22]
netstat -naputeo | awk '{if(($3)>0) print $3;}'
That is all wrong because it somehow produces all the lines of that field.
All I need is that number and line sent to a csv and that's all. I can deal with error checking later and maybe refine it.
Any suggestions??
Used this and it worked for now but there is room for improvement
filterQs() {
while read recv send address pid_program; do
ip=${address%%:*}
port=${address##*:}
pid=${pid_program%%/*}
program=${pid_program#*/}
echo "recv=${recv} send=${send} ip=${ip} port=${port} pid=${pid} program=${program}"
if [[ ${port} -eq 35487|| ${port} -eq 65485|| ${port} -eq CalorisPort || ${port} -eq 22 ]]
then
echo "recv=${recv} send=${send} ip=${ip} port=${port} pid=${pid} program=${program}" >> Qmonitor.txt
fi
done < <(netstat -napute 2>/dev/null | awk '$1 ~ /^(tcp|udp)/ && ($2 > 500 || $3 > 500) { print $2, $3, $4, $9 }')
}
Thanks all

Something like
$ netstat -naputeo 2>/dev/null | awk -v OFS=';' '$1 ~ /^tcp/ && $3 > 4000 { sub(/^.+:/, "", $4); print $3, $4, $9 }'
?
That would output the 3rd column (Send-Q), the port part of the 4th column (Local Address) and the 9th column (PID/Program name) if Send-Q > 4000, separated by semicolons so you can pipe it into your CSV.
E.g. (for Send-Q > 0 on my box)
$ netstat -naputeo 2>/dev/null | awk -v OFS=';' '$1 ~ /^tcp/ && $3 > 0 { sub(/^.+:/, "", $4); print $3, $4, $9 }'
52;22;4363/sshd:
EDIT:
If you really need to further process the values in bash, then you can just print the respective columns via awk and iterate over the lines like this:
#!/bin/bash
while read recv send address pid_program; do
ip=${address%%:*}
port=${address##*:}
pid=${pid_program%%/*}
program=${pid_program#*/}
echo "recv=${recv} send=${send} ip=${ip} port=${port} pid=${pid} program=${program}"
# do stuff here
done < <(netstat -naputeo 2>/dev/null | awk '$1 ~ /^(tcp|udp)/ && ($2 > 4000 || $3 > 4000) { print $2, $3, $4, $9 }')
E.g.:
$ ./t.sh
recv=0 send=52 ip=x.x.x.x port=22 pid=12345 program=sshd:
Note: I don't understand why you need the -o switch to netstat since you don't seem to be interested in the timers output, so you could probably drop that.

Try this:
netstat -naputeo | awk '{ if (($3 + 0) >= 4000) { sub(/.*:/, "", $4); print $3, $4, $9;} }'
This filters out the header line, and extracts the port number from the field $4.

Pure bash solution:
#!/bin/bash
filterHuge() {
while read -r -a line; do
if (( line[2] > 4000 )) && [[ ${line[3]##*:} == '22' ]]; then # if Send-Q is higher than 4000 and port number is 22
echo "Size: ${line[2]} Whole line: ${line[#]}"
fi
done
}
netstat -naputeo | filterHuge

I have a lineage2 server and have some problems with sent-q
I use your script and ....:
Size: 84509 Whole line: tcp 0 84509 144.217.255.80:6254 179.7.212.0:35176 ESTABLISHED 0 480806 2286/java on (46.42/11/0)
Size: 12130 Whole line: tcp 0 12130 144.217.255.80:6254 200.120.203.238:52295 ESTABLISHED 0 410043 2286/java on (0.69/0/0)
Size: 13774 Whole line: tcp 0 13774 144.217.255.80:6254 190.30.75.253:63749 ESTABLISHED 0 469361 2286/java on (0.76/0/0)
Size: 12319 Whole line: tcp 0 12319 144.217.255.80:6254 200.120.203.238:52389 ESTABLISHED 0 487569 2286/java on (0.37/0/0)
Size: 9800 Whole line: tcp 0 9800 144.217.255.80:6254 186.141.200.7:63572 ESTABLISHED 0 478974 2286/java on (0.38/0/0)
Size: 12150 Whole line: tcp 0 12150 144.217.255.80:6254 200.120.203.238:52298 ESTABLISHED 0 410128 2286/java on (0.26/0/0)
Size: 9626 Whole line: tcp 0 9626 144.217.255.80:6254 186.141.200.7:63569 ESTABLISHED 0 482721 2286/java on (0.44/0/0)
Size: 11443 Whole line: tcp 0 11443 144.217.255.80:6254 200.120.203.238:52291 ESTABLISHED 0 411061 2286/java on (0.89/0/0)
Size: 79254 Whole line: tcp 0 79254 144.217.255.80:6254 179.7.212.0:6014 ESTABLISHED 0 501998 2286/java on (89.42/10/0)
Size: 10722 Whole line: tcp 0 10722 144.217.255.80:6254 179.7.111.208:12925 ESTABLISHED 0 488352 2286/java on (0.23/0/0)
Size: 126708 Whole line: tcp 0 126708 144.217.255.80:6254 190.11.106.181:3481 ESTABLISHED 0 487867 2286/java on (85.32/7/0)
Problem are in one port : 6254
Which I could place for connections that are greater than 4000 in sent to the restart to 0 or dropping them

Related

How to kill all specific java processes

I have following search result coming from netstat:
netstat -nap | grep java | grep :::300*
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::30008 :::* LISTEN 81159/java
tcp6 0 0 :::30010 :::* LISTEN 81164/java
tcp6 0 0 :::30001 :::* LISTEN 81155/java
tcp6 0 0 :::30002 :::* LISTEN 81162/java
tcp6 0 0 :::30003 :::* LISTEN 81156/java
tcp6 0 0 :::30004 :::* LISTEN 81161/java
tcp6 0 0 :::30005 :::* LISTEN 81158/java
tcp6 0 0 :::30006 :::* LISTEN 81157/java
tcp6 0 0 :::30007 :::* LISTEN 81160/java
now I need to iterate over that result, get the process id ad kill it. How can I do it ?
PIDS=( $(netstat -nap | grep java | grep :::300*| awk '{print $6}' | cut -d/ -f2 | xargs) ); for p in "${PIDS[#]}"; do kill -9 ${p}; done
i think you can do like this but i think can be more easy way
It's not good code, but the following does what you asked for (newlines added for clarity, would be equally correct without them):
kill $(netstat -nap | awk '
/java/ && /:::300[[:digit:]]{2}/ {
gsub(/\/.*$/, "", $6);
gsub(/[^[:digit:]], "", $6);
print $6
}
')
The second gsub is a safety measure, to ensure that only digits are printed; this reduces the chances of shell globbing or string splitting having unintended effects should the column ordering be off.
Consider using systemd or another process supervision framework to manage services, instead of doing this kind of thing with hand-written code.

How to get process/PID responsible for creating outbound connection request (linux) (command line)

I've got an alert from my firewall that a Debian virtual machine I have tried to download a miner virus.
tcpdump shows every minute it reaching out to:
07:55:01.379558 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 7300
07:55:01.379566 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.379576 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 2920
07:55:01.379584 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.379593 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 5840
07:55:01.379601 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.379609 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 8760
07:55:01.379617 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.379657 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 7300
07:55:01.379669 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.379680 IP 185.191.32.198.80 > 192.168.1.205.49126: tcp 4380
07:55:01.380974 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:55:01.381264 IP 192.168.1.205.49126 > 185.191.32.198.80: tcp 0
07:56:01.900223 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.900517 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 0
07:56:01.900553 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.900826 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 146
07:56:01.900967 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 0
07:56:01.901642 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 2920
07:56:01.901667 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901684 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 4380
07:56:01.901696 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901705 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 4380
07:56:01.901714 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901725 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 2920
07:56:01.901738 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901814 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 5840
07:56:01.901835 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901848 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 8760
07:56:01.901858 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901868 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 2920
07:56:01.901880 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901891 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 4380
07:56:01.901905 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901915 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 2920
07:56:01.901922 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901932 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 5840
07:56:01.901939 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.901949 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 2920
07:56:01.901955 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.902010 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.902039 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 4380
07:56:01.902065 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:56:01.902076 IP 185.191.32.198.80 > 192.168.1.205.49128: tcp 4380
07:56:01.902084 IP 192.168.1.205.49128 > 185.191.32.198.80: tcp 0
07:57:01.909829 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.910130 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 0
07:57:01.910157 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.910245 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 146
07:57:01.910375 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 0
07:57:01.911050 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 2920
07:57:01.911076 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.911096 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 4380
07:57:01.911108 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.911120 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 4380
07:57:01.911130 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.911141 IP 185.191.32.198.80 > 192.168.1.205.49130: tcp 2920
07:57:01.911414 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
07:57:01.911507 IP 192.168.1.205.49130 > 185.191.32.198.80: tcp 0
When I look at the firewall logs I can see it reaching out to: http://185.191.32.198/lr.sh
I can block it via the firewall, but what I'm interested in is understanding which PROCESS on my server is doing such queries, as these are outbound queries. So there is some kind of exploit or virus reaching out from the server to try and download this script.
I've tried various netstat and lsof commands I've found on here, but they don't catch the traffic when it's actually happening, they just dump out and so no active connections. Also, bear in mind, I have no local ports actively listening, these new outbound requests once a minute.
So how would one set something up to see which process / PID is making these outbound requests each minute?
netstat can be used in continuous mode with the "-p" option to log the process initiating the connections, as described here: https://unix.stackexchange.com/questions/56453/how-can-i-monitor-all-outgoing-requests-connections-from-my-machine
Use the following command to log the connection attempts and pinpoint the initiating process:
sudo netstat -nputwc | grep 185.191.32.198 | tee /tmp/nstat.txt
Interrupt with Ctrl-C when you think the connection was logged.
less /tmp/nstat.txt
Then you can analyze the <PID> (replace with the pid of the process), its environment and threads with ps:
sudo ps -ef | grep <PID>
sudo ps eww <PID>
sudo ps -T <PID>
Using mbax's & Dude Boy input you could do this:
#!/bin/bash
while true
do
PID=$(netstat -nputw | grep 185.191.32.198)
if [ $? -ne 0 ]; then
:
else
ps -ajxf
echo "PID: ${PID}"
exit
fi
done
As a oneliner:
while true; do PID=$(netstat -nputw | grep 185.191.32.198); if [ $? -ne 0 ]; then :; else ps -ajxf; echo "PID: ${PID}"; break; fi; done
Edit: The original while timer 0.1 did not detect every attempt I tested, 0.01 did.
Edit 2: Using true uses up to 2% CPU, worth it when hunting ;)
Suggesting to research your problem using nethogs traffic monitoring tool. https://www.geeksforgeeks.org/linux-monitoring-network-traffic-with-nethogs/
It might take a while to catch the offending process .
And even if you catch it, it is possible the offending process is a transient vanishing script/program that is recreated with random names.
If your system is infected, then probably you will identify the infection is applied on a legitimated process or service.
Suggesting to scan your system with anti-virus as well.

Which PID is using a PORT inside a k8s pod without net tools

Sorry about the long question post, but I think it can be useful to others to learn how this works.
What I know:
On any linux host (not using docker container), I can look at /proc/net/tcp to extract information tcp socket related.
So, I can detect the ports in LISTEN state with:
cat /proc/net/tcp |
grep " 0A " |
sed 's/^[^:]*: \(..\)\(..\)\(..\)\(..\):\(....\).*/echo $((0x\4)).$((0x\3)).$((0x\2)).$((0x\1)):$((0x\5))/g' |
bash
Results:
0.0.0.0:111
10.174.109.1:53
127.0.0.53:53
0.0.0.0:22
127.0.0.1:631
0.0.0.0:8000
/proc/net/tcp gives UID, GID, unfortunately does not provides the PID. But returns the inode. That I can use to discover the PID using it as file descriptor.
So one way is to search /proc looking for the inode socket. It's slow, but works on host:
cat /proc/net/tcp |
grep " 0A " |
sed 's/^[^:]*: \(..\)\(..\)\(..\)\(..\):\(....\).\{72\}\([^ ]*\).*/echo $((0x\4)).$((0x\3)).$((0x\2)).$((0x\1)):$((0x\5))\\\t$(find \/proc\/ -type d -name fd 2>\/dev\/null \| while read f\; do ls -l $f 2>\/dev\/null \| grep -q \6 \&\& echo $f; done)/g' |
bash
output:
0.0.0.0:111 /proc/1/task/1/fd /proc/1/fd /proc/924/task/924/fd /proc/924/fd
10.174.109.1:53 /proc/23189/task/23189/fd /proc/23189/fd
127.0.0.53:53 /proc/923/task/923/fd /proc/923/fd
0.0.0.0:22 /proc/1194/task/1194/fd /proc/1194/fd
127.0.0.1:631 /proc/13921/task/13921/fd /proc/13921/fd
0.0.0.0:8000 /proc/23122/task/23122/fd /proc/23122/fd
Permission tip 1: You will only see what you have permission to look at.
Permission tip 2: fake root used in containers does not have access to all file descriptors in /proc/*/fd. You need to query it for each user.
If you run as normal user the results are:
0.0.0.0:111
10.174.109.1:53
127.0.0.53:53
0.0.0.0:22
127.0.0.1:631
0.0.0.0:8000 /proc/23122/task/23122/fd /proc/23122/fd
Using unshare to isolate environment it works as expected:
$ unshare -r --fork --pid unshare -r --fork --pid --mount-proc -n bash
# ps -fe
UID PID PPID C STIME TTY TIME CMD
root 1 0 2 07:19 pts/6 00:00:00 bash
root 100 1 0 07:19 pts/6 00:00:00 ps -fe
# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
# python -m SimpleHTTPServer &
[1] 152
# Serving HTTP on 0.0.0.0 port 8000 ...
netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 152/python
# cat /proc/net/tcp |
> grep " 0A " |
> sed 's/^[^:]*: \(..\)\(..\)\(..\)\(..\):\(....\).\{72\}\([^ ]*\).*/echo $((0x\4)).$((0x\3)).$((0x\2)).$((0x\1)):$((0x\5))\\\t$(find \/proc\/ -type d -name fd 2>\/dev\/null \| while read f\; do ls -l $f 2>\/dev\/null \| grep -q \6 \&\& echo $f; done)/g' |
> bash
0.0.0.0:8000 /proc/152/task/152/fd /proc/152/fd
# ls -l /proc/152/fd
total 0
lrwx------ 1 root root 64 mai 25 07:20 0 -> /dev/pts/6
lrwx------ 1 root root 64 mai 25 07:20 1 -> /dev/pts/6
lrwx------ 1 root root 64 mai 25 07:20 2 -> /dev/pts/6
lrwx------ 1 root root 64 mai 25 07:20 3 -> 'socket:[52409024]'
lr-x------ 1 root root 64 mai 25 07:20 7 -> /dev/urandom
# cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:1F40 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 52409024 1 0000000000000000 100 0 0 10 0
Inside a docker container in my host, it seems to work in same way.
The problem:
I have a container inside a kubernetes pod running jitsi. Inside this container, I am unable to get the PID of the service listening the ports.
Nor after installing netstat:
root#jitsi-586cb55594-kfz6m:/# netstat -ntpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5222 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5269 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5347 0.0.0.0:* LISTEN -
tcp6 0 0 :::5222 :::* LISTEN -
tcp6 0 0 :::5269 :::* LISTEN -
tcp6 0 0 :::5280 :::* LISTEN -
# ps -fe
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May22 ? 00:00:00 s6-svscan -t0 /var/run/s6/services
root 32 1 0 May22 ? 00:00:00 s6-supervise s6-fdholderd
root 199 1 0 May22 ? 00:00:00 s6-supervise jicofo
jicofo 203 199 0 May22 ? 00:04:17 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/ -Dnet.java.sip.communicator.SC_HOME_DIR_NAME=config -Djava
root 5990 0 0 09:48 pts/2 00:00:00 bash
root 10926 5990 0 09:57 pts/2 00:00:00 ps -fe
Finally the Questions:
a) Why can't I read the file descriptors of the proccess listening port 5222 ?
root#jitsi-586cb55594-kfz6m:/# cat /proc/net/tcp | grep " 0A "
0: 00000000:1466 00000000:0000 0A 00000000:00000000 00:00000000 00000000 101 0 244887827 1 ffff9bd749145800 100 0 0 10 0
...
root#jitsi-586cb55594-kfz6m:/# echo $(( 0x1466 ))
5222
root#jitsi-586cb55594-kfz6m:/# ls -l /proc/*/fd/* 2>/dev/null | grep 244887827
root#jitsi-586cb55594-kfz6m:/# echo $?
1
root#jitsi-586cb55594-kfz6m:/# su - svc
svc#jitsi-586cb55594-kfz6m:~$ id -u
101
svc#jitsi-586cb55594-kfz6m:~$ ls -l /proc/*/fd/* 2>/dev/null | grep 244887827
svc#jitsi-586cb55594-kfz6m:~$ echo $?
1
b) There is another way to list inode and link it to a pid without searching /proc/*/fd ?
Update 1:
Based on Anton Kostenko tip, I looked to AppArmor. It's not the case because the server don't use AppArmor, but searching, took me to SELinux.
In a ubuntu machine where AppArmor is running, I got:
$ sudo apparmor_status | grep dock
docker-default
In the OKE(Oracle Kubernetes Engine, my case) node there is no AppArmor. I got SELinux instead:
$ man selinuxenabled | grep EXIT -A1
EXIT STATUS
It exits with status 0 if SELinux is enabled and 1 if it is not enabled.
$ selinuxenabled && echo $?
0
Now, I do believe that SELinux is blocking the /proc/*/fd listing from root inside the container. But I don't know yet how to unlock it.
References:
https://jvns.ca/blog/2016/10/10/what-even-is-a-container/
The issue is solved by adding the POSIX capability: CAP_SYS_PTRACE
I'm my case the container are under kubernetes orchestration.
this reference explains about kubectl and POSIX Capabilities
So I have
root#jitsi-55584f98bf-6cwpn:/# cat /proc/1/status | grep Cap
CapInh: 00000000a80425fb
CapPrm: 00000000a80425fb
CapEff: 00000000a80425fb
CapBnd: 00000000a80425fb
CapAmb: 0000000000000000
So I careful read the POSIX Capabilities Manual. But even adding CAP_SYS_ADMIN, the PID does not appear on netstat. So I tested all capabilities. CAP_SYS_PTRACE is The Chosen One
root#jitsi-65c6b5d4f7-r546h:/# cat /proc/1/status | grep Cap
CapInh: 00000000a80c25fb
CapPrm: 00000000a80c25fb
CapEff: 00000000a80c25fb
CapBnd: 00000000a80c25fb
CapAmb: 0000000000000000
So here my deployment spec change:
...
spec:
...
template:
...
spec:
...
containers:
...
securityContext:
capabilities:
add:
- SYS_PTRACE
...
Yet I don't know what security reasons selinux use to do it. But for now it's good enough for me.
References:
https://man7.org/linux/man-pages/man7/capabilities.7.html
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Line manipulation & sorting

I am alright at writing Linux scripts but could use some advice. I know the problem is sort of vague, so if you can provide any help whatsoever I will appreciate it!
The following issue is for personal growth, and because I am writing some network tools for fun/learning. No homework involved (I'm a senior in college, none of my classes require this stuff!)
I am using tshark to get information about packet captures. This is what it looks like:
rachel#Ubuntu-1:~/PCAP$ tshark -r LargeTorrent.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
eth frames:4309 bytes:3984321
ip frames:4119 bytes:3969006
icmp frames:1316 bytes:1308988
udp frames:1408 bytes:1350786
data frames:1368 bytes:1346228
dns frames:16 bytes:1176
nbns frames:14 bytes:1300
http frames:8 bytes:1596
nbdgm frames:2 bytes:486
smb frames:2 bytes:486
mailslot frames:2 bytes:486
browser frames:2 bytes:486
tcp frames:1395 bytes:1309232
data frames:1300 bytes:1294800
http frames:6 bytes:3763
data-text-lines frames:2 bytes:324
xml frames:2 bytes:3205
tcp.segments frames:1 bytes:787
nbss frames:34 bytes:5863
smb frames:17 bytes:3047
pipe frames:4 bytes:686
lanman frames:4 bytes:686
smb2 frames:13 bytes:2444
bittorrent frames:10 bytes:1709
tcp.segments frames:2 bytes:433
bittorrent frames:2 bytes:433
bittorrent frames:1 bytes:258
bittorrent frames:2 bytes:221
bittorrent frames:2 bytes:221
arp frames:146 bytes:8760
ipv6 frames:44 bytes:6555
udp frames:40 bytes:6211
dns frames:18 bytes:1711
dhcpv6 frames:14 bytes:2114
http frames:6 bytes:1014
data frames:2 bytes:1372
icmpv6 frames:4 bytes:344
===================================================================
What I would like for it to look like:
rachel#Ubuntu-1:~/PCAP$ tshark -r LargeTorrent.pcap -q -z io,phs
===================================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
=====================================
eth 984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
===================================================================
Edit: I am going to add the original question for the purpose of making sense of the (great) answer that was provided.
Originally, I wanted to only print statistics for "leaves" because eth, ip, etc. are all parents and their statistics are not necessary for my purposes. In addition, instead of having a god-awful block of text with only spaces to show hierarchy, I wanted to erase all the statistics for parents, and show them as breadcrumbs behind the child.
Example:
eth frames:4309 bytes:3984321
ip frames:4119 bytes:3969006
icmp frames:1316 bytes:1308988
udp frames:1408 bytes:1350786
data frames:1368 bytes:1346228
dns frames:16 bytes:1176
Should become
eth:ip:icmp - 1308988 bytes
eth:ip:udp:data - 1346228 bytes
eth:ip:udp:dns - 1176 bytes
To preserve the hierarchy and avoid printing useless statistics.
Anyway, the approved answer by Etan solved this perfectly! And for those of you who are on my level who are unsure of how to proceed after this answer, this will help you finish up:
Save the given script as a filename.awk file
Save the block of text you want to manipulate as a filename.txt file
Call awk -f filename.awk filename.txt
Optionally pipe the output to a file ( awk -f filename.awk filename.txt >> output.txt )
The output I originally thought you wanted could be achieved with this awk script. (I think this can probably be done cleaner but this seems to work well enough.)
function entry() {
# Don't want to print empty entries.
if (ind[0]) {
printf "%s", ind[0]
for (i = 1; i <= ls; i++) {
printf ":%s", ind[i]
}
split(b, a, /:/)
printf " - %s %s\n", a[2], a[1]
}
}
# Found our data marker. Note that and print the current line.
$1 == "Filter:" {d=1; print; next}
# Print lines until we see our data marker.
!d {print; next}
# Print empty lines.
!NF {print; next}
# Save our trailing line for later.
/===/ {suf=$0; next}
{
# Save our previous indentation level.
ls = s
# Find our new indentation level (by where the first field starts).
s = (match($0, /[^[:space:]]/)-1) / 2
# If the current line is at or below the last indent level print the last line.
if (s <= ls) {
entry()
}
# Save the current line's byte count.
b=$NF
# Save the current line's field name.
ind[s] = $1
}
END {
# Print a final line if we had one.
entry()
# Print the suffix line if we have one.
if (suf) {
print suf
}
}
Which, on the sample input, gets you this output.
===================================================================
Protocol Hierarchy Statistics
Filter:
eth:ip:icmp - 1308988 bytes
eth:ip:udp:data - 1346228 bytes
eth:ip:udp:dns - 1176 bytes
eth:ip:udp:nbns - 1300 bytes
eth:ip:udp:http - 1596 bytes
eth:ip:udp:nbdgm:smb:mailslot:browser - 486 bytes
eth:ip:tcp:data - 1294800 bytes
eth:ip:tcp:http:data-text-lines - 324 bytes
eth:ip:tcp:http:xml:tcp.segments - 787 bytes
eth:ip:tcp:nbss:smb:pipe:lanman - 686 bytes
eth:ip:tcp:nbss:smb2 - 2444 bytes
eth:ip:tcp:bittorrent:tcp.segments:bittorrent:bittorrent - 258 bytes
eth:ip:tcp:bittorrent:bittorrent:bittorrent - 221 bytes
eth:arp - 8760 bytes
eth:ipv6:udp:dns - 1711 bytes
eth:ipv6:udp:dhcpv6 - 2114 bytes
eth:ipv6:udp:http - 1014 bytes
eth:ipv6:udp:data - 1372 bytes
eth:ipv6:icmpv6:data - 344 bytes
===================================================================
Output like what you edited to indicate you want is probably more easily handled with sed though.
/Filter:/a \
Protocol Bytes \
=====================================
s/frames:[^ ]*//
s/ b/b/
s/bytes:\([^ ]*\)/\1/
Which ends up with output.
===================================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
=====================================
eth 3984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
===================================================================
A simple script with sed will work as well.
$ printf "\n==========================================================\n"; printf "Protocol Hierarchy Statistics\nFilter:\n\n";printf "\nProtocol\t\t\t\t Bytes\n================================================\n" && sed -e 's/\(frames[:].*bytes[:]\)\(.*$\)/\2/' dat/tshark.txt | tail -n+4 | head -n-1 && printf "================================================\n"
broken down into script form (where dat/tshark.txt is the filename holding the tshark output):
printf "\n==========================================================\n"
printf "Protocol Hierarchy Statistics\nFilter:\n\n"
printf "\nProtocol\t\t\t\t Bytes\n================================================\n"
sed -e 's/\(frames[:].*bytes[:]\)\(.*$\)/\2/' dat/tshark.txt | tail -n+4 | head -n-1
printf "================================================\n"
Output
==========================================================
Protocol Hierarchy Statistics
Filter:
Protocol Bytes
================================================
eth 3984321
ip 3969006
icmp 1308988
udp 1350786
data 1346228
dns 1176
nbns 1300
http 1596
nbdgm 486
smb 486
mailslot 486
browser 486
tcp 1309232
data 1294800
http 3763
data-text-lines 324
xml 3205
tcp.segments 787
nbss 5863
smb 3047
pipe 686
lanman 686
smb2 2444
bittorrent 1709
tcp.segments 433
bittorrent 433
bittorrent 258
bittorrent 221
bittorrent 221
arp 8760
ipv6 6555
udp 6211
dns 1711
dhcpv6 2114
http 1014
data 1372
icmpv6 344
================================================
Formatting
Following on from your comment on how to align the bytes info given the variable length of the protocol tags, you can make use of printf to format the output as you have indicated. Like Ethan, I started working on your original question that had the tags consolidated. My initial approach was to read the different levels into different associative arrays that could be combined into what you initially specified. Doing so, I had to produce the output lined up using printf. Here is the first attempt I made working with the first 4-levels of your tshark data:
declare -i ln=0
declare -A l1 l2 l3 l4
## read each line in file and assing to associative arrays for each level
while read -r line; do
ln=${#line} # base level on length of line read
[ $ln -gt 66 ] && continue;
[ $ln -eq 66 ] && { iface="${line%% *}"; l1[${iface}]="${line##* }"; }
[ $ln -eq 64 ] && { proto="${iface}:${line%% *}"; l2[${proto}]="${line##* }"; }
[ $ln -eq 62 ] && { ptype="${proto}:${line%% *}"; l3[${ptype}]="${line##* }"; }
[ $ln -le 60 ] && { data="${ptype}:${line%% *}"; l4[${data}]="${line##* }"; }
done < "$1"
## output a summary of the file
printf "\n4-level deep summary of file '%s':\n\n" "$1"
for i in "${!l1[#]}"; do
for j in "${!l2[#]}"; do
printf " %-32s %s\n" "$j" "${l2[$j]}"
for k in "${!l3[#]}"; do
printf " %-32s %s\n" "$k" "${l3[$k]}"
for l in "${!l4[#]}"; do
[ "${l%:*}" == "$k" ] && printf " %-32s %s\n" "$l" "${l4[$l]}"
done
done
done
done
The output it produced was for example:
eth:ip frames:4119 bytes:3969006
eth:ip:udp frames:1408 bytes:1350786
eth:ip:udp:data frames:1368 bytes:1346228
eth:ip:udp:nbdgm frames:2 bytes:486
eth:ip:udp:nbns frames:14 bytes:1300
You can look at the various printf statements in the code above and see how the alignment is handled. Let me know if you have further questions.
I'm a little surprised that tshark doesn't have a JSON or machine-readable way to get the -z io,phs info, when it has so many ways to extract packet info.
I tried playing with some of the above, but bash seems to have changed over the years (or has different defaults depending on the environment). I am also not sure which shell or version of it was used to produce the above.
The line lengths/output from tshark have also changed: My debugging showed different line lengths, so the trick above using line lengths, e.g. [ $ln -gt 66 ] didn't work for me.
It seems that read -r strips out leading/trailing whitespaces. If you actually want it, you need IFS= to make it give you the spaces:
## read each line in file
while IFS= read -r line ; do
...
done
The "nested" levels associative arrays is clever, but hard to work with - it shows what rabbit holes you can go down with bash - although now when iterating through it, bash produces it in "hash" order and not the order they were added.
Since I actually needed the data in the rest of my script, the nested arrays made it particularly fiddly to deal with. Fine for printf purposes where you just print the line, but what if you actually want to get the frames count for each item and do then do something with it.
Here was my attempt that simplified it a bit. I implemented it as a bash function which gets a few other bits of info from the sample file:
TSHARK=/usr/bin/tshark
CAPINFOS=/usr/bin/capinfos
declare -A fcount
declare -A bcount
declare -A capinfo
function loadcapinfo
{
local sample=$1
local statstofile=$2
local bytes
local frames
local key
if [ ! -f "$sample" ] ; then
echo "FATAL: loadcapinfo: file does not exist: $sample"
exit 1
fi
capinfo[start_time_epoch]=$($CAPINFOS -Tr -Sa $sample | cut -f2)
capinfo[start_time]=$($CAPINFOS -Tr -a $sample | cut -f2)
capinfo[end_time_epoch]=$($CAPINFOS -Tr -Se $sample | cut -f2)
capinfo[end_time]=$($CAPINFOS -Tr -e $sample | cut -f2)
capinfo[size]=$($CAPINFOS -Tr -s $sample | cut -f2)
declare -i ln=0
while IFS= read -r line ; do
ln=${#line} # base level on length of line read
[ $ln -le 1 ] && continue;
pat=".*frames:([0-9]+)\s+bytes:([0-9]+)"
pat_1="^(\w+)"
pat_2="^\s{2}(\w+)"
pat_3="^\s{4}(\w+)"
pat_4="^\s{6}(\w+)"
ethertype="ethertype"
[[ $line =~ $pat ]] && { frames=${BASH_REMATCH[1]}; bytes=${BASH_REMATCH[2]}; } || continue;
[[ $line =~ $pat_1 ]] && { encap="${BASH_REMATCH[1]}:${ethertype}"; key="${encap}"; }
[[ $line =~ $pat_2 ]] && { proto=${BASH_REMATCH[1]}; key="${encap}:${proto}"; }
[[ $line =~ $pat_3 ]] && { ptype=${BASH_REMATCH[1]}; key="${encap}:${proto}:${ptype}"; }
[[ $line =~ $pat_4 ]] && { data=${BASH_REMATCH[1]}; key="${encap}:${proto}:${ptype}:${data}"; }
[ "$proto" = "llc" ] && { key=${key/eth:ethertype:llc/eth:llc} ; }
fcount[${key}]=${frames:=0}
bcount[${key}]=${bytes:=0}
if [ -n "$statstofile" ] ; then
echo "${capinfo[start_time_epoch]},${key},${frames},${bytes}" >> $statstofile
fi
done < <($TSHARK -qr $sample -z io,phs)
unset fcount[0]
}
Now, after this in the script, we can do:
loadcapinfo /my/sample/file.pcap /tmp/stats.txt
Optionally write the counts to a file, /tmp/stats.txt
This uses one associative array for each count, and puts other info into capinfo so now we can do things like:
echo "IPv4 Packet Count is: ${fcount[eth:ethertype:ip]}"
echo "IPv6 Packet Count is: ${fcount[eth:ethertype:ipv6]}"
echo "ARP Count is: ${fcount[eth:ethertype:arp]}"
echo "STP Count is: ${fcount[eth:llc:stp]}"
echo "Start time: ${capinfo[start_time]}"
echo "End time: ${capinfo[end_time]}"
echo "File size: ${capinfo[size]}"
I made the keys match Wireshark's frame.protocols field, which inserts some "pseudo protocol" for most things called "ethertype". This way, if you want to then iterate through the associative array to find the packet(s) in the pcap file, you can use the information to find packets with a given protocol.
tshark -r /my/sample/file.pcap -Y "frame.protocols == eth:ethertype:ip:udp:snmp" -Tfields -e frame.number -e eth.src_resolved -e eth.dst_resolved -e ip.src -e ip.dst -e frame.protocols
for i in "${!fcount[#]}"; do
tshark -r /my/sample/file.pcap -Y "frame.protocols == $i" -Tfields -e frame.number -e eth.src_resolved -e eth.dst_resolved -e ip.src -e ip.dst -e frame.protocols > /tmp/$i.txt
done

How can you obtain a queue field from netstat -i in Linux?

In Solaris, the output of 'netstat -i' gives something like the following:
root# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 136799 0 136799 0 0 0
igb0 1500 vulture vulture 1272272 0 347277 0 0 0
Note that there is a Queue field on the end.
In Linux, 'netstat -i' gives output with no Queue field:
[root#roseate ~]# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 2806170 0 0 0 791768 0 0 0 BMRU
eth1 1500 0 0 0 0 0 0 0 0 0 BMU
eth2 1500 0 0 0 0 0 0 0 0 0 BMU
eth3 1500 0 0 0 0 0 0 0 0 0 BMU
lo 16436 0 1405318 0 0 0 1405318 0 0 0 LRU
I've figured out how to get collisions in Linux by adding the -e option, but is there a way to get the Queue in Linux?
The only reference to queue I ever saw in netstat on Linux was when using -s, but that's probably too garrulous for your use-case?
$ netstat -na | awk 'BEGIN { RecvQ=0; SendQ=0; } { RecvQ+=$2; SendQ+=$3; } END { print "RecvQ " RecvQ/1024; print "SendQ " SendQ/1024; }'
RecvQ 255.882
SendQ 0.0507812
For per interface, I have dirty way
[spatel#us04 ~]$ for qw in `/sbin/ifconfig | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}'`; do echo `/sbin/ip addr | grep $qw | awk '{print $7}'` : ; echo `netstat -na | grep $qw | awk 'BEGIN { RecvQ=0; SendQ=0; } { RecvQ+=$2; SendQ+=$3; } END { print "RecvQ " RecvQ/1024; print "SendQ " SendQ/1024; }'`; done
eth0 :
RecvQ 0 SendQ 0
eth2 :
RecvQ 0.0703125 SendQ 1.56738
:
RecvQ 0 SendQ 0
I ended up using
tc -s -d qdisc
[root#roseate ~]# tc -s -d qdisc
qdisc mq 0: dev eth2 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth3 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 218041403 bytes 1358829 pkt (dropped 0, overlimits 0 requeues 1)
rate 0bit 0pps backlog 0b 0p requeues 1
qdisc mq 0: dev eth1 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
which gives backlog bytes and packets.
Source

Resources