I have several questions regarding a Ubuntu machine which I am working on.
First of all here is the output of max open files:
$ cat /proc/sys/fs/file-max
1024000
However when checking how many are currently open:
$ lsof | wc -l
2002129
How is this number so much higer than the max?
Second, I had a look at what these open files were. 99% of them are these enteries:
ruby2.7 2749468 2750970 worker-14 fluentd 62u IPv4 1356781066 0t0 TCP {URL REDACTED}:5044->ip-10-153-7-71.eu-central-1.compute.internal:13637 (CLOSE_WAIT)
There is a mix of CLOSE_WAIT and ESTABLISHED connections. But when I check with netstat or ss it does not show that many connections:
ss -a | wc -l
1447
What else can I do to investigate why fluentd is rejecting connections and the machine is killing connections which cause the log delivery to fail?
As a final example, this one is a little crude. Listing fd's in /proc doesnt show as many as how many lsof shows:
ls -l /proc/*/fd | wc -l
1109
Related
I am not able to find an answer to a simple thing I will try to achive:
once a tcp connection is established to my linux server, let's say ssh / tcp 22 or x11 / tcp 6000 display -> how do I close this connection without killing the process (sshd / x11 display server).
I saw also some suggestoin to use iptables, but it does not work for me, the connection is still visible in netstat -an.
would be good if someone can point me to the right direction.
what I tried so far
tcpkill: kills the process, not good for me
iptables: does not close the established connection, but prevent further connections.
Thanks in adavnce
DJ
Ok, I found at least one solution (killcx) which is working. Maybe we will be able to find an easier solution.
Also, i saw the comment from "zb" - thanks - which might also work, but I was not able to find a working syntax, since this tool seems to be really useful but complex.
So here is an example how to work with the 1. solution which is working for me:
netstat -anp | grep 22
output: tcp 0 0 192.168.0.82:22 192.168.0.77:33597 VERBUNDEN 25258/0
iptables -A INPUT -j DROP -s 192.168.0.77 (to prevent reconnect)
perl killcx.pl 192.168.0.77:33597 (to kill the tcp connection)
killcx can be found here: http://killcx.sourceforge.net/
it "steals" the connection from the foreign host (192.168.0.77) and close it. So that solution is working fine, but to complex to setup quickly if you are under stress. Here are the required packages:
apt-get install libnetpacket-perl libnet-pcap-perl libnet-rawip-perl
wget http://killcx.sourceforge.net/killcx.txt -O killcx.pl
however, would be good to have an easier solution.
tcpkill wont work, since it will only kill any new connection, it doesnt kill existing ESTABLISHED connections
heres how you remove an Established TCP connection
find the PID of the process and the IP of the client connecting,
lets say you are on serverA and someone is connecting from serverB
root#A> netstat -tulpan | grep ssh | grep serverB
should see something like,
tcp 0 0 <serverA IP>:<port> <serverB>:<port> ESTABLISHED 221955/sshd
use lsof utility to get the File Descriptor of this connection using the parent PID
root#A> lsof -np 221995 | grep serverB IP
should see something like this
sshd 221955 <user> 17u IPv4 2857516568 0t0 TCP <serverA IP>:<port>-><serverB IP>:<port> (ESTABLISHED)
get the File Descriptor number (4th column) = 17u
use GDB to shut down this connection, w/out killing sshd
root#A> gdb -p 211955 --batch -ex 'call shutdown(17u, 2)'
should see something similar,
0x00007f0b138c0b40 in __read_nocancel () from /usr/lib64/libc.so.6
$1 = 0
[Inferior 1 (process 211955) detached]
that TCP connection should now be closed
I have a relatively high traffic linux/apache webserver running Wordpress (oh the headaches). I think our developer configured the memcache settings incorrectly because when I run this command to look at all incoming httpd connections.
sudo netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
I get:
1 68.106.x.x
1 74.125.x.x
1 74.125.x.x
1 74.125.x.x
1 74.125.x.x
15 0.0.0.0
70 173.0.x.x
194 127.0.0.1
...I see that I have 194 connections from 127.0.0.1, and VERY few from actual public IP's. looking at netstat further I can see those are going to port 11211 (memcache). Even if I restart httpd, it only takes a few seconds for the open memcached connections from 127.0.0.1 to skyrocket up again and almost immediately we are pushing our max httpd process limit (currently MaxClients = 105).
Here are the details for those connections:
tcp 0 0 127.0.0.1:26210 127.0.0.1:11211 ESTABLISHED -
cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""
We have an linux application (we don't have the source) that seems to be hanging. The socket between the two processes is reported as ESTABLISHED, and there is some data in the kernel socket buffer (although nowhere near the configured 16M via wmem/rmem). Both ends of the socket seem to be stuck on a sendto().
Below is some investigation using netstat/lsof and strace:
HOST A (10.152.20.28)
[root#hosta ~]# lsof -n -u df01 | grep 12959 | grep 12u
q 12959 df01 12u IPv4 4398449 TCP 10.152.20.28:38521->10.152.20.29:gsigatekeeper (ESTABLISHED)
[root#hosta ~]# netstat -anp | grep 38521
tcp 268754 90712 10.152.20.28:38521 10.152.20.29:2119 ESTABLISHED 12959/q
[root#hosta ~]# strace -p 12959
Process 12959 attached - interrupt to quit
sendto(12, "sometext\0somecode\0More\0exJKsss"..., 542, 0, NULL, 0 <unfinished ...>
Process 12959 detached
[root#hosta~]#
HOST B (10.152.20.29)
[root#hostb ~]# netstat -anp | grep 38521
tcp 72858 110472 10.152.20.29:2119 10.152.20.28:38521 ESTABLISHED 25512/q
[root#hostb ~]# lsof -n -u df01 | grep 38521
q 25512 df01 14u IPv4 6456715 TCP 10.152.20.29:gsigatekeeper->10.152.20.28:38521 (ESTABLISHED)
[root#hostb ~]# strace -p 25512
Process 25512 attached - interrupt to quit
sendto(14, "\0\10\0\0\0Owner\0sym\0Type\0Ctpy\0Time\0Lo"..., 207, 0, NULL, 0 <unfinished ...>
Process 25512 detached
[root#hostb~]#
We have upgraded the NIC driver to the latest and greatest. The systems are running RHEL 5.6 x64 (2.6.18-238.el5), I have checked the eratta for RHEL 5.7 and 5.8 but I can see no mention of bugs with the bnx2 driver or the kernel.
Does anyone have any ideas of how to debug this further?
Is either side actually reading? If not, it could be that both sides' receive buffers are full, leading to not sending data (due to the receive window being filled), leading to both send buffers being filled, which will cause sendto to block. (It's possible that this could happen despite your setting of wmem/rmem if the application is setting the SO_RCVBUF and SO_SNDBUF socket options.)
To debug this, I'd synchronize both machine's clocks, then run both applications under strace with the -e trace=network and -tt options, so you can compare the logs and see if the application isn't reading.
You could also use a network analyzer (such as Wireshark) to determine if the TCP receive window gets stuck on 0.
If this is the case, you could probably work around this by creating a small caching proxy, which would recv/send from both sides, buffering whatever can't be sent at the time.
If I do a
sudo netstat -tup | awk '{print $5}' | sort | uniq -c | sort -n
I can get a list and count of how many connections I have to each host. But how can I tell how many times I've been connecting to that host ever minute/hour/day? Is there a TCP log somewhere that I can grep through to see how and when I'm connecting to what?
I'm guessing that there is no such log because it would be massive in a short period of time. How would I create such a log with just my desired information? I don't think that using netstat would work for this since it only lists the connections at a give time. without timestamps, I don't have a reliable way to count the connections.
iptables -A OUTPUT -m state --state NEW -j LOG --log-prefix "New connection: "
This will log all new outgoing connections, not just TCP. If you only want TCP, add:
-p tcp
You can grep 'New connection: ' out of syslog.
IPTables can be used to log connections.
http://linuxdynasty.org/59/how-to-log-in-iptables/
Is there a way to tie a network connection to a PID (process ID) without forking to lsof or netstat?
Currently lsof is being used to poll what connections belong which process ID. However lsof or netstat can be quite expensive on a busy host and would like to avoid having to fork to these tools.
Is there someplace similar to /proc/$pid where one can look to find this information? I know what the network connections are by examining /proc/net but can't figure out how to tie this back to a pid. Over in /proc/$pid, there doesn't seem to be any network information.
The target hosts are Linux 2.4 and Solaris 8 to 10. If possible, a solution in Perl, but am willing to do C/C++.
additional notes:
I would like to emphasize the goal here is to tie a network connection to a PID. Getting one or the other is trivial, but putting the two together in a low cost manner appears to be difficult. Thanks for the answers to so far!
I don't know how often you need to poll, or what you mean with "expensive", but with the right options both netstat and lsof run a lot faster than in the default configuration.
Examples:
netstat -ltn
shows only listening tcp sockets, and omits the (slow) name resolution that is on by default.
lsof -b -n -i4tcp:80
omits all blocking operations, name resolution, and limits the selection to IPv4 tcp sockets on port 80.
On Solaris you can use pfiles(1) to do this:
# ps -fp 308
UID PID PPID C STIME TTY TIME CMD
root 308 255 0 22:44:07 ? 0:00 /usr/lib/ssh/sshd
# pfiles 308 | egrep 'S_IFSOCK|sockname: '
6: S_IFSOCK mode:0666 dev:326,0 ino:3255 uid:0 gid:0 size:0
sockname: AF_INET 192.168.1.30 port: 22
For Linux, this is more complex (gruesome):
# pgrep sshd
3155
# ls -l /proc/3155/fd | fgrep socket
lrwx------ 1 root root 64 May 22 23:04 3 -> socket:[7529]
# fgrep 7529 /proc/3155/net/tcp
6: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7529 1 f5baa8a0 300 0 0 2 -1
00000000:0016 is 0.0.0.0:22. Here's the equivalent output from netstat -a:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
Why don't you look at the source code of netstat and see how it get's the information? It's open source.
For Linux, have a look at the /proc/net directory
(for example, cat /proc/net/tcp lists your tcp connections). Not sure about Solaris.
Some more information here.
I guess netstat basically uses this exact same information so i don't know if you will be able to speed it up a whole lot. Be sure to try the netstat '-an' flags to NOT resolve ip-adresses to hostnames realtime (as this can take a lot of time due to dns queries).
The easiest thing to do is
strace -f netstat -na
On Linux (I don't know about Solaris). This will give you a log of all of the system calls made. It's a lot of output, some of which will be relevant. Take a look at the files in the /proc file system that it's opening. This should lead you to how netstat does it. Indecently, ltrace will allow you to do the same thing through the c library. Not useful for you in this instance, but it can be useful in other circumstances.
If it's not clear from that, then take a look at the source.
Take a look at these answers which thoroughly explore the options available:
How I can get ports associated to the application that opened them?
How to do like "netstat -p", but faster?