I have read article https://tailscale.com/blog/case-of-spiky-file-descriptors/ where author investigates anomalies with the amount of open file descriptors.
He said that made snapshot of open files using lsof which allowed him to track lifetime of file:
01:12:17.24 tailcontr 142810 ubuntu 273u IPv6 5725049 0t0 TCP 172.31.10.244:https->1.2.3.4:57666 (ESTABLISHED)
01:13:32.17 tailcontr 142810 ubuntu 273u IPv6 5725049 0t0 TCP 172.31.10.244:https->1.2.3.4:57666 (CLOSE_WAIT)
01:14:03.22 tailcontr 142810 ubuntu 273u sock 0,8 0t0 5725049 protocol: TCPv6
01:16:53.93 (socket is removed)
What flags should be used to track lifetime of file as given in the example above?
P.S: I want to investigate similar problem with TCPv6 connections. According to lsof -u <user> I have a lot of files looking like:
01:14:03.22 tailcontr 142810 ubuntu 273u sock 0,8 0t0 5725049 protocol: TCPv6
And I want to understand where were these connections made to.
Related
I'm binding UDP socket to the INADDR_ANY (0.0.0.0) with a port. The bind would succeed, but for some reason, the binding seems to be lost after some unknown time.
I noticed that by running lsof -i 4 to check the open network fd , and saw the UDP binding disappeared after sometime.
The bind port is "mdns", i.e. 5353.
$ lsof -i 4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
foofoo 3642 pi 34u IPv4 1903802 0t0 UDP *:mdns
foofoo 3642 pi 35u IPv4 1907783 0t0 UDP *:47531
after a while,
$ lsof -i 4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
foofoo 3642 pi 35u IPv4 1907783 0t0 UDP *:47531
AFAIK, the code did not close the socket for "mdns" binding. Is there any case this could happen?
Thanks.
I'm running a server (Ubuntu Server 14.04) which allows the clients to make a ssh tunnel from their device (Raspberry Pi) so they can access their web server from the internet (as a mean to traverse NATs). I can get a list of processes owned by the user (which is the same for all the devices) using ps -u username (this user only runs sshd to forward ports), but I can't filter those processes by the port they're forwarding. So the question is, how can I get the pid of the sshd that is forwarding port #N?
You can make use of lsof command since everything is a file on linux.
Something like lsof -Pan -i | grep :PORT will get you what you ask. It has an output like this when i run it for port 80 on my machine:
Command PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 1104 root 6u IPv4 23348 0t0 TCP *:80 (LISTEN)
nginx 1105 www-data 6u IPv4 23348 0t0 TCP *:80 (LISTEN)
nginx 1106 www-data 6u IPv4 23348 0t0 TCP *:80 (LISTEN)
nginx 1107 www-data 6u IPv4 23348 0t0 TCP *:80 (LISTEN)
nginx 1108 www-data 6u IPv4 23348 0t0 TCP *:80 (LISTEN)
More on lsof can be found here
I have storm running on 2 machines.
Each machine runs nimbus process (fancy for master process) and worker processes.
And I wanted to see the communication between them - what ports are open and how they connect to each other.
$ netstat -tulpn | grep -w 10669
tcp 0 0 :::6700 :::* LISTEN 10669/java
udp 0 0 :::42405 :::* 10669/java
$ lsof -i :6700
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 10669 storm 25u IPv6 57830 0t0 TCP host1:50778->host2:6700 (ESTABLISHED)
java 10669 storm 26u IPv6 57831 0t0 TCP host1:6700->host2:57339 (ESTABLISHED)
java 10669 storm 29u IPv6 57843 0t0 TCP host1:6700->host1:50847 (ESTABLISHED)
java 10669 storm 53u IPv6 57811 0t0 TCP *:6700 (LISTEN)
java 10681 storm 53u IPv6 57841 0t0 TCP host1:50780->host2:6700 (ESTABLISHED)
java 10681 storm 54u IPv6 57842 0t0 TCP host1:50847->host1:6700 (ESTABLISHED)
What I dont understand from the above output is that why netstat does not show port 50778 being open in the process with PID=10669 where as lsof clearly shows that the same process has an established connection as host1:50778->host2:6700
netstat -l limits the results to listening sockets, and prevents the display of sockets in other states.
Try this instead:
netstat -anp | egrep :6700
It's very strange.
I write a message distribute server upon nanomsg.
But after some time,when i restart the server, i failed because the listening port has been used.
Here is the situation:
[root#vsmHost12 src]# lsof -n -i:3333
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
hsmcs 105013 root **20u** IPv4 **24845821** 0t0 TCP :dec-notes (LISTEN)
hsmcs 105013 root 66u IPv4 25366582 0t0 TCP 192.168.167.1:dec-notes->192.168.167.1:47826 (ESTABLISHED)
java 111946 root **20u** IPv4 **24845821** 0t0 TCP *:dec-notes (LISTEN)
java 111946 root 34u IPv6 25366581 0t0 TCP 192.168.167.1:47826->192.168.167.1:dec-notes (ESTABLISHED)
It's not because of the java, some other daemon may also have the problem.
Look at the FD number and DEVICE number, it's the SAME!
Can anyone explain it ?
I was wondering, if the output of
lsof -i
sshd 21880 root 3r IPv4 4843515 TCP somehost.lu.isp.com:ssh->d-XX-XXX.ITS.SOMEWHERE.COM:45037 (ESTABLISHED)
sshd 21882 mike 3u IPv4 4843515 TCP somehost.lu.isp.com:ssh->d-XX-XXX.ITS.SOMEWHERE.COM:45037 (ESTABLISHED)
sshd 23853 root 3u IPv6 960417 TCP *:ssh (LISTEN)
sshd 23853 root 4u IPv4 960419 TCP *:ssh (LISTEN)
sshd 24043 root 3r IPv4 4871654 TCP somehost.lu.isp.com:ssh->XXX.XX.XXX.XXX:42104 (ESTABLISHED)
sshd 24044 sshd 3u IPv4 4871654 TCP somehost.lu.isp.com:ssh->XXX.XX.XXX.XXX:42104 (ESTABLISHED)
Does that imply that somebody has logged in on the system and is currently doing something? or means it's just trying to log in? I'm not quite sure about it.
Any clues? Thanks
According to this
lsof -i only shows you active tcp connections. So it doesn't tell you if there logged in or still attempting to authenticate.
if you want to check to see who's logged in and from where you can run the "who" command.
which will give you a list of the users logged in and where there logged in from (e.g. ssh, tty, etc)
The 'ESTABLISHED' means the TCP connection is established, ie the handshake has been performed on TCP/IP level. This is needed before the ssh process sees any data at all. Theoretically, the connection could be quite long in ESTABLISHED mode without sending any data depending on the timeouts set (on TCP level and/or sshd config). Expect login to occur after it.
To look into it more, use 'iptraf' for monitoring the amount of traffic, or see /var/log/auth.log (at least, on a Debian system) for seeing who succesfully logged on.