I'm trying to figure out what process is holding the other end of a unix domain socket. In some strace output I've identified a given file descriptor which is involved in the problem I'm currently debugging, and I'd like to know which process is on the other end of that. As there are multiple connections to that socket, simply going by path name won't work.
lsof provides me with the following information:
dbus-daem 4175 mvg 10u unix 0xffff8803e256d9c0 0t0 12828 #/tmp/dbus-LyGToFzlcG
So I know some address (“kernel address”?), I know some socket number, and I know the path. I can find that same information in other places:
$ netstat -n | grep 12828
unix 3 [ ] STREAM CONNECTED 12828 #/tmp/dbus-LyGToFzlcG
$ grep -E '12828|ffff8803e256d9c0' /proc/net/unix
ffff8803e256d9c0: 00000003 00000000 00000000 0001 03 12828 #/tmp/dbus-LyGToFzlcG
$ ls -l /proc/*/fd/* 2>/dev/null | grep 12828
lrwx------ 1 mvg users 64 10. Aug 09:08 /proc/4175/fd/10 -> socket:[12828]
However, none of this tells me what the other end of my socket connection is. How can I tell which process is holding the other end?
Similar questions have been asked on Server Fault and Unix & Linux. The accepted answer is that this information is not reliably available to the user space on Linux.
A common suggestion is to look at adjacent socket numbers, but ls -l /proc/*/fd/* 2>/dev/null | grep 1282[79] gave no results here. Perhaps adjacent lines in the output from netstat can be used. It seems like there was a pattern of connections with and without an associated socket name. But I'd like some kind of certainty, not just guesswork.
One answer suggests a tool which appears to be able to address this by digging through kernel structures. Using that option requires debug information for the kernel, as generated by the CONFIG_DEBUG_INFO option and provided as a separate package by some distributions. Based on that answer, using the address provided by lsof, the following solution worked for me:
# gdb /usr/src/linux/vmlinux /proc/kcore
(gdb) p ((struct unix_sock*)0xffff8803e256d9c0)->peer
This will print the address of the other end of the connection. Grepping lsof -U for that number will provide details like the process id and the file descriptor number.
If debug information is not available, it might be possible to access the required information by knowing the offset of the peer member into the unix_sock structure. In my case, on Linux 3.5.0 for x86_64, the following code can be used to compute the same address without relying on debugging symbols:
(gdb) p ((void**)0xffff8803e256d9c0)[0x52]
I won't make any guarantees about how portable that solution is.
Update: It's been possible to to do this using actual interfaces for a while now. Starting with Linux 3.3, the UNIX_DIAG feature provides a netlink-based API for this information, and lsof 4.89 and later support it. See https://unix.stackexchange.com/a/190606/1820 for more information.
Related
I was wondering what possibilities are available to detect a connection on a tty. My goal is to create an alert in case someone tries to watch what I am doing through the console
First I though about who, which allows to see wether someone is connected and on what tty, but let's say this user isn't logged in, is there still a way of detecting that a tty is opened? Maybe with /dev/tty? Or is it possible to know how many file descriptors are pointing to the file /dev/console and what processes are using the hardware/io? Or maybe using hardware detection with vcs? I actually have no idea how to use/test those.
let's say this user isn't logged in
Each process belongs to some user. The only way to run some code and not being logged in is to write this code in kernel.
I was wondering what possibilities are available to detect a connection on a tty
Try this:
$ lsof /dev/tty0
lsof is a tool for observe open files.
For example, serial console for my development board is /dev/ttyUSB0. So I opened minicom session and also made cat for my TTY file:
$ minicom -D /dev/ttyUSB0
$ cat /dev/ttyUSB0
and checked this:
$ lsof /dev/ttyUSB0
which gave me:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
minicom 13299 joe 3u CHR 188,0 0t0 135832 /dev/ttyUSB0
cat 13310 joe 3r CHR 188,0 0t0 135832 /dev/ttyUSB0
From this output you can figure out, who (USER) connected to your TTY, and whic program he used (COMMAND, PID).
After closing minicom session and cat, lsof doesn't print anything.
Also it can be done using w command:
$ w | grep ttyUSB0
Output:
joe pts/2 :0 17:57 2:47 0.12s 0.00s minicom -D /dev/ttyUSB0
joe pts/3 :0 17:58 2:39 0.08s 0.00s cat /dev/ttyUSB0
UPDATE
If you don't want to be watched in the way described above (i.e. via open descriptor), you can do next.
Let's say you are using /dev/ttyUSB0. To connect to your console via this file, you must open it first (no matter which way, e.g. using cat or minicom etc.). Once you have opened it, your session can be seen by other users (e.g. by root) just looking if there are open file descriptors for that file (lsof /dev/ttyUSB0).
Now, TTY devices are just character devices, and you can create your own files (nodes) for those devices. Let's see closely to /dev/ttyUSB0 file:
$ ls -l /dev/ttyUSB0
Output:
crw-rw---- 1 root dialout 188, 0 Mar 4 16:07 /dev/ttyUSB0
Here c indicates that this is character device, and we can see that it has major number 188 and minor number 0. Let's create our own node (file) for this device.
$ cd /tmp
$ sudo mknod some_tricky_name c 188 0
$ sudo chown root:dialout some_tricky_name
$ sudo chmod 644 some_trickyname
Now you can connect to this file instead of /dev/ttyUSB0 and nobody's gonna see you:
$ minicom -D /tmp/some_tricky_name
Still, if you are gonna run some shell in this TTY, good administrator still can catch you, looking to /var/log/auth.log. But I believe this technique prevents you from being caught using w or lsof commands.
My mother tongue is not English, sorry for my English.
I use pppd with a GPRS module.
I use like pppd record record.pcap call tdscdma command to access Internet.And pppdump record.pcap or wireshark to show the record.pcap.
when pppd run ,the record.pcap will save all data and the file size getting bigger and bigger.
Now I am just want save last(Newest) 1Mb(for example,or quantity) message.And how can I limit the file size.
I am more concerned about the recent network conditions. FIFO is not necessary.if the file bigger than 1Mb, truncate it to zero is OK too.
[root#AT91SAM9-RT9x5 logs]# pppd -v
pppd: unrecognized option '-v'
pppd version 2.4.5
[root#AT91SAM9-RT9x5 logs]# uname -a
Linux AT91SAM9-RT9x5 2.6.39 #34 Wed Jun 4 16:12:41 CST 2014 armv5tejl GNU/Linux
Use wireshark looks like this:
Can you use tcpdump program for capturing traffic of ppp0 interface?
There are -C and -W options for limiting size of output files.
Example:
tcpdump -i ppp0 -C 1 -W 2 -w file.pcap
See more from man page: tcpdump(8).
I'm running a long-running socket connection, locally. When I kill the server application it closes cleanly. I now want to see how the client behaves when it does not close cleanly. If they were on different servers I could pull out the network cable. When the connection is local I have nothing to pull. [Insert rude joke here.]
I can do netstat -an |grep tcp | grep :80 and see the connection, which looks something like this:
tcp 0 0 10.1.2.3:80 10.1.2.3:40494 ESTABLISHED
Can I use that 40494 port number to kill or hang the socket, the same way I can use a program's PID to kill it?
Very similar question: How can I simulate a 'plugged network cable' (TCP/IP)? (That question is asking about Windows, I am on Linux)
Unload the kernel module for the network driver might kill it the way you want.
Pick your network interface from the list of active (eth0, eth1, etc...)
ifconfig -a
eth0 Link encap:Ethernet HWaddr 18:A9:05:68:6F:B0
inet6 addr: fe80::1aa9:5ff:fe68:6fb0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
To find the driver module for eth0 interface
ls -l /sys/class/net/eth0/device/driver/module
lrwxrwxrwx 1 root root 0 Aug 15 18:39 module -> ../../../../module/bnx2
To list the kernel modules for your system run
lsmod | grep bnx
bnx2i 48158 0
cnic 55511 1 bnx2i
bnx2 81890 0
To remove the module from memory will require root or sudo access
rmmod -f -w bnx2
This is not a safe operation and you messing with the operating system in an unsafe way. Expect bad things(tm) to happen when you do this. Don't do it in a production environment. Also this may not work depending on the driver behavior itself.
That said, this is an interesting problem and I'm interested in hearing how it turns out.
It appears the answer is "no", there is no way to kill a particular connection.
What I ended up doing was sudo ifdown eth0 (then afterwards sudo ifup eth0 to restore the network). Obviously this is a sledgehammer approach, as it takes down all connections.
For completeness, another approach was to guess which Apache process was running the socket connection, and kill just that process. But apart from involving guessing, that is also more application specific than the question I was asking.
I am running kernel 2.6.33.7.
Previously, I was running v2.6.18.x. On 2.6.18, the flush processes were named pdflush.
After upgrading to 2.6.33.7, the flush processes have a format of "flush-:".
For example, currently I see flush process "flush-8:32" popping up in top.
In doing a google search to try to determine an answer to this question, I saw examples of "flush-8:38", "flush-8:64" and "flush-253:0" just to name a few.
I understand what the flush process itself does, my question is what is the significance of the numbers on the end of the process name? What do they represent?
Thanks
Device numbers used to identify block devices. A kernel thread may be spawned to handle a particular device.
(On one of my systems, block devices are currently numbered as shown below. They may change from boot to boot or hotplug to hotplug.)
$ grep ^ /sys/class/block/*/dev
/sys/class/block/dm-0/dev:254:0
/sys/class/block/dm-1/dev:254:1
/sys/class/block/dm-2/dev:254:2
/sys/class/block/dm-3/dev:254:3
/sys/class/block/dm-4/dev:254:4
/sys/class/block/dm-5/dev:254:5
/sys/class/block/dm-6/dev:254:6
/sys/class/block/dm-7/dev:254:7
/sys/class/block/dm-8/dev:254:8
/sys/class/block/dm-9/dev:254:9
/sys/class/block/loop0/dev:7:0
/sys/class/block/loop1/dev:7:1
/sys/class/block/loop2/dev:7:2
/sys/class/block/loop3/dev:7:3
/sys/class/block/loop4/dev:7:4
/sys/class/block/loop5/dev:7:5
/sys/class/block/loop6/dev:7:6
/sys/class/block/loop7/dev:7:7
/sys/class/block/md0/dev:9:0
/sys/class/block/md1/dev:9:1
/sys/class/block/sda/dev:8:0
/sys/class/block/sda1/dev:8:1
/sys/class/block/sda2/dev:8:2
/sys/class/block/sdb/dev:8:16
/sys/class/block/sdb1/dev:8:17
/sys/class/block/sdb2/dev:8:18
/sys/class/block/sdc/dev:8:32
/sys/class/block/sdc1/dev:8:33
/sys/class/block/sdc2/dev:8:34
/sys/class/block/sdd/dev:8:48
/sys/class/block/sdd1/dev:8:49
/sys/class/block/sdd2/dev:8:50
/sys/class/block/sde/dev:8:64
/sys/class/block/sdf/dev:8:80
/sys/class/block/sdg/dev:8:96
/sys/class/block/sdh/dev:8:112
/sys/class/block/sdi/dev:8:128
/sys/class/block/sr0/dev:11:0
/sys/class/block/sr1/dev:11:1
/sys/class/block/sr2/dev:11:2
You should also be able to figure this out by searching for those numbers in /proc/self/mountinfo, eg:
$ grep 8:32 /proc/self/mountinfo
25 22 8:32 / /var rw,relatime - ext4 /dev/mapper/sysvg-var rw,barrier=1,data=ordered
This has the side benefit of working with nfs as well:
$ grep 0:73 /proc/self/mountinfo
108 42 0:73 /foo /mnt/foo rw,relatime - nfs host.domain.com:/volume/path rw, ...
Note, the data I included here is fabricated, but the mechanism works just fine.
Is there a way to tie a network connection to a PID (process ID) without forking to lsof or netstat?
Currently lsof is being used to poll what connections belong which process ID. However lsof or netstat can be quite expensive on a busy host and would like to avoid having to fork to these tools.
Is there someplace similar to /proc/$pid where one can look to find this information? I know what the network connections are by examining /proc/net but can't figure out how to tie this back to a pid. Over in /proc/$pid, there doesn't seem to be any network information.
The target hosts are Linux 2.4 and Solaris 8 to 10. If possible, a solution in Perl, but am willing to do C/C++.
additional notes:
I would like to emphasize the goal here is to tie a network connection to a PID. Getting one or the other is trivial, but putting the two together in a low cost manner appears to be difficult. Thanks for the answers to so far!
I don't know how often you need to poll, or what you mean with "expensive", but with the right options both netstat and lsof run a lot faster than in the default configuration.
Examples:
netstat -ltn
shows only listening tcp sockets, and omits the (slow) name resolution that is on by default.
lsof -b -n -i4tcp:80
omits all blocking operations, name resolution, and limits the selection to IPv4 tcp sockets on port 80.
On Solaris you can use pfiles(1) to do this:
# ps -fp 308
UID PID PPID C STIME TTY TIME CMD
root 308 255 0 22:44:07 ? 0:00 /usr/lib/ssh/sshd
# pfiles 308 | egrep 'S_IFSOCK|sockname: '
6: S_IFSOCK mode:0666 dev:326,0 ino:3255 uid:0 gid:0 size:0
sockname: AF_INET 192.168.1.30 port: 22
For Linux, this is more complex (gruesome):
# pgrep sshd
3155
# ls -l /proc/3155/fd | fgrep socket
lrwx------ 1 root root 64 May 22 23:04 3 -> socket:[7529]
# fgrep 7529 /proc/3155/net/tcp
6: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7529 1 f5baa8a0 300 0 0 2 -1
00000000:0016 is 0.0.0.0:22. Here's the equivalent output from netstat -a:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
Why don't you look at the source code of netstat and see how it get's the information? It's open source.
For Linux, have a look at the /proc/net directory
(for example, cat /proc/net/tcp lists your tcp connections). Not sure about Solaris.
Some more information here.
I guess netstat basically uses this exact same information so i don't know if you will be able to speed it up a whole lot. Be sure to try the netstat '-an' flags to NOT resolve ip-adresses to hostnames realtime (as this can take a lot of time due to dns queries).
The easiest thing to do is
strace -f netstat -na
On Linux (I don't know about Solaris). This will give you a log of all of the system calls made. It's a lot of output, some of which will be relevant. Take a look at the files in the /proc file system that it's opening. This should lead you to how netstat does it. Indecently, ltrace will allow you to do the same thing through the c library. Not useful for you in this instance, but it can be useful in other circumstances.
If it's not clear from that, then take a look at the source.
Take a look at these answers which thoroughly explore the options available:
How I can get ports associated to the application that opened them?
How to do like "netstat -p", but faster?