I'm writing some code to replace TCP sockets with POSIX message queues. Sometimes the program will crash (still in development) and the queues that were created are not deleted (did not execute: mq_close() + mq_unlink()). This causes issues when I run the code again.
Is there a way to delete/remove these queues using the command line? I tried using: ipcs -q. This failed to list any queues.
I tried: lsof | grep queue-name. They did show up here.
Ideally, I'd like to use: ipcrm.
POSIX IPC objects are implemented as files in virtual file systems. These files can be listed and removed with ls and rm. To do this with POSIX message queues, we must mount the message queue file system using the following commands:
$ su
Password:
# mkdir /dev/mqueue
# mount -t mqueue none /dev/mqueue
# exit
Related
Consider the following, observed with bash 4.4 on a Linux 3.19 kernel:
# in reality, this may access files "nobody" isn't allowed
get_a_secret() { printf '%s\n' "This is a secret"; }
# attach a process substitution reading the secret to FD 10
exec 10< <(get_a_secret)
# run a less-privileged program that needs the secret, passing it the file descriptor.
chpst -u nobody:nobody -- cat /dev/fd/10
...or the shorter/simpler:
chpst -u nobody:nobody -- cat <(get_a_secret)
Either fails in a manner akin to the following:
cat: /proc/self/fd/10: Permission denied
So, two branches to this question:
What's going on here?
Is there a way to get the desired behavior (passing the ability to read the secret through to the single child process being invoked in a way that doesn't persistently expose that secret to other processes running as "nobody") without exposing the FIFO's output to other processes?
(Yes, I'm well aware that I need to lock down ptrace and /proc/*/mem to prevent another process running as "nobody" from pulling the secret out of the client as it's being read; that said, that's (1) something I can do, and (2) when the process is only run before any potentially-attacker-controlled executables are invoked, less exposure than allowing any process running as nobody to pull the secret out of /proc/*/environ for the full duration of that process).
The following workaround avoids this issue:
exec 10< <(get_a_secret)
chpst -u nobody:nobody -- sh -c 'cat <&10'
Note the redirection being written as <&10 -- not </dev/fd/10 or </proc/self/fd/10 (on platforms which provide /dev/fd -- on platforms without this facility, bash rewrites it into a fdup2() call).
An answer with an explanation of the behavior (and perhaps a workaround that allows programs that don't accept a FD number as input to act on the read side?) would be in a position to supercede this one. :)
I want to create a Java console application that runs as a daemon on Linux, I have created the application and the script to run the application as a background daemon. The application runs and waits for command line input.
My question:
Is it possible to pass command line input to a running daemon?
On Linux, all running processes have a special directory under /proc containing information and hooks into the process. Each subdirectory of /proc is the PID of a running process. So if you know the PID of a particular process you can get information about it. E.g.:
$ sleep 100 & ls /proc/$!
...
cmdline
...
cwd
environ
exe
fd
fdinfo
...
status
...
Of note is the fd directory, which contains all the file descriptors associated with the process. 0, 1, and 2 exist for (almost?) all processes, and 0 is the default stdin. So writing to /proc/$PID/fd/0 will write to that process' stdin.
A more robust alternative is to set up a named pipe connected to your process' stdin; then you can write to that pipe and the process will read it without needing to rely on the /proc file system.
See also Writing to stdin of background process on ServerFault.
The accepted answer above didn't quite work for me, so here's my implementation.
For context I'm running a Minecraft server on a Linux daemon managed with systemctl. I wanted to be able to send commands to stdin (StandardInput).
First, use mkfifo /home/user/server_input to create a FIFO file somewhere (also known as the 'named pipe' solution mentioned above).
[Service]
ExecStart=/usr/local/bin/minecraft.sh
StandardInput=file:/home/user/server_input
Then, in your daemon *.service file, execute the bash script that runs your server or background program and set the StandardInput directive to the FIFO file we just created.
In minecraft.sh, the following is the key command that runs the server and gets input piped into the console of the running service.
tail -f /home/user/server_input| java -Xms1024M -Xmx4096M -jar /path/to/server.jar nogui
Finally, run systemctl start your_daemon_service and to pass input commands simply use:
echo "command" > /home/user/server_input
Creds to the answers given on ServerFault
I am using strace to capture calls to open(), close() and read() on Linux. The target process is the jetty web server. As far as I can tell, strace is not logging all calls to open(). Maybe the others too, I have not tried to correlate the file descriptors to open() calls.
For example, starting strace:
strace -f -e trace=open,close,read -o/tmp/strace.out -p62881
I then use wget to fetch 100 static files; all were retrieved successfully. In one run, only 56 open events were logged; on another run of 100 different files, I got 66 open events.
I believe that using "-f" results in strace attaching to all the LWPIDs for the threads ("Process 62881 attached with 25 threads - interrupt to quit
"); when I try to explicitly attach to all using multiple "-p" options, I get a single "attach" success message, but multiple "Operation not permitted messages", one for each child PID.
I restarted Jetty to clear its cache before my tests.
Kernel version is 2.6.32-504.3.3.el6.x86_64 (Red Hat). Strace package version is strace-4.5.19-1.19.el6.x86_64.
What am I missing?
Thanks
On some systems you have to use openat() instead of open().
Try:
strace -f -e trace=openat,close,read -o/tmp/strace.out -p62881
Try -ff (in addition to -f):
-ff: If the -o filename option is in effect, each processes trace is written to filename.pid where pid is
the numeric process id of each process. This is incompatible with -c, since no per-process counts are
kept.
I have a Java program after 2 weeks of running in average will become stuck and produce the following error:
Caused by: java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:415)
at sun.nio.ch.Net.socket(Net.java:408)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
That hints to me that many sockets are opened but never closed.
Before diving into programmatic instrumentation i started to inspect what information i could draw from linux itself. I am using Redhat.
And then, a few questions came up as follows:
Why the following commands do not give the same output?
See
[ec2-user#ip-172-22-28-102 ~]$ sudo ls /proc/32085/fd | wc -l
592
[ec2-user#ip-172-22-28-102 ~]$ sudo lsof -a -p 32085 | wc -l
655
Is there a way to know from the proc stat info which thread created which file descriptor?
It seems like there is not because if i do the following, i am getting the same information:
[ec2-user#ip-172-22-28-102 ~]$ sudo ls /proc/32085/task/22386/fd | wc -l
592
[ec2-user#ip-172-22-28-102 ~]$ sudo ls /proc/32085/fd | wc -l
592
Same if i go to the thread directly from under /proc/ .
Thx
Is there a way to know from the proc stat info which thread created which file descriptor?
I am pretty sure the answer here is "no". File descriptors are opened by processes, not threads (and will be visible to all threads spawned by the same process).
Why the following commands do not give the same output?
First, the -a argument to lsof appears to be a no-op in this case. Specfically, the man says that it "causes list selection options to be ANDed, as described above". So you are really just running:
sudo lsof -p 32085
And that will print things other than open file descriptors (such as memory-mapped files, current working directory, etc), while /proc/<PID>/fd contains only open file descriptors. So you're getting different results because you're asking for different information.
The only reason you can receive that message is that you have opened files and you didn't close them after use. You have a file descriptor leak in your java application. Java programmers normally don't check memory as the garbage collector copes with unreferenced objects. If you save file descriptors without closing in some data structure or you don't close the files after using, you can reach the maximum limit allowed to a process (this is controlled per process and can be changed by the ulimit shell command)
But if your problem is a file descriptor leak, pushing up the ulimit will only delay the problem some time. File descriptors must be closed, or you'll run into trouble.
I've just ran across this difference today, the explanation is that lsof takes into account more types of files, like memory-mapped objects, run-time libraries etc
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
Linux: Which process is causing "device busy" when doing umount?
Look at the lsof command (list open files) -- it can tell you which processes are holding what open. Sometimes it's tricky but often something as simple as sudo lsof | grep (your device name here) could do it for you.
Just in case... sometimes happens that you are calling umount from the terminal, and your current directory belongs to the mounted filesystem.
You should use the fuser command.
Eg. fuser /dev/cdrom will return the pid(s) of the process using /dev/cdrom.
If you are trying to unmount, you can kill theses process using the -k switch (see man fuser).
Check for open loop devices mapped to a file on the filesystem with "losetup -a". They wont show up with either lsof or fuser.
Also check /etc/exports. If you are exporting paths within the mountpoint via NFS, it will give this error when trying to unmount and nothing will show up in fuser or lsof.
lsof +f -- /mountpoint
(as lists the processes using files on the mount mounted at /mountpoint. Particularly useful for finding which process(es) are using a mounted USB stick or CD/DVD.
lsof and fuser are indeed two ways to find the process that keeps a certain file open.
If you just want umount to succeed, you should investigate its -f and -l options.
That's exactly why the "fuser -m /mount/point" exists.
BTW, I don't think "fuser" or "lsof" will indicate when a resource is held by kernel module, although I don't usually have that issue..
Open files
Processes with open files are the usual culprits. Display them:
lsof +f -- <mountpoint or device>
There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.
fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.
List files on <mountpoint> (see caveat above):
fuser -vmM <mountpoint>
Interactively kill only processes with files open for writing:
fuser -vmMkiw <mountpoint>
After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:
fuser -vmMk <mountpoint>
Mountpoints
The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:
mount | grep <mountpoint>/
For loopback mounts, also check the output of:
losetup -la
Anonymous inodes (Linux)
Anonymous inodes can be created by:
Temporary files (open with O_TMPFILE)
inotify watches
[eventfd]
[eventpoll]
[timerfd]
These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).
They won't appear in lsof +f -- /dev/<device>, so you'll need to:
lsof | grep a_inode
For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).
lsof and fuser didn't give me anything either.
After a process of renaming all possible directories to .old and rebooting the system every time after I made changes I found one particular directory (relating to postfix) that was responsible.
It turned out that I had once made a symlink from /var/spool/postfix to /disk2/pers/mail/postfix/varspool in order to minimise disk writes on an SDCARD-based root filesystem (Sheeva Plug).
With this symlink, even after stopping the postfix and dovecot services (both ps aux as well as netstat -tuanp didn't show anything related) I was not able to unmount /disk2/pers.
When I removed the symlink and updated the postfix and dovecot config files to point directly to the new dirs on /disk2/pers/ I was able to successfully stop the services and unmount the directory.
Next time I will look more closely at the output of:
ls -lR /var | grep ^l | grep disk2
The above command will recursively list all symbolic links in a directory tree (here starting at /var) and filter out those names that point to a specific target mount point (here disk2).
If you still can not unmount or remount your device after stopping all services and processes with open files, then there may be a swap file or swap partition keeping your device busy. This will not show up with fuser or lsof. Turn off swapping with:
sudo swapoff -a
You could check beforehand and show a summary of any swap partitions or swap files with:
swapon -s
or:
cat /proc/swaps
As an alternative to using the command sudo swapoff -a, you might also be able to disable the swap by stopping a service or systemd unit. For example:
sudo systemctl stop dphys-swapfile
or:
sudo systemctl stop var-swap.swap
In my case, turning off swap was necessary, in addition to stopping any services and processes with files open for writing, so that I could remount my root partition as read only in order to run fsck on my root partition without rebooting. This was necessary on a Raspberry Pi running Raspbian Jessie.
Filesystems mounted on the filesystem you're trying to unmount can cause the target is busy error in addition to any files that are in use. (For example when you mount -o bind /dev /mnt/yourmount/dev in order to use chroot there.)
To find which file systems are mounted on the filesystem run the following:
mount | grep '/mnt/yourmount'
To find which files are in use the advice already suggested by others here:
lsof | grep '/mnt/yourmount'