In userspace Linux, I have a process blocking on a semaphore, as found by strace. Once the error condition occurs, the blocking is repeatable, so there must be another process that holds the semaphore and did not release it.
Is there a way to know which other process is currently holding the semaphore?
ipcs lists the semaphore, so does /proc/sysvipc/sem. Where can I find info on the holding process?
Semaphores aren't mutexes. You don't "hold" them. If the process is blocked, that means it's waiting for someone else to do an "up" or "V" operation on it in the future. There's no kernel tool that will tell you what the future behavior of software will be.
To find the pids associated with the list of Semaphore Arrays listed by ipcs -s you can run this:
for pid in $( for semid in $( sudo ipcs -s | awk '/0x/{ print $2 }' ); do sudo ipcs -s -i $semid | tail -2 | head -1 | awk '{print $5}'; done | sort -u ); do ps uh -p $pid; done
There may be an easier way but you can use the semctl() call with the GETPID cmd. That should return the process that executed the last semop() call for the semaphore. This may or may not be your rogue process but it is probably a good hint.
"ipcs -p " can not show the semaphores of the process holding, that must be a bug, or it's a limit because it's hard to show.
You have to query by yourslef.
run "ipcs -s" to get all semid
for each semid run "ipcs -s -i "
for each semnum, to get owner pid,
if the owner pid is you wish, then show the current semid and semnum.
Note: if the process just read semaphores, then you may cannot get such information via ipcs command.
Did you try
ipcs -p
Related
I am trying to see what would happen about system call when I running one command, but it seems those command after | can't be shown? like:
strace -f cat a.txt| cat
It seems strace and -f perimeter can show the whole process. I think the last part is in the child progress created by fork. Why and how to make it?
From the strace manual (emphasis mine).
-f Trace child processes as they are created by
currently traced processes as a result of the fork(2),
vfork(2) and clone(2) system calls.
The traced process in your case is the first cat process. The second cat process is not a child of the first cat process. The fork is done by the shell.
One way to achieve what you want is to trace the shell:
strace -f bash -c "cat a.txt| cat"
I need to track the process status ps axf during executable lifetime.
Let's say I have executable main.exec and want to store into a file all subprocess which are called during main.exec execution.
$ main.exec &
$ echo $! # and redirect every ps change for PID $! in a file.
strace - trace system calls and signals
$ main.exec &
$ strace -f -p $! -o child.txt
-f Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls. Note that -p PID -f will attach all threads of process PID if it is multi-threaded, not only thread with thread_id = PID.
If you can't recompile and instrument main.exec, ps in a loop is a simple option that may work for you:
while true; do ps --ppid=<pid> --pid=<pid> -o pid,ppid,%cpu,... >> mytrace.txt; sleep 0.2; done
Then parse the output accordingly.
top may also work, and can run in batch mode but not sure if you can get it to dynamically monitor child processes like ps. Don't think so.
Currently, I am taking up the long method of doing this by getting a list of processes using the following command
sudo ps -eo pid,command | grep -v grep | awk '{print $1}' > pids.txt
And then iterating through the process ids and executing in background the strace of each process and generating logs for each process with the process id in the log's extension
filename="$1"
while read -r line
do
chmod +x straceProgram.sh
./straceProgram.sh $line &
done < "$filename"
straceProgram.sh
pid="$1"
sudo strace -p $pid -o log.$pid
However, the problem with this approach is that if there is any new process which gets started, it will not be straced since the strace is on the process ids stored in the pids.txt during the first run.
The list of pids.txt can be updated with new process ids, however, I was inquisitive on running a strace at an operating system level which would strace all the activities being performed.
Could there be a better way to do this?
If your resulting filesystem is going to be a kernel filesystem driver, I would recommend using tracefs to gather the information you require. I would recommend against making this a kernel filesystem unless you have a lot of time and a lot of testing resources. It is not trivial.
If you want an easier, safer alternative, write your filesystem using fuse. The downside is that performance is not quite as good and there are a few places where it cannot be used, but it is often acceptable. Note that there is already an implementation of a logging filesystem under fuse.
use the strace -f (fork) option, also I suggest the -s 9999 for more details
I would like to monitor the number of threads used by a specific process on Linux.
Is there an easy way to get this information without impacting the performance of the process?
try
ps huH p <PID_OF_U_PROCESS> | wc -l
or htop
To get the number of threads for a given pid:
$ ps -o nlwp <pid>
Where nlwp stands for Number of Light Weight Processes (threads). Thus ps aliases nlwp to thcount, which means that
$ ps -o thcount <pid>
does also work.
If you want to monitor the thread count, simply use watch:
$ watch ps -o thcount <pid>
To get the sum of all threads running in the system:
$ ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'
Each thread in a process creates a directory under /proc/<pid>/task. Count the number of directories, and you have the number of threads.
cat /proc/<PROCESS_PID>/status | grep Threads
ps -eLf on the shell shall give you a list of all the threads and processes currently running on the system.
Or, you can run top command then hit 'H' to toggle thread listings.
$ ps H p pid-id
H - Lists all the individual threads in a process
or
$cat /proc/pid-id/status
pid-id is the Process ID
eg.. (Truncated the below output)
root#abc:~# cat /proc/8443/status
Name: abcdd
State: S (sleeping)
Tgid: 8443
VmSwap: 0 kB
Threads: 4
SigQ: 0/256556
SigPnd: 0000000000000000
If you use:
ps uH p <PID_OF_U_PROCESS> | wc -l
You have to subtract 1 to the result, as one of the lines "wc" is counting is the headers of the "ps" command.
My answer is more gui, but still within terminal. Htop may be used with a bit of setup.
Start htop.
Enter setup menu by pressing F2.
From leftmost column choose "Columns"
From rightmost column choose the column to be added to main monitoring output, "NLWP" is what you are looking for.
Press F10.
JStack is quite inexpensive - one option would be to pipe the output through grep to find active threads and then pipe through wc -l.
More graphically is JConsole, which displays the thread count for a given process.
Here is one command that displays the number of threads of a given process :
ps -L -o pid= -p <pid> | wc -l
Unlike the other ps based answers, there is here no need to substract 1 from its output as there is no ps header line thanks to the -o pid=option.
Newer JDK distributions ship with JConsole and VisualVM. Both are fantastic tools for getting the dirty details from a running Java process. If you have to do this programmatically, investigate JMX.
If you're looking for thread count for multiple processes, the other answers won't work well for you, since you won't see the process names or PIDs, which makes them rather useless. Use this instead:
ps -o pid,nlwp,args -p <pid_1> <pid_2> ... <pid_N>
In order to watch the changes live, just add watch:
watch ps -o pid,nlwp,args -p <pid_1> <pid_2> ... <pid_N>
jvmtop can show the current jvm thread count beside other metrics.
The easiest way is using "htop". You can install "htop" (a fancier version of top) which will show you all your cores, process and memory usage.
Press "Shift+H" to show all process or press again to hide it.
Press "F4" key to search your process name.
Installing on Ubuntu or Debian:
sudo apt-get install htop
Installing on Redhat or CentOS:
yum install htop
dnf install htop [On Fedora 22+ releases]
If you want to compile "htop" from source code, you will find it here.
If you are trying to find out the number of threads using cpu for a given pid I would use:
top -bc -H -n2 -p <pid> | awk '{if ($9 != "0.0" && $1 ~ /^[0-9]+$/) print $1 }' | sort -u | wc -l
If you want the number of threads per user in a linux system then you should use:
ps -eLf | grep <USER> | awk '{ num += $6 } END { print num }'
where as <USER> use the desired user name.
If you're interested in those threads which are really active -- as in doing something (not blocked, not timed_waiting, not reporting "thread running" but really waiting for a stream to give data) as opposed to sitting around idle but live -- then you might be interested in jstack-active.
This simple bash script runs jstack then filters out all the threads which by heuristics seem to be idling, showing you stack traces for those threads which are actually consuming CPU cycles.
First get the process ID (pid) by executing below command:
ps -ef | grep (for e.g ps -ef | grep java)
Now replace the pid in below command and execute to get the total thread count of a process.
ps huH p | wc -l
VisualVM can show clear states of threads of a given JVM process
I would like a "system" that monitors a process and would kill said process if:
the process exceeds some memory requirements
the process does not respond to a message from the "system" in some period of time
I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem.
For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.
Monitoring daemons exist for this sort of thing as well. God is a recent example.
I wrote a script that runs as a cron job and can be customized to kill problem processes:
#!/usr/local/bin/perl
use strict;
use warnings;
use Proc::ProcessTable;
my $table = Proc::ProcessTable->new;
for my $process (#{$table->table}) {
# skip root processes
next if $process->uid == 0 or $process->gid == 0;
# skip anything other than Passenger application processes
#next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/;
# skip any using less than 1 GiB
next if $process->rss < 1_073_741_824;
# document the slaughter
(my $cmd = $process->cmndline) =~ s/\s+\z//;
print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n";
# try first to terminate process politely
kill 15, $process->pid;
# wait a little, then kill ruthlessly if it's still around
sleep 5;
kill 9, $process->pid;
}
https://www.endpointdev.com/blog/2012/08/automatically-kill-process-using-too/
To limit memory usage of processes, check /etc/security/limits.conf
Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.
There's a few other monitoring scripts there you might find interesting too.
If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.
That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.
I have a shell script here that could be your start point. I did it because I also had some issues with processes exceeding memory limit. Actually it just checks a given limit of CPU usage, but you can easily change to watch memory, or the jobs list for an idle process.
file: pkill.sh
#!/bin/bash
if [ -z "$1" ]
then
maxlimit=99
else
maxlimit=$1
fi
ps axo user,%cpu,pid,vsz,rss,uid,gid --sort %cpu,rss\
| awk -v max=$maxlimit '$6 != 0 && $7 != 0 && $2 > max'\
| awk '{print $3}'\
| while read line;\
do\
ps u --no-headers -p $line;\
echo "$(date) - $(ps u --no-headers -p $line)" >> pkill.log;\
notify-send 'Killing proccess!' $(ps -p $line -o command --no-headers | awk '{print $1}') -u normal -i dialog-warning -t 3000;\
kill $line;\
done;
Simple run it once like: sh ./pkill.sh <limit-cpu>
Or, to keep it running: watch -n 10 sh ./pkill.sh 90
In the case above it will keep running each 10 seconds, killing processes that exceeds 90% of CPU
Are the monitored processes ones you're writing, or just any process?
If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.