I am trying to see what would happen about system call when I running one command, but it seems those command after | can't be shown? like:
strace -f cat a.txt| cat
It seems strace and -f perimeter can show the whole process. I think the last part is in the child progress created by fork. Why and how to make it?
From the strace manual (emphasis mine).
-f Trace child processes as they are created by
currently traced processes as a result of the fork(2),
vfork(2) and clone(2) system calls.
The traced process in your case is the first cat process. The second cat process is not a child of the first cat process. The fork is done by the shell.
One way to achieve what you want is to trace the shell:
strace -f bash -c "cat a.txt| cat"
I want to monitor all the running processes using strace and when a process ends the output of the strace should be sent to a file.
And how to find every running proc PID. I also want to include process name in the output file.
$ sudo strace -p 1725 -o firefox_trace.txt
$ tail -f firefox_trace.txt
1725 would be the PID of the proccess you want to monitor (you can find the PID with "ps -C firefox-bin", for firefox in the example)
And firefox_trace.txt would be the output file !
The way to got would be to find every running proc PID, and use the command to write them in the output file !
Considering the doc,
-p pid
Attach to the process with the process ID pid and begin tracing. The
trace may be terminated at any time by a keyboard interrupt signal (
CTRL -C). strace will respond by detaching itself from the traced
process(es) leaving it (them) to continue running. Multiple -p options
can be used to attach to up to 32 processes in addition to command
(which is optional if at least one -p option is given).
Use -o to store the output to the file, or 2>&1 to redirect standard error to output, so you can filter it (grep) or redirect it into file (> file).
To monitor process without knowing its PID, but name, you can use pgrep command, e.g.
strace -p $(pgrep command) -o file.out
where command is your name of process (e.g. php, Chrome, etc.).
To learn more about parameters, check man strace.
I'm pretty inexperienced with Linux bash. That being said, I have a CentOS7 machine that runs a COTS application server. This application server runs other processes that sometimes hang. Since I have no control over the start of these processes, I'm looking for a script that runs every 2 minutes that kills processes of the name "spicer" that have been running for longer than 10 minutes. I've looked around and have only been able to find answers for processes that are run and owned by me.
I use the command ps -eo pid, command,etime | grep spicer to get all the spicer processes. The output of this command looks like:
18216 spicer -l/opt/otmm-10.5/Spi 14:20
18415 spicer -l/opt/otmm-10.5/Spi 11:49
etc...
18588 grep --color=auto spicer
I don't know if there's a way to parse this directly in bash. I'm also not well-versed at all in other Linux tools. I know that awk (or gawk) could possibly help.
EDIT
I have no control over the data that the process is working on.
What about wrapping the executable of spicer and start it using the timeout command? Let's say it is installed in /usr/bin/spicer. Then issue:
cp /usr/bin/spicer{,.orig}
echo '#!/bin/bash' > /usr/bin/spicer
echo 'timeout 10m spicer.orig "$#"' >> /usr/bin/spicer
Another approach would be to create a cronjob defintion into /etc/cron.d/kill_spicer. Like this:
* * * * * root kill $(ps --no-headers -C spicer -o pid,etimes | awk '$2>=600{print $1}')
The cronjob will get executed minutely and uses ps to obtain a list of spicer processes that run longer than 10minutes and passes them to kill.
Probably you even want kill -9 if the process is hanging.
You can use the -C option of ps to select processes by name.
ps --no-headers -C spicer -o pid,etime
Then you can use cut to filter the results, if the spacing is consistent. On my system the pid field takes up 8 characters, so I'd use
kill $(ps --no-headers -C spicer -o pid,etime | cut -c-8)
If the spacing is inconsistent (but if so, what kind of messed up ps are you using? :-P), you can use awk { print $1 } instead of cut.
Currently, I am taking up the long method of doing this by getting a list of processes using the following command
sudo ps -eo pid,command | grep -v grep | awk '{print $1}' > pids.txt
And then iterating through the process ids and executing in background the strace of each process and generating logs for each process with the process id in the log's extension
filename="$1"
while read -r line
do
chmod +x straceProgram.sh
./straceProgram.sh $line &
done < "$filename"
straceProgram.sh
pid="$1"
sudo strace -p $pid -o log.$pid
However, the problem with this approach is that if there is any new process which gets started, it will not be straced since the strace is on the process ids stored in the pids.txt during the first run.
The list of pids.txt can be updated with new process ids, however, I was inquisitive on running a strace at an operating system level which would strace all the activities being performed.
Could there be a better way to do this?
If your resulting filesystem is going to be a kernel filesystem driver, I would recommend using tracefs to gather the information you require. I would recommend against making this a kernel filesystem unless you have a lot of time and a lot of testing resources. It is not trivial.
If you want an easier, safer alternative, write your filesystem using fuse. The downside is that performance is not quite as good and there are a few places where it cannot be used, but it is often acceptable. Note that there is already an implementation of a logging filesystem under fuse.
use the strace -f (fork) option, also I suggest the -s 9999 for more details
I would like to monitor the number of threads used by a specific process on Linux.
Is there an easy way to get this information without impacting the performance of the process?
try
ps huH p <PID_OF_U_PROCESS> | wc -l
or htop
To get the number of threads for a given pid:
$ ps -o nlwp <pid>
Where nlwp stands for Number of Light Weight Processes (threads). Thus ps aliases nlwp to thcount, which means that
$ ps -o thcount <pid>
does also work.
If you want to monitor the thread count, simply use watch:
$ watch ps -o thcount <pid>
To get the sum of all threads running in the system:
$ ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'
Each thread in a process creates a directory under /proc/<pid>/task. Count the number of directories, and you have the number of threads.
cat /proc/<PROCESS_PID>/status | grep Threads
ps -eLf on the shell shall give you a list of all the threads and processes currently running on the system.
Or, you can run top command then hit 'H' to toggle thread listings.
$ ps H p pid-id
H - Lists all the individual threads in a process
or
$cat /proc/pid-id/status
pid-id is the Process ID
eg.. (Truncated the below output)
root#abc:~# cat /proc/8443/status
Name: abcdd
State: S (sleeping)
Tgid: 8443
VmSwap: 0 kB
Threads: 4
SigQ: 0/256556
SigPnd: 0000000000000000
If you use:
ps uH p <PID_OF_U_PROCESS> | wc -l
You have to subtract 1 to the result, as one of the lines "wc" is counting is the headers of the "ps" command.
My answer is more gui, but still within terminal. Htop may be used with a bit of setup.
Start htop.
Enter setup menu by pressing F2.
From leftmost column choose "Columns"
From rightmost column choose the column to be added to main monitoring output, "NLWP" is what you are looking for.
Press F10.
JStack is quite inexpensive - one option would be to pipe the output through grep to find active threads and then pipe through wc -l.
More graphically is JConsole, which displays the thread count for a given process.
Here is one command that displays the number of threads of a given process :
ps -L -o pid= -p <pid> | wc -l
Unlike the other ps based answers, there is here no need to substract 1 from its output as there is no ps header line thanks to the -o pid=option.
Newer JDK distributions ship with JConsole and VisualVM. Both are fantastic tools for getting the dirty details from a running Java process. If you have to do this programmatically, investigate JMX.
If you're looking for thread count for multiple processes, the other answers won't work well for you, since you won't see the process names or PIDs, which makes them rather useless. Use this instead:
ps -o pid,nlwp,args -p <pid_1> <pid_2> ... <pid_N>
In order to watch the changes live, just add watch:
watch ps -o pid,nlwp,args -p <pid_1> <pid_2> ... <pid_N>
jvmtop can show the current jvm thread count beside other metrics.
The easiest way is using "htop". You can install "htop" (a fancier version of top) which will show you all your cores, process and memory usage.
Press "Shift+H" to show all process or press again to hide it.
Press "F4" key to search your process name.
Installing on Ubuntu or Debian:
sudo apt-get install htop
Installing on Redhat or CentOS:
yum install htop
dnf install htop [On Fedora 22+ releases]
If you want to compile "htop" from source code, you will find it here.
If you are trying to find out the number of threads using cpu for a given pid I would use:
top -bc -H -n2 -p <pid> | awk '{if ($9 != "0.0" && $1 ~ /^[0-9]+$/) print $1 }' | sort -u | wc -l
If you want the number of threads per user in a linux system then you should use:
ps -eLf | grep <USER> | awk '{ num += $6 } END { print num }'
where as <USER> use the desired user name.
If you're interested in those threads which are really active -- as in doing something (not blocked, not timed_waiting, not reporting "thread running" but really waiting for a stream to give data) as opposed to sitting around idle but live -- then you might be interested in jstack-active.
This simple bash script runs jstack then filters out all the threads which by heuristics seem to be idling, showing you stack traces for those threads which are actually consuming CPU cycles.
First get the process ID (pid) by executing below command:
ps -ef | grep (for e.g ps -ef | grep java)
Now replace the pid in below command and execute to get the total thread count of a process.
ps huH p | wc -l
VisualVM can show clear states of threads of a given JVM process