How can I see which CPU core a thread is running in? - linux

In Linux, supposing a thread's pid is [pid], from the directory /proc/[pid] we can get many useful information. For example, these proc files, /proc/[pid]/status,/proc/[pid]/stat and /proc/[pid]/schedstat are all useful. But how can I get the CPU core number that a thread is running in? If a thread is in sleep state, how can I know which core it will run after it is scheduled again?
BTW, is there a way to dump the process(thread) list of running and sleeping tasks for each CPU core?

The "top" command may help towards this, it does not have CPU-grouped list of threads but rather you can see the list of threads (probably for a single process) and which CPU cores the threads are running on by
top -H -p {PROC_ID}
then pressing f to go into field selection, j to enable the CPU core column, and Enter to display.

The answer below is no longer accurate as of 2014
Tasks don't sleep in any particular core. And the scheduler won't know ahead of time which core it will run a thread on because that will depend on future usage of those cores.
To get the information you want, look in /proc/<pid>/task/<tid>/status. The third field will be an 'R' if the thread is running. The sixth from the last field will be the core the thread is currently running on, or the core it last ran on (or was migrated to) if it's not currently running.
31466 (bc) S 31348 31466 31348 34819 31466 4202496 2557 0 0 0 5006 16 0 0 20 0 1 0 10196934 121827328 1091 18446744073709551615 4194304 4271839 140737264235072 140737264232056 217976807456 0 0 0 137912326 18446744071581662243 0 0 17 3 0 0 0 0 0
Not currently running. Last ran on core 3.
31466 (bc) R 31348 31466 31348 34819 31466 4202496 2557 0 0 0 3818 12 0 0 20 0 1 0 10196934 121827328 1091 18446744073709551615 4194304 4271839 140737264235072 140737264231824 4235516 0 0 0 2 0 0 0 17 2 0 0 0 0 0
Currently running on core 2.
To see what the rest of the fields mean, have a look at the Linux kernel source -- specifically the do_task_stat function in fs/proc/array.c or Documentation/filesystems/stat.txt.
Note that all of this information may be obsolete by the time you get it. It was true at some point between when you made the open call on the file in proc and when that call returned.

You can also use ps, something like this:
ps -mo pid,tid,%cpu,psr -p `pgrep BINARY-NAME`

The threads are not necessary to bound one particular Core (if you did not pin it). Therefore to see the continuous switching of the core you can use (a modified answer of Dmitry):
watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p \`pgrep BINARY-NAME\`
For example:
watch -tdn0.5 ps -mo pid,tid,%cpu,psr -p \`pgrep firefox\`

This can be done with top command. The default top command output does not show these details. To view this detail you will have to press f key while on top command interface and then press j(press Enter key after you pressed j). Now the output will show you details regarding a process and which processor its running. A sample output is shown below.
top - 04:24:03 up 96 days, 13:41, 1 user, load average: 0.11, 0.14, 0.15
Tasks: 173 total, 1 running, 172 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.1%us, 0.2%sy, 0.0%ni, 88.4%id, 0.1%wa, 0.0%hi, 0.0%si, 4.2%st
Mem: 1011048k total, 950984k used, 60064k free, 9320k buffers
Swap: 524284k total, 113160k used, 411124k free, 96420k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
12426 nginx 20 0 345m 47m 29m S 77.6 4.8 40:24.92 7 php-fpm
6685 mysql 20 0 3633m 34m 2932 S 4.3 3.5 63:12.91 4 mysqld
19014 root 20 0 15084 1188 856 R 1.3 0.1 0:01.20 4 top
9 root 20 0 0 0 0 S 1.0 0.0 129:42.53 1 rcu_sched
6349 memcache 20 0 355m 12m 224 S 0.3 1.2 9:34.82 6 memcached
1 root 20 0 19404 212 36 S 0.0 0.0 0:20.64 3 init
2 root 20 0 0 0 0 S 0.0 0.0 0:30.02 4 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:12.45 0 ksoftirqd/0
The P column in the output shows the processor core number where the process is currently being executed. Monitoring this for a few minutes will make you understand that a pid is switching processor cores in between. You can also verify whether your pid for which you have set affinity is running on that particular core only
top f navigation screen ( a live system example ) :
Fields Management for window 1:Def, whose current sort field is forest view
Navigate with Up/Dn, Right selects for move then <Enter> or Left commits,
'd' or <Space> toggles display, 's' sets sort. Use 'q' or <Esc> to end!
* PID = Process Id
* USER = Effective User Name
* PR = Priority
* NI = Nice Value
* VIRT = Virtual Image (KiB)
* RES = Resident Size (KiB)
* SHR = Shared Memory (KiB)
* S = Process Status
* %CPU = CPU Usage
* %MEM = Memory Usage (RES)
* TIME+ = CPU Time, hundredths
* COMMAND = Command Name/Line
PPID = Parent Process pid
UID = Effective User Id
RUID = Real User Id
RUSER = Real User Name
SUID = Saved User Id
SUSER = Saved User Name
GID = Group Id
GROUP = Group Name
PGRP = Process Group Id
TTY = Controlling Tty
TPGID = Tty Process Grp Id
SID = Session Id
nTH = Number of Threads
* P = Last Used Cpu (SMP)
TIME = CPU Time
SWAP = Swapped Size (KiB)
CODE = Code Size (KiB)
DATA = Data+Stack (KiB)
nMaj = Major Page Faults
nMin = Minor Page Faults
nDRT = Dirty Pages Count
WCHAN = Sleeping in Function
Flags = Task Flags <sched.h>
CGROUPS = Control Groups
SUPGIDS = Supp Groups IDs
SUPGRPS = Supp Groups Names
TGID = Thread Group Id
ENVIRON = Environment vars
vMj = Major Faults delta
vMn = Minor Faults delta
USED = Res+Swap Size (KiB)
nsIPC = IPC namespace Inode
nsMNT = MNT namespace Inode
nsNET = NET namespace Inode
nsPID = PID namespace Inode
nsUSER = USER namespace Inode
nsUTS = UTS namespace Inode

Accepted answer is not accurate. Here are the ways to find out which CPU is running the thread (or was the last one to run) at the moment of inquiry:
Directly read /proc/<pid>/task/<tid>/stat. Before doing so, make sure format didn't change with latest kernel. Documentation is not always up to date, but at least you can try https://www.kernel.org/doc/Documentation/filesystems/proc.txt. As of this writing, it will be the 14th value from the end.
Use ps. Either give it -F switch, or use output modifiers and add code PSR.
Use top with Last Used Cpu column (hitting f gets you to column selection)
Use htop with PROCESSOR column (hitting F2 gets you to setup screen)

To see the threads of a process :
ps -T -p PID
To see the thread run info
ps -mo pid,tid,%cpu,psr -p PID
Example :
/tmp # ps -T -p 3725
PID SPID TTY TIME CMD
3725 3725 ? 00:00:00 Apps
3725 3732 ? 00:00:10 t9xz1d920
3725 3738 ? 00:00:00 XTimer
3725 3739 ? 00:00:05 Japps
3725 4017 ? 00:00:00 QTask
3725 4024 ? 00:00:00 Kapps
3725 4025 ? 00:00:17 PTimer
3725 4026 ? 00:01:17 PTask
3725 4027 ? 00:00:00 RTask
3725 4028 ? 00:00:00 Recv
3725 4029 ? 00:00:00 QTimer
3725 4033 ? 00:00:01 STask
3725 4034 ? 00:00:02 XTask
3725 4035 ? 00:00:01 QTimer
3725 4036 ? 00:00:00 RTimer
3725 4145 ? 00:00:00 t9xz1d920
3725 4147 ? 00:00:02 t9xz1d920
3725 4148 ? 00:00:00 t9xz1d920
3725 4149 ? 00:00:00 t9xz1d920
3725 4150 ? 00:00:00 t9xz1d920
3725 4865 ? 00:00:02 STimer
/tmp #
/tmp #
/tmp # ps -mo pid,tid,%cpu,psr -p 3725
PID TID %CPU PSR
3725 - 1.1 -
- 3725 0.0 2
- 3732 0.1 0
- 3738 0.0 0
- 3739 0.0 0
- 4017 0.0 6
- 4024 0.0 3
- 4025 0.1 0
- 4026 0.7 0
- 4027 0.0 3
- 4028 0.0 7
- 4029 0.0 0
- 4033 0.0 4
- 4034 0.0 1
- 4035 0.0 0
- 4036 0.0 2
- 4145 0.0 2
- 4147 0.0 0
- 4148 0.0 5
- 4149 0.0 2
- 4150 0.0 7
- 4865 0.0 0
/tmp #

Related

What is using so much memory on an idle linux server? Comparing output of "htop" and "ps aux"

I am trying to understand and compare the output I see from htop (sorted by mem%) and "ps aux --sort=-%mem | grep query.jar" and determine why 24.2G out of 32.3G is in use on an idle server.
The ps command shows a single parent (not child process I assume):
ps aux --sort=-%mem | grep query.jar
1000 67970 0.4 4.4 6721304 1452512 ? Sl 2020 163:55 java -Djava.security.egd=file:/dev/./urandom -Xmx700m -Xss256k -jar ./query.jar
Whereas htop shows PID 6790 as well as many other PIDs for query.jar below. I am trying to grasp what this means for memory usage. I also wonder if this has anything to do with open file handlers.
I ran this file handler command on the server: ls -la /proc/$$/fd
which produces this output (although I am not sure if this is showing me any issues):
total 0
lrwx------. 1 ziggy ziggy 64 Jan 2 09:14 0 -> /dev/pts/1
lrwx------. 1 ziggy ziggy 64 Jan 2 09:14 1 -> /dev/pts/1
lrwx------. 1 ziggy ziggy 64 Jan 2 09:14 2 -> /dev/pts/1
lrwx------. 1 ziggy ziggy 64 Jan 2 11:39 255 -> /dev/pts/1
lr-x------. 1 ziggy ziggy 64 Jan 2 09:14 3 -> /var/lib/sss/mc/passwd
Obviously the mem% output in htop (if totaled) exceeds 100% so I am guessing that despite there being different pids, the repetitive mem% values shown of 9.6 and 4.4 are not necessarily unique. Any clarification is appreciated here. I am trying to determine the best method to accurately report what is using 24.GB of memory on this server.
The complete output of the ps aux command is here which shows me all the different pids using memory. Again, I am confused by how this output differs form htop.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1000 40268 0.2 9.5 3432116 3143516 ? Sl 2020 73:33 /usr/local/bin/node --max-http-header-size=65000 index.js
1000 67970 0.4 4.4 6721304 1452516 ? Sl 2020 164:05 java -Djava.security.egd=file:/dev/./urandom -Xmx700m -Xss256k -jar ./query.jar
root 86212 2.6 3.0 15208548 989928 ? Ssl 2020 194:18 dgraph alpha --my=dgraph-public:9080 --lru_mb 2048 --zero dgraph-public:5080
1000 68027 0.2 2.9 6295452 956516 ? Sl 2020 71:43 java -Djava.security.egd=file:/dev/./urandom -Xmx512m -Xss256k -jar ./build.jar
1000 88233 0.3 2.9 6415084 956096 ? Sl 2020 129:25 java -Djava.security.egd=file:/dev/./urandom -Xmx500m -Xss256k -jar ./management.jar
1000 66554 0.4 2.4 6369108 803632 ? SLl 2020 159:23 ./TranslationService thrift sisense-zookeeper.sisense:2181 S1
polkitd 27852 1.2 2.3 2111292 768376 ? Ssl 2020 417:24 mongod --config /data/configdb/mongod.conf --bind_ip_all
root 52493 3.3 2.3 8361444 768188 ? Ssl 2020 1107:53 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --storage.tsdb.retention.size=7G
B --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://sisense-prom-oper
ator-prom-prometheus.monitoring:9090 --web.route-prefix=/
1000 54574 0.0 1.9 901996 628900 ? Sl 2020 13:47 /usr/local/bin/node dist/index.js
root 78245 0.9 1.9 11755696 622940 ? Ssl 2020 325:03 /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
root 5838 4.4 1.4 781420 484736 ? Ssl 2020 1488:26 kube-apiserver --advertise-address=10.1.17.71 --allow-privileged=true --anonymous-auth=True --apiserver-count=1 --authorization-mode=Node,RBAC --bind-addre
ss=0.0.0.0 --client-ca-file=/etc/kubernetes/ssl/ca.crt --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=False --enable-bootstrap-token-auth=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/ssl
/etcd/ssl/ca.pem --etcd-certfile=/etc/ssl/etcd/ssl/node-dev-analytics-2.pem --etcd-keyfile=/etc/ssl/etcd/ssl/node-dev-analytics-2-key.pem --etcd-servers=https://10.1.17.71:2379 --insecure-port=0 --kubelet-client-certificat
e=/etc/kubernetes/ssl/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP --profiling=
False --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key --request-timeout=1m0s --requestheader-allowed-names=front-proxy-client --request
header-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --runtime-config
= --secure-port=6443 --service-account-key-file=/etc/kubernetes/ssl/sa.pub --service-cluster-ip-range=10.233.0.0/18 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/etc/kubernetes/ssl/apiserve
r.crt --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
1000 91921 0.1 1.2 7474852 415516 ? Sl 2020 41:04 java -Xmx4G -server -Dfile.encoding=UTF-8 -Djvmp -DEC2EC -cp /opt/sisense/jvmConnectors/jvmcontainer_1_1_0.jar com.sisense.container.launcher.ContainerLaunc
herApp /opt/sisense/jvmConnectors/connectors/ec2ec/com.sisense.connectors.Ec2ec.jar sisense-zookeeper.sisense:2181 connectors.sisense
1000 21035 0.3 0.8 2291908 290568 ? Ssl 2020 111:23 /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /zookeeper-3.4.12/bin/../build/classes:/zookeeper-
3.4.12/bin/../build/lib/*.jar:/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/zookeeper-3.4.12/bin/../lib/log4j-1
.2.17.jar:/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/conf: -XX:MaxRA
MFraction=2 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMa
in /conf/zoo.cfg
1000 91955 0.1 0.8 7323208 269844 ? Sl 2020 40:40 java -Xmx4G -server -Dfile.encoding=UTF-8 -Djvmp -DGenericJDBC -cp /opt/sisense/jvmConnectors/jvmcontainer_1_1_0.jar com.sisense.container.launcher.Containe
rLauncherApp /opt/sisense/jvmConnectors/connectors/genericjdbc/com.sisense.connectors.GenericJDBC.jar sisense-zookeeper.sisense:2181 connectors.sisense
1000 92076 0.1 0.8 8302704 262772 ? Sl 2020 52:11 java -Xmx4G -server -Dfile.encoding=UTF-8 -Djvmp -Dsql -cp /opt/sisense/jvmConnectors/jvmcontainer_1_1_0.jar com.sisense.container.launcher.ContainerLaunche
rApp /opt/sisense/jvmConnectors/connectors/mssql/com.sisense.connectors.MsSql.jar sisense-zookeeper.sisense:2181 connectors.sisense
1000 91800 0.1 0.7 9667560 259928 ? Sl 2020 39:38 java -Xms128M -jar connectorService.jar jvmcontainer_1_1_0.jar /opt/sisense/jvmConnectors/connectors sisense-zookeeper.sisense:2181 connectors.sisense
1000 91937 0.1 0.7 7326312 253708 ? Sl 2020 40:14 java -Xmx4G -server -Dfile.encoding=UTF-8 -Djvmp -DExcel -cp /opt/sisense/jvmConnectors/jvmcontainer_1_1_0.jar com.sisense.container.launcher.ContainerLaunc
herApp /opt/sisense/jvmConnectors/connectors/excel/com.sisense.connectors.ExcelConnector.jar sisense-zookeeper.sisense:2181 connectors.sisense
1000 92085 0.1 0.7 7323660 244160 ? Sl 2020 39:53 java -Xmx4G -server -Dfile.encoding=UTF-8 -Djvmp -DSalesforceJDBC -cp /opt/sisense/jvmConnectors/jvmcontainer_1_1_0.jar com.sisense.container.launcher.Conta
inerLauncherApp /opt/sisense/jvmConnectors/connectors/salesforce/com.sisense.connectors.Salesforce.jar sisense-zookeeper.sisense:2181 connectors.sisense
1000 16326 0.1 0.7 3327260 243804 ? Sl 2020 12:21 /opt/sisense/monetdb/bin/mserver5 --zk_system_name=S1 --zk_address=sisense-zookeeper.sisense:2181 --external_server=ec-devcube-qry-10669921-96e0-4.sisense -
-instance_id=qry-10669921-96e0-4 --dbname=aDevCube --farmstate=Querying --dbfarm=/tmp/aDevCube_2020.12.28.16.46.23.280/dbfarm --set mapi_port 50000 --set gdk_nr_threads 4
100 64158 20.4 0.7 1381624 232548 ? Sl 2020 6786:08 /usr/local/lib/erlang/erts-11.0.3/bin/beam.smp -W w -K true -A 128 -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000
-stbt db -zdbbl 128000 -B i -- -root /usr/local/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa -noshell -noinput -s rabbit boot -boot start_sasl -lager crash_log false -lager handlers []
root 1324 11.3 0.7 5105748 231100 ? Ssl 2020 3773:15 /usr/bin/dockerd --iptables=false --data-root=/var/lib/docker --log-opt max-size=50m --log-opt max-file=5 --dns 10.233.0.3 --dns 10.1.22.68 --dns 10.1.22.6
9 --dns-search default.svc.cluster.local --dns-search svc.cluster.local --dns-opt ndots:2 --dns-opt timeout:2 --dns-opt attempts:2
Adding more details:
$ free -m
Mem: 31993 23150 2602 1677 6240 6772
Swap: 0 0 0
$ top -b -n1 -o "%MEM"|head -n 20
top - 13:46:18 up 23 days, 3:26, 3 users, load average: 2.26, 1.95, 2.10
Tasks: 2201 total, 1 running, 2199 sleeping, 1 stopped, 0 zombie
%Cpu(s): 4.4 us, 10.3 sy, 0.0 ni, 85.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32761536 total, 2639584 free, 23730688 used, 6391264 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 6910444 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
40268 1000 20 0 3439284 3.0g 8228 S 0.0 9.6 73:39.94 node
67970 1000 20 0 6721304 1.4g 7216 S 0.0 4.4 164:24.16 java
86212 root 20 0 14.5g 996184 13576 S 0.0 3.0 197:36.83 dgraph
68027 1000 20 0 6295452 956516 7256 S 0.0 2.9 71:52.15 java
88233 1000 20 0 6415084 956096 9556 S 0.0 2.9 129:40.80 java
66554 1000 20 0 6385500 803636 8184 S 0.0 2.5 159:42.44 TranslationServ
27852 polkitd 20 0 2111292 766860 11368 S 0.0 2.3 418:26.86 mongod
52493 root 20 0 8399864 724576 15980 S 0.0 2.2 1110:34 prometheus
54574 1000 20 0 905324 631708 7656 S 0.0 1.9 13:48.66 node
78245 root 20 0 11.2g 623028 1800 S 0.0 1.9 325:43.74 fluent-bit
5838 root 20 0 781420 477016 22944 S 7.7 1.5 1492:08 kube-apiserver
91921 1000 20 0 7474852 415516 3652 S 0.0 1.3 41:10.25 java
21035 1000 20 0 2291908 290484 3012 S 0.0 0.9 111:38.03 java
The primary difference between htop and ps aux is that htop shows each individual thread belonging to a process rather than the process only - this is similar to ps auxm. Using the htop interactive command H, you can hide threads to get to a list that more closely corresponds to ps aux.
In terms of memory usage, those additional entries representing individual threads do not affect the actual memory usage total because threads share the address space of the associated process.
RSS (resident set size) in general is problematic because it does not adequately represent shared pages (due to shared memory or copy-on-write) for your purpose - the sum can be higher than expected in those cases. You can use smem -t to get a better picture with the PSS (proportional set size) column. Based on the facts you provided, that is not your issue, though.
In your case, it might make sense to dig deeper via smem -tw to get a memory usage breakdown that includes (non-cache) kernel resources. /proc/meminfo provides further details.

Which one is PID1: /sbin/init or systemd

I'm using Arch Linux. I have read about systemd, and as I understand it, systemd is the first process, and it starts the rest of the processes.
But when I use:
ps -aux
The result shows that /sbin/init has PID 1. And when I use:
pstree -Apn
The result shows that systemd has PID 1. Which is correct? Is /sbin/init starting systemd?
They're probably both right.
$ sudo ls -ltrh /proc/1/exe
[sudo] password for user:
lrwxrwxrwx 1 root root 0 May 30 21:22 /proc/1/exe -> /lib/systemd/systemd
$ echo $(tr '\0' ' ' < /proc/1/cmdline )
/sbin/init splash
$ stat /sbin/init
File: '/sbin/init' -> '/lib/systemd/systemd'
Size: 20 Blocks: 0 IO Block: 4096 symbolic link
Device: 801h/2049d Inode: 527481 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-05-30 21:27:12.058023583 -0500
Modify: 2016-10-26 08:04:58.000000000 -0500
Change: 2016-11-19 11:38:45.749226284 -0600
Birth: -
The commands above show us:
what is the file corresponding to pid 1's executable image?
what was invoked (passed to exec()) when pid 1 was started?
what are the characteristics of the path at /sbin/init?
On my system, /sbin/init is a symlink to "/lib/systemd/systemd". This is likely similar to your system. We can see what information ps -aux is using by straceing it.
$ strace ps -aux
...
open("/proc/1/cmdline", O_RDONLY) = 6
read(6, "/sbin/init\0splash\0", 131072) = 18
read(6, "", 131054) = 0
close(6) = 0
...
and likewise for pstree:
$ strace pstree -Apn
...
getdents(3, /* 332 entries */, 32768) = 8464
open("/proc/1/stat", O_RDONLY) = 4
stat("/proc/1", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(4, "1 (systemd) S 0 1 1 0 -1 4194560"..., 8192) = 192
read(4, "", 7168) = 0
open("/proc/1/task", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5
...
So the difference in output is because they use different sources of information. /proc/1/cmdline tells us how the process was invoked. Whereas /proc/1/stat shows that the process' name is systemd.
$ cat /proc/1/stat
1 (systemd) S 0 1 1 0 -1 4194560 34371 596544 1358 3416 231 144 298 1758 20 0 1 0 4 190287872 772 18446744073709551615 1 1 0 0 0 0 671173123 4096 1260 0 0 0 17 2 0 0 12188 0 0 0 0 0 0 0 0 0 0

CPU and HDD information

I searched but I found nothing for my problem.
I would like to have in Linux command line the information about the CPU usage and the local HDDs with formatting text like exactly as the examples below for my program.
These examples are command line outputs on MS-Windows.
I hope it is possible on Linux, too.
Thank you
wmic logicaldisk where drivetype=3 get caption,freespace,size
Caption FreeSpace Size
C: 135314194432 255953203200
D: 126288519168 128033222656
E: 336546639872 1000194015232
F: 162184503296 1000194015232
wmic cpu get loadpercentage
LoadPercentage
4
You won't find anything exactly like the output you provided.
The only option is to use for disk space df:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 73216256 27988724 41485276 41% /
devtmpfs 8170164 0 8170164 0% /dev
tmpfs 8203680 544 8203136 1% /dev/shm
tmpfs 8203680 12004 8191676 1% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 8203680 0 8203680 0% /sys/fs/cgroup
/dev/sdb1 482922 83939 374049 19% /boot
and for cpu you have many more options, e.g.
vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 11865304 149956 1474172 0 0 53 46 126 707 3 0 96 0 0
or top -b | head:
top - 21:48:43 up 54 min, 1 user, load average: 0.13, 0.17, 0.22
Tasks: 188 total, 1 running, 187 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.0 us, 0.4 sy, 0.1 ni, 96.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16407364 total, 11848936 free, 2888844 used, 1669584 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 13230972 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 40544 6440 3780 S 0.0 0.0 0:01.15 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
There is no command that gives you a load percentage of the cpu. It's actually impossible to get that with a system call (nor in linux neither in Windows). What you can get is the number of ticks currently executed (for each field, user, system, io,irq idle)..., then call it again a certain amount of time later and calculate it. That way is how work all the commands for reading the cpu percentage.
Here a script bash that gives you that: (just create a file named for example cpu.sh paste this code and execute to see the results)
_estado()
{
cat /proc/stat | grep "cpu " | sed -e 's/ */:/g' -e 's/^cpux//'
}
_ticksconcretos()
{
echo $1 | cut -d ':' -f $2
}
while true ; do
INICIAL=$(_estado)
sleep 1
FINAL=$(_estado)
UsuarioI=$(_ticksconcretos $INICIAL 1)
UsuarioF=$(_ticksconcretos $FINAL 1)
NiceI=$(_ticksconcretos $INICIAL 2)
NiceF=$(_ticksconcretos $FINAL 2)
SistemaI=$(_ticksconcretos $INICIAL 3)
SistemaF=$(_ticksconcretos $FINAL 3)
idleI=$(_ticksconcretos $INICIAL 4)
idleF=$(_ticksconcretos $FINAL 4)
IOI=$(_ticksconcretos $INICIAL 5)
IOF=$(_ticksconcretos $FINAL 5)
IRQI=$(_ticksconcretos $INICIAL 6)
IRQF=$(_ticksconcretos $FINAL 6)
SOFTIRQI=$(_ticksconcretos $INICIAL 7)
SOFTIRQF=$(_ticksconcretos $FINAL 7)
STEALI=$(_ticksconcretos $INICIAL 8)
STEALF=$(_ticksconcretos $FINAL 8)
InactivoF=$(( $idleF + $IOF ))
InactivoI=$(( $idleI + $IOI ))
ActivoI=$(( $UsuarioI + $NiceI + $SistemaI + $IRQI + $SOFTIRQI + $STEALI ))
ActivoF=$(( $UsuarioF + $NiceF + $SistemaF + $IRQF + $SOFTIRQF + $STEALF ))
TOTALI=$(( $ActivoI + $InactivoI ))
TOTALF=$(( $ActivoF + $InactivoF ))
PORC=$(( ( ( ( $TOTALF - $TOTALI ) - ( $InactivoF - $InactivoI ) ) * 100 / ( $TOTALF - $TOTALI ) ) ))
clear
echo "CPU: $PORC %"
done
For the free space You could use something like this:
df -h -x tmpfs -x devtmpfs | awk -F " " '{print $1 " " $4 " " $2}'
wich will give you this output:
Filesystem Free Size
/dev/sda1 16G 25G
/dev/sda5 46G 79G
/dev/sdb8 130G 423G
sda represents the first disk, sda1 the first partition, sda2, the second one etc. you can add (or change) $6 inside the print to get the mount points instead of the partitions, change the order or even more things.

perf : How to check processess running on particular cpu

Is there any option in perf to look into processes running on a particular cpu /core, and how much percentage of that core is taken by each process.
Reference links would be helpful.
perf is intended to do a profiling which is not good fit for your case. You may try to do sampling /proc/sched_debug (if it is compiled in your kernel). For example you may check which process is currently running on CPU:
egrep '^R|cpu#' /proc/sched_debug
cpu#0, 917.276 MHz
R egrep 2614 37730.177313 ...
cpu#1, 917.276 MHz
R bash 2023 218715.010833 ...
By using his PID as a key, you may check how many CPU time in milliseconds it consumed:
grep se.sum_exec_runtime /proc/2023/sched
se.sum_exec_runtime : 279346.058986
However, as #BrenoLeitão mentioned, SystemTap is quite useful for your script. Here is script for your task.
global cputimes;
global cmdline;
global oncpu;
global NS_PER_SEC = 1000000000;
probe scheduler.cpu_on {
oncpu[pid()] = local_clock_ns();
}
probe scheduler.cpu_off {
if(oncpu[pid()] == 0)
next;
cmdline[pid()] = cmdline_str();
cputimes[pid(), cpu()] <<< local_clock_ns() - oncpu[pid()];
delete oncpu[pid()];
}
probe timer.s(1) {
printf("%6s %3s %6s %s\n", "PID", "CPU", "PCT", "CMDLINE");
foreach([pid+, cpu] in cputimes) {
cpupct = #sum(cputimes[pid, cpu]) * 10000 / NS_PER_SEC;
printf("%6d %3d %3d.%02d %s\n", pid, cpu,
cpupct / 100, cpupct % 100, cmdline[pid]);
}
delete cputimes;
}
It traces moments when process is running on CPU and stops execution on that (due to migration or sleeping) by attaching to scheduler.cpu_on and scheduler.cpu_off probes. Second probe calculates time difference between these events and saves it to cputimes aggregation along with process command line arguments.
timer.s(1) fires once per second -- it walks over aggregation and calculates percentage. Here is sample output for Centos 7 with bash running infinite loop:
0 0 100.16
30 1 0.00
51 0 0.00
380 0 0.02 /usr/bin/python -Es /usr/sbin/tuned -l -P
2016 0 0.08 sshd: root#pts/0 "" "" "" ""
2023 1 100.11 -bash
2630 0 0.04 /usr/libexec/systemtap/stapio -R stap_3020c9e7ba76838179be68cd2390a10c_2630 -F3
I understand that perf is not the proper way to do it, although you can limit perf per CPU, as using perf record -C <cpulist> or even perf stat -c <cpulist>.
The close you are going to see is the context-switch event, but, this is not going to provide you the application names at all.
I think you are going to need something more powerful, as systemtap.

kipmi0 kernel thread schedule class

What's the schedule class and priority of kipmi0 supposed to be?
I checked some servers and the ps output of kipmi0 was:
STARTED USER PID PPID PGID TT TID %MEM %CPU VSZ CLS NI PRI STAT WCHAN COMMAND
Nov 03 root 52 2 0 ? 52 0.0 0.0 0 TS 19 0 SN ipmi_thread \_ [kipmi0]
Seems the kipmi0 kernel thread should be TS schedule which means SCHED_OTHER.
While for one of my server(Redhat5.x 2.6.18-238.el5), it's RR which means SCHED_RR:
STARTED USER PID PPID PGID TT TID %MEM %CPU VSZ CLS NI PRI STAT WCHAN COMMAND
Dec 04 root 1755 79 1 ? 1755 0.0 0.6 0 RR - 139 SN ? \_ [kipmi0]
Does anyone know any factor which may cause such situation?
Answer to my own question:
The default schedule class of Kipmi is TS while it could be changed by chrt. We just find out this command from an product initial script: chrt -p --rr 99 $pid_of_kipmi
It's added for some reason, and regarding our finial solution,
echo 100 > /sys/module/ipmi_si/parameters/kipmid_max_busy_us
cat /etc/modprobe.d/ipmi.conf
options ipmi_si kipmid_max_busy_us=100

Resources