I'm trying to configure and enable ftrace on my ubuntu machine but its doesn't work. I also tried the instructions from the link here. I'm sure I'm missing something minor but I can't tell what.
Configuration:
$ cat /boot/config-5.15.0-56-generic | grep CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_TRACER=y
$ cat /boot/config-5.15.0-56-generic | grep CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_FUNCTION_GRAPH_TRACER=y
$ cat /boot/config-5.15.0-56-generic | grep CONFIG_STACK_TRACER
CONFIG_STACK_TRACER=y
$ cat /boot/config-5.15.0-56-generic | grep CONFIG_DYNAMIC_FTRACE
CONFIG_DYNAMIC_FTRACE=y
$ uname -a
Linux <redacted> 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Enable tracing (with root user)
$ echo function > /sys/kernel/debug/tracing/current_trace
$ echo do_sys_open > /sys/kernel/debug/tracing/set_ftrace_filter
$ echo 1 > tracing_on
debugfs is already mounted:
$ mount
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
After enabling the tracing I was expecting to see the traces in /sys/kernel/debug/tracing/trace or /sys/kernel/debug/tracing/trace_pipe files but I didn't see any traces.
Update: Instructions to generate the traces are working as expected. I was able to generate the traces successfully for do_sys_openat2 but with do_sys_open I don't see any traces.
Related
# cat /etc/oracle-release
Oracle Linux Server release 7.8
uname -a
Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cat 99-custom.rules
ACTION!="add|change", GOTO="rule_end"
ENV{ID_VENDOR}=="NETAPP*", RUN+="/bin/sh -c 'echo 4096 > /sys%p/queue/max_sectors_kb'"
LABEL="rule_end"
----2nd option using KERNEL option just for dm-* devices
ACTION!="add|change", GOTO="rule_end"
KERNEL=="dm-[0-9]*",ENV{ID_VENDOR}=="NETAPP*", RUN+="/bin/sh -c 'echo 4096 > /sys%p/queue/max_sectors_kb'"
LABEL="rule_end"
udevadm control --reload-rules && udevadm trigger
/sbin/udevadm trigger --type=devices --action=change
Issue : We don't see udev rule applied to all dm-* device
/sys/block/dm-19/queue/max_sectors_kb:512
/sys/block/dm-9/queue/max_sectors_kb:512
We even tried to remove ENV{ID_VENDOR}=="NETAPP*" from rule
What we need to apply max_sectors_kb to 4096 to all dm-* ( multipath devices )
and all sd* devices can be left default of 512
Any idea what could be wrong ?
I have already seen Cannot enable kernel dynamic debugging on linux ; https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html .
I have rebuilt the Raspbian 9 kernel with CONFIG_DYNAMIC_DEBUG, and booted into it; the file /sys/kernel/debug/dynamic_debug/control and is populated with 2k+ dynamic debug rule statements:
pi#raspberrypi:~ $ sudo ls -la /sys/kernel/debug/dynamic_debug/control
-rw-r--r-- 1 root root 0 Jan 1 1970 /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | wc -l
2358
pi#raspberrypi:~ $ sudo grep 'snd_device' /sys/kernel/debug/dynamic_debug/control
sound/core/device.c:132 [snd]snd_device_disconnect =_ "device disconnect %p (from %pS), not found\012"
sound/core/device.c:156 [snd]snd_device_free =_ "device free %p (from %pS), not found\012"
Ok, so I want to trace the is_connected_output_ep function, which is in sound/soc/soc-dapm.c. So I do this:
pi#raspberrypi:~ $ sudo bash -c "echo -n 'func is_connected_output_ep +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep is_conn
pi#raspberrypi:~ $
pi#raspberrypi:~ $ sudo bash -c "echo 'file sound/soc/soc-dapm.c line 1175 +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep dapm
pi#raspberrypi:~ $
... and I get no errors - but seemingly, nothing "sticks". (and yeah, I don't see this function being traced either).
The documentation says that +p does:
p enables the pr_debug() callsite.
I'm not sure what they mean by this - does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this? If so, what happens in the case when there are no such statements in the function - as is the case with is_connected_output_ep? Can I still setup dynamic debug to somehow trace this function - without having to manually insert printk or other statements and recompiling the kernel module?
Well, I did some more reading, and it seems the answer to:
does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this?
... is likely "yes" - so you cannot do dynamic debug of a function that does not have pr_debug statements in it already.
Also, it seems that the /sys/kernel/debug/dynamic_debug/control (upon read) is actually a list of all possible dynamic debug "probe points" if you will, along with their status (enabled or not), though I'm not sure about this.
Anyways, here is some more reading where this stuff is mentioned:
The dynamic debugging interface [LWN.net] 2011
Dynamic Debug, conference paper, 2009
So I cannot trace is_connected_output_ep with dynamic debug - so maybe I should look into ftrace or kprobes (dynamic probes) facilities of the Linux kernel...
EDIT: It turns out, dynamic_debug/control lists debuggable statements ONLY from currently loaded modules in the kernel! Example, there is a dev_dbg in the dpcm_path_get function in the soc-pcm.c source file, which ends up in the snd_soc_core kernel module (snd-soc-core.ko). This module by default is not loaded by Raspbian 9, so we get this:
pi#raspberrypi:~ $ lsmod | grep snd
snd_bcm2835 32768 1
snd_pcm 98304 1 snd_bcm2835
snd_timer 32768 1 snd_pcm
snd 69632 5 snd_timer,snd_bcm2835,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $
Ok, now if the kernel module is loaded with modprobe, now suddenly the debuggable callsites appear in dynamic_debug/control:
pi#raspberrypi:~ $ sudo modprobe snd_soc_core
pi#raspberrypi:~ $ lsmod | grep snd
snd_soc_core 200704 0
snd_compress 20480 1 snd_soc_core
snd_pcm_dmaengine 16384 1 snd_soc_core
snd_bcm2835 32768 1
snd_pcm 98304 3 snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer 32768 1 snd_pcm
snd 69632 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1367 [snd_soc_core]dpcm_prune_paths =_ "ASoC: pruning %s BE %s for %s\012"
sound/soc/soc-pcm.c:1373 [snd_soc_core]dpcm_prune_paths =_ "ASoC: found %d old BE paths for pruning\012"
...
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =_ "ASoC: found %d audio %s paths\012"
And finally, we can now enable this print statement:
pi#raspberrypi:~ $ sudo bash -c "echo 'func dpcm_path_get +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =p "ASoC: found %d audio %s paths\012"
Apparently, the disabled lines have a =_ symbol in the line, and the enabled lines have =p ...
Now all that I'd want, is to enable some statements before the driver is loaded, so I could monitor printouts in _probe functions of kernel module drivers...
You can add following argument to insmod
dyndbg="+p"
I'm trying to configure Suspend/Hibernate with my laptop and I issued some troubles.
I noticed the following troubles:
Suspend :
- Switching the lid off of my laptop : When I switch the lid on of the laptop, nothing happens. I must push down the power button to force the shutdown and then switch on the laptop.
- Typing systemctl suspend : same as previously.
Hibernate :
- Typing systemctl hibernate : the laptop seems to shutdown
I read those following links for helping me :
Hibernate with swap file
Suspend and hibernate
My system :
4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 GNU/Linux
My swap :
$ cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=403661c1-c7b4-47a8-9493-c5c0262ce14e / ext4 errors=remount-ro 0 1
UUID=BF40-CD4F /boot/efi vfat umask=0077 0 1
/swapfile none swap sw 0 0
/swap swap swap defaults 0 0
What I've done :
Create swap file
$ fallocate -l 8g /swap
$ mkswap /swap
Add swappiness
sysctl -w vm.swappiness=1
$ cat /etc/sysctl.conf | grep swappiness
vm.swappiness=1
$ swapon /swap
Configure grub
$ cat /etc/default/grub | grep -i grub_cmdline_linux_default
GRUB_CMDLINE_LINUX_DEFAULT="resume=/swap resume_offset=60354560 quiet splash"
$ sudo filefrag -v /swap | head -n4
Filesystem type is: ef53
File size of /swap is 8589934592 (2097152 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected: flags:
0: 0.. 0: 60354560.. 60354560: 1:
According to previous links I could configure /etc/mkinitcpio.conf but there is no file like this in my system.
So, I don't really know how to configure my initramfs.
Here is my configuration from /sys/power :
$ cat /sys/power/disk
[platform] shutdown reboot suspend test_resume
$ cat /sys/power/mem_sleep
s2idle [deep]
$ cat /sys/power/image_size
3261943808
$ cat /sys/power/resume
0:0
Could you give me some hints to make a step forward. Thanks you.
When I run
uname -r
I get 3.2.0-4-686-pae on my Debian7 and 4.4.26-gentoo on my Gentoo.
But what I can't find anywhere is what Linux kernel source versions do these distributions use. If I understand right all distributions take the kernel source code from here https://www.kernel.org. So how can I find out what Linux kernel versions from the site do use 4.4.26-gentoo or 3.2.0-4-686-pae, for example?
Try man uname and find -v for the kernel version used. ;)
So it is uname -v or uname -a if you just dont care about specifics and want to see it all. :p
$ uname -v
#1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
$ uname -a
Linux Phobos 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
Check out the difference between the uname -r part and the uname -v part in uname -a (the first version number in -a)
$ uname -r
3.16.0-4-amd64
you have some way to know linux kernel version :
good way
uname -r
bad way to see kernel version ( you should see version of vmlinu* initrd files if they have version number of kernel )
ls /boot | grep "vmlinu*"; ls /boot | grep "initrd*"
you can try this way to know all of kernel versions on your machine :
ls /lib/modules/
uname -r is the easiest way.
though there are utilities like screenfetch or neofetch that give you more information in a clear and pretty manner.
Just run screenfetch and it'll display details.
(screenfetch is usually pre-installed on most distros these days).
If it is not installed, just run sudo apt install screenfetch (on debian based distros like ubuntu) or sudo pacman -S screenfetch ( on Arch based distros.
If you are looking for original Linux kernel version that matches browse-able
https://elixir.bootlin.com/linux/vx.y.z/
or LINUX_KERNEL_CODE you check when developing kernel code, "uname" is not the right answer!
There are two methods (I use my Ubuntu 16.04 as example):
dmesg
dmesg | grep "Linux\ version"
[ 0.000000] Linux version 4.4.0-210-generic (buildd#lgw01-
amd64-009) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-
6ubuntu1~16.04.12) ) #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021
(Ubuntu 4.4.0-210.242-generic 4.4.262)
The original kernel version is 4.4.262! And 4.4.0-210-generic is Ubuntu's kernel number.
Install the kernel source use command like
sudo apt-get install linux-source
Look at the newly installed file ('uname -r' => 4.4.0-210-generic)
/lib/modules/4.4.0-210-generic/build/include/generated/uapi/linux/version.h
#define LINUX_VERSION_CODE 263423
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))
That version code is 0x0404FF indicating 4.4.[>=255] (where FF is for when patch exceeds 255).
-------- interesting reading below--------
It is interesting to look at the file
/lib/modules/4.4.0-210-generic/build/include/generated/utsrelease.h
#define UTS_RELEASE "4.4.0-210-generic"
#define UTS_UBUNTU_RELEASE_ABI 210
Now we know where uname and /proc/version get their number.
On my Ubuntu 20.04 5.4.0-80-generic
dmesg | grep "Linux\ versin"
[ 0.000000] Linux version 5.4.0-80-generic (buildd#lcy01-
amd64-030) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04))
#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021
(Ubuntu 5.4.0-80.90-generic **5.4.124**)
and LINUX_KERNEL_CODE is
$ cat /lib/modules/5.4.0-80-generic/build/include/generated/uapi/linux/version.h
#define LINUX_VERSION_CODE 328828
where 328828 = 0x05047C, indicating 5.4.124.
When I am trying to set kernel.softlockup_panic value to 30 instead of 0. I am getting Invalid argument error
# sysctl -w kernel.softlockup_panic=30
Error message
# sysctl: setting key "kernel.softlockup_panic": Invalid argument
kernel.softlockup_panic = 30
Can anyone please guide?
I am using SLES12 64bit OS
# uname -a
Linux linux-r1qg 3.12.28-4-default #1 SMP Thu Sep 25 17:02:34 UTC 2014 (9879bd4) x86_64 x86_64 x86_64 GNU/Linux
I Have used syctl for other purposes. Guess this params is not supported by kernel .
you can check this thru either of the commands
sudo sysctl -a | grep softlockup
cat /proc/sys /etc/sysctl.conf