Cannot set kernel dynamic debug on Linux? - linux

Question

I have already seen Cannot enable kernel dynamic debugging on linux ; https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html .
I have rebuilt the Raspbian 9 kernel with CONFIG_DYNAMIC_DEBUG, and booted into it; the file /sys/kernel/debug/dynamic_debug/control and is populated with 2k+ dynamic debug rule statements:
pi#raspberrypi:~ $ sudo ls -la /sys/kernel/debug/dynamic_debug/control
-rw-r--r-- 1 root root 0 Jan 1 1970 /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | wc -l
2358
pi#raspberrypi:~ $ sudo grep 'snd_device' /sys/kernel/debug/dynamic_debug/control
sound/core/device.c:132 [snd]snd_device_disconnect =_ "device disconnect %p (from %pS), not found\012"
sound/core/device.c:156 [snd]snd_device_free =_ "device free %p (from %pS), not found\012"
Ok, so I want to trace the is_connected_output_ep function, which is in sound/soc/soc-dapm.c. So I do this:
pi#raspberrypi:~ $ sudo bash -c "echo -n 'func is_connected_output_ep +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep is_conn
pi#raspberrypi:~ $
pi#raspberrypi:~ $ sudo bash -c "echo 'file sound/soc/soc-dapm.c line 1175 +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep dapm
pi#raspberrypi:~ $
... and I get no errors - but seemingly, nothing "sticks". (and yeah, I don't see this function being traced either).
The documentation says that +p does:
p enables the pr_debug() callsite.
I'm not sure what they mean by this - does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this? If so, what happens in the case when there are no such statements in the function - as is the case with is_connected_output_ep? Can I still setup dynamic debug to somehow trace this function - without having to manually insert printk or other statements and recompiling the kernel module?

Answer 1

Well, I did some more reading, and it seems the answer to:
does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this?
... is likely "yes" - so you cannot do dynamic debug of a function that does not have pr_debug statements in it already.
Also, it seems that the /sys/kernel/debug/dynamic_debug/control (upon read) is actually a list of all possible dynamic debug "probe points" if you will, along with their status (enabled or not), though I'm not sure about this.
Anyways, here is some more reading where this stuff is mentioned:
The dynamic debugging interface [LWN.net] 2011
Dynamic Debug, conference paper, 2009
So I cannot trace is_connected_output_ep with dynamic debug - so maybe I should look into ftrace or kprobes (dynamic probes) facilities of the Linux kernel...
EDIT: It turns out, dynamic_debug/control lists debuggable statements ONLY from currently loaded modules in the kernel! Example, there is a dev_dbg in the dpcm_path_get function in the soc-pcm.c source file, which ends up in the snd_soc_core kernel module (snd-soc-core.ko). This module by default is not loaded by Raspbian 9, so we get this:
pi#raspberrypi:~ $ lsmod | grep snd
snd_bcm2835 32768 1
snd_pcm 98304 1 snd_bcm2835
snd_timer 32768 1 snd_pcm
snd 69632 5 snd_timer,snd_bcm2835,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $
Ok, now if the kernel module is loaded with modprobe, now suddenly the debuggable callsites appear in dynamic_debug/control:
pi#raspberrypi:~ $ sudo modprobe snd_soc_core
pi#raspberrypi:~ $ lsmod | grep snd
snd_soc_core 200704 0
snd_compress 20480 1 snd_soc_core
snd_pcm_dmaengine 16384 1 snd_soc_core
snd_bcm2835 32768 1
snd_pcm 98304 3 snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer 32768 1 snd_pcm
snd 69632 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1367 [snd_soc_core]dpcm_prune_paths =_ "ASoC: pruning %s BE %s for %s\012"
sound/soc/soc-pcm.c:1373 [snd_soc_core]dpcm_prune_paths =_ "ASoC: found %d old BE paths for pruning\012"
...
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =_ "ASoC: found %d audio %s paths\012"
And finally, we can now enable this print statement:
pi#raspberrypi:~ $ sudo bash -c "echo 'func dpcm_path_get +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =p "ASoC: found %d audio %s paths\012"
Apparently, the disabled lines have a =_ symbol in the line, and the enabled lines have =p ...
Now all that I'd want, is to enable some statements before the driver is loaded, so I could monitor printouts in _probe functions of kernel module drivers...

Answer 2

You can add following argument to insmod
dyndbg="+p"