I am going to monitor dmesg by calling:
dmesg -c
every 1 second. I will look for specific strings in the output and then I will execute an action depending on the found strings.
However, I am worried about this scenario:
The Kernel prints a 20-chars message into the dmesg ring buffer, for example:
printk("Hello World !!!!!!!!"))
Right when the Kernel is printing a specific char... let's say the 10th char, the user calls dmesg
The last line was not completely written by the Kernel when we read the buffer
What will "dmesg -c" print?
a) A truncated line with only its first 10 chars (Hello Worl)
b) Omits the pending line
And then, in a next call of "dmesg -c", will it...
a) print the missing 10 chars of the pending line (d !!!!!!!!), or
b) print the full pending line?
This worries me because I will look for specific strings and if they get truncated then my idea will not work.
I am using Linux 2.6.31 and dmesg is run through busybox:
rwxrwxrwx 1 root root 7 Jun 3 07:36 dmesg -> busybox
BusyBox v1.19.3 () multi-call binary.
Usage: dmesg [-c] [-n LEVEL] [-s SIZE]
I have run dmesg several times and I have never noticed a truncated line, but it's better to know if it has only been luck.
Thanks.
Related
I know that we can get the value at a memory address through a C program or gdb.
Is there any way to get it by bash shell or something like "one-line" perl instruction?
Short answer... No
If you were very careful it MIGHT be possible to pass instructions into GDB and interpret the output. Look at rocky's answer if you want to try.
Its part of the unix process model that one process cannot see inside another and cannot read each other's memory. This is for security. There is a special kernal API which can be used to see into programs. But there are very few clients for this API. Off the top of my head GDB is the only one I know.
I suspect that what you're trying to achieve is either a really bad idea or can be done without reading a program's memory.
The following possibly gets close to what you may want. And if not, hopefully you'll be able to adapt.
I'll say at the outset though that you are extremely vague at what you mean by "memory address". Below I am going to take that to mean the address of the current instruction executed which I will use the gdb command "backtrace" to get. Adapt the gdb command from "bt" to whatever it is you were thinking of to get the "memory address".
Let's say your program is sleeper-for-pid with process id 3963
sudo gdb --ex 'bt 1' --batch -nx -p 3963
will run gdb attached to the process and give a backtrace of the most recent entry. Here is an example:
sudo gdb --ex 'bt 1' --batch -nx -p 3963
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) #0 0x00007f5dee71f2f0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
If you really just want the line with the address, you'd grep for #0, e.g.
sudo gdb --ex 'bt 1' --batch -nx -p 3963 | grep '#0'
(gdb) #0 0x00007f5dee71f2f0 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:84
And if you just want the address you might have to do a further cut, e.g.
$ sudo gdb --ex 'bt 1' -nx -p 3963| grep '#0' | cut -d' ' -f 4
0x00007f5dee71f2f0
If you don't know the process id, you might be able to use pidof. For example if the command name is sleeper-for-pid:
$ sudo gdb --ex 'bt 1' --batch -nx -p $(pidof sleeper-for-pid)
And "sudo" may or may not work for you or be needed.
Personally, instead of just the most recent backtrace entry I prefer more than that. So bt 1 might get adjusted to bt 3 or bt for a more full back trace.
Hopefully this is enough to get you started .
I know that we can get the value at a memory address through a C program or gdb Are there any way to get it by bash shell or something like "one-line" perl instruction??
A shell is just a program. What you asking to do is rather meaningless. Assuming a shell did allow you to inspect a memory address (and you could easily modify a shell or write your own to do so), The value of a memory address is for the shell's process. Any other process is going to have a different memory value at the same address.
I would like to explore the memory of a living process, and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
Therefore I would like to get this info from /proc/kcore (if you know of another way to do this please let me know).
So I made a little experiment. I created a file called TEST with only "EXTRATESTEXTRA" inside.
Then I opened it with less
$ less TEST
I got the PID of this process with
$ ps aux | grep TEST
user 7785 0.0 0.0 17944 992 pts/8 S+ 16:15 0:00 less TEST
user 7798 0.0 0.0 13584 904 pts/9 S+ 16:16 0:00 grep TEST
And then I used this script to create a dump of all files :
#!/bin/bash
grep rw-p /proc/$1/maps | sed -n 's/^\([0-9a-f]*\)-\([0-9a-f]*\) .*$/\1 \2/p' | while read start stop; do gdb --batch --pid $1 -ex "dump memory $1-$start-$stop.dump 0x$start 0x$stop"; done
(I found it on this site https://serverfault.com/questions/173999/dump-a-linux-processs-memory-to-file)
$ sudo ./dump_all_pid_memory.sh 7785
After this, I looked for "TRATESTEX" in all dumped files :
$ grep -a -o -e '...TRATESTEX...' ./*.dump
./7785-00624000-00628000.dump:HEXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
So I concluded that there must be an occurance of this string somewhere between 0x00624000 and 0x00628000 .
Therefore I converted the offsets into decimal numbers and used dd to get the memory from /proc/kcore :
$ sudo dd if="/proc/kcore" of="./y.txt" skip="0" count="1638400" bs=1
To my surprise, the file y.txt was full of zeros (I didn't find the string I was looking for in it).
As a bonus surprise, I ran a simmilar test at the same time with a different test file and found that the other test string i was using
(both processes with less were running at the same time) should be found at the same location (the dumping and greping gave the same offset).
So there must be something I don't understand clearly.
Isn't the /proc/pid/maps supposed to show the offset of the memory (i.e. : if it would say "XXX" is at offset 0x10, another program could not be using the same offset am I right? - this is the source of my second surprise)
How can I read /proc/kmap to get the memory that belongs to a process which's pid I know ?
If you have root access and are on a linux system, you can use the following linux script (adapted from Gilles' excellent unix.stackexchange.com answer and the answer originally given in the question above but including SyntaxErrors and not being pythonic):
#!/usr/bin/env python
import re
import sys
def print_memory_of_pid(pid, only_writable=True):
"""
Run as root, take an integer PID and return the contents of memory to STDOUT
"""
memory_permissions = 'rw' if only_writable else 'r-'
sys.stderr.write("PID = %d" % pid)
with open("/proc/%d/maps" % pid, 'r') as maps_file:
with open("/proc/%d/mem" % pid, 'r', 0) as mem_file:
for line in maps_file.readlines(): # for each mapped region
m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r][-w])', line)
if m.group(3) == memory_permissions:
sys.stderr.write("\nOK : \n" + line+"\n")
start = int(m.group(1), 16)
if start > 0xFFFFFFFFFFFF:
continue
end = int(m.group(2), 16)
sys.stderr.write( "start = " + str(start) + "\n")
mem_file.seek(start) # seek to region start
chunk = mem_file.read(end - start) # read region contents
print chunk, # dump contents to standard output
else:
sys.stderr.write("\nPASS : \n" + line+"\n")
if __name__ == '__main__': # Execute this code when run from the commandline.
try:
assert len(sys.argv) == 2, "Provide exactly 1 PID (process ID)"
pid = int(sys.argv[1])
print_memory_of_pid(pid)
except (AssertionError, ValueError) as e:
print "Please provide 1 PID as a commandline argument."
print "You entered: %s" % ' '.join(sys.argv)
raise e
If you save this as write_mem.py, you can run this (with python2.6 or 2.7) or early in python2.5 (if you add from __future__ import with_statement) as:
sudo python write_mem.py 1234 > pid1234_memory_dump
to dump pid1234 memory to the file pid1234_memory_dump.
For process 1234 you can get its memory map by reading sequentially /proc/1234/maps (a textual pseudo-file) and read the virtual memory by e.g. read(2)-ing or mmap(2)-ing appropriate segments of the /proc/1234/mem sparse pseudo-file.
However, I believe you cannot avoid some kind of synchronization (perhaps with ptrace(2), as gdb does), since the process 1234 can (and does) alter its address space at any time (with mmap & related syscalls).
The situation is different if the monitored process 1234 is not arbitrary, but if you could improve it to communicate somehow with the monitoring process.
I'm not sure to understand why do you ask this. And gdb is able to watch some location without stopping the process.
Since the 3.2 version of the kernel. You can use the process_vm_readv system call to read process memory without interruption.
ssize_t process_vm_readv(pid_t pid,
const struct iovec *local_iov,
unsigned long liovcnt,
const struct iovec *remote_iov,
unsigned long riovcnt,
unsigned long flags);
These system calls transfer data between the address space of the
calling process ("the local process") and the process identified by
pid ("the remote process"). The data moves directly between the
address spaces of the two processes, without passing through kernel
space.
You'll have to use /proc/pid/mem to read process memory. I wouldn't recommend trying to read /proc/kcore or any of the kernel memory functions (which is time consuming).
If you just want to get the value of a global variable or a specified address, you can use my tool gvardump instead of reading the entire memory.
gvardump will parse the variable address and print its value nicely without causing process interruption.
For example:
root#ubuntu:/home/u/trace_test# ./gvardump.py 53670 -a 1 '*g_ss[0].sss[0].ps'
*((g_ss[0]).sss[0]).ps = {
.a = 6,
.sss = {
{
.bbb = 0,
.ps = 0x563ca42a2020,
.bs = {
.m = 0,
},
},
// other 9 elements are omit
},
...
and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
I have modified gdb to avoid attaching.
With this modified gdb, you can run gdb -m <PID> to explore the memory without stopping the process.
i achieved this by issuing the below command
[root#stage1 ~]# echo "Memory usage for PID [MySql]:"; for mem in {Private,Rss,Shared,Swap,Pss};do grep $mem /proc/ps aux |grep mysql |awk '{print $2}'|head -n 1/smaps | awk -v mem_type="$mem" '{i=i+$2} END {print mem_type,"memory usage:"i}' ;done
Result Output
Memory usage for PID [MySql]:
Private memory usage:204
Rss memory usage:1264
Shared memory usage:1060
Swap memory usage:0
Pss memory usage:423
I wrote a simple hello-world module on Ubuntu 10.04 machine. When loading and unloading the module, printk should log messages from the following hello_world and bye_world functions.
static int hello_world()
{
printk(KERN_INFO "Hello, beautiful world");
return 0;
}
static void bye_world()
{
printk(KERN_INFO "Good-bye kernel uncle");
}
module_init(hello_world);
module_exit(bye_world);
However, when actually inserting and removing the hello.ko module, I see that the message printed on the terminal (I redirected it to the current terminal) is exactly one message behind.
# rmmod hello.ko
# Jul 8 16:34:02 panchavati kernel: [64599.954113] Hello, beautiful world
# insmod hello.ko
# Jul 8 16:34:57 panchavati kernel: [65456.367422] Good-bye kernel uncle
From dmesg output, I can see the next message entry (corresponding to insmod) has been already logged, just that it is not being printed yet.
chanakya#panchavati:~$ dmesg | tail -2
[65456.367422] Good-bye kernel uncle
[65511.198299] Hello, beautiful world
The entry [65511.198299] is there in the kernel log, but only the previous entry [65456.367422] Good-bye kernel uncle was printed. Why is it so?
I earlier thought that '-' specifier used in /etc/syslog.conf might have to do something with this (it prevents synchronization after every write), but removing it didn't work either.
Have you tried using tailf (or tail -f)? It will print out the messages as they arrive. For example:
tailf /var/log/messages
Try ending your strings with a newline ("\n").
This question already has answers here:
Redirect STDERR / STDOUT of a process AFTER it's been started, using command line?
(8 answers)
Closed 6 years ago.
Normally I would start a command like
longcommand &;
I know you can redirect it by doing something like
longcommand > /dev/null;
for instance to get rid of the output or
longcommand 2>&1 > output.log
to capture output.
But I sometimes forget, and was wondering if there is a way to capture or redirect after the fact.
longcommand
ctrl-z
bg 2>&1 > /dev/null
or something like that so I can continue using the terminal without messages popping up on the terminal.
See Redirecting Output from a Running Process.
Firstly I run the command cat > foo1 in one session and test that data from stdin is copied to the file. Then in another session I redirect the output.
Firstly find the PID of the process:
$ ps aux | grep cat
rjc 6760 0.0 0.0 1580 376 pts/5 S+ 15:31 0:00 cat
Now check the file handles it has open:
$ ls -l /proc/6760/fd
total 3
lrwx—— 1 rjc rjc 64 Feb 27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 Feb 27 15:32 1 -> /tmp/foo1
lrwx—— 1 rjc rjc 64 Feb 27 15:32 2 -> /dev/pts/5
Now run GDB:
$ gdb -p 6760 /bin/cat
GNU gdb 6.4.90-debian
[license stuff snipped]
Attaching to program: /bin/cat, process 6760
[snip other stuff that's not interesting now]
(gdb) p close(1)
$1 = 0
(gdb) p creat("/tmp/foo3", 0600)
$2 = 1
(gdb) q
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: /bin/cat, process 6760
The p command in GDB will print the value of an expression, an expression can be a function to call, it can be a system call… So I execute a close() system call and pass file handle 1, then I execute a creat() system call to open a new file. The result of the creat() was 1 which means that it replaced the previous file handle. If I wanted to use the same file for stdout and stderr or if I wanted to replace a file handle with some other number then I would need to call the dup2() system call to achieve that result.
For this example I chose to use creat() instead of open() because there are fewer parameter. The C macros for the flags are not usable from GDB (it doesn’t use C headers) so I would have to read header files to discover this – it’s not that hard to do so but would take more time. Note that 0600 is the octal permission for the owner having read/write access and the group and others having no access. It would also work to use 0 for that parameter and run chmod on the file later on.
After that I verify the result:
ls -l /proc/6760/fd/
total 3
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 2008-02-27 15:32 1 -> /tmp/foo3 <====
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 2 -> /dev/pts/5
Typing more data in to cat results in the file /tmp/foo3 being appended to.
If you want to close the original session you need to close all file handles for it, open a new device that can be the controlling tty, and then call setsid().
You can also do it using reredirect (https://github.com/jerome-pouiller/reredirect/).
The command bellow redirects the outputs (standard and error) of the process PID to FILE:
reredirect -m FILE PID
The README of reredirect also explains other interesting features: how to restore the original state of the process, how to redirect to another command or to redirect only stdout or stderr.
The tool also provides relink, a script allowing to redirect the outputs to the current terminal:
relink PID
relink PID | grep usefull_content
(reredirect seems to have same features than Dupx described in another answer but, it does not depend on Gdb).
Dupx
Dupx is a simple *nix utility to redirect standard output/input/error of an already running process.
Motivation
I've often found myself in a situation where a process I started on a remote system via SSH takes much longer than I had anticipated. I need to break the SSH connection, but if I do so, the process will die if it tries to write something on stdout/error of a broken pipe. I wish I could suspend the process with ^Z and then do a
bg %1 >/tmp/stdout 2>/tmp/stderr
Unfortunately this will not work (in shells I know).
http://www.isi.edu/~yuri/dupx/
Screen
If process is running in a screen session you can use screen's log command to log the output of that window to a file:
Switch to the script's window, C-a H to log.
Now you can :
$ tail -f screenlog.2 | grep whatever
From screen's man page:
log [on|off]
Start/stop writing output of the current window to a file "screenlog.n" in the window's default directory, where n is the number of the current window. This filename can be changed with the 'logfile' command. If no parameter is given, the state of logging is toggled. The session log is appended to the previous contents of the file if it already exists. The current contents and the contents of the scrollback history are not included in the session log. Default is 'off'.
I'm sure tmux has something similar as well.
I collected some information on the internet and prepared the script that requires no external tool: See my response here. Hope it's helpful.
I have some scripts that ought to have stopped running but hang around forever. Is there some way I can figure out what they're writing to STDOUT and STDERR in a readable way?
I tried, for example, to do:
$ tail -f /proc/(pid)/fd/1
but that doesn't really work. It was a long shot anyway.
Any other ideas?
strace on its own is quite verbose and unreadable for seeing this.
Note: I am only interested in their output, not in anything else. I'm capable of figuring out the other things on my own; this question is only focused on getting access to stdout and stderr of the running process after starting it.
Since I'm not allowed to edit Jauco's answer, I'll give the full answer that worked for me (Russell's page relies on un-guaranteed behaviour that, if you close file descriptor 1 for STDOUT, the next creat call will open FD 1.
So, run a simple endless script like this:
import time
while True:
print 'test'
time.sleep(1)
Save it to test.py, run with
$ python test.py
Get the PID:
$ ps auxw | grep test.py
Now, attach gdb:
$ gdb -p (pid)
and do the fd magic:
(gdb) call creat("/tmp/stdout", 0600)
$1 = 3
(gdb) call dup2(3, 1)
$2 = 1
Now you can tail /tmp/stdout and see the output that used to go to STDOUT.
There's several new utilities that wrap up the "gdb method" and add some extra touches. The one I use now is called "reptyr" ("Re-PTY-er"). In addition to grabbing STDERR/STDOUT, it will actually change the controlling terminal of a process (even if it wasn't previously attached to a terminal).
The best use of this is to start up a screen session, and use it to reattach a running process to the terminal within screen so you can safely detach from it and come back later.
It's packaged on popular distros (Ex: 'apt-get install reptyr').
http://onethingwell.org/post/2924103615/reptyr
GDB method seems better, but you can do this with strace, too:
$ strace -p <PID> -e write=1 -s 1024 -o file
Via the man page for strace:
-e write=set
Perform a full hexadecimal and ASCII dump of all the
data written to file descriptors listed in the spec-
ified set. For example, to see all output activity
on file descriptors 3 and 5 use -e write=3,5. Note
that this is independent from the normal tracing of
the write(2) system call which is controlled by the
option -e trace=write.
This prints out somewhat more than you need (the hexadecimal part), but you can sed that out easily.
I'm not sure if it will work for you, but I read a page a while back describing a method that uses gdb
I used strace and de-coded the hex output to clear text:
PID=some_process_id
sudo strace -f -e trace=write -e verbose=none -e write=1,2 -q -p $PID -o "| grep '^ |' | cut -c11-60 | sed -e 's/ //g' | xxd -r -p"
I combined this command from other answers.
strace outputs a lot less with just -ewrite (and not the =1 suffix). And it's a bit simpler than the GDB method, IMO.
I used it to see the progress of an existing MythTV encoding job (sudo because I don't own the encoding process):
$ ps -aef | grep -i handbrake
mythtv 25089 25085 99 16:01 ? 00:53:43 /usr/bin/HandBrakeCLI -i /var/lib/mythtv/recordings/1061_20111230122900.mpg -o /var/lib/mythtv/recordings/1061_20111230122900.mp4 -e x264 -b 1500 -E faac -B 256 -R 48 -w 720
jward 25293 20229 0 16:30 pts/1 00:00:00 grep --color=auto -i handbr
$ sudo strace -ewrite -p 25089
Process 25089 attached - interrupt to quit
write(1, "\rEncoding: task 1 of 1, 70.75 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.76 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.77 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.78 % "..., 73) = 73^C
You can use reredirect (https://github.com/jerome-pouiller/reredirect/).
Type
reredirect -m FILE PID
and outputs (standard and error) will be written in FILE.
reredirect README also explains how to restore original state of process, how to redirect to another command or to redirect only stdout or stderr.
You don't state your operating system, but I'm going to take a stab and say "Linux".
Seeing what is being written to stderr and stdout is probably not going to help. If it is useful, you could use tee(1) before you start the script to take a copy of stderr and stdout.
You can use ps(1) to look for wchan. This tells you what the process is waiting for. If you look at the strace output, you can ignore the bulk of the output and identify the last (blocked) system call. If it is an operation on a file handle, you can go backwards in the output and identify the underlying object (file, socket, pipe, etc.) From there the answer is likely to be clear.
You can also send the process a signal that causes it to dump core, and then use the debugger and the core file to get a stack trace.