Reading living process memory without interrupting it - linux

Question

I would like to explore the memory of a living process, and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
Therefore I would like to get this info from /proc/kcore (if you know of another way to do this please let me know).
So I made a little experiment. I created a file called TEST with only "EXTRATESTEXTRA" inside.
Then I opened it with less
$ less TEST
I got the PID of this process with
$ ps aux | grep TEST
user 7785 0.0 0.0 17944 992 pts/8 S+ 16:15 0:00 less TEST
user 7798 0.0 0.0 13584 904 pts/9 S+ 16:16 0:00 grep TEST
And then I used this script to create a dump of all files :
#!/bin/bash
grep rw-p /proc/$1/maps | sed -n 's/^\([0-9a-f]*\)-\([0-9a-f]*\) .*$/\1 \2/p' | while read start stop; do gdb --batch --pid $1 -ex "dump memory $1-$start-$stop.dump 0x$start 0x$stop"; done
(I found it on this site https://serverfault.com/questions/173999/dump-a-linux-processs-memory-to-file)
$ sudo ./dump_all_pid_memory.sh 7785
After this, I looked for "TRATESTEX" in all dumped files :
$ grep -a -o -e '...TRATESTEX...' ./*.dump
./7785-00624000-00628000.dump:HEXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
So I concluded that there must be an occurance of this string somewhere between 0x00624000 and 0x00628000 .
Therefore I converted the offsets into decimal numbers and used dd to get the memory from /proc/kcore :
$ sudo dd if="/proc/kcore" of="./y.txt" skip="0" count="1638400" bs=1
To my surprise, the file y.txt was full of zeros (I didn't find the string I was looking for in it).
As a bonus surprise, I ran a simmilar test at the same time with a different test file and found that the other test string i was using
(both processes with less were running at the same time) should be found at the same location (the dumping and greping gave the same offset).
So there must be something I don't understand clearly.
Isn't the /proc/pid/maps supposed to show the offset of the memory (i.e. : if it would say "XXX" is at offset 0x10, another program could not be using the same offset am I right? - this is the source of my second surprise)
How can I read /proc/kmap to get the memory that belongs to a process which's pid I know ?

Answer 1

If you have root access and are on a linux system, you can use the following linux script (adapted from Gilles' excellent unix.stackexchange.com answer and the answer originally given in the question above but including SyntaxErrors and not being pythonic):
#!/usr/bin/env python
import re
import sys
def print_memory_of_pid(pid, only_writable=True):
"""
Run as root, take an integer PID and return the contents of memory to STDOUT
"""
memory_permissions = 'rw' if only_writable else 'r-'
sys.stderr.write("PID = %d" % pid)
with open("/proc/%d/maps" % pid, 'r') as maps_file:
with open("/proc/%d/mem" % pid, 'r', 0) as mem_file:
for line in maps_file.readlines(): # for each mapped region
m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r][-w])', line)
if m.group(3) == memory_permissions:
sys.stderr.write("\nOK : \n" + line+"\n")
start = int(m.group(1), 16)
if start > 0xFFFFFFFFFFFF:
continue
end = int(m.group(2), 16)
sys.stderr.write( "start = " + str(start) + "\n")
mem_file.seek(start) # seek to region start
chunk = mem_file.read(end - start) # read region contents
print chunk, # dump contents to standard output
else:
sys.stderr.write("\nPASS : \n" + line+"\n")
if __name__ == '__main__': # Execute this code when run from the commandline.
try:
assert len(sys.argv) == 2, "Provide exactly 1 PID (process ID)"
pid = int(sys.argv[1])
print_memory_of_pid(pid)
except (AssertionError, ValueError) as e:
print "Please provide 1 PID as a commandline argument."
print "You entered: %s" % ' '.join(sys.argv)
raise e
If you save this as write_mem.py, you can run this (with python2.6 or 2.7) or early in python2.5 (if you add from __future__ import with_statement) as:
sudo python write_mem.py 1234 > pid1234_memory_dump
to dump pid1234 memory to the file pid1234_memory_dump.

Answer 2

For process 1234 you can get its memory map by reading sequentially /proc/1234/maps (a textual pseudo-file) and read the virtual memory by e.g. read(2)-ing or mmap(2)-ing appropriate segments of the /proc/1234/mem sparse pseudo-file.
However, I believe you cannot avoid some kind of synchronization (perhaps with ptrace(2), as gdb does), since the process 1234 can (and does) alter its address space at any time (with mmap & related syscalls).
The situation is different if the monitored process 1234 is not arbitrary, but if you could improve it to communicate somehow with the monitoring process.
I'm not sure to understand why do you ask this. And gdb is able to watch some location without stopping the process.

Answer 3

Since the 3.2 version of the kernel. You can use the process_vm_readv system call to read process memory without interruption.
ssize_t process_vm_readv(pid_t pid,
const struct iovec *local_iov,
unsigned long liovcnt,
const struct iovec *remote_iov,
unsigned long riovcnt,
unsigned long flags);
These system calls transfer data between the address space of the
calling process ("the local process") and the process identified by
pid ("the remote process"). The data moves directly between the
address spaces of the two processes, without passing through kernel
space.

Answer 4

You'll have to use /proc/pid/mem to read process memory. I wouldn't recommend trying to read /proc/kcore or any of the kernel memory functions (which is time consuming).

Answer 5

If you just want to get the value of a global variable or a specified address, you can use my tool gvardump instead of reading the entire memory.
gvardump will parse the variable address and print its value nicely without causing process interruption.
For example:
root#ubuntu:/home/u/trace_test# ./gvardump.py 53670 -a 1 '*g_ss[0].sss[0].ps'
*((g_ss[0]).sss[0]).ps = {
.a = 6,
.sss = {
{
.bbb = 0,
.ps = 0x563ca42a2020,
.bs = {
.m = 0,
},
},
// other 9 elements are omit
},
...

Answer 6

and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
I have modified gdb to avoid attaching.
With this modified gdb, you can run gdb -m <PID> to explore the memory without stopping the process.

Answer 7

i achieved this by issuing the below command
[root#stage1 ~]# echo "Memory usage for PID [MySql]:"; for mem in {Private,Rss,Shared,Swap,Pss};do grep $mem /proc/ps aux |grep mysql |awk '{print $2}'|head -n 1/smaps | awk -v mem_type="$mem" '{i=i+$2} END {print mem_type,"memory usage:"i}' ;done
Result Output
Memory usage for PID [MySql]:
Private memory usage:204
Rss memory usage:1264
Shared memory usage:1060
Swap memory usage:0
Pss memory usage:423