I have a long-running process with stdout redirected to a file. E.g.:
./my-script.sh > file.txt &
Part of the stdout is still cached, but I would like to flush it to the file, to see the results earlier. Is there a way to do it?
The caching is handled by the libc. You can use the stdbuf command to change the buffer size:
stdbuf -o0 ./my-script.sh > file.txt &
-o0 sets the buffer size for stdout to 0. Probably you also want -e0 for stderr.
You can inspect the /proc/ filesystem and alter the file descriptor of stdout. For example:
gerard#droole ~$ bash -c '
while [ true ]; do echo "."; sleep .5; done
' > ./myfile.txt &
[1] 3816
gerard#droole ~$ ls -la /proc/3816/fd/1
l-wx------ 1 gerard gerard 64 May 30 14:55 /proc/3816/fd/1 -> /home/gerard/myfile.txt
You can see that stdout is symlinked to the file I specified on the command line. If you want to change it, you can simply link it to something else.
If you want to reroute this output, you can start a tee process, symlink the stdout of the process you're watching to a the stdin of the new process. You can reroute basically anything you want this way.
However, this is not very stable, as your programs output will be broken if you do not carefully restore its stdout file descriptor before the tee process is terminated.
But it is not impossible ;)
Related
I want in Linux, get the stdout of a NodeJS program that is opened, from other NodeJS program or bash.
I have the PID, or name of program and the data, put to function in real time.
Maybe touching files in /proc?
This is possible?
You can use strace -e write -p <pid> to see what output is the program writing to stdout (or other FDs) in real time. It does not show what has been written earlier and it needs a little parsing to extract clean stdout contents.
By default, it truncates shown writes to only 32 characters. To show more, use -s switch:
strace -e write -s 9999 -p <pid>
I'm trying to display the status progression of a file copied with dd command:
I run this command : dd if=/dev/zero of=file.txt count=1024 bs=10485760
Then in another terminal, I run this command to display the status of progression:
watch -n 1 kill -USR1 $pid_dd
My problem I have tried to redirect the output of watch to a file but without success.
I have tried this solution proposed on this link How to save output of "watch" to file
while true
do
watch -n 1 kill -USR1 $pid_dd | tee -a output_watch.txt
sleep 2
done
I don't know how to redirect the output of this command to a file. My solution doesn't work.
I have tested that loop above without the "watch" command like this :
while true
do
kill -USR1 $pid_dd | tee -a output_watch.txt
sleep 2
done
The problem is the "output_watch.txt" file is empty. I don't understand why.
kill -USR1 $pid_dd doesn't output anything. You have indeed successfully captured its output, which is empty. It sends a signal that causes the dd process -- an entirely different process -- to output progress data on its stderr, which you didn't redirect.
If you simply redirect dd ... 2>&1 | tee -a file it will make it more difficult to determine the PID of dd which you need. Depending on your shell consider instead
dd ... 2> >(tee -a file) & pid=$!
watch -n1 kill -USR1 $pid
# you don't need a loop around watch, it already _is_ a loop
# or _instead_
while kill -USR1 $pid; do sleep 1; done
Alternatively, see pv which already does almost what you want (if available). Although described as monitoring the copying of data from one process to another, it actually does stdin to stdout which can (both) be files instead of piped processes.
Exactly #dave_thompson ! It works, I have written this code :
dd ... 2>>/home/file.txt & pid_dd=$!
watch -n 1 kill -USR1 $pid_dd
The file "file.txt" has all information about the dd progress.
Thanks so much.
I want to monitor all the running processes using strace and when a process ends the output of the strace should be sent to a file.
And how to find every running proc PID. I also want to include process name in the output file.
$ sudo strace -p 1725 -o firefox_trace.txt
$ tail -f firefox_trace.txt
1725 would be the PID of the proccess you want to monitor (you can find the PID with "ps -C firefox-bin", for firefox in the example)
And firefox_trace.txt would be the output file !
The way to got would be to find every running proc PID, and use the command to write them in the output file !
Considering the doc,
-p pid
Attach to the process with the process ID pid and begin tracing. The
trace may be terminated at any time by a keyboard interrupt signal (
CTRL -C). strace will respond by detaching itself from the traced
process(es) leaving it (them) to continue running. Multiple -p options
can be used to attach to up to 32 processes in addition to command
(which is optional if at least one -p option is given).
Use -o to store the output to the file, or 2>&1 to redirect standard error to output, so you can filter it (grep) or redirect it into file (> file).
To monitor process without knowing its PID, but name, you can use pgrep command, e.g.
strace -p $(pgrep command) -o file.out
where command is your name of process (e.g. php, Chrome, etc.).
To learn more about parameters, check man strace.
I have some scripts that ought to have stopped running but hang around forever. Is there some way I can figure out what they're writing to STDOUT and STDERR in a readable way?
I tried, for example, to do:
$ tail -f /proc/(pid)/fd/1
but that doesn't really work. It was a long shot anyway.
Any other ideas?
strace on its own is quite verbose and unreadable for seeing this.
Note: I am only interested in their output, not in anything else. I'm capable of figuring out the other things on my own; this question is only focused on getting access to stdout and stderr of the running process after starting it.
Since I'm not allowed to edit Jauco's answer, I'll give the full answer that worked for me (Russell's page relies on un-guaranteed behaviour that, if you close file descriptor 1 for STDOUT, the next creat call will open FD 1.
So, run a simple endless script like this:
import time
while True:
print 'test'
time.sleep(1)
Save it to test.py, run with
$ python test.py
Get the PID:
$ ps auxw | grep test.py
Now, attach gdb:
$ gdb -p (pid)
and do the fd magic:
(gdb) call creat("/tmp/stdout", 0600)
$1 = 3
(gdb) call dup2(3, 1)
$2 = 1
Now you can tail /tmp/stdout and see the output that used to go to STDOUT.
There's several new utilities that wrap up the "gdb method" and add some extra touches. The one I use now is called "reptyr" ("Re-PTY-er"). In addition to grabbing STDERR/STDOUT, it will actually change the controlling terminal of a process (even if it wasn't previously attached to a terminal).
The best use of this is to start up a screen session, and use it to reattach a running process to the terminal within screen so you can safely detach from it and come back later.
It's packaged on popular distros (Ex: 'apt-get install reptyr').
http://onethingwell.org/post/2924103615/reptyr
GDB method seems better, but you can do this with strace, too:
$ strace -p <PID> -e write=1 -s 1024 -o file
Via the man page for strace:
-e write=set
Perform a full hexadecimal and ASCII dump of all the
data written to file descriptors listed in the spec-
ified set. For example, to see all output activity
on file descriptors 3 and 5 use -e write=3,5. Note
that this is independent from the normal tracing of
the write(2) system call which is controlled by the
option -e trace=write.
This prints out somewhat more than you need (the hexadecimal part), but you can sed that out easily.
I'm not sure if it will work for you, but I read a page a while back describing a method that uses gdb
I used strace and de-coded the hex output to clear text:
PID=some_process_id
sudo strace -f -e trace=write -e verbose=none -e write=1,2 -q -p $PID -o "| grep '^ |' | cut -c11-60 | sed -e 's/ //g' | xxd -r -p"
I combined this command from other answers.
strace outputs a lot less with just -ewrite (and not the =1 suffix). And it's a bit simpler than the GDB method, IMO.
I used it to see the progress of an existing MythTV encoding job (sudo because I don't own the encoding process):
$ ps -aef | grep -i handbrake
mythtv 25089 25085 99 16:01 ? 00:53:43 /usr/bin/HandBrakeCLI -i /var/lib/mythtv/recordings/1061_20111230122900.mpg -o /var/lib/mythtv/recordings/1061_20111230122900.mp4 -e x264 -b 1500 -E faac -B 256 -R 48 -w 720
jward 25293 20229 0 16:30 pts/1 00:00:00 grep --color=auto -i handbr
$ sudo strace -ewrite -p 25089
Process 25089 attached - interrupt to quit
write(1, "\rEncoding: task 1 of 1, 70.75 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.76 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.77 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.78 % "..., 73) = 73^C
You can use reredirect (https://github.com/jerome-pouiller/reredirect/).
Type
reredirect -m FILE PID
and outputs (standard and error) will be written in FILE.
reredirect README also explains how to restore original state of process, how to redirect to another command or to redirect only stdout or stderr.
You don't state your operating system, but I'm going to take a stab and say "Linux".
Seeing what is being written to stderr and stdout is probably not going to help. If it is useful, you could use tee(1) before you start the script to take a copy of stderr and stdout.
You can use ps(1) to look for wchan. This tells you what the process is waiting for. If you look at the strace output, you can ignore the bulk of the output and identify the last (blocked) system call. If it is an operation on a file handle, you can go backwards in the output and identify the underlying object (file, socket, pipe, etc.) From there the answer is likely to be clear.
You can also send the process a signal that causes it to dump core, and then use the debugger and the core file to get a stack trace.
I have some scripts that ought to have stopped running but hang around forever. Is there some way I can figure out what they're writing to STDOUT and STDERR in a readable way?
I tried, for example, to do:
$ tail -f /proc/(pid)/fd/1
but that doesn't really work. It was a long shot anyway.
Any other ideas?
strace on its own is quite verbose and unreadable for seeing this.
Note: I am only interested in their output, not in anything else. I'm capable of figuring out the other things on my own; this question is only focused on getting access to stdout and stderr of the running process after starting it.
Since I'm not allowed to edit Jauco's answer, I'll give the full answer that worked for me (Russell's page relies on un-guaranteed behaviour that, if you close file descriptor 1 for STDOUT, the next creat call will open FD 1.
So, run a simple endless script like this:
import time
while True:
print 'test'
time.sleep(1)
Save it to test.py, run with
$ python test.py
Get the PID:
$ ps auxw | grep test.py
Now, attach gdb:
$ gdb -p (pid)
and do the fd magic:
(gdb) call creat("/tmp/stdout", 0600)
$1 = 3
(gdb) call dup2(3, 1)
$2 = 1
Now you can tail /tmp/stdout and see the output that used to go to STDOUT.
There's several new utilities that wrap up the "gdb method" and add some extra touches. The one I use now is called "reptyr" ("Re-PTY-er"). In addition to grabbing STDERR/STDOUT, it will actually change the controlling terminal of a process (even if it wasn't previously attached to a terminal).
The best use of this is to start up a screen session, and use it to reattach a running process to the terminal within screen so you can safely detach from it and come back later.
It's packaged on popular distros (Ex: 'apt-get install reptyr').
http://onethingwell.org/post/2924103615/reptyr
GDB method seems better, but you can do this with strace, too:
$ strace -p <PID> -e write=1 -s 1024 -o file
Via the man page for strace:
-e write=set
Perform a full hexadecimal and ASCII dump of all the
data written to file descriptors listed in the spec-
ified set. For example, to see all output activity
on file descriptors 3 and 5 use -e write=3,5. Note
that this is independent from the normal tracing of
the write(2) system call which is controlled by the
option -e trace=write.
This prints out somewhat more than you need (the hexadecimal part), but you can sed that out easily.
I'm not sure if it will work for you, but I read a page a while back describing a method that uses gdb
I used strace and de-coded the hex output to clear text:
PID=some_process_id
sudo strace -f -e trace=write -e verbose=none -e write=1,2 -q -p $PID -o "| grep '^ |' | cut -c11-60 | sed -e 's/ //g' | xxd -r -p"
I combined this command from other answers.
strace outputs a lot less with just -ewrite (and not the =1 suffix). And it's a bit simpler than the GDB method, IMO.
I used it to see the progress of an existing MythTV encoding job (sudo because I don't own the encoding process):
$ ps -aef | grep -i handbrake
mythtv 25089 25085 99 16:01 ? 00:53:43 /usr/bin/HandBrakeCLI -i /var/lib/mythtv/recordings/1061_20111230122900.mpg -o /var/lib/mythtv/recordings/1061_20111230122900.mp4 -e x264 -b 1500 -E faac -B 256 -R 48 -w 720
jward 25293 20229 0 16:30 pts/1 00:00:00 grep --color=auto -i handbr
$ sudo strace -ewrite -p 25089
Process 25089 attached - interrupt to quit
write(1, "\rEncoding: task 1 of 1, 70.75 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.76 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.77 % "..., 73) = 73
write(1, "\rEncoding: task 1 of 1, 70.78 % "..., 73) = 73^C
You can use reredirect (https://github.com/jerome-pouiller/reredirect/).
Type
reredirect -m FILE PID
and outputs (standard and error) will be written in FILE.
reredirect README also explains how to restore original state of process, how to redirect to another command or to redirect only stdout or stderr.
You don't state your operating system, but I'm going to take a stab and say "Linux".
Seeing what is being written to stderr and stdout is probably not going to help. If it is useful, you could use tee(1) before you start the script to take a copy of stderr and stdout.
You can use ps(1) to look for wchan. This tells you what the process is waiting for. If you look at the strace output, you can ignore the bulk of the output and identify the last (blocked) system call. If it is an operation on a file handle, you can go backwards in the output and identify the underlying object (file, socket, pipe, etc.) From there the answer is likely to be clear.
You can also send the process a signal that causes it to dump core, and then use the debugger and the core file to get a stack trace.