it is a negative testing to check if the program could handle the FD exhausting issue, and I am not able to modify the code. I need to check how the problem behave when hit the limit. it at least not being allowed to crash
I want to consume all the FDs of a running process, I tried the followed command to reduce the softlimit and hardlimit of the process to simulate this, however, I want to real consume the opened FD to its original limit, is there any other ways to achieve this?
# ls /proc/169607/fd|wc -l
503
# prlimit --pid 169607 --nofile=503:503
I also have other follow up questions need to confirm, I tried to attach the process in gdb and invoke open to consume the FD, it works like in below:
# this way works
(gdb) p open("/dev/null")
$11 = 80
# shell does not work
(gdb) shell for i in `seq 1 10`; do echo $i>"/dev/null"; done
$1 = 5
(gdb) define lopen
Type commands for definition of "lopen".
End with a line saying just "end".
>set $total = $arg0
>set $i = 0
> while($i<$total)
> set $i = $i + 1
> p open("/dev/null")
> end
>end
(gdb) lopen 10 # this works in loop
I found I could still open files even if reached the hard limit and soft limit of the current consumed amount, why is that?
E.g.,
# ls /proc/169607/fd|wc -l
503
# prlimit --pid 169607 --nofile=503:503
# In gdb I could still open files, and I found the fd keep increasing
(gdb) p open("/dev/null")
Related
Considering the following example, emulating a command which gives output after 10 seconds: exec 5< <(sleep 10; pwd)
In Solaris, if I check the file descriptor earlier than 10 seconds, I can see that it has a size of 0 and this tells me that it hasn't been populated with data yet. I can simply check every second until the file test condition is met (different from 0) and then pull the data:
while true; do
if [[ -s /proc/$$/fd/5 ]]; then
variable=$(cat <&5)
break
fi
sleep 1
done
But in Linux I can't do this (RedHat, Debian etc). All file descriptors appear with a size of 64 bytes no matter if they hold data or not. For various commands that will take a variable amount of time to dump their output, I will not know when I should read the file descriptor. No, I don't want to just wait for cat <&5 to finish, I need to know when I should perform the cat in the first place. Because I am using this mechanism to issue simultaneous commands and assign their output to corresponding file descriptors. As mentioned already, this works great in Solaris.
Here is the skeleton of an idea :
#!/bin/bash
exec 5< <(sleep 4; pwd)
while true
do
if
read -t 0 -u 5 dummy
then
echo Data available
cat <&5
break
else
echo No data
fi
sleep 1
done
From the Bash reference manual :
If timeout is 0, read returns immediately, without trying to read and
data. The exit status is 0 if input is available on the specified file
descriptor, non-zero otherwise.
The idea is to use read with -t 0 (to have zero timeout) and -u 5 (read from file descriptor 5) to instantly check for data availability.
Of course this is just a toy loop to demonstrate the concept.
The solution given by User Fred using only bash builtins works fine, but is a tiny bit non-optimal due to polling for the state of a file descriptor. If calling another interpreter (for example Python) is not a no-go, a non-polling version is possible:
#! /bin/bash
(
sleep 4
echo "This is the data coming now"
echo "More data"
) | (
python3 -c 'import select;select.select([0],[],[])'
echo "Data is now available and can be processed"
# Replace with more sophisticated real-world processing, of course:
cat
)
The single line python3 -c 'import select;select.select([0],[],[])' waits until STDIN has data ready. It uses the standard select(2) system call, for which I have not found a direct shell equivalent or wrapper.
I am trying to figure out a way to monitor the files I am dumping from my script. If there is no increment seen in child files then kill my script.
I am doing this to free up the resources when not needed. Here is what I think of , but I think my apporch is going to add burden to CPU. Can anyone please suggest more efficent way of doing this?
Below script is suppose to poll in every 15 sec and collect two file size of same file, if the size of the two samples are same then exit.
checkUsage() {
while true; do
sleep 15
fileSize=$(stat -c%s $1)
sleep 10;
fileSizeNew=$(stat -c%s $1)
if [ "$fileSize" == "$fileSizeNew" ]; then
echo -e "[Error]: No activity noted on this window from 5 sec. Exiting..."
kill -9 $$
fi
done
}
And I am planning to call it as follow (in background):
checkUsage /var/log/messages &
I can also get solution if, someone suggest how to monitor tail command and if nothing printing on tail then exit. NOT SURE WHY PEOPLE ARE CONFUSED. End goal of this question is to ,check if the some file is edited in last 15 seconds. If not exit or throw some error.
I have achived this by above script,but I don't know if this is the smartest way of achiveing this. I have asked this question to know views from other if there is any alternative way or better way of doing it.
I would based the check on file modification time instead of size, so something like this (untested code):
checkUsage() {
while true; do
# Test if file mtime is 'second arg' seconds older than date, default to 10 seconds
if [ $(( $(date +"%s") - $(stat -c%Y /var/log/message) )) -gt ${2-10} ]; then
echo -e "[Error]: No activity noted on this window from ${2-10} sec. Exiting..."
return 1
fi
#Sleep 'first arg' second, 15 seconds by default
sleep ${1-15}
done
}
The idea is to compare the file mtime with current time, if it's greater than second argument seconds, print the message and return.
And then I would call it like this later (or with no args to use defaults):
[ checkusage 20 10 ] || exit 1
Which would exit the script with code 1 as when the function return from it's infinite loop (as long as the file is modified)
Edit: reading me again, the target file could be a parameter too, to allow a better reuse of the function, left as an exercise to the reader.
If on Linux, in a local file system (Ext4, BTRFS, ...) -not a network file system- then you could consider inotify(7) facilities: something could be triggered when some file or directory changes or is accessed.
In particular, you might have some incron job thru incrontab(5) file; maybe it could communicate with some other job ...
PS. I am not sure to understand what you really want to do...
I suppose an external programme is modifying /var/log/messages.
If this is the case, below is my script (with minor changes to yours)
#Bash script to monitor changes to file
#!/bin/bash
checkUsage() # Note that U is in caps
{
while true
do
sleep 15
fileSize=$(stat -c%s $1)
sleep 10;
fileSizeNew=$(stat -c%s $1)
if [ "$fileSize" == "$fileSizeNew" ]
then
echo -e "[Notice : ] no changes noted in $1 : gracefully exiting"
exit # previously this was kill -9 $$
# changing this to exit would end the program gracefully.
# use kill -9 to kill a process which is not under your control.
# kill -9 sends the SIGKILL signal.
fi
done
}
checkUsage $1 # I have added this to your script
#End of the script
Save the script as checkusage and run it like :
./checkusage /var/log/messages &
Edit :
Since you're looking for better solutions I would suggest inotifywait, thanks for the suggestion from the other answerer.
Below would be my code :
while inotifywait -t 10 -q -e modify $1 >/dev/null
do
sleep 15 # as you said the polling would happen in 15 seconds.
done
echo "Script exited gracefully : $1 has not been changed"
Below are the details from the inotifywait manpage
-t <seconds>, --timeout <seconds> Exit if an appropriate event has not occurred within <seconds> seconds. If <seconds> is zero (the default),
wait indefinitely for an event.
-e <event>, --event <event> Listen for specific event(s) only. The events which can be listened for are listed in the EVENTS section.
This option can be specified more than once. If omitted, all events
are listened for.
-q, --quiet If specified once, the program will be less verbose. Specifically, it will not state when it has completed establishing all
inotify watches.
modify(Event) A watched file or a file within a watched directory was
written to.
Notes
You might have to install the inotify-tools first to make use of the inotifywait command. Check the inotify-tools page at Github.
So I have a bash command to start a server and it outputs some lines before getting to the point where it outputs something like "Server started, Press Control+C to exit". How do I pipe this output so when this line occurs i put this process in the background and continue with another script/function (i.e to do stuff that needs to wait until the server starts such as run tests)
I want to end up with 3 functions
start_server
run_tests
stop_server
I've got something along the lines of:
function read_server_output{
while read data; do
printf "$data"
if [[ $data == "Server started, Press Control+C to exit" ]]; then
# do something here to put server process in the background
# so I can run another function
fi
done
}
function start_server{
# start the server and pipe its output to another function to check its running
start-server-command | read_server_output
}
function run_test{
# do some stuff
}
function stop_server{
# stop the server
}
# run the bash script code
start_server()
run_tests()
stop_tests()
related question possibly SH/BASH - Scan a log file until some text occurs, then exit. How?
Thanks in advance I'm pretty new to this.
First, a note on terminology...
"Background" and "foreground" are controlling-terminal concepts, i.e., they have to do with what happens when you type ctrl+C, ctrl+Z, etc. (which process gets the signal), whether a process can read from the terminal device (a "background" process gets a SIGTTIN that by default causes it to stop), and so on.
It seems clear that this has little to do with what you want to achieve. Instead, you have an ill-behaved program (or suite of programs) that needs some special coddling: when the server is first started, it needs some hand-holding up to some point, after which it's OK. The hand-holding can stop once it outputs some text string (see your related question for that, or the technique below).
There's a big potential problem here: a lot of programs, when their output is redirected to a pipe or file, produce no output until they have printed a "block" worth of output, or are exiting. If this is the case, a simple:
start-server-command | cat
won't print the line you're looking for (so that's a quick way to tell whether you will have to work around this issue as well). If so, you'll need something like expect, which is an entirely different way to achieve what you want.
Assuming that's not a problem, though, let's try an entirely-in-shell approach.
What you need is to run the start-server-command and save the process-ID so that you can (eventually) send it a SIGINT signal (as ctrl+C would if the process were "in the foreground", but you're doing this from a script, not from a controlling terminal, so there's no key the script can press). Fortunately sh has a syntax just for this.
First let's make a temporary file:
#! /bin/sh
# myscript - script to run server, check for startup, then run tests
TMPFILE=$(mktemp -t myscript) || exit 1 # create /tmp/myscript.<unique>
trap "rm -f $TMPFILE" 0 1 2 3 15 # arrange to clean up when done
Now start the server and save its PID:
start-server-command > $TMPFILE & # start server, save output in file
SERVER_PID=$! # and save its PID so we can end it
trap "kill -INT $SERVER_PID; rm -f $TMPFILE" 0 1 2 3 15 # adjust cleanup
Now you'll want to scan through $TMPFILE until the desired output appears, as in the other question. Because this requires a certain amount of polling you should insert a delay. It's also probably wise to check whether the server has failed and terminated without ever getting to the "started" point.
while ! grep '^Server started, Press Control+C to exit$' >/dev/null; do
# message has not yet appeared, is server still starting?
if kill -0 $SERVER_PID 2>/dev/null; then
# server is running; let's wait a bit and try grepping again
sleep 1 # or other delay interval
else
echo "ERROR: server terminated without starting properly" 1>&2
exit 1
fi
done
(Here kill -0 is used to test whether the process still exists; if not, it has exited. The "cleanup" kill -INT will produce an error message, but that's probably OK. If not, either redirect that kill command's error-output, or adjust the cleanup or do it manually, as seen below.)
At this point, the server is running and you can do your tests. When you want it to exit as if the user hit ctrl+C, send it a SIGINT with kill -INT.
Since there's a kill -INT in the trap set for when the script exits (0) as well as when it's terminated by SIGHUP (1), SIGINT (2), SIGQUIT (3), and SIGTERM (15)—that's the:
trap "do some stuff" 0 1 2 3 15
part—you can simply let your script exit at this point, unless you want to specifically wait for the server to exit too. If you want that, perhaps:
kill -INT $SERVER_PID; rm -f $TMPFILE # do the pre-arranged cleanup now
trap - 0 1 2 3 15 # don't need it arranged anymore
wait $SERVER_PID # wait for server to finish exit
would be appropriate.
(Obviously none of the above is tested, but that's the general framework.)
Probably the easiest thing to do is to start it in the background and block on reading its output. Something like:
{ start-server-command & } | {
while read -r line; do
echo "$line"
echo "$line" | grep -q 'Server started' && break
done
cat &
}
echo script continues here after server outputs 'Server started' message
But this is a pretty ugly hack. It would be better if the server could be modified to perform a more specific action which the script could wait for.
I would like to explore the memory of a living process, and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
Therefore I would like to get this info from /proc/kcore (if you know of another way to do this please let me know).
So I made a little experiment. I created a file called TEST with only "EXTRATESTEXTRA" inside.
Then I opened it with less
$ less TEST
I got the PID of this process with
$ ps aux | grep TEST
user 7785 0.0 0.0 17944 992 pts/8 S+ 16:15 0:00 less TEST
user 7798 0.0 0.0 13584 904 pts/9 S+ 16:16 0:00 grep TEST
And then I used this script to create a dump of all files :
#!/bin/bash
grep rw-p /proc/$1/maps | sed -n 's/^\([0-9a-f]*\)-\([0-9a-f]*\) .*$/\1 \2/p' | while read start stop; do gdb --batch --pid $1 -ex "dump memory $1-$start-$stop.dump 0x$start 0x$stop"; done
(I found it on this site https://serverfault.com/questions/173999/dump-a-linux-processs-memory-to-file)
$ sudo ./dump_all_pid_memory.sh 7785
After this, I looked for "TRATESTEX" in all dumped files :
$ grep -a -o -e '...TRATESTEX...' ./*.dump
./7785-00624000-00628000.dump:HEXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
./7785-00b8f000-00bb0000.dump:EXTRATESTEXTRA
So I concluded that there must be an occurance of this string somewhere between 0x00624000 and 0x00628000 .
Therefore I converted the offsets into decimal numbers and used dd to get the memory from /proc/kcore :
$ sudo dd if="/proc/kcore" of="./y.txt" skip="0" count="1638400" bs=1
To my surprise, the file y.txt was full of zeros (I didn't find the string I was looking for in it).
As a bonus surprise, I ran a simmilar test at the same time with a different test file and found that the other test string i was using
(both processes with less were running at the same time) should be found at the same location (the dumping and greping gave the same offset).
So there must be something I don't understand clearly.
Isn't the /proc/pid/maps supposed to show the offset of the memory (i.e. : if it would say "XXX" is at offset 0x10, another program could not be using the same offset am I right? - this is the source of my second surprise)
How can I read /proc/kmap to get the memory that belongs to a process which's pid I know ?
If you have root access and are on a linux system, you can use the following linux script (adapted from Gilles' excellent unix.stackexchange.com answer and the answer originally given in the question above but including SyntaxErrors and not being pythonic):
#!/usr/bin/env python
import re
import sys
def print_memory_of_pid(pid, only_writable=True):
"""
Run as root, take an integer PID and return the contents of memory to STDOUT
"""
memory_permissions = 'rw' if only_writable else 'r-'
sys.stderr.write("PID = %d" % pid)
with open("/proc/%d/maps" % pid, 'r') as maps_file:
with open("/proc/%d/mem" % pid, 'r', 0) as mem_file:
for line in maps_file.readlines(): # for each mapped region
m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r][-w])', line)
if m.group(3) == memory_permissions:
sys.stderr.write("\nOK : \n" + line+"\n")
start = int(m.group(1), 16)
if start > 0xFFFFFFFFFFFF:
continue
end = int(m.group(2), 16)
sys.stderr.write( "start = " + str(start) + "\n")
mem_file.seek(start) # seek to region start
chunk = mem_file.read(end - start) # read region contents
print chunk, # dump contents to standard output
else:
sys.stderr.write("\nPASS : \n" + line+"\n")
if __name__ == '__main__': # Execute this code when run from the commandline.
try:
assert len(sys.argv) == 2, "Provide exactly 1 PID (process ID)"
pid = int(sys.argv[1])
print_memory_of_pid(pid)
except (AssertionError, ValueError) as e:
print "Please provide 1 PID as a commandline argument."
print "You entered: %s" % ' '.join(sys.argv)
raise e
If you save this as write_mem.py, you can run this (with python2.6 or 2.7) or early in python2.5 (if you add from __future__ import with_statement) as:
sudo python write_mem.py 1234 > pid1234_memory_dump
to dump pid1234 memory to the file pid1234_memory_dump.
For process 1234 you can get its memory map by reading sequentially /proc/1234/maps (a textual pseudo-file) and read the virtual memory by e.g. read(2)-ing or mmap(2)-ing appropriate segments of the /proc/1234/mem sparse pseudo-file.
However, I believe you cannot avoid some kind of synchronization (perhaps with ptrace(2), as gdb does), since the process 1234 can (and does) alter its address space at any time (with mmap & related syscalls).
The situation is different if the monitored process 1234 is not arbitrary, but if you could improve it to communicate somehow with the monitoring process.
I'm not sure to understand why do you ask this. And gdb is able to watch some location without stopping the process.
Since the 3.2 version of the kernel. You can use the process_vm_readv system call to read process memory without interruption.
ssize_t process_vm_readv(pid_t pid,
const struct iovec *local_iov,
unsigned long liovcnt,
const struct iovec *remote_iov,
unsigned long riovcnt,
unsigned long flags);
These system calls transfer data between the address space of the
calling process ("the local process") and the process identified by
pid ("the remote process"). The data moves directly between the
address spaces of the two processes, without passing through kernel
space.
You'll have to use /proc/pid/mem to read process memory. I wouldn't recommend trying to read /proc/kcore or any of the kernel memory functions (which is time consuming).
If you just want to get the value of a global variable or a specified address, you can use my tool gvardump instead of reading the entire memory.
gvardump will parse the variable address and print its value nicely without causing process interruption.
For example:
root#ubuntu:/home/u/trace_test# ./gvardump.py 53670 -a 1 '*g_ss[0].sss[0].ps'
*((g_ss[0]).sss[0]).ps = {
.a = 6,
.sss = {
{
.bbb = 0,
.ps = 0x563ca42a2020,
.bs = {
.m = 0,
},
},
// other 9 elements are omit
},
...
and when I do so, the process must not get disturbed - so attaching gdb to the process (which would stop it) is not an option.
I have modified gdb to avoid attaching.
With this modified gdb, you can run gdb -m <PID> to explore the memory without stopping the process.
i achieved this by issuing the below command
[root#stage1 ~]# echo "Memory usage for PID [MySql]:"; for mem in {Private,Rss,Shared,Swap,Pss};do grep $mem /proc/ps aux |grep mysql |awk '{print $2}'|head -n 1/smaps | awk -v mem_type="$mem" '{i=i+$2} END {print mem_type,"memory usage:"i}' ;done
Result Output
Memory usage for PID [MySql]:
Private memory usage:204
Rss memory usage:1264
Shared memory usage:1060
Swap memory usage:0
Pss memory usage:423
I want to limit the execution time of a program I am running under Linux. I put in my scons script a line like:
Command("com","","ulimit -t 1; myprogram")
and tested it with an infinite loop program: it did not work and the program ran forever.
Am I missing something?
-- tsf
ulimit -t 1 means that the limit is set to 1 second of CPU time. If your infinite loop program uses any sort of sleep in its inner loop then it will use practically no CPU time. This means it will not get killed in 1 second of real, on the clock time. In fact it may take minutes or hours to use up its 1 second allocation.
What happens if you run the command outside of SCons? Perhaps you don't have permission to change the limit at all...
ulimit -t 1; ./myprogram
For example, it may say the following if the limit is already set to 0:
bash: ulimit: cpu time: cannot modify limit: Operation not permitted
Edit: it seems that the -t option is broken on Ubuntu 9.04. A fix has been committed 05 June 2009, but it may take a while to trickle into the updates - it may not be fixed until 9.10.
As an historical note, this problem no longer exists in Ubuntu 10.04.
You can also use this script:
(taken from http://newsgroups.derkeiler.com/Archive/Comp/comp.sys.mac.system/2005-12/msg00247.html)
#!/bin/sh
# timeout script
#
usage()
{
echo "usage: timeout seconds command args ..."
exit 1
}
[[ $# -lt 2 ]] && usage
seconds=$1; shift
timeout()
{
sleep $seconds
kill -9 $pid >/dev/null 2>/dev/null
}
eval "$#" &
pid=$!
timeout &
wait $pid
.