I want to find how much it takes to run my mysqldump and compare it with my I/O rate at the end of mysqldump command.
looking for someting like :
bash:> time .dumpscript
--and at the end he will calculate my I/O rate from starting point to finish point giving me something like :
Dumpsize Time I/O per sec
30 gb 30 min 5mb/sec
Thx all!
You can use the time command in bash to see how long your command takes. This will give the execution time in seconds:
{ time -p ./dumpscript; } 2>&1 | tail -3 | awk 'NR==1{print $2}'
Presumably you know the location of the dump file, so you can find the size of that with stat. Since you now know the size of the file, and the time it took to create it, you can calculate the I/O rate with some basic arithmetic.
Related
I'm running a command-line utility and it's producing 60 records(CSV) at a time and sleeps for one minute and producing 60 records again and so on.
I can redirect the output in a file but I want only the last 60 records to be saved (overwrite) for every minute.
Please help me to achieve this.
This can easily be done using the tail command.
Although tail -60 exists, I advise you to use tail -n 60.
I have been playing around with the bashrc and one of the thing I want to see at all time is my cpu usage in percentage. I decided to set this data in my PS1. The problem is that to have an accurate estimation of my CPU usage I need to do operations that require waiting for at least 0.5 seconds.
As a result of this, my new command line only displays at the end of the CPU calculation, 0.5 seconds later, which is really not acceptable. To deal with this I thought that I could maybe use a thread to do the CPU calculation and only display it at the end but I don't know how to do so.
One of the problem is that I display other information after CPU percentage so I don't know if it is even possible to delay the CPU display while still displaying the rest of the command line. I thought that maybe I could display a temporary string such as ??.?? and then replace it by the real value but I am not sure how to do so since if I type commands fast the position of the ??.?? can change (for example typing ls 5 times in a row very fast).
Maybe there is an even simpler solution to my problem such as calculating the CPU percentage in an other way ?
My CPU percentage calculating function:
function cpuf(){
NonIdle=0;Idle=0;Total=0;TotalD=0;Idled=0
NonIdle=$((`cat /proc/stat | awk '/^cpu / {print$2+$3+$4+$7+$8+$9}'` - $NonIdle))
Idle=$((`cat /proc/stat | awk '/^cpu / {print$5+$6}'` - $Idle))
sleep 0.5
NonIdle=$((`cat /proc/stat | awk '/^cpu / {print$2+$3+$4+$7+$8+$9}'` - $NonIdle))
Idle=$((`cat /proc/stat | awk '/^cpu / {print$5+$6}'` - $Idle))
Total=$((Idle+NonIdle))
CPU=$(((Total-Idle)/Total))
echo `echo "scale=2;($Total*100-$Idle*100)/$Total" | bc -l`
}
How I call it in the bashrc:
alias cpu="cpuf"
PS1+="(\[${MAGENTA}\]CPU $(cpu)%"
There is no need to reinvent the wheel here, linux already takes care of capturing system stats in /proc/loadavg. The first number is average load in the last minute across all cpus, so we just need to divide by the number of cpus, which we can determine by reading /proc/cpuinfo. Rolling this into .bashrc we get:
.bashrc
...
# My fancy prompt, adjust as you like...
FANCY_PROMPT="$GREEN\u$YELLOW#\h:$PURPLE\w$BLUE$ $RESET"
CPUS=$( grep -c bogomips /proc/cpuinfo )
_prompt_command() {
LOAD_AVG_1_MIN=$( cut -d ' ' -f 1 /proc/loadavg )
PERCENT=$( echo "scale=0; $LOAD_AVG_1_MIN * 100 / $CPUS" | bc -l )
PS1="CPU $PERCENT% $FANCY_PROMPT"
true
}
PROMPT_COMMAND="_prompt_command"
In Use:
SO linux /proc/loadavg
I have a bash script that I plan to run every 5 or 15 mins using crontab based on the load it puts on server.
I can find time of running the script, but load, memory usage and CPU usage I am not sure how to find.
Can someone help me?
Also any suggestions of rough benchmark that will help me decide if the script puts too much load and should be run every 15 mins and not 5 mins.
Thanks in Advance!
You can use "top -b", top gives the CPU usage, memory usage etc,
Insert these lines in your script, this will process in background and will terminate the process as soon as your testing overs.
ssh server_name "nohup top -b -d 0.5 >> file_name &"
\top process will run in background because of &, -d 0.5 will give you the cpu status at every 0.5 secs, redirect the output in file_name for latter analysis.
for killing the process after your test, insert following in your script,
ssh server_name "kill \`ps -elf | grep 'top -b' | grep -v grep | sed 's/ */ /g' |cut -d ' ' -f4\`"
Your main testing script should be between top command and command for killing top.
I presumed you are running the script from client side, if not ignore "ssh server_name".
If you are running it from client side, because of "ssh", you will be asked for the password, for avoiding this follow these 3 simple steps
This will definitely solve the issue.
You can check following utilities
pidstat for CPU load, man page
pmap for memory load, man page
Although you might need to make measurements also for the child processes of your executable, in order to collect summarizing information
For memory, use free -m. Your actual memory available is the second number next to +/- buffers/cache (in megabytes with -m) (source).
For CPU, it's a bit more complicated. Start by looking at cat /proc/stat | grep 'cpu ' (note the space). You'll see something like this:
cpu 2255 34 2290 22625563 6290 127 456
The columns are from left to right, "user, nice, system, idle". CPU usage is usually calculated as (user+nice+system) / (user+nice+system+idle). However, these numbers show the number of "time units" that the CPU has spent doing that thing since boot, and thus are always increasing. If you were to do the aforementioned calculation, you'd get the CPU usage average since boot. To get a point-in-time usage, you have to take 2 samples, find their difference, and calculate the usage from that. To be clear, that will be the average CPU usage between your samples. (source)
The following will display the time it took for a given command to execute. How would I do the same but with better precision (i.e 5.23 seconds)?
[root#localhost ~]# start=`date +%s`; sleep 5 && echo execution time is $(expr `date +%s` - $start) seconds
execution time is 5 seconds
[root#localhost ~]#
You could try using the time command.
time sleep 5
In addition to elapsed wall clock time it will tell you how much CPU time the process consumed, and how much of the CPU time was spent in the application and how much in operating system calls.
Use the time command:
time COMMAND
You can also use the SECONDS variable:
In the Bash Variables section of the reference manual you'll read:
SECONDS This variable expands to the number of seconds since the shell was started. Assignment to this variable resets the count to the value assigned, and the expanded value becomes the value assigned plus the number of seconds since the assignment.
Hence, the following:
SECONDS=0; sleep 5; echo "I slept for $SECONDS seconds"
should output 5. It's only good for measuring times with precision of 1 second, but it sure is much better than the way you showed using the date command.
If you need more precision, you can use the time command, but it's a bit tricky to get the output of this command in a variable. You'll find all the information in the BashFAQ/032: How can I redirect the output of time to a variable or file?.
I'm running some JMeter tests against a Java process to determine how responsive a web application is under load (500+ users). JMeter will give the response time for each web request, and I've written a script to ping the Tomcat Manager every X seconds which will get me the current size of the JVM heap.
I'd like to collect stats on the server of the % of CPU being used by Tomcat. I tried to do it in a shell script using ps like this:
PS_RESULTS=`ps -o pcpu,pmem,nlwp -p $PID`
...running the command every X seconds and appending the results to a text file. (for anyone wondering, pmem = % mem usage and nlwp is number of threads)
However I've found that this gives a different definition of "% of CPU Utilization" than I'd like - according to the manpages for ps, pcpu is defined as:
cpu utilization of the process in "##.#" format. It is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage.
In other words, pcpu gives me the % CPU utilization for the process for the lifetime of the process.
Since I want to take a sample every X seconds, I'd like to be collecting the CPU utilization of the process at the current time only - similar to what top would give me
(CPU utilization of the process since the last update).
How can I collect this from within a shell script?
Use top -b (and other switches if you want different outputs). It will just dump to stdout instead of jumping into a curses window.
The most useful tool I've found for monitoring a server while performing a test such as JMeter on it is dstat. It not only gives you a range of stats from the server, it outputs to csv for easy import into a spreadsheet and lets you extend the tool with modules written in Python.
User load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $2}' |sed 's/.[^.]*$//'
System load: top -b -n 2 |grep Cpu |tail -n 1 |awk '{print $3}' |sed 's/.[^.]*$//'
Idle load: top -b -n 1 |grep Cpu |tail -n 1 |awk '{print $5}' |sed 's/.[^.]*$//'
Every outcome is a round decimal.
Off the top of my head, I'd use the /proc filesystem view of the system state - Look at man 5 proc to see the format of the entry for /proc/PID/stat, which contains total CPU usage information, and use /proc/stat to get global system information. To obtain "current time" usage, you probably really mean "CPU used in the last N seconds"; take two samples a short distance apart to see the current rate of CPU consumption. You can then munge these values into something useful. Really though, this is probably more a Perl/Ruby/Python job than a pure shell script.
You might be able to get the rough data you're after with /proc/PID/status, which gives a Sleep average for the process. Pretty coarse data though.
also use 1 as iteration count, so you will get current snapshot without waiting to get another one in $delay time.
top -b -n 1
This will not give you a per-process metric, but the Stress Terminal UI is super useful to know how badly you're punishing your boxes. Add -c flag to make it dump the data to a CSV file.