I have a shell script which backups my MySQL database every hour. It's a basic script:
mysqldump --user=$USER --password=$PASS $DB > /$PATH/$DATE.sql &&
7z a -t7z -mx=9 /$PATH/$DATE.sql.7z /$PATH/$DATE.sql &&
rm /$PATH/$DATE.sql
I'm using a 7z compression, because:
file permissions and owner/group aren't required to be kept
the space saved by 7z compared to gzip is essential to me
What's troubling me is that the 7z part (line 2) takes about 30 seconds and uses quite a lot of CPU during that time. There are 50% peaks on my server load graphs every hour from this and I'd like to get rid of those peaks.
I'm already executing this script with nice:
0 * * * * /usr/bin/nice -n 19 /path/to/backup.sh > /dev/null
The only solution I can come up with at this point is to somehow pause the execution of the 7z part, let's say every second for 1 second. This could free up the CPU time for other processes.
Is this possible?
I'm doing something similar in my .php scripts, although the pauses are in loops, for example:
while($d=mysqli_fetch_assoc($q)){
usleep(250000);
// process $d
}
I'd like the 7z execution to sleep every X (something) for 1 second.
You can use
prlimit --cpu=10 7z a -t7z -mx=9 /$PATH/$DATE.sql.7z /$PATH/$DATE.sql
Which will limit the cpu usage to 10%
How about configuring a limit for a user/group using the /etc/security/limits.conf file? And running the script/compression as specific user? After setting the limit you can test it using fork bomb.
Related
Is there a way to limit the absolute CPU time (in CPU seconds) spend in a process group?
ulimit -t 10; ./my-process looks like a good option but if my-process forks then each process in the process group gets its own limit. The whole process group can use an arbitrary amount of time by forking every 9 seconds.
The accepted answer on a similar question is to use cgroups but doesn't explain how. However, there are other answers (Limit total CPU usage with cgroups) saying that this is not possible in cgroups and only relative cpu usage can be limited (for example, 0.2 seconds out of every 1 second).
Liran Funaro suggested using a long period for cpu.cfs_period_us (https://stackoverflow.com/a/43660834/892961) but the parameter for the quota can be at most 1 second. So even with a long period I don't see how to set a CPU time limit of 10 seconds or an hour.
If ulimit and cgroups cannot do this, is there another way?
you can do it with cgroups. Do as root:
# Create cgroup
cgcreate -g cpu:/limited
# set shares (cpu limit)
cgset -r cpu.shares=256 limited
# run your program
cgexec -g cpu:limited /my/hungry/program
Alternatively you can use the cpulimit program which can freeze your code periodically. cgroups is the most advanced method though.
to set fixed cpu share :
cgcreate -g cpu:/fixedlimit
# allow fix 25% cpu usage (1 cpu)
cgset -r cpu.cfs_quota_us=25000,cpu.cfs_period_us=100000 fixedlimit
cgexec -g cpu:fixedlimit /my/hungry/program
It turned out, the goal is to limit runtime to certain seconds while measuring it. After setting the desired cgroup limits (in order to get a fair sandbox) you can achieve this goal by running:
((time -p timeout 20 cgexec -g cpu:fixedlimit /program/to/test ) 2>&1) | grep user
After 20 seconds the program will be stopped no matter what, and we can parse for user time (or system or real time) to evaluate it's performance.
This not directly answer the question but refers to the discussion on the actual need of the OP.
If your competition ignores everything except CPU time, it may be fundamentally flawed. One can simply, for example, cache results in the primary storage device. Since you do not count storage access time, it may have the least CPU cycles, but the worse actual performance.
A perfect crime would be to simply send the data via the Internet to another computer, which calculate the task then return the answer. This would finish the task with what appear to be zero cycles.
You actually want to measure "real" time and give this process the highest priority in your system (or actually running it secludedly).
When checking students' homework, we simply used an unrealistic time limit (e.g., 5 minutes for what should be a 10 seconds program), then killing the process if it has not finished in time and failing this submission.
If you want to pick a winner, then simply re-run the best competitors multiple times to ensure the validity of their results.
I found a solution that works for me. It is still far from perfect (read the caveats before using it). I'm somewhat new to bash scripting so any comments about this are welcome.
#!/bin/bash
#
# This script tries to limit the CPU time of a process group similar to
# ulimit but counting the time spent in spawned processes against the
# limit. It works by creating a temporary cgroup to run the process in
# and checking on the used CPU time of that process group. Instead of
# polling in regular intervals, the monitoring process assumes that no
# time is lost to I/O (i.e., wall clock time = CPU time) and checks in
# after the time limit. It then updates its assumption by comparing the
# actual CPU usage to the time limit and waiting again. This is repeated
# until the CPU usage exceeds its limit or the monitored process
# terminates. Once the main process terminates, all remaining processes
# in the temporary cgroup are killed.
#
# NOTE: this script still has some major limitations.
# 1) The monitored process can exceed the limit by up to one second
# since every iteration of the monitoring process takes at least that
# long. It can exceed the limit by an additional second by ignoring
# the SIGXCPU signal sent when hitting the (soft) limit but this is
# configurable below.
# 2) It assumes there is only one CPU core. On a system with n cores
# waiting for t seconds gives the process n*t seconds on the CPU.
# This could be fixed by figuring out how many CPUs the process is
# allowed to use (using the cpuset cgroup) and dividing the remaining
# time by that. Since sleep has a resolution of 1 second, this would
# still introduce an error of up to n seconds.
set -e
if [ "$#" -lt 2 ]; then
echo "Usage: $(basename "$0") TIME_LIMIT_IN_S COMMAND [ ARG ... ]"
exit 1
fi
TIME_LIMIT=$1
shift
# To simulate a hard time limit, set KILL_WAIT to 0. If KILL_WAIT is
# non-zero, TIME_LIMIT is the soft limit and TIME_LIMIT + KILL_WAIT is
# the hard limit.
KILL_WAIT=1
# Update as necessary. The script needs permissions to create cgroups
# in the cpuacct hierarchy in a subgroup "timelimit". To create it use:
# sudo cgcreate -a $USER -t $USER -g cpuacct:timelimit
CGROUPS_ROOT=/sys/fs/cgroup
LOCAL_CPUACCT_GROUP=timelimit/timelimited_$$
LOCAL_CGROUP_TASKS=$CGROUPS_ROOT/cpuacct/$LOCAL_CPUACCT_GROUP/tasks
kill_monitored_cgroup() {
SIGNAL=$1
kill -$SIGNAL $(cat $LOCAL_CGROUP_TASKS) 2> /dev/null
}
get_cpu_usage() {
cgget -nv -r cpuacct.usage $LOCAL_CPUACCT_GROUP
}
# Create a cgroup to measure the CPU time of the monitored process.
cgcreate -a $USER -t $USER -g cpuacct:$LOCAL_CPUACCT_GROUP
# Start the monitored process. In case it fails, we still have to clean
# up, so we disable exiting on errors.
set +e
(
set -e
# In case the process doesn't fork a ulimit is more exact. If the
# process forks, the ulimit still applies to each child process.
ulimit -t $(($TIME_LIMIT + $KILL_WAIT))
ulimit -S -t $TIME_LIMIT
cgexec -g cpuacct:$LOCAL_CPUACCT_GROUP --sticky $#
)&
MONITORED_PID=$!
# Start the monitoring process
(
REMAINING_TIME=$TIME_LIMIT
while [ "$REMAINING_TIME" -gt "0" ]; do
# Wait $REMAINING_TIME seconds for the monitored process to
# terminate. On a single CPU the CPU time cannot exceed the
# wall clock time. It might be less, though. In that case, we
# will go through the loop again.
sleep $REMAINING_TIME
CPU_USAGE=$(get_cpu_usage)
REMAINING_TIME=$(($TIME_LIMIT - $CPU_USAGE / 1000000000))
done
# Time limit exceeded. Kill the monitored cgroup.
if [ "$KILL_WAIT" -gt "0" ]; then
kill_monitored_cgroup XCPU
sleep $KILL_WAIT
fi
kill_monitored_cgroup KILL
)&
MONITOR_PID=$!
# Wait for the monitored job to exit (either on its own or because it
# was killed by the monitor).
wait $MONITORED_PID
EXIT_CODE=$?
# Kill all remaining tasks in the monitored cgroup and the monitor.
kill_monitored_cgroup KILL
kill -KILL $MONITOR_PID 2> /dev/null
wait $MONITOR_PID 2>/dev/null
# Report actual CPU usage.
set -e
CPU_USAGE=$(get_cpu_usage)
echo "Total CPU usage: $(($CPU_USAGE / 1000000))ms"
# Clean up and exit with the return code of the monitored process.
cgdelete cpuacct:$LOCAL_CPUACCT_GROUP
exit $EXIT_CODE
I am totally new to this forum and also new to bash, so please bear with me :).
I would like to write a bash script to conduct a CPU stress test while observing the clock speed. Therefore, I have done the following:
1.) For the CPU stress test, I have created a script named "bernoulli" with the following code:
#!/bin/bash
# argument 1: n
function bernoulli()
{
if (( $1 < 3 ))
then
echo 1
else
echo $(( $(bernoulli $(( $1 - 1 ))) + $(bernoulli $(( $1 - 2 ))) ))
fi
}
bernoulli $1
2.) I have figured out that by using the "timeout" command I can kill a task after a specified time. For example,
timeout 30s ./bernoulli 35
starts a task calculating the 35th bernoulli number and the task is killed after 30 seconds.
3.) I also found out that by typing
timeout 30s watch grep \"cpu MHz\" /proc/cpuinfo
I can watch the clock speed of my cores (updated every 2 seconds) for 30 seconds (at which point "timeout 30s" kills this task).
What I want: I would like to do the above stress test and simultaneously observe the clock speed. In other words, I would somehow run the two commands
timeout 30s ./bernoulli 35
timeout 30s watch grep \"cpu MHz\" /proc/cpuinfo
"at the same time". I hope I could make it clear what I would like to achieve. Can anyone help with my issue? Thanks a lot for every comment!
How about
timeout 30s ./bernoulli 35 &
timeout 30s watch grep \"cpu MHz\" /proc/cpuinfo
& at the end will make command to run at background, so that second timeout will be executed almost instantly after the first one.
PS: this is rather poor way to test modern CPU. You will be exercising only single core and most likely only limited part of your CPU (no sse, etc). It is not trivial to write CPU benchmark, so you might want to use one of already available. For example, you can take a look at sysbench with something like sysbench --test=cpu --cpu-max-prime=20000 run.
You can run them in a dedicated shell:
timeout 30s bash -c './bernoulli 35 & watch grep \"cpu MHz\" /proc/cpuinfo'
Note that the single & is not a typo. It is not a logical and, it runs the bernoulli script in background.
I would like to keep a process running for a long time (e.g., more than half an hour). My program is gpg. If I encrypt a 500MB file using gpg elgamal encryption, it takes around 1-2 minutes (compression is turned off). To increase the running time, I can only create a file with a few GB, which is not desirable. Is there any other way to make this gpg program run for longer time?
I believe, by default, gpg takes its input from standard input and sends output to standard output.
So you could make it run forever with something like:
gpg --encrypt </dev/urandom >/dev/null
To use that for consuming CPU for an hour (for example), you could create a script like:
gpg --encrypt </dev/urandom >/dev/null &
pid=$!
sleep 3600
kill -9 ${pid}
https://unix.stackexchange.com/questions/18869/slow-down-a-process-without-affecting-other-processes
CPULimit might do what you need without affecting other processes:
http://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/
You start the program, then run cpulimit against the program name or PID, specifying what percentage you want it limited. Note the percentage is of all cores; so if you have 4 cores, you could use 400%.
Am getting the real-time memory stats from vmstat command. I did this using following steps:
$nohup vmstat 60 > vmstatrecord.app &
the command executes in background and writes the log to the file vmstatrecord.app. When i see use the command
$ps -A | grep stat
I could see the vmstat running in the background and i could also access the log using tail command as:
$tail -f vmstatrecord.app
the file updates every 60sec interval.
Now my question is
1. process continues to write to the file so what will happen if i leave for days ?
Assumption:
If the process writes the file forever am afraid that the file size might grow too large
If my assumption is correct and my steps are inefficient. Is there any alternatives to achieve what am trying to achieve from my above steps ?
This question should better be asked on superuser.com or maybe serverfault.com, as it's not about programming.
Yes, your file will keep growing. That's what the 2nd parameter of vmstat is for - run vmstat 60 1440 to stop after a day (note 1440 = 60 minutes * 24 hours). Once when i had this problem, i made a crontab entry:
0 0 * * * vmstat 60 1440 > /some/where/vmstat.out
to restart the output every day.
I want to execute something in a linux shell under a few different conditions, and be able to output the execution time of each execution.
I know I could write a perl or python script that would do this, but is there a way I can do it in the shell? (which happens to be bash)
Use the built-in time keyword:
$ help time
time: time [-p] PIPELINE
Execute PIPELINE and print a summary of the real time, user CPU time,
and system CPU time spent executing PIPELINE when it terminates.
The return status is the return status of PIPELINE. The `-p' option
prints the timing summary in a slightly different format. This uses
the value of the TIMEFORMAT variable as the output format.
Example:
$ time sleep 2
real 0m2.009s
user 0m0.000s
sys 0m0.004s
You can get much more detailed information than the bash built-in time (i.e time(1), which Robert Gamble mentions). Normally this is /usr/bin/time.
Editor's note:
To ensure that you're invoking the external utility time rather than your shell's time keyword, invoke it as /usr/bin/time.
time is a POSIX-mandated utility, but the only option it is required to support is -p.
Specific platforms implement specific, nonstandard extensions: -v works with GNU's time utility, as demonstrated below (the question is tagged linux); the BSD/macOS implementation uses -l to produce similar output - see man 1 time.
Example of verbose output:
$ /usr/bin/time -v sleep 1
Command being timed: "sleep 1"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 1%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.05
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 210
Voluntary context switches: 2
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
#!/bin/bash
START=$(date +%s)
# do something
# start your script work here
ls -R /etc > /tmp/x
rm -f /tmp/x
# your logic ends here
END=$(date +%s)
DIFF=$(( $END - $START ))
echo "It took $DIFF seconds"
For a line-by-line delta measurement, try gnomon.
$ npm install -g gnomon
$ <your command> | gnomon --medium=1.0 --high=4.0 --ignore-blank --real-time=100
A command line utility, a bit like moreutils's ts, to prepend timestamp information to the standard output of another command. Useful for long-running processes where you'd like a historical record of what's taking so long.
You can also use the --high and/or --medium options to specify a length threshold in seconds, over which gnomon will highlight the timestamp in red or yellow. And you can do a few other things, too.
Should you want more precision, use %N with date (and use bc for the diff, because $(()) only handles integers).
Here's how to do it:
start=$(date +%s.%N)
# do some stuff here
dur=$(echo "$(date +%s.%N) - $start" | bc)
printf "Execution time: %.6f seconds" $dur
Example:
start=$(date +%s.%N); \
sleep 0.1s; \
dur=$(echo "$(date +%s.%N) - $start" | bc); \
printf "Execution time: %.6f seconds\n" $dur
Result:
Execution time: 0.104623 seconds
If you intend to use the times later to compute with, learn how to use the -f option of /usr/bin/time to output code that saves times. Here's some code I used recently to get and sort the execution times of a whole classful of students' programs:
fmt="run { date = '$(date)', user = '$who', test = '$test', host = '$(hostname)', times = { user = %U, system = %S, elapsed = %e } }"
/usr/bin/time -f "$fmt" -o $timefile command args...
I later concatenated all the $timefile files and pipe the output into a Lua interpreter. You can do the same with Python or bash or whatever your favorite syntax is. I love this technique.
If you only need precision to the second, you can use the builtin $SECONDS variable, which counts the number of seconds that the shell has been running.
while true; do
start=$SECONDS
some_long_running_command
duration=$(( SECONDS - start ))
echo "This run took $duration seconds"
if some_condition; then break; fi
done
You can use time and subshell ():
time (
for (( i=1; i<10000; i++ )); do
echo 1 >/dev/null
done
)
Or in same shell {}:
time {
for (( i=1; i<10000; i++ )); do
echo 1 >/dev/null
done
}
The way is
$ > g++ -lpthread perform.c -o per
$ > time ./per
output is >>
real 0m0.014s
user 0m0.010s
sys 0m0.002s
one possibly simple method ( that may not meet different users needs ) is the use of shell PROMPT.it is a simple solution that can be useful in some cases. You can use the bash prompting feature as in the example below:
export PS1='[\t \u#\h]\$'
The above command will result in changing the shell prompt to :
[HH:MM:SS username#hostname]$
Each time you run a command (or hit enter) returning back to the shell prompt, the prompt will display current time.
notes:
1) beware that if you waited for sometime before you type your next command, then this time need to be considered, i.e the time displayed in the shell prompt is the timestamp when the shell prompt was displayed, not when you enter command. some users choose to hit Enter key to get a new prompt with a new timestamp before they are ready for the next command.
2) There are other available options and modifiers that can be used to change the bash prompt, refer to ( man bash ) for more details.
perf stat Linux CLI utility
This tool is overkill for just getting time. But it can do so much more for you to help profile and fix slowness that it is worth knowing about. Ubuntu 22.04 setup:
sudo apt install linux-tools-common linux-tools-generic
echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid
Usage:
perf stat <mycmd>
Sample run with stress-ng:
perf stat stress-ng --cpu 1 --cpu-method matrixprod -t 5
Sample output:
Performance counter stats for 'stress-ng --cpu 1 --cpu-method matrixprod -t 5':
5,005.46 msec task-clock # 0.999 CPUs utilized
88 context-switches # 17.581 /sec
1 cpu-migrations # 0.200 /sec
1,188 page-faults # 237.341 /sec
18,847,667,167 cycles # 3.765 GHz
26,544,261,897 instructions # 1.41 insn per cycle
3,239,655,001 branches # 647.225 M/sec
25,393,369 branch-misses # 0.78% of all branches
5.012218939 seconds time elapsed
4.998051000 seconds user
0.009122000 seconds sys
perf can also do a bunch more advanced things, e.g. here I show how to use it to profile code: How do I profile C++ code running on Linux?