Hegle Jensens' wrote a great SnapBtr script that makes snapshot-based backups with smart algorithm that chooses which old backup(s) to delete, when the free space become scarce.
Unfortunately the BTRFS file system has a peculiarity that after any delete command it doesn't immediately free the disk space; instead it merely schedules the deletion of each node. The actual process of freeing the disk space happens in the background, and only after it is finished, we know how much free space became available.
That's why I'd like to improve this script so after deletion of a spare subvolume it will wait until there will be no hard drive activity in order to get the actual free disk space statistics.
The question: knowing that there is so many Python libraries around, do you know any that would return something, that I can use to get hard drive activity saturation in %?
If that helps, I've already made a Bash script wait-for-disk-idle.sh, which relies on iostat for the disk activity information. But I guess that calling external Bash process for something so simple is quite inefficient and error prone (what if the iostat is not installed?):
#! /bin/bash
USAGE="Usage: `basename $0` [-t sample time] [-p disk IO percent threshold] disk-device"
time=4
percent=10
# Parse command line options.
while getopts ":t:" OPT; do
case "$OPT" in
t)
time=$OPTARG
;;
:)
# getopts issues an error message
echo "`basename $0` version 0.1"
echo $USAGE >&2
exit 1
;;
\?)
# getopts issues an error message
echo "`basename $0` version 0.1"
echo $USAGE >&2
exit 1
;;
esac
done
while getopts ":p:" OPT; do
case "$OPT" in
p)
percent=$OPTARG
;;
:)
;;
\?)
# getopts issues an error message
echo "`basename $0` version 0.1"
echo $USAGE >&2
exit 1
;;
esac
done
# Remove the switches we parsed above.
shift `expr $OPTIND - 1`
# We want at least one non-option argument.
# Remove this block if you don't need it.
if [ $# -eq 0 ]; then
# getopts issues an error message
echo "`basename $0` version 0.1"
echo $USAGE >&2
exit 1
fi
echo percent: $percent, time: $time, disk: $1
while [[ $(iostat -d -x $time 2 $1 |
sed -n 's/.*[^0-9]\([0-9][0-9]*\)[\.,][^,^\.]*$/\1/p' | tail -1) > $percent
]]; do
echo wait
done
Here is an answer I've got from the (former) maintainer of the script:
I am no longer cleaning up using the SnapBtr.py script, however you
may be able to wait for the delete to complete with a btrfs filesystem sync.
Related
Is there a way like that you can use the "if" branch to check whether a USB-Stick is connected? And if the stick is not connected, a message should then be issued.
Something like:
if [-e /sdb1]
then
cp /home/backup/* /sdb1
rm -r /home/backup/*
echo "your files are successfully transferred."
else
echo "please be sure, if your USB-Stick is connected"
fi
No, the mount point exists whether or not a USB device is connected.
Something like this should suffice:
if [[ $(df | grep "/sdb1") && -d /sdb1 && -w /sdb1 ]]
That is if, of course, you have actually created the directory /sdb1/.
Use the findmnt utility (installed by default on RH and Ubuntu, at least)
findmnt /backup
echo $?
1
Return of 1 means not mounted.
Here is some sample code using it with an if statment
findmnt /backup >/dev/null
if [ $? = 0 ]; then
echo "It's mounted all right"
else
r=$(( RANDOM % 4 ))
echo "USB is not mounted."
case $r in
0) echo "Check the couch cushions."
;;
1) echo "I think I saw it in the kitchen."
;;
2) echo "Sign up for Prime and get free shipping!"
;;
3) echo "The dog ate it."
;;
esac
fi
I've been trying to understand a problem that's cropped up with some of the scripts we use at work.
To generate many of our script logs, we utilize the exec command and file redirects to print all output from the script to both the terminal and a log file. Occasionally, for information that doesn't need to be displayed to the user, we do a straight redirect to the log file.
The issue we're seeing occurs on the last line of output to the file when we're printing the number of errors that occurred during that execution: The text doesn't get printed to the file.
In an attempt to diagnose the problem, I wrote a simplified version of our production script (script1.bash) and a test script (script2.bash) to try to tease out the problem.
script1.bash
#!/bin/bash
log_name="${USER}_`date +"%Y%m%d-%H%M%S"`_${HOST}_${1}.log"
log="/tmp/${log_name}"
log_tmp="/tmp/temp_logs"
err_count=0
finish()
{
local ecode=0
if [ $# -eq 1 ]; then
ecode=${1}
fi
# This is the problem line
echo "Error Count: ${err_count}" >> "${log}"
mvlog
local success=$?
exec 1>&3 2>&4
if [ ${success} -ne 0 ]; then
echo ""
echo "WARNING: Failed to save log file to ${log_tmp}"
echo ""
ecode=$((ecode+1))
fi
exit ${ecode}
}
mvlog()
{
local ecode=1
if [ ! -d "${log_tmp}" ]; then
mkdir -p "${log_tmp}"
chmod 775 "${log_tmp}"
fi
if [ -d "${log_tmp}" ]; then
rsync -pt --bwlimit=4096 "${log}" "${log_tmp}/${log_name}" 2> /dev/null
[ $? -eq 0 ] && ecode=0
if [ ${ecode} -eq 0 ]; then
rm -f "${log}"
fi
fi
}
exec 3>&1 4>&2 >(tee "${log}") 2>&1
ecode=0
echo
echo "Some text"
echo
finish ${ecode}
script2.bash
#!/bin/bash
runs=10000
logdir="/tmp/temp_logs"
if [ -d "${logdir}" ]; then
rm -rf "${logdir}"
fi
for i in $(seq 1 ${runs}); do
echo "Conducting run #${i}/${runs}..."
${HOME}/bin/script1.bash ${i}
done
echo "Scanning logs from runs..."
total_count=`find "${logdir}" -type f -name "*.log*" | wc -l`
missing_count=`grep -L 'Error Count:' ${logdir}/*.log* | grep -c /`
echo "Number of runs performed: ${runs}"
echo "Number of log files generated: ${total_count}"
echo "Number of log files missing text: ${missing_count}"
My first test indicated roughly 1% of the time the line isn't written to the log file. I then proceeded to try several different methods of handling this line of output.
Echo and Wait
echo "Error Count: ${err_count}" >> "${log}"
wait
Alternate print method
printf "Error Count: %d\n" ${err_count} >> "${log}"
No Explicit File Redirection
echo "Error Count: ${err_count}"
Echo and Sleep
echo "Error Count: ${err_count}" >> "${log}"
sleep 0.2
Of these, #1 and #2 each had a 1% fail rate while #4 had a staggering 99% fail rate. #3 was the only methodology that had a 0% fail rate.
At this point, I'm at a loss for why this behavior is occurring, so I'm asking the gurus here for any insight.
(Note that the simple solution is to implement #3, but I want to know why this is happening.)
Without testing, this looks like a race condition between your script and tee. It's generally better to avoid multiple programs writing to the same file at the same time.
If you do insist on having multiple writers, make sure they are all in append mode, in this case by using tee -a. Appends to the local filesystem are atomic, so all writes should make it (this is not necessarily true for networked file systems).
I want know what am I doing wrong in this code:
#!/bin/sh
SERVICE_NAME=neocloud
PATH_TO_JAR=/etc/neocloud/cloud.jar
PID_PATH_NAME=/tmp/neocloud-pid
case $1 in
start)
echo "Starting $SERVICE_NAME ..."
if [ ! -f $PID_PATH_NAME ]; then
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is already running ..."
fi
;;
stop)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stoping ..."
kill $PID;
echo "$SERVICE_NAME stopped ..."
rm $PID_PATH_NAME
else
echo "$SERVICE_NAME is not running ..."
fi
;;
restart)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
echo "$SERVICE_NAME stopping ...";
kill $PID;
echo "$SERVICE_NAME stopped ...";
rm $PID_PATH_NAME
echo "$SERVICE_NAME starting ..."
nohup java -jar $PATH_TO_JAR /tmp 2>> /dev/null >> /dev/null &
echo $! > $PID_PATH_NAME
echo "$SERVICE_NAME started ..."
else
echo "$SERVICE_NAME is not running ..."
fi
;;
esac
executing via sh I get the following error message:
Syntax error: word unexpected (expecting "in")
But in the case command I've the in word.
Anyone know how to fix this bug?
Thanks a lot!
As posted here, your code is syntactically valid POSIX-like shell code, which you can verify at shellcheck.net (which will, however, warn you about potentially unwanted side effects due to not double-quoting your variable references (e.g., echo $PID_PATH_NAME rather than echo "$PID_PATH_NAME"), which however does NOT apply to $1 in the case statement[1]
).
Similarly, copying the code in your question and pasting it into a new file and using sh on Ubuntu (which is Dash) to execute it, works fine too.
Thus - unless your sh is not what it should be - I suspect that you have "weird" characters in your shell file, such as Unicode whitespace outside the standard ASCII range, which may look like normal whitespace, but isn't; the Unicode no-break space character (U_00A0, UTF8-encoded as 0xC2 0xA0) is an example.
To look for such characters, run the following (where script represents your script):
LC_ALL=C cat -e script
and look for M- and ^<letter> sequences in the output; for instance, the aforementioned no-break space shows up as M-BM-.
[1] Double-quoting the argument given to the case statement doesn't hurt, but is not necessary.
While unquoted parameter/variable references are word-split and pathname-expanded in most places in POSIX-like shells, case is a curious exception.
The following demonstrates this, and works with all major POSIX-like shells (dash, bash, ksh, zsh):
$ sh -c 'case $1 in "foo *") echo "match";; *) echo "nomatch"; esac' - 'foo *'
match
Literal argument foo * matches the case branch, even though $1 is unquoted.
(Contrast this with the typical situation (e.g., echo $1), where the value of $1 would be subject to both word-splitting and pathname expansion (globbing).)
Have you tried quoting the argument?
case "$1" in
...
I'm new to shell scripting, my script appears to be okay, but its the flow that I'm having an issue controlling. Could someone point out what silly mistake I've made please.
#! /bin/sh
echo "Are you sure youx want to delete $1? Answer y or n"
read ans
echo $ans
if $ans = "y"|"Y"
then
mv $1 /home/parallels/dustbin
echo "File $1 has been deleted"
else echo "File $1 has not been deleted"
fi
Make your if condition like this:
if [ "$ans" = "y" -o "$ans" = "Y" ]
There are a few things wrong with your script. Some are serious, some are less so.
First, the serious problems.
As guru suggested, you need to use square brackets to surround your if condition. This is because if only tests for the output of a condition, it doesn't perform actual string comparisons. Traditionally, a program called /bin/test, which was also called /bin/[ took care of that. These days, that functionality is built in to the shell, but /bin/sh still behaves as if it's a separate program.
In fact, you can do interesting things with if when you don't use square brackets for your condition. For example, if grep -q 'RE' /path/to/file; then is quite common. The grep -q command issues no output, but simply returns a "success" or "fail" that is detected by if.
Second serious problem is that you are echoing a status that may or may not be true. I call this a serious problem because ... well, log messages simply shouldn't make false claims. If the permissions are wrong for the file in $1, or the filename contains a space, then your mv command will fail, but the message will claim that it did not. More on this later.
Next, the less serious problems.
These are mostly style and optimization things.
First off, read on most platforms includes a -p option that lets you specify a prompt. Use this, and you don't need to include an echo command.
Second, your indenting makes it hard to see what the if construct is wrapping. This isn't a huge problem in a program this small, but as you grow, you REALLY want to follow consistent standards.
Third, you can probably get more flexibility in multiple-choice questions like this if you use case statements instead of if.
After all that, here's how I'd write this script:
#!/bin/sh
if [ "$1" = "-y" ]; then
ans=y
shift
elif [ -t 0 ]; then
read -p "Are you sure you want to delete '$1' (y/N) ? " ans
fi
case "$ans" in
Y*|y*)
retval=0
if [ -z "$1" ]; then
retval=64
echo "ERROR: you didn't specify a filename." >&2
if [ ! -f "$1" ]; then
retval=66
echo "ERROR: file '$1' not found!" >&2
elif mv "$1" /home/parallels/dustbin/; then
echo "File '$1' has been deleted" >&2
else
retval=$?
echo "ERROR: file '$1' could not be deleted!" >&2
fi
;;
*)
echo "ABORT: file '$1' has not been deleted" >&2
retval=4
;;
esac
exit $retval
Aside from what's mentioned above, here are some things in this code snippet:
[ "$1" = "-y" ] - if the user specifies a -y option, then we behave as if the question was answered with a "yes".
[ -t 0 ] - this tests whether we are on an interactive terminal. If we are, then it makes sense to ask questions with read.
Y*|y*) - in a case statement, this matches any string that begins with an upper or lower case "y". Valid affirmative responses would therefore be "Y", "yes", "yellow", etc.
[ ! -f "$1" ] - this tests whether the file exists. You can man test or man sh to see the various tests available in shell. (-f may not be the most appropriate for you.)
>&2 - at the end of a line, sends its output to "standard error" instead of "standard out". This changes how output will be handled by pipes, cron, etc. Errors and log data are often sent to stderr, so that stdout can be dedicated to a program's actual output.
mv "$1" ... - The filename is in quotes. This protects you in case the filename has special characters like spaces in it.
$retval - the values for this came from a best guess of the closest item in man sysexits.
retval=$? - this is the exit status of the most recently executed command. In this case, that means we're assigning mv's exit status to the variable $retval, so that if mv failed, the whole script reports the reason for the fail, as far as mv is concerned.
You can also convert the user response to either case and just check it for respective case like
read ans
ans=${ans,,} # make 'ans' lowercase, or use ${ans^^} for making it uppercase
if [ "$ans" = "y" ]
then
....
fi
Below is the perfect code with error handling included
#!/bin/sh
echo "Are you sure you want to delete $1? Answer y or n"
read ans
echo $ans
if [ $ans == "y" ] || [ $ans == "Y" ]
then
if [ -f $1 ]
then
mv $1 /home/parallels/dustbin
echo "File $1 has been deleted"
else
echo " File $1 is not found"
fi
else
echo "File $1 has not been deleted"
fi
I am using the following to keep a single instance of a script running on my server. I have a cronjob to run this every minute.
How do I daemonize an arbitrary script in unix?
#!/bin/bash
if [[ $# < 1 ]]; then
echo "Name of pid file not given."
exit
fi
# Get the pid file's name.
PIDFILE=$1
shift
if [[ $# < 1 ]]; then
echo "No command given."
exit
fi
echo "Checking pid in file $PIDFILE."
#Check to see if process running.
PID=$(cat $PIDFILE 2>/dev/null)
if [[ $? = 0 ]]; then
ps -p $PID >/dev/null 2>&1
if [[ $? = 0 ]]; then
echo "Command $1 already running."
exit
fi
fi
# Write our pid to file.
echo $$ >$PIDFILE
# Get command.
COMMAND=$1
shift
# Run command
$COMMAND "$*"
Now I found out that my script had hung for some reason and therefore it was stuck. I'd like a way to check if the $PIDFILE is "old" and if so, kill the process. I know that's possible (check the timestamp on the file) but I don't know the syntax or if this is even a good idea. Also, when this script is running, the CPU should be pretty heavily used. If it hangs (rare but it happened at least once so far), the CPU usage drops to 0%. It would be nice if I could check that the process is really hung/not active, but I don't know if there's an easy way to do that (and I don't want to have many false positives where it gets killed but it's running fine).
To answer the question in your title, which seems quite different from your problem, use timeout.
Now, for your problem, I don't see where it could hang, unless you gave it a fifo queue for the pid file. Now, to run and respawn, you can just run this script once, on startup:
#!/bin/bash
while /bin/true; do
"$#"
wait
done
Which brings up another bug in the code you got from the other question: "$*" will pass all the arguments to the script as a single argument; without the quotes it'll split arguments with white space. "$#" will pass them individually and handling white space properly.
Call with /path/to/script command [argument]....