I'm trying to calculate time difference stored inside of two variables inside of a shell script, I'm observing the following pattern:
hhmm -> 0950
so:
time1=1333
time2=0950
Now I need to calculate the difference in time between time1 and time2, as for now I have tried:
deltaTime=$(($time1-$time2))
but I'm facing the following error message
1333-0950: value too great for base (error token is "0950")
I'm expecting as a result: $deltaTime=0343
Unfortunately, I am strictly bound to use this time pattern. I have already researched for a solution online, some of them propose to use date -d... but I couldn't get it to work :(
Your approach has two issues.
First issue: bash recognizes numbers with leading zeroes as octal. You can force base10 by adding 10# prefix.
Second issue: it is incorrect to consider strings in hhmm format as numbers and substract them. e.g. 1333-950=383 but difference between 09:50 and 13:33 is 3 hours and 43 minutes. You should convert string values to common units, e.g. to minutes, substract them and convert back to hhmm format.
time1=1333
time2=0950
str2min()
{
printf "%u" $((10#${1%??} * 60 + 10#${1#??}))
}
min2str()
{
printf "%02u%02u" $(($1 / 60)) $(($1 % 60))
}
time1m=$(str2min $time1)
time2m=$(str2min $time2)
timediff=$(($time1m - $time2m))
deltaTime=$(min2str $timediff)
You could use this implementation maybe?
#!/usr/bin/env bash
diff_hhmm() {
local -r from=$1
local -i from_hh=10#${from:0:2} # skip 0 chars, read 2 chars (`${from:0:2}`) using base 10 (`10#`)
local -ri from_mm=10#${from:2:2} # skip 2 chars, read 2 chars (`${from:0:2}`) using base 10 (`10#`)
local -r upto=$2
local -ri upto_hh=10#${upto:0:2}
local -ri upto_mm=10#${upto:2:2}
local -i diff_hh
local -i diff_mm
# Compute difference in minutes
(( diff_mm = from_mm - upto_mm ))
# If it's negative, we've "breached" into the previous hour, so adjust
# the `diff_mm` value to be modulo 60 and compensate the `from_hh` var
# to reflect that we've already subtracted some of the minutes there.
if (( diff_mm < 0 )); then
(( diff_mm += 60 ))
(( from_hh -= 1 ))
fi
# Compute difference in hours
(( diff_hh = from_hh - upto_hh ))
# Ensure the result is modulo 24, the number of hours in a day.
if (( diff_hh < 0 )); then
(( diff_hh += 24 ))
fi
# Print the values with 0-padding if necessary.
printf '%02d%02d\n' "$diff_hh" "$diff_mm"
}
$ diff_hhmm 1333 0950
0343
$ diff_hhmm 0733 0950
2143
$ diff_hhmm 0733 0930
2203
Or an even shorter implementation using a big arithmetic compound command ((( ... )) ) and inlining some variables:
diff_hhmm_terse() {
local -i diff_hh diff_mm
((
diff_mm = 10#${1:2:2} - 10#${2:2:2},
diff_hh = 10#${1:0:2} - 10#${2:0:2},
diff_hh -= diff_mm < 0 ? 1 : 0,
diff_mm += diff_mm < 0 ? 60 : 0,
diff_hh += diff_hh < 0 ? 24 : 0
))
printf '%02d%02d\n' "$diff_hh" "$diff_mm"
}
Do you have the possibility to drop the leading zero?
As you can see from my prompt:
Prompt> echo $((1333-0950))
-bash: 1333-0950: value too great for base (error token is "0950")
Prompt> echo $((1333-950))
383
Other proposal:
date '+%s'
Let me give you some examples:
date '+%s'
1662357975
... (after some time)
date '+%s'
1662458180
=>
echo $((1662458180-1662357975))
100205 (amount of seconds)
=>
echo $(((1662458180-1662357975)/3600))
27 (amount of hours)
This bash one-liner may be used if time difference is not negative (that is, time1 >= time2):
printf '%04d\n' $(( 10#$time1 - 10#$time2 - (10#${time1: -2} < 10#${time2: -2} ? 40 : 0) ))
Related
I have DB error log file, it will grow continuously.
Now i want to set some error monitoring on that file for every 5 minutes.
The problem is i don’t want to scan whole file for every 5 minutes(when monitoring cron executed), because it may grow very big in future. Scanning through whole(big) file for every 5 mins will consume bit more resources.
So i just want to scan only the lines which were inserted/written to the log during last 5 mins interval.
Each error recorded in log will have Timestamp prepend to it like below:
180418 23:45:00 [ERROR] mysql got signal 11.
So i want to search with pattern [ERROR] only on lines which were added from last 5 mins(not whole file) and place the output to another file.
Please help me here.
Feel free if u need more clarification on my question.
I’m using RHEL 7 and i’m trying to implement above monitoring through bash shell script
Serializing the Byte Offset
This picks up where the last instance left off. If you run it every 5 minutes, then, it'll scan 5 minutes of data.
Note that this implementation knowingly can scan data added during an invocation's run twice. This is a little sloppy, but it's much safer to scan overlapping data twice than to never read it at all, which is a risk that can be run if relying on cron to run your program on schedule (likewise, sleeps can run over the requested time if the system is busy).
#!/usr/bin/env bash
file=$1; shift # first input: filename
grep_opts=( "$#" ) # remaining inputs: grep options
dir=$(dirname -- "$file") # extract directory name to use for offset storage
basename=${file##*/} # pick up file name w/o directory
size_file="$dir/.$basename.size" # generate filename to use to store offset
if [[ -s $size_file ]]; then # ...if we already have a file with an offset...
old_size=$(<"$size_file") # ...read it from that file
else
old_size=0 # ...otherwise start at the front.
fi
new_size=$(stat --format=%s -- "$file") || exit # Figure out current size
if (( new_size < old_size )); then
old_size=0 # file was truncated, so we can't trust old_size
elif (( new_size == old_size )); then
exit 0 # no new contents, so no point in trying to search
fi
# read starting at old_size and grep only that content
dd iflag=skip_bytes skip="$old_size" if="$file" | grep "${grep_opts[#]}"; grep_retval=$?
# if the read failed, don't store an updated offset
(( ${PIPESTATUS[0]} != 0 )) && exit 1
# create a new tempfile to store offset in
tempfile=$(mktemp -- "${size_file}.XXXXXX") || exit
# write to that temporary file...
printf '%s\n' "$new_size" > "$tempfile" || { rm -f "$tempfile"; exit 1; }
# ...and if that write succeeded, overwrite the last place where we serialized output.
mv -- "$tempfile" "$new_size" || exit
exit "$grep_retval"
Alternate Mode: Bisect For The Timestamp
Note that this can miss content if you're relying on, say, cron to invoke your code every 5 minutes on-the-dot; storing byte offsets can thus be more accurate.
Using the bsearch tool by Ole Tange:
#!/usr/bin/env bash
file=$1; shift
start_date=$(date -d 'now - 5 minutes' '+%y%m%d %H:%M:%S')
byte_offset=$(bsearch --byte-offset "$file" "$start_date")
dd iflag=skip_bytes skip="$byte_offset" if="$file" | grep "$#"
Another approach could be something like this:
DB_FILE="FULL_PATH_TO_YOUR_DB_FILE"
current_db_size=$(du -b "$DB_FILE" | cut -f 1)
if [[ ! -a SOME_PATH_OF_YOUR_CHOICE/last_size_db_file ]] ; then
tail --bytes $current_db_size $DB_FILE > SOME_PATH_OF_YOUR_CHOICE/log-file_$(date +%Y-%m-%d_%H-%M-%S)
else
if [[ $(cat last_size_db_file) -gt $current_db_size ]] ; then
previously_readed_bytes=0
else
previously_readed_bytes=$(cat last_size_db_file)
fi
new_bytes=$(($current_db_size - $previously_readed_bytes))
tail --bytes $new_bytes $DB_FILE > SOME_PATH_OF_YOUR_CHOICE/log-file_$(date +%Y-%m-%d_%H-%M-%S)
fi
printf $current_db_size > SOME_PATH_OF_YOUR_CHOICE/last_size_db_file
this prints all bytes of DB_FILE not previously printed to SOME_PATH_OF_YOUR_CHOICE/log-file_$(date +%Y-%m-%d_%H-%M-%S)
Note that $(date +%Y-%m-%d_%H-%M-%S) will be the current 'full' date at the time of creating the log file
you can make this an script, and use cron to execute that script every five minutes; something like this:
*/5 * * * * PATH_TO_YOUR_SCRIPT
Here is my approach:
First, read the whole log once so far.
If you reach the end, collect and read new lines for a timespan (in my example 9 seconds, for faster testing, while my dummy server appends to the logfile every 3 seconds).
After the timespan, echo the cache, clear the cache (an array arr), loop and sleep for some time, so that this process doesn't consume all CPU time.
First, my dummy logfile writer:
#!/bin/bash
#
# dummy logfile writer
#
while true
do
s=$(( $(date +%s) % 3600))
echo $s server msg
sleep 3
done >> seconds.log
Startet via ./seconds-out.sh &.
Now the more complicated part:
#!/bin/bash
#
# consume a logfile as written so far. Then, collect every new line
# and show it in an interval of $interval
#
interval=9 # 9 seconds
#
printf -v secnow '%(%s)T' -1
start=$(( secnow % (3600*24*365) ))
declare -a arr
init=false
while true
do
read line
printf -v secnow '%(%s)T' -1
now=$(( secnow % (3600*24*365) ))
# consume every line created in the past
if (( ! init ))
then
# assume reading a line might not take longer than a second (rounded to whole seconds)
while (( ${#line} > 0 && (now - start) < 2 ))
do
read line
start=$now
echo -n "." # for debugging purpose, remove
printf -v secnow '%(%s)T' -1
now=$(( secnow % (3600*24*365) ))
done
init=1
echo "init=$init" # for debugging purpose, remove
# collect new lines, display them every $interval seconds
else
if ((${#line} > 0 ))
then
echo -n "-" # for debugging purpose, remove
arr+=("read: $line \n")
fi
if (( (now - start) > interval ))
then
echo -e "${arr[#]]}"
arr=()
start=$now
fi
fi
sleep .1
done < seconds.log
Output with logfile generator in 3 seconds, running for some time, then starting the read-seconds.sh script, with debugging output activated:
./read-seconds.sh
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................init=1
---read: 1688 server msg
read: 1691 server msg
read: 1694 server msg
---read: 1697 server msg
read: 1700 server msg
read: 1703 server msg
----read: 1706 server msg
read: 1709 server msg
read: 1712 server msg
read: 1715 server msg
^C
Every dot represents a logfile line from the past and therefor skipped.
Every dash represents a logfile line collected.
I have string variables MIN and SEC (minute and seconds).
MIN = "1"
SEC = "34"
I want to do calculations on these.
TOTSEC = MIN*60 + SEC
I tried:
expr $SEC + $MIN * 60
Result:
expr: non-numeric argument
Let it be known I am running busybox on a custom microcomputer and so have no access to bash,bc, and that other solution provides.
In sh, by which I'll assume you mean a POSIX shell, your best option is to use Arithmetic Expansion:
$ MIN=1
$ SEC=34
$ TOTSEC=$(( MIN * 60 + SEC ))
$ printf '%d\n' "$TOTSEC"
94
In csh however, the built-in math works quite differently:
% set MIN = 1
% set SEC = 34
% # TOTSEC = ( $MIN * 60 + $SEC )
% printf '%d\n' "$TOTSEC"
94
According to the man page, the # command permits numeric calculations to be performed and the result assigned to a variable.
Note that the expr command is external to the shell, so it should be usable from either one.
In sh:
$ TOTSEC=$(expr "$MIN" \* 60 + "$SEC")
And in csh:
% set TOTSEC = `expr "$MIN" \* 60 + "$SEC"`
Note: your sh may not be POSIX compliant. Most likely, it's ash, which is the ancestor of dash and FreeBSD's /bin/sh. You'll need to test in your environment.
I'm looking for a bash script that can parse a time duration.
If three arguments are given, they represent hours, minutes, and seconds. If two arguments are given, they represent minutes and seconds, with the hours zero.
What about the following:
#!/bin/bash
h=0
if [ "$#" -ge 3 ]
then
h=$1
shift
fi
sec=$((3600*$h+60*$1+$2))
echo "The total number of seconds is $sec"
Since the question does not specify what you aim to do with the given time, the program calculates the total number of seconds. Furthermore perhaps it is useful to do a check if at least two arguments are given.
The script uses the shift operation, the shift makes makes $1 := $2; $2 := $3, etc. In other words, the first argument is processed, and then you "pretend" it never existed.
By default you set h to zero, and only if the number of arguments is greater than or equal to 3, it will set h.
This is a more or less general solution for that type of task. Sorry, if it is a monkeycode, but I think it is sufficient:
gettime() {
params=(
years months weeks days hours minutes seconds
)
for i in `seq ${#params}`; do
param_i=$((${#params} - i + 1)) # reversed params index
[ $i -le $# ] && {
eval "local ${params[$param_i]}=\$$(($# - i + 1))"
} || {
eval "local ${params[$param_i]}=0"
}
eval "echo ${params[$param_i]} '==' \$${params[$param_i]}" # debug output
done
}
Here's the sample output:
$ gettime 3 4 5 6 7
seconds == 7
minutes == 6
hours == 5
days == 4
weeks == 3
months == 0
years == 0
Note, that the shell you are using must be not only support POSIX standards, but also arrays.
First Argument: $1
Second Argument: $2
Third Argument: $3
and so on...
Example:
bash-2.05a$ ./parseDuration.sh 13 25 25
13 hours and 25 minutes and 25 seconds
bash-2.05a$ cat ./parseDuration.sh
#!/bin/bash
echo "$1 hours and $2 minutes and $3 seconds"
I have a piece of script that basically calculates the amount of space the directories in the current directory use but I want help understanding some of the syntax and language etiquette.
Here is the script:
#!/bin/bash
# This script prints a little histogram of how much space
# the directories in the current working directory use
error () {
echo "Error: $1"
exit $2
} >&2
# Create a tempfile (in a BSD- and Linux-friendly way)
my_mktemp () {
mktemp || mktemp -t hist
} 2> /dev/null
# check we are using bash 4
(( BASH_VERSINFO[0] < 4 )) && error "This script can only be run by bash 4 or higher" 1
# An array to keep all the file sizes
declare -A file_sizes
declare -r tempfile=$(my_mktemp) || error "Cannot create tempfile" 2
# How wide is the terminal?
declare -ir term_cols=$(tput cols)
# Longest file name, Largest file, total file size
declare -i max_name_len=0 max_size=0 total_size=0
# A function to draw a line
drawline () {
declare line=""
declare char="-"
for (( i=0; i<$1; ++i )); do
line="${line}${char}"
done
printf "%s" "$line"
}
# This reads the output from du into an array
# And calculates total size and maximum size, max filename length
read_filesizes () {
while read -r size name; do
file_sizes["$name"]="$size"
(( total_size += size ))
(( max_size < size )) && (( max_size=size ))
(( max_file_len < ${#name} )) && (( max_file_len=${#name} ))
done
}
# run du to get filesizes
# Using a temporary file for output from du
{ du -d 0 */ || du --max-depth 0 *; } 2>/dev/null > "$tempfile"
read_filesizes < "$tempfile"
# The length for each line and percentage for each file
declare -i length percentage
# How many columns may the lines take up?
declare -i cols="term_cols - max_file_len - 10"
for k in "${!file_sizes[#]}"; do
(( length=cols * file_sizes[$k] / max_size ))
(( percentage=100 * file_sizes[$k] / total_size ))
printf "%-${max_file_len}s | %3d%% | %s\n" "$k" "$percentage" $(drawline $length)
done
printf "%d Directories\n" "${#file_sizes[#]}"
printf "Total size: %d blocks\n" "$total_size"
# clean up
rm "$tempfile"
exit 0
In the first and second line of the read_filesizes() function that I highlighted in bold, why are two variables (size name) being created if the name is being assigned to size in the array?
In the same function, (( max_size < size )) && (( max_size=size )) this line seems odd to me because how can the two expressions both be true?
Then in the first line of the for loop, (( **length=cols** * file_sizes[$k] / max_size )) I don't understand why the variable length is assigned to cols..why were they defined separately to begin with?
While I'm not 100% sure of the syntax, it seems clear enough to answer your questions :
First Question
why are two variables (size name) being created if the name is being assigned to size in the array?
It looks like name holds the file name and size holds the file size. Then the assignment file_sizes["$name"]="$size" stores the file sizes indexed by the file names.
Second Question
(( max_size < size )) && (( max_size=size ))
I believe this line assigns size to max_size if the previous value of max_size is smaller than size. The goal is that at the end max_size would hold the size of the largest file.
Third Question
(( length=cols * file_sizes[$k] / max_size ))
This calculates the length of the line that would be displayed for each file (whose goal is probably to illustrate the relative size of the file compared to the largest file). The length of the line is relative to the size of the file. cols is the length of the line that would be displayed for the largest file (the one whose size is max_size). cols = the lengh of the terminal - the length of the longest file name - 10.
I want to generate a series of files in which the file name of each file shall be increased by 1 (File1.txt, File2.txt, File3.txt, ... FileN.txt) where N = 250
Each file has 2 lines.
AAAXXX (where XXX = 001 to 250 - automatic increased for each file)
BBBYYY (where YYY = 3 digit random number )
Example:
File1.txt:
AAA001
BBB175
File5.txt:
AAA005
BBB067
File102.txt:
AAA102
BBB765
I'm a newbie using Ubuntu Linux 12.04 - but I'm hoping someone can assist.
You can do it as follows:
#!/bin/bash
for i in {1..250}
do
printf "AAA%03d\nBBB%03d" ${i} $(($RANDOM % 1000)) > File${i}.txt
done
Explanation:
for i in {1..250} - bash way of specifying iteration from 1 to 250, increment size of 1.
printf - shell printf command - used to print formatted string
AAA - string literal (means "exactly as written")
%03d - formatted string, this prints a decimal number padded with 3 zero's in front.
\n - newline
BBB - another string literal
%03d - same as before
${i} - this is the value used in the first formatted string (%03d)
$(($RANDOM % 1000)) - $RANDOM is a system variable that provides a random number for you each time you access it. The % 1000 to take the modulo so you get a range betwee 0-999. This is used in the 2nd formatted string (%03d)
> File${i}.txt: output redirection; creates and saves to a file (overwrites if file already exists.
Here's a quick one-liner that might start you off:
for i in {1..250}; do printf "AAA%03d\nBBB%03d" $i $(($RANDOM % 1000)) > "File${i}.txt"; done
Using bash:
for i in {1..250}; do printf "AAA%03d\nBBB%03d\n" "$i" "$((RANDOM%1000))" > "File$i.txt"; done
You can write a bash script for this
#!/bin/bash
for (( i=1; i<=250; i++ ))
do
NUMBER=$[ ( $RANDOM % 999 ) + 100 ]
echo "AAA$i BBB$NUMBER" > File$i.txt
done