How can I change my current Ping timestamps to include milliseconds? Currently it gives me date which only has seconds fine-grained.
ping google.com | while read pong; do echo " $pong #$(date)"; done
If note that the answers should be executable on shell as I am running it on a phone using ADB.
If you're happy with Nanoseconds ... (milliseconds would require some extra maths that would impact on execution time) this would work:
ping google.com | while read pong; do echo " $pong #$(date '+%Y%m%d%H%M%S.%N')"; done
And dissecting the format string we threw at date
%Y = 4-digit year
%m = month
%d = day
%H = hour
%M = minute
%S = seconds
%N = nanoseconds
For details:
man date
Related
I need to subtract the below 2 times using a shell script
var1=2019-11-14-03.00.02.000000
var2=2019-11-14-03.00.50.000000
The output should be 00-00-00-00.00.48.00000
First convert var1 and var2 to date in seconds (since epoch) with:
sec1=$(date --date $var1 +%s)
...
Use bash math operators to calculate the difference
delta=$((sec1 - sec2))
Finally convert it back to a readable format
date --date #1502938801"$delta"
As mentioned in my comment, var1 and var2 are not valid date formats for passing to GNU date with the -d option. Before you can convert the times to seconds from epoch, you must
remove the '-' between the date and time portions of each variable,
isolate the time alone,
remove the milliseconds
replace all '.' in the time with ':'
restore the milliseconds separated from the time with '.'
pass the reformatted string for each to date -d"$dt $tm" +%s with the reformatted date and time space-separated.
Bash provides parameter substitutions to handle each very easily.
After computing the time since epoch for both and taking the difference, you then have to manually compute the difference in years, months, days, hours, minutes, seconds and milliseconds in order to output the difference and you must format the output using printf and the integer conversion specifier using both the appropriate field-width and leading-zero modifiers.
Putting it altogether, (and using a 365-day/year and 30-day/mo approximation) you could do:
#!/bin/bash
var1=2019-11-14-03.00.02.000000
var2=2019-11-14-03.00.50.000000
dt1=${var1%-*} ## isolate date portion of variables
dt2=${var2%-*}
tm1=${var1##*-} ## isolate time portion of variables
tm2=${var2##*-}
ms1=${tm1##*.} ## isolate milliseconds portion of variables
ms2=${tm2##*.}
tm1=${tm1%.*} ## remove milliseconds from time
tm2=${tm2%.*}
tm1=${tm1//./:} ## substitute all . with : in times w/o milliseconds
tm2=${tm2//./:}
tm1=${tm1}.$ms1 ## restore milliseconds
tm2=${tm2}.$ms2
epoch1=$(date -d"$dt1 $tm1" +%s) ## get time since epoch for both
epoch2=$(date -d"$dt2 $tm2" +%s)
epochdiff=$((epoch2-epoch1)) ## get difference in epoch times
## Approximates w/o month or leap year considerations
y=$((epochdiff/(3600*24*365))) ## years difference
rem=$((epochdiff-y)) ## remainder
m=$((rem/(3600*24*30))) ## months difference (based on 30 day mo)
rem=$((rem-m)) ## remainder
d=$((rem/(3600*24))) ## days difference
rem=$((rem-m)) ## remainder
H=$((rem/3600)) ## hours difference
rem=$((rem-H)) ## remainder
M=$((rem/60)) ## minutes difference
S=$((rem-M)) ## secnds difference
ms=$((ms2-ms1)) ## millisecond difference
## format output
printf "%04d-%02d-%02d-%02d:%02d:%02d.%04d\n" $y $m $d $H $M $S $ms
(note: you can further fine-tune the month and year/leap-year calculations -- that is left to you)
Example Use/Output
$ bash ~/scr/dtm/fulltimediff.sh
0000-00-00-00:00:48.0000
Look things over and let me know if you have further questions.
I have a backup script which is written in bash/shell scripting language. I calculate the total runtime/execution time by doing this:
#!/bin/bash
# start time
res1=$(date +%s.%N)
### do some work here
# end time & calculate
res2=$(date +%s.%N)
dt=$(echo "$res2 - $res1" | bc)
dd=$(echo "$dt/86400" | bc)
dt2=$(echo "$dt-86400*$dd" | bc)
dh=$(echo "$dt2/3600" | bc)
dt3=$(echo "$dt2-3600*$dh" | bc)
dm=$(echo "$dt3/60" | bc)
ds=$(echo "$dt3-60*$dm" | bc)
# finished
printf " >>> Process Completed - Total Runtime (d:h:m:s) : %d:%02d:%02d:%02.4f\n" $dd $dh $dm $ds
echo " "
exit 0
This outputs something like this:
How do you format the result, so it looks something like this:
0 Days, 0 Hours, 0 Minutes and 0.0968 Seconds
If it can intelligently show only values > 0, like these examples - it would be abonus:
7 Minutes and 5.126 Seconds
or
2 hours, 4 Minutes and 1.106 Seconds
or
7.215 Seconds etc...
You can use your last printf like this:
printf " >>> Process Completed - Total Runtime (d:h:m:s) : %d Days, %02d Hours, %02d Minutes, %02.4f Seconds\n" $dd $dh $dm $ds
However I would suggest you to use awk and do all calculations and formatting in awk itself so that you can avoid many invocations of bc.
Suggested awk script:
awk -v res1="$res1" -v res2="$res2" 'BEGIN {dt=res2-res1; dd=dt/86400; dt2=dt-86400*dd;
dh=dt2/3600; dt3=dt2-3600*dh; dm=dt3/60; ds=dt3-60*dm;
printf " >>> Process Completed - Total Runtime (d:h:m:s) : %d Days, %02d Hours, %02d Minutes, %02.4f Seconds\n",
dt/86400, dd, dh, dm, ds}'
In a Linux script: I have a file that has two time entries for each message within the file. A 'received time' and a 'source time'. there are hundreds of messages within the file.
I want to calculate the elapsed time between the two times.
2014-07-16T18:40:48Z (received time)
2014-07-16T18:38:27Z (source time)
The source time is 3 lines after the received time, not that it matters.
info on the input data:
The input has a lines are as follows:
TimeStamp: 2014-07-16T18:40:48Z
2 lines later: a bunch of messages in one line and within each line, multiple times is:
sourceTimeStamp="2014-07-16T18:38:27Z"
If you have GNU's date (not busybox's), you can give difference in seconds with:
#!/bin/bash
A=$(date -d '2014-07-16T18:40:48Z' '+%s')
B=$(date -d '2014-07-16T18:38:27Z' '+%s')
echo "$(( A - B )) seconds"
For busybox's date and ash (modern probably / BusyBox v1.21.0):
#!/bin/ash
A=$(busybox date -d '2014-07-16 18:40:48' '+%s')
B=$(busybox date -d '2014-07-16 18:38:27' '+%s')
echo "$(( A - B )) seconds"
you should be able to use date like this (e.g.)
date +%s --date="2014-07-16T18:40:48Z"
to convert both timestamps into a unix timestamp. Getting the time difference between them is then reduced to a simple subtraction.
Does this help?
I would use awk. The following script searches for the lines of interest, converts the time value into a UNIX timestamp and saves them in the start, end variables. At the end of the script the difference will get calculated and printed:
timediff.awk:
/received time/ {
"date -d "$1" +%s" | getline end
}
/source time/ {
"date -d "$1" +%s" | getline start
exit
}
END {
printf "%s seconds in between", end - start
}
Execute it like this:
awk -f timediff.awk log.file
Output:
141 seconds in between
This question already has answers here:
Filter log file entries based on date range
(5 answers)
Closed 6 years ago.
I have agent.log file. This file is updating as regular interval.
Entries are as follows 2014-01-07 03:43:35,223 INFO ...some data
I want to extract data of last 3 minutes, Is there any way so that I will get this data using bash script?
Try the solution below:
awk \
-v start="$(date +"%F %R" --date=#$(expr `date +%s` - 180))" \
-v end="$(date "+%F %R")" \
'$0 ~ start, $0 ~ end' \
agent.log
In the start variable there is the time stamp 3 minutes (180 seconds) before the current time.
In the end there is the current time.
$0 ~ start, $0 ~ end selects the lines between start and end
date +"%F %R" gives you the current time down to the minute.
grep '^'"$(date +"%F %R")" agent.log will select the last minute from the file
Now for the previous two minutes it's more tricky... I have developed some scripts that can do complete time manipulation in relative or absolute, and it may be simpler than fiddling with date...
2 minutes ago in the right format: date --date="#$(($(date +"%s") - 2*60))" +"%F %R"
Merge all 3:
NOW=$(date +"%F %R")
M1=$(date --date="#$(($(date +"%s") - 1*60))" +"%F %R")
M2=$(date --date="#$(($(date +"%s") - 2*60))" +"%F %R")
grep '^'"$NOW\|$M1\|$M2" agent.log
my answer considers the followings:
using bash and UNIX/Linux commands
the last log line is the start time not the actual server time
there is no expectation about the lines' date (minutes, days, years, etc.)
the whole script should be expandable to the inverse, or a specified from-to interval
#!/bin/bash
# this script expects descending dates in a log file (reverse as real life examples)!!!
FILE=$1
INTV=180 # sec
while read LINE
do
if [ -z $LAST_LOG_LINE ]
then
# interval stat line
LAST_LOG_LINE=$(date --date="$( echo "$LINE" | sed -e 's/INFO.*//')" +%s)
# mod
#continue
fi
ACT_LOG_LINE=$(date --date="$( echo "$LINE" | sed -e 's/INFO.*//')" +%s)
# print line if not greater than $INTV (180s)
# else break the reading and exit
if [ $(($LAST_LOG_LINE-$ACT_LOG_LINE)) -gt $INTV ]
then
break
fi
# actual print
echo "$LINE"
done < $FILE
Testing:
2014-01-07 03:43:35,223 INFO ...some data
2014-01-07 03:42:35,223 INFO ...some data
2014-01-07 03:41:35,223 INFO ...some data
2014-01-07 03:40:35,223 INFO ...some data
2014-01-07 02:43:35,223 INFO ...some data
2014-01-07 01:43:35,223 INFO ...some data
2014-01-06 03:43:35,223 INFO ...some data
$ /tmp/stack.sh /tmp/log
2014-01-07 03:42:35,223 INFO ...some data
2014-01-07 03:41:35,223 INFO ...some data
2014-01-07 03:40:35,223 INFO ...some data
$
I think you may be somewhat better off using Python in this case. Even if this script doesn't find a date exactly 3 minutes ago, it will still get any log entries in between the time the script was called and 3 minutes ago. This is both concise and more robust than some of the previous solutions offered.
#!/usr/bin/env python
from datetime import datetime, timedelta
with open('agent.log') as f:
for line in f:
logdate = datetime.strptime(line.split(',')[0], '%Y-%m-%d %H:%M:%S')
if logdate >= datetime.now() - timedelta(minutes=3):
print(line)
A Ruby solution (tested on ruby 1.9.3)
You can pass days, hours, minutes or seconds as a parameter and it will search for the expression and on the file specified (or directory, in which case it will append '/*' to the name):
In your case just call the script like so: $0 -m 3 "expression" log_file
Note: Also if you know the location of 'ruby' change the shebang (first line of the script),
for security reasons.
#! /usr/bin/env ruby
require 'date'
require 'pathname'
if ARGV.length != 4
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
begin
total_amount = Integer ARGV[1]
rescue ArgumentError
$stderr.print "error: parameter 'time' must be an Integer\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
if ARGV[0] == "-m"
gap = Rational(60, 86400)
time_str = "%Y-%m-%d %H:%M"
elsif ARGV[0] == "-s"
gap = Rational(1, 86400)
time_str = "%Y-%m-%d %H:%M:%S"
elsif ARGV[0] == "-h"
gap = Rational(3600, 86400)
time_str = "%Y-%m-%d %H"
elsif ARGV[0] == "-d"
time_str = "%Y-%m-%d"
gap = 1
else
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
pn = Pathname.new(ARGV[3])
if pn.exist?
log = (pn.directory?) ? ARGV[3] + "/*" : ARGV[3]
else
$stderr.print "error: file '" << ARGV[3] << "' does not exist\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
search_str = ARGV[2]
now = DateTime.now
total_amount.times do
now -= gap
system "cat " << log << " | grep '" << now.strftime(time_str) << ".*" << search_str << "'"
end
I can do that in php with the following code:
$dt1 = '2011-11-11 11:11:11';
$t1 = strtotime($dt1);
$dt2 = date('Y-m-d H:00:00');
$t2 = strtotime($dt2);
$tDiff = $t2 - $t1;
$hDiff = round($tDiff/3600);
$hDiff will give me the result in hours.
How do I implement the above in bash shell?
You could use date command to achieve this. man date will provide you with more details. A bash script could be something on these lines (seems to work fine on Ubuntu 10.04 bash 4.1.5):
#!/bin/bash
# Date 1
dt1="2011-11-11 11:11:11"
# Compute the seconds since epoch for date 1
t1=$(date --date="$dt1" +%s)
# Date 2 : Current date
dt2=$(date +%Y-%m-%d\ %H:%M:%S)
# Compute the seconds since epoch for date 2
t2=$(date --date="$dt2" +%s)
# Compute the difference in dates in seconds
let "tDiff=$t2-$t1"
# Compute the approximate hour difference
let "hDiff=$tDiff/3600"
echo "Approx hour diff b/w $dt1 & $dt2 = $hDiff"
Hope this helps!