linux shell date question - linux

shell>/bin/date -d "091029 20:18:02" +%s
1256827682
Similarly I created shell script:
#My.sh
myDate=`date +'%y%m%d_%H%M%S'`
myDate1=`echo $myDate | sed 's/_/ /g'`
myDate2=`echo $myDate1 | sed 's/\([0-9][0-9][0-9][0-9][0-9][0-9]\) \([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)/\/bin\/date -d "\1 \2:\3:\4" +%s/'`
print $myDate2
`$myDate2`
But it doesn't execute above command. WHy?

The backticks in the last line executes the command and then executes the output of that command, which is probably not what you are trying to do -- when I run this I get the output -bash: 1256829614: command not found. Remove the backticks on the last line so that it just says
$myDate2

In addition to the solutions above, I would like to offer assistant to get to myDate2:
$ myDate=$(date +'/bin/date -d "%y%m%d %H:%M:%S" +%%s')
$ echo $myDate
/bin/date -d "091029 09:06:29" +%s

The notation:
`$myDate2`
expands $myDate2 and executes the command (and I'll come back to why there are problems with that), and then captures the output - and tries to run the output.
What you are looking for is eval:
eval $myDate2
Handling quotes is tricky - and eval is often a part of the answer. When you build up a string with internal quotes, you need to use eval to get the shell to put the quotes back together.
One very useful tool that I have is a program called al - for argument list.
#include <stdio.h>
int main(int argc, char **argv)
{
while (*++argv != 0)
puts(*argv);
return(0);
}
It prints each separate argument on a separate line. It was almost the first thing I did when looking at what you are up to.
myDate=`date +'%y%m%d_%H%M%S'`
myDate1=`echo $myDate | sed 's/_/ /g'`
myDate2=`echo $myDate1 | sed 's/\([0-9][0-9][0-9][0-9][0-9][0-9]\) \([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)/\/bin\/date -d "\1 \2:\3:\4" +%s/'`
print $myDate2
#`$myDate2`
al $myDate2
eval al $myDate2
eval $myDate2
The trace output from this was:
+ date +%y%m%d_%H%M%S
+ myDate=091029_082546
+ sed 's/_/ /g'
+ echo 091029_082546
+ myDate1='091029 082546'
+ sed 's/\([0-9][0-9][0-9][0-9][0-9][0-9]\) \([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)/\/bin\/date -d "\1 \2:\3:\4" +%s/'
+ echo 091029 082546
+ myDate2='/bin/date -d "091029 08:25:46" +%s'
+ print /bin/date -d '"091029' '08:25:46"' +%s
/bin/date -d "091029 08:25:46" +%s
+ al /bin/date -d '"091029' '08:25:46"' +%s
/bin/date
-d
"091029
08:25:46"
+%s
+ eval al /bin/date -d '"091029' '08:25:46"' +%s
+ al /bin/date -d '091029 08:25:46' +%s
/bin/date
-d
091029 08:25:46
+%s
+ eval /bin/date -d '"091029' '08:25:46"' +%s
+ /bin/date -d '091029 08:25:46' +%s
usage: date [-jnu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Note how when I ran 'al $myDate2' the date string was split into two arguments, but when I ran 'eval al $myDate2', the string was one argument - as required. I was testing on MacOS X, where the data command does not accept the date string format you supplied - that is a whole separate problem. But getting the string healed requires 'eval'.
I didn't even address the issue of what the shell script was trying to do.
I gather from Hai Vu's answer that we're really after the current time in seconds since the epoch; I can sort of see how that might be.
On MacOS X, that is obtained trivially:
date +'%s'
(where the single quotes really aren't needed). The MacOS X manual page also includes the example:
date -j -f "%a %b %d %T %Z %Y" "`date`" "+%s"
This seems a bit convoluted - but would allow you to find the seconds since the epoch for any date previously given by the date command - or a date that will be given at some time in the future (by replacing the back-quoted date with the previous string).
An æon or so ago, I wrote programs 'systime' to print the current time as the number of seconds past the epoch, and also a program 'timestamp' to convert such values back into formatted dates - because none of the standard versions of the 'date' command supported such mechanisms back then (before the C standard was standard, and therefore before strftime() was widely available). I also have a program 'strptime' for converting a formatted date into a timestamp. Ah well - nice to know that the standard programs can now do it.
However, I note that the MacOS 'date' command is a superset of the POSIX standard version; I suspect that the Linux (GNU) 'date' command is a different superset of the POSIX standard, and so on for each platform.

Related

Count occurrences of string in logfile in last 5 minutes in bash

I have log file containing logs like this:
[Oct 13 09:28:15] WARNING.... Today is good day...
[Oct 13 09:28:15] Info... Tommorow will be...
[Oct 13 09:28:15] WARNING.... Yesterday was...
I need shell command to count occurrences of certain string in last 5 minutes.
I have tried this:
$(awk -v d1="$(date --date="-5 min" "+%b %_d %H:%M:%S")" -v d2="$(date "+%b %_d %H:%M:%S")" '$0 > d1 && $0 < d2 || $0 ~ d2' "$1" |
grep -ci "$2")
and calling script like this: sh ${script} /var/log/message "day" but it does not work
Your immediate problem is that you are comparing dates in random string format. To Awk (and your computer generally) a string which starts with "Dec" is "less than" a string which starts with "Oct" (this is what date +%b produces). Generally, you would want both your log files and your programs to use dates in some standard computer-readable format, usually ISO 8601.
Unfortunately, though, sometimes you can't control that, and need to adapt your code accordingly. The solution then is to normalize the dates before comparing them.
awk -v d1=$(date -d "-5 min" +"%F-%T") -v d2=$(date +"%F-%T") '
BEGIN { split("Jan:Feb:Mar:Apr:May:Jun:Jul:Aug:Sep:Oct:Nov:Dec", m, ":")
for (i=1; i<=12; ++i) mon["[" m[i]] = i }
{ timestamp = substr(d1, 1, 5) mon[$1] "-" $2 "-" $3 }
timestamp > d1 && timestamp <= d2' "$1" | grep -ci "$2
This will not work across New Year boundaries, but should hopefully at least help get you started in the right direction. (I suppose you could check if the year in d2 is different, and then check if the month in $1 is January, and then add 1 to the year from d1 in timestamp; but I leave this as an exercise for the desperate. This still won't work across longer periods of time, but the OP calls for a maximum period of 5 minutes, so the log can't straddle multiple years. Or if it does, you have a more fundamental problem.)
Perhaps note as well that date -d is a GNU extension which is not portable to POSIX (so this will not work e.g. on MacOS without modifications).
(Also, for production use, I would refactor the grep -ci into the Awk script; see also useless use of grep.)
Finally, the command substitution $(...) around your entire command line is wrong; this would instruct your shell to use the output from Awk and run it as a command.

Rounding of the millisecond part in Linux datetime

So I have the date format like this : 2019-10-19 23:55:42.797 and I want the millisecond part to be round of into the second so the output should look something like this: 2019-10-19 23:55:43
I have tried
date -d "2019-10-19 23:55:42.797" "+%Y-%m-%d %H:%M:%S"
but it's giving me output like 2019-10-19 23:55:42
How should I do this in Linux bash shell?
This can be done in a single awk like this:
s='2020-12-31 23:59:59.501'
awk -F. 'gsub(/[-:]/, " ", $1) {
dt = mktime($1)
if ($2 >= 500) dt++
print strftime("%F %X", dt)
}' <<< "$s"
2021-01-01 00:00:00
The behaviour you observe is as expected. The format specifiers represent the actual quantity without rounding. Imagine you would include rounding and you have the time "2019-10-19 23:55:42.797" but you are not interested in seconds and set the format to "%F %H:%M", do you want to see "2019-10-19 23:55" or "2019-10-19 23:56", and even further. Imagine you have the time "2020-12-31 23:59:59.501" with format "%F %T", do you want it to show "2021-01-01 00:00:00" or "2020-12-31 23:59:59". While we all want 2020 to finish as soon as possible, the latter still remains the correct time representation.
Rounding in times is only relevant when you look at time differences and not at absolute times. Hence, I strongly recommend not to implement any rounding and just use the output that date provides you.
However, if, for whatever reason you actually need to round the time to the nearest second, then you can do this:
epoch_ms=$(date -d "2019-10-19 23:55:42.797" "+%s%3N")
epoch=$(( (epoch_ms + 500)/1000 ))
date -d "#$epoch" "%F %T"
Or in a single line:
date -d "#$(( ( $(date -d "2019-10-19 23:55:42.797" "+%s%3N") + 500 )/1000 ))" "+%F %T"

Read datetime from a file and add one second

I'm reading a timestamp from a file and I'd like to copy the value to the START_DATE variable, and add 1 second to that value.
export START_DATE=`cat ${WINDOW_END_FILE}`
Timestamp format
2019-04-03-23.59.59
In the end, I'd like the date to be
2019-04-04-00.00.00
Convert the date to epoch time, then add 1:
fmt='%Y-%m-%d-%H.%M.%S'
date -j -f %s $(( $(date -j -f "$fmt" 2019-04-03-23.59.59 +%s) + 1 )) +"$fmt"
Ok here goes, this isn't pretty, but fun:
If I understood, let's just say START_DATE=2019-04-03-23.59.59
# Pull Date section
date=$(echo $START_DATE | cut -d - -f 1-3)
# Pull Time section
time=$(echo $START_DATE | cut -d - -f 4 | sed 's/\./:/g')
# Use bash builtin date function for conversion
date -d "$cal $time + 1 second" +"%Y-%m-%d-%H-%M-%S"
Output:
2019-04-04-00-00-00
Using GNU awk:
$ gawk '{
gsub(/[^0-9]/," ") # change the format mktime friendly
print strftime("%F-%H.%M.%S",mktime($0)+1) # to epoch, add one, reformat back
}' file # a timestamp from a file
2019-04-04-00.00.00 # output this time
mktime turns datespec (YYYY MM DD HH MM SS [DST]) to number of seconds since the system epoch. strftime formats the timestamp to given format.

split a string variable through shell script

i have a string containing date and time as timestamp= 12-12-2012 16:45:00
I need to reformat it into timestamp= 16:45:00 12-12-2012
How to achieve this in shell script?
Note Please : variable's value is 12-12-2012 16:45:00 and timestamp is the name of variable
#!usr/bin/expect
set timestamp "16:45:00 12-12-2012"
Now what should i do so value of timestamp will become 12-12-2012 16:45:00
script extention is .tcl example test.tcl
You could use variable patterned removal. ## means "greedily remove everything that matches the pattern, starting from the left". %% means the same from the right:
tm=${timestamp##* }
dt=${timestamp%% *}
result="$tm $dt"
or you could use cut to do the same, giving a field delimiter:
tm=$(echo $timestamp | cut -f2 -d' ')
dt=$(echo $timestamp | cut -f1 -d' ')
result="$tm $dt"
or you could use sed to swap them with a regex (see other post).
or if you are pulling the date from the date command, you could ask it to format it for you:
result=$(date +'%r %F')
and for that matter, you might have a version of date that will parse your date and then let you express it however you want:
result=$(date -d '12/12/2012 4:45 pm' +'%r %F')
admittedely, this last one is picky about date input...see "info date" for information on accepted inputs.
If you want to use regex, I like Perl's...they are cleaner to write:
echo $timestamp | perl -p -e 's/^(\S+)\s+(\S+)/$2 $1/'
where \S matches non-space characters, + means "one or more", and \s matches spaces. The parens do captures of the parts matched.
EDIT:
Sorry, didn't realize that the "timestamp=" was part of the actual data. All of the above example work if you first strip that bit out:
var='timestamp=2012-12-12 16:45:11'
timestamp=${var#timestamp=}
... then as above ...
Using sed:
sed 's/\([0-9]*-[0-9]*-[0-9]*\)\([ \t]*\)\(.*\)/\3\2\1/' input
this command works on lines containing the pattern number-number-number whitespace antyhing. It simply swaps the number-number-number part \([0-9]*-[0-9]*-[0-9]*\) with the anything part \(.*\), also keeping the original whitespaces \([ \t]*\). So the replace part of sed is \3\2\1, which means the third part, white spaces, and the first part.
Same logic with tcl:
set timestamp "12-12-2012 16:45:00"
set s [regsub {([0-9]*-[0-9]*-[0-9]*)([ \t]*)(.*)} $timestamp \\3\\2\\1]
puts $s
awk solution here:
string="timestamp= 12-12-2012 16:45:00"
awk '{print $1, $3, $2}' <<< "$string"
In bash (and similar shells):
$ timestamp="12-12-2012 16:45:00"
$ read -a tsarr <<< "$timestamp"
$ echo "${tsarr[1]} ${tsarr[0]}"
16:45:00 12-12-2012

Is it possible to use the matched string in sed as an input to shell date command?

I have a file with records having timestamp fields that include GMT offset. I want to use the sed command to replace the value on the record to a regular timestamp (without GMT offset).
For example:
`$date -d '2012/11/01 00:50:22 -0800' '+%Y-%m-%d %H:%M:%S'`
returns this value which is what I am looking to do:
2012-11-01 01:50:22
Except I want to perform that operation on every line of this file and apply the date command to the timestamp value. Here is a sample record:
"SB","6GV96644X48128125","","","","T0006",2012/10/03 13:08:43 -0700,"NJ"
Here is my code:
head -1 myfile | sed 's/,[0-9: /\-]\{25\},/,'"`date -d \1 '+%Y-%m-%d %H:%M:%S'`"',/
which doesn't work: it just ignores \1 and replaces the matched pattern with today's date:
"SB","6GV96644X48128125","","","","T0006",2012-11-14 01:00:00,"NJ"
I hoped that \1 would result in the matched patterns be passed to the date function and return a regular timestamp value (as in the example I provided above showing how the date functions applies the GMT offset and returns a regular stimestamp string) and would replace the old value on the record.
I would use awk instead. For example:
awk '{cmd="date -d \""$7"\" \"+%Y-%m-%d %H:%M:%S\"";
cmd | getline k; $7=k; print}' FS=, OFS=, myFile
This will replace the 7th field with the results of running the date command on the original contents of the 7th field.
In sed:
head -1 datefile |
sed '
# handle % in input correctly
s/%/%%/g
# execute date(1) command
s/\(.*,\)\([0-9: /\-]\{25\}\)\(,.*\)/'"date -d '\2' '+\1%Y-%m-%d %H:%M:%S\3'"'/e'
'
This might work for you (GNU sed):
sed -r 's/^(([^,]*,){6})([^,]*)(.*)/printf "%s%s%s" '\''\1'\'' $(date -d '\''\3'\'' '\''+%Y-%m-%d %H:%M:%S'\'') '\''\4'\''/e;q' file
use
head -1 datefile | sed -e 's?\(..\)/\(..\)/\(.... ..:..:..\)?'"date -d '\2/\1/\3' '+%s'"'?e'

Resources