Get/extract the data from log file of last 3 minutes? [duplicate] - linux

Question

This question already has answers here:
Filter log file entries based on date range
(5 answers)
Closed 6 years ago.
I have agent.log file. This file is updating as regular interval.
Entries are as follows 2014-01-07 03:43:35,223 INFO ...some data
I want to extract data of last 3 minutes, Is there any way so that I will get this data using bash script?

Answer 1

Try the solution below:
awk \
-v start="$(date +"%F %R" --date=#$(expr `date +%s` - 180))" \
-v end="$(date "+%F %R")" \
'$0 ~ start, $0 ~ end' \
agent.log
In the start variable there is the time stamp 3 minutes (180 seconds) before the current time.
In the end there is the current time.
$0 ~ start, $0 ~ end selects the lines between start and end

Answer 2

date +"%F %R" gives you the current time down to the minute.
grep '^'"$(date +"%F %R")" agent.log will select the last minute from the file
Now for the previous two minutes it's more tricky... I have developed some scripts that can do complete time manipulation in relative or absolute, and it may be simpler than fiddling with date...
2 minutes ago in the right format: date --date="#$(($(date +"%s") - 2*60))" +"%F %R"
Merge all 3:
NOW=$(date +"%F %R")
M1=$(date --date="#$(($(date +"%s") - 1*60))" +"%F %R")
M2=$(date --date="#$(($(date +"%s") - 2*60))" +"%F %R")
grep '^'"$NOW\|$M1\|$M2" agent.log

Answer 3

my answer considers the followings:
using bash and UNIX/Linux commands
the last log line is the start time not the actual server time
there is no expectation about the lines' date (minutes, days, years, etc.)
the whole script should be expandable to the inverse, or a specified from-to interval
#!/bin/bash
# this script expects descending dates in a log file (reverse as real life examples)!!!
FILE=$1
INTV=180 # sec
while read LINE
do
if [ -z $LAST_LOG_LINE ]
then
# interval stat line
LAST_LOG_LINE=$(date --date="$( echo "$LINE" | sed -e 's/INFO.*//')" +%s)
# mod
#continue
fi
ACT_LOG_LINE=$(date --date="$( echo "$LINE" | sed -e 's/INFO.*//')" +%s)
# print line if not greater than $INTV (180s)
# else break the reading and exit
if [ $(($LAST_LOG_LINE-$ACT_LOG_LINE)) -gt $INTV ]
then
break
fi
# actual print
echo "$LINE"
done < $FILE
Testing:
2014-01-07 03:43:35,223 INFO ...some data
2014-01-07 03:42:35,223 INFO ...some data
2014-01-07 03:41:35,223 INFO ...some data
2014-01-07 03:40:35,223 INFO ...some data
2014-01-07 02:43:35,223 INFO ...some data
2014-01-07 01:43:35,223 INFO ...some data
2014-01-06 03:43:35,223 INFO ...some data
$ /tmp/stack.sh /tmp/log
2014-01-07 03:42:35,223 INFO ...some data
2014-01-07 03:41:35,223 INFO ...some data
2014-01-07 03:40:35,223 INFO ...some data
$

Answer 4

I think you may be somewhat better off using Python in this case. Even if this script doesn't find a date exactly 3 minutes ago, it will still get any log entries in between the time the script was called and 3 minutes ago. This is both concise and more robust than some of the previous solutions offered.
#!/usr/bin/env python
from datetime import datetime, timedelta
with open('agent.log') as f:
for line in f:
logdate = datetime.strptime(line.split(',')[0], '%Y-%m-%d %H:%M:%S')
if logdate >= datetime.now() - timedelta(minutes=3):
print(line)

Answer 5

A Ruby solution (tested on ruby 1.9.3)
You can pass days, hours, minutes or seconds as a parameter and it will search for the expression and on the file specified (or directory, in which case it will append '/*' to the name):
In your case just call the script like so: $0 -m 3 "expression" log_file
Note: Also if you know the location of 'ruby' change the shebang (first line of the script),
for security reasons.
#! /usr/bin/env ruby
require 'date'
require 'pathname'
if ARGV.length != 4
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
begin
total_amount = Integer ARGV[1]
rescue ArgumentError
$stderr.print "error: parameter 'time' must be an Integer\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
if ARGV[0] == "-m"
gap = Rational(60, 86400)
time_str = "%Y-%m-%d %H:%M"
elsif ARGV[0] == "-s"
gap = Rational(1, 86400)
time_str = "%Y-%m-%d %H:%M:%S"
elsif ARGV[0] == "-h"
gap = Rational(3600, 86400)
time_str = "%Y-%m-%d %H"
elsif ARGV[0] == "-d"
time_str = "%Y-%m-%d"
gap = 1
else
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
pn = Pathname.new(ARGV[3])
if pn.exist?
log = (pn.directory?) ? ARGV[3] + "/*" : ARGV[3]
else
$stderr.print "error: file '" << ARGV[3] << "' does not exist\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
search_str = ARGV[2]
now = DateTime.now
total_amount.times do
now -= gap
system "cat " << log << " | grep '" << now.strftime(time_str) << ".*" << search_str << "'"
end