How to grep the logs between two date range in Unix - linux

I have a log file abc.log in which each line is a date in date +%m%d%y format:
061019:12
062219:34
062319:56
062719:78
I want to see the all the logs between this date range (7 days before date to current date) i.e (from 062019 to 062719 in this case). The result should be:
062219:34
062319:56
062719:78
I have tried few things from my side to achieve:
awk '/062019/,/062719' abc.log
This gives me correct answer, but if i don't want to hard-code the date value and try achieving the same it does not give the correct value.
awk '/date --date "7 days ago" +%m%d%y/,/date +%m%d%y' abc.log
Note:
date --date "7 days ago" +%m%d%y → 062019 (7 days back date)
date +%m%d%y → 062719 (Current date)
Any suggestions how this can be achieved?

Your middle-endian date format is unfortunate for sorting and comparison purposes. Y-m-d would have been much easier.
Your approach using , ranges in awk requires exactly one log entry per day (and that the log entries are sorted chronologically).
I would use perl, e.g. something like:
perl -MPOSIX=strftime -ne '
BEGIN { ($start, $end) = map strftime("%y%m%d", localtime $_), time - 60 * 60 * 24 * 7, time }
print if /^(\d\d)(\d\d)(\d\d):/ && "$3$1$2" ge $start && "$3$1$2" le $end' abc.log
Use strftime "%y%m%d" to get the ends of the date range in big-endian format (which allows for lexicographical comparisons).
Use a regex to extract day/month/year from each line into separate variables.
Compare the rearranged date fields of the current line to the ends of the range to determine whether to print the line.

To get around the issue of looking for dates that may not be there, you could generate a pattern that matches any of the dates (since there are only 8 of them it doesn’t get too big, if you want to look for the last year it might not work as well):
for d in 7 6 5 4 3 2 1 0
do
pattern="${pattern:+${pattern}\\|}$(date --date "${d} days ago" +%m%d%y)"
done
grep "^\\(${pattern}\\)" abc.log

Related

Extract 1 month older data using Shell scripts [duplicate]

This question already has answers here:
Filter log file entries based on date range
(5 answers)
Closed 5 years ago.
I need to find the entries from a huge log file which are 1 month older than the current date using shell scripts. The log entry date is in the 3rd field of each line.
This is my sample log file:
0114374510,OK,10/23/2017 9:22:50 AM,0016692200,OK
0112364510,OK,10/23/2017 9:22:50 AM,0016692200,OK
0112364510,TX,10/21/2017 9:10:00 AM,0016692200,OK
0115364510,TX,10/21/2017 10:52:00 AM,0016692200,OK
0112368979,OK,7/29/2016 10:25:00 AM,0000718374,OK
0113368979,OK,7/29/2016 12:50:00 PM,0000718374,OK
0112368979,TX,9/16/2015 10:57:00 AM,0033545820,OK
Thanks!
Here is a solution, using grep to match certain lines in your file and date to manipulate dates.
Linux version
grep -E "$(date --date='-1 month' '+%m')/[0-9]+/$(date --date='-1 month' +'%Y') <your_file>"
date --date='-1 month' '+%m' - Subtract 1 month to the current date and print the month
[0-9]+ - Regex matching any number (any day in our case)
$(date --date='-1 month' +'%Y') - Subtract 1 month to the current date and print the year
macOS version
grep -E "$(date -v -1m '+%m')/[0-9]+/$(date -v -1m +'%Y') <your_file>"
date -v -1m '+%m' - Subtract 1 month to the current date and print the month
[0-9]+ - Regex matching any number (any day in our case)
date -v -1m +'%Y' - Subtract 1 month to the current date and print the year

How to get Date Month values in linux date command as integers to work on

I want to convert date and month as integers.
for example.
if the current date as per the command "Date +%m-%d-%y" output, is this
09-11-17
Then I am storing
cur_day=`date +%d`
cur_month=`date +%m`
the $cur_day will give me 11 and $cur_month will give me 09.
I want to do some operations on the month as 09. like i want to print all the numbers up to 09.
like this 01,02,03,04,05,06,07,08,09
Same way I want to display all the numbers up to cur_day
like 01,02,03,04,05,06,07,08,09,10,11
Please tell me how can i do it.
Thanks in Advance.
For months:
$ printf ',%02d' $(seq 1 $(date +%m)) | sed 's/,/like this /; s/$/\n/'
like this 01,02,03,04,05,06,07,08,09
For days:
$ printf ',%02d' $(seq 1 $(date +%d)) | sed 's/,/like /; s/$/\n/'
like 01,02,03,04,05,06,07,08,09,10,11
printf will print according to a format. In this case, the format ,%02d formats the numbers with commas and leading zeros.
The sed command puts the string you want at the beginning of the line and adds a newline at the end.

How to get logs of last hour in linux using awk command

I have a logs file named source.log having time format like :-
Fri, 09 Dec 2016 05:03:29 GMT 127.0.0.1
and i am using script to get logs from a logs file for last 1 hour.
Script:-
awk -vDate=`date -d'now-1 hour' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' source.log > target.log
But this script gives the result same as like the source file.
There is something wrong in time format matching, due to which it is not giving last hour records.
I know I'm late to help the OP, but maybe this answer can help anyone else in this situation.
First it's necessary to compare the whole date and not only the time part, because times near midnight.
Note that awk can only compare strings and numbers. Some awk implementations have the mktime() function that converts a specifically formatted string into UNIX timestamp, in order to make datetime comparisons, but it doesn't support any datetime format, so we can't use it.
The best way would be changing (if possible) the datetime format of the log entries, using 'YYMMDDhhmmss' datetime format or ISO format. In this way, comparing two datetimes is simple as compare strings or numbers.
But let's assume that we can't change log entries date format, so we'll need to convert ourselves inside awk:
awk -vDate="`date -d'now-1 hour' +'%Y%m%d%H%M%S'`" '
BEGIN{
for(i=0; i<12; i++)
MON[substr("JanFebMarAprMayJunJulAugSepOctNovDec", i*3+1, 3)] = sprintf("%02d", i+1);
}
toDate() > Date
function toDate(){
time = $5; gsub(/:/, "", time);
return $4 MON[$3] $2 time;
}' source.log
Explanation
-vDate=... sets the Date awk variable with the initial datetime (one hour ago).
BEGIN section creates an array indexed by the month abbreviation (it's especific to english)
toDate() function converts the line's fields into a string with the same format as Date variable (YYYMMDDhhmmss).
Finally when the condition toDate() > Date is true, awk prints the current line (log entry).

Extract month and day from linux "date" command

I would like to modify the following statement to extract the Month, as well as day:
testdate=$(date -d "2 days" +%d| sed 's/0([1-9])/ \1/')
right now it only extracts the day...
I'm currently googling how sed works. looks like a regular expression of sorts that's being used but... I thought I'd check with the forum for certain.
Thanks.
EDIT 1
now I have:
testdate=$(date -d "2 days" "+%b %_d %Y")
Which is great... except when the leading zeros in the day are stripped out. then i end up with two spaces between the Month and day
for example, for April 29, 2014, I get:
Apr 29 2014
which is great. But for May 01, 2014 I get:
May 1, 2014
where there's a double space between "May" and "1". I guess i can do it in two steps... by doing the date command, followed by sed to find and replace double spaces.
Just wondering if there's a way to do it all in one command.
Thanks.
date accepts a FORMAT arg which should be used here. sed is not necessary:
date +%d.%m.
Outputs:
03.04.
in European time format. If you want the month's name printed use
date +'%d. %B'
Output:
03. April
To read the month and day into shell variables (assuming bash/ksh/zsh)
read month day < <(date -d "2 days" "+%m %d")
If you're planning to do arithmetic on these values, be of numbers that would be treated as octal but are invalid octal numbers 08 and 09. If you want to strip off the leading zero, use "+%_m %_d"
Using read, the shell takes care of the excess whitespace for you:
read mon day year < <(date -d "2 days" "+%b %_d %Y")
testdate="$mon $day $year"
If you don't want to use the temp vars:
testdate="Apr 5 2014" # $(date -d "2 days" "+%b %_d %Y")
testdate=${testdate// / } # globally replace 2 spaces with 1 space
echo "$testdate"
For date format %d would print only the day. It's unlikely to extract month unless you specify that in the format.
Specify the format for obtaining the month too.
%b locale's abbreviated month name (e.g., Jan)
%B locale's full month name (e.g., January)
%m month (01..12)
For example:
$ date -d "2 days" +%d/%b
05/Apr
$ date -d "2 days" +%d/%B
05/April
$ date -d "2 days" +%d/%m
05/04
Seems there is a simpler way to extract [the] month and day from [the] linux “date” command, then all the previous answers.
From $ date --help:
By default, date pads numeric fields with zeroes.
The following optional flags may follow `%':
- (hyphen) do not pad the field
Which gives us the below, without the use of sed or read to remove any kind of padding:
testdate=$(date -d "3 days" "+%b %-d %Y") (days adjusted to show single-digit day number, from today's date)
And outputs the below:
$ echo $testdate
Sep 1 2014

change the call from a bash script in a month to a week

I have 2 script in bash, and i have some files:
transaction-2012-01-01.csv.bz2
transaction-2012-01-02.csv.bz2
transaction-2012-01-03.csv.bz2
transaction-2012-01-04.csv.bz2
.
.
transaction-2012-01-31.csv.bz2
transaction-2012-02-01.csv.bz2
.
.
transaction-2012-02-28.csv.bz2
I have a script called script.sh
cat script.sh
YEAR_MONTH=$1
FILEPATH="transaction-$YEAR_MONTH*.csv.bz2"
bzcat $FILEPATH|strings|grep -v "code" >> output
And if you need call the script you can use other script
cat script2.sh
LAST_MONTH=$(date -d -1month +%Y"-"%m)
if [ $# -eq 1 ]; then
DATE=$1
else
DATE=$LAST_MONTH
fi
script.sh $DATE 1>output$DATE.csv 2>> log.txt
And it do cat the files in a month, but now i need call the script with a specific week in a year:
bash script2.sh 2012-01
where 2012 is the year and 01 is the month
Now i need call the script with:
bash script2.sh 2012 13
where 2012 is the year and 13 is the week in a year
Now i need the cat only to the files in the year and week that the user specified, no per month per week
But the format of the files do not help me!!!! because the name is transaction-year-month-day.csv.bz2, not transaction-year-week.csv.bz2
Take a look at the manpage for strftime. These are date format codes. For example:
$ date +"%A, %B %e, %Y at %I:%m:%S %p"
Will print out a date like:
Thursday, May 30, 2013 at 02:05:31 PM
Try to see why this works.
On some systems, the date command will have a -j switch. This means, don't set the date, but reformat the given date. This allows you to convert one date to another:
$ date -f"$input_format" "$string_date" +"$output_format"
The $input_format is the format of your input date. $string_date is the string representation of the date in your $input_format. And, $output_format is the format you want your date in.
The first two fields are easy. Your date is in YY-MM-DD format:
$ date -f"%Y-%m-%d" "$my_date_string"
The question is what can you do for the final format. Fortunately, there is a format for the week in the year. %V which represents the weeks at 01-53 and %W which represents the weeks as 00-53.
What you need to do is find the date string on your file name, then convert that to the year and week number. If that's the same as the input, you need to concatenate this file.
find $dir -type f | while read transaction_file
do
file_date=${transaction_file#transaction-} #Removes the prefix
file_date=${file_date%.csv.bz2} #Removes the suffix
weekdate=$(date -j -f"%Y-%m-%d" "$file_date" +"%Y %W")
[ "$weekdate" -eq "$desired_date" ] || continue
...
done
For example, someone puts in 2013 05 as the desired date, you will go through all of your files and find ones with dates in the range you want. NOTE: That the week of the year is zero filled. You may need to zero fill the input of the week number to match.

Resources