bash and awk script get to previous month date in format YYYY-MM-DD - linux

I need to get last month date in format YYYY-MM-DD, suppose if run script in May on any date then it should return me Last date of April month in format 2018-04-30.
Would be good if can get the same work done with awk statement also.
I tried below code and able to get output in below form but i need in format YYYY-MM-DD
Apr 30 2018
Code:
#!/bin/bash
cur_month=`date +%m`
cur_year=`date +%Y`
prev_month=$(($cur_month-1))
# Check to see if this is January
if [ $prev_month -lt 1 ]
then
prev_year=$(($cur_year-1))
prev_month=12
LastDayOfpreviousMonth=`cal $prev_month $cur_year | grep -v "^$" | sed -n '1p;$p' | tr "\n" " " | awk '{print substr($1,1,3),$NF,$2}'`
else
LastDayOfpreviousMonth=`cal $prev_month $cur_year | grep -v "^$" | sed -n '1p;$p' | tr "\n" " " | awk '{print substr($1,1,3),$NF,$2}'`
fi
echo $LastDayOfpreviousMonth
Input file: In this file in column14 need to put value of date for last month in form YYYY-MM-DD(2018-04-30)
EPOS|EPOS_OUTWARD_B2B_OSR_TTML_TTSL_$LDPM_20180504_V1.0.txt|||AD|Outward|ANP|Receipt cum invoice|42018|37AAACT2438A1ZS|INV|TAX|IN37201800001452|2018-04-17||||1||||SOHANRAJ RAJ KUMAR||||37||||||9984||Telecommunication services||||300|0||9|27|9|27|||||354||||||||||
EPOS|EPOS_OUTWARD_B2B_OSR_TTML_TTSL_$LDPM_20180504_V1.0.txt|||AD|Outward|ANP|Receipt cum invoice|42018|37AAACT2438A1ZS|INV|TAX|IN37201800001426|2018-04-12||||1||||MOLLITE SIVARAMA KRISHNA||||37||||||9984||Telecommunication services||||300|0||9|27|9|27|||||354||||||||||
EPOS|EPOS_OUTWARD_B2B_OSR_TTML_TTSL_$LDPM_20180504_V1.0.txt|||GJ|Outward|GUJ|Receipt cum invoice|42018|24AAACT2438A1ZZ|INV|TAX|IN24201800000651|2018-04-07||||1||||AIREN LTD ||||24||||||9984||Telecommunication services||||500|0||9|45|9|45|||||590||||||||||

GNU date itself is capable of handling this task perfectly and reliably, no need for other magic.
$ date -d "$(date +%Y-%m-01) - 1 day" "+%Y-%m-%d"
2018-04-30

Even though it is evident that the OP's dateformat is unix based, it is worth mentioning that there are two general calendars in play. The Julian calendar and the Gregorian calendar. Both calendars have a different concept of what a leap year is.
note: at this time, most countries adopt the Gregorian calendar. However, it was not at once officially adopted by all countries. This should be kept in mind when making historical research. In Great Britain, for instance , the change was made as late as in A.D. 1752, and in Turkey not before 1927. (Ref Astronomical Algorithms by Jean Meeus)
With this in mind, and depending whether your digital file is from Turkey from 1920 or 2011 or Great Britain from 1740 or 1789, I present two awk solutions
In a leap year, the month February has 29 days, where in a common year it has 28 days.
In the Julian calendar, a year is a leap (or bissextile) year of 366 days if its numerical designation is divisible by 4. All other years are common years (365 days).
Using awk, as this was the OP's request, we can write the following function that would do the requested conversion :
awk 'function julian(str, y,m,d) {
y=substr(str,1,4); m=substr(str,6,2)+0;
if (m==1) {y=y-1; m=12} else {m=m-1}
if (((m-1)%7)%2==0) {d=31} else {d=30}
if (m==2) { d = (y%4==0) ? 29 : 28 }
return sprintf("%0.4d-%0.2d-%0.2d",y,m,d)
}
{ print $14, julian($14) }' <file>
In the Gregorian calendar, the same rule holds as in the Julian calendar, with the following exception: the centurial years that are not divisible by 400 are common years.
awk 'function gregorian(str, y,m,d) {
y=substr(str,1,4); m=substr(str,6,2)+0;
if (m==1) {y=y-1; m=12} else {m=m-1}
if (((m-1)%7)%2==0) {d=31} else {d=30}
if (m==2) { d = (y%4!=0) ? 28 : (y%100==0 && y%400!=0 ? 28 : 29 ) }
return sprintf("%0.4d-%0.2d-%0.2d",y,m,d)
}
{ print $14, gregorian($14) }' <file>
As the UNIX calendar is Gregorian, we can also use the GNU Awk Time Functions. This however will only work from 1970-02-01 till 2038-01-19 for 32-bit systems and 292277026596-04-01 for 64-bit systems (see here)
awk 'function unix(str, tmp) {
tmp=substr(str,1,7)" 01 00 00 00"; gsub("-"," ",tmp);
return strftime("%Y-%m-%d",mktime(tmp) - 1) }
{ print $14, unix($14) }' <file>

Related

Number of Mondays Falls on the First of the month

I want a command line can display number of Monday(s) which fall(s) on the first of the month in a given year without using sed or awk commands
I have this command that display the first date of the current month
date -d "-0 month -$(($(date +%d)-1)) days"
With GNU date, you can read input from a file (or standard input):
printf '%s\n' 2021-{01..12}-01 | date -f- +%u | grep -c 1
This prints dates for the first of each month in a year, then formats them as "weekday" (where 1 is "Monday"), then counts the number of Mondays.
To parametrize the year, replace 2021 with a variable containing the year; wrapped in a function:
mondays() {
local year=$1
printf '%s\n' "$year"-{01..12}-01 | date -f- +%u | grep -c 1
}
Using a for loop, this can be accomplished as follows.
for mon in {01..12}; do date -d "2021-$mon-01" +%u; done | grep -c 1
Breakdown
We iterate through the numbers 01 to 12 representing the months.
We call date passing in the custom date value with the first date of each month in the year. We use +%u to return the day of week where 1 represents Monday.
Lastly we count the number of 1s using grep -c or grep --count
Note, the desired year has been hard coded as 2021. The current year can be used as:
for mon in {01..12}; do date -d "$(date +%Y)-$mon-01" +%u; done | grep -c 1
This can also all be put into a function and the desired year passed in as an argument:
getMondays() {
for mon in {01..12}; do date -d "$1-$mon-01" +%u; done | grep -c 1
}
I implemented it as:
for ((i=1,year=2021,mondays=0; i< 12; i++)) {
if [ $(date -d "$i/1/$year" +%u) -eq 1 ]
then
let "mondays++"
fi
}
echo "There are $mondays Mondays in $year."
That said, I like Mushfiq's answer. Quite elegant.

How to grep the logs between two date range in Unix

I have a log file abc.log in which each line is a date in date +%m%d%y format:
061019:12
062219:34
062319:56
062719:78
I want to see the all the logs between this date range (7 days before date to current date) i.e (from 062019 to 062719 in this case). The result should be:
062219:34
062319:56
062719:78
I have tried few things from my side to achieve:
awk '/062019/,/062719' abc.log
This gives me correct answer, but if i don't want to hard-code the date value and try achieving the same it does not give the correct value.
awk '/date --date "7 days ago" +%m%d%y/,/date +%m%d%y' abc.log
Note:
date --date "7 days ago" +%m%d%y → 062019 (7 days back date)
date +%m%d%y → 062719 (Current date)
Any suggestions how this can be achieved?
Your middle-endian date format is unfortunate for sorting and comparison purposes. Y-m-d would have been much easier.
Your approach using , ranges in awk requires exactly one log entry per day (and that the log entries are sorted chronologically).
I would use perl, e.g. something like:
perl -MPOSIX=strftime -ne '
BEGIN { ($start, $end) = map strftime("%y%m%d", localtime $_), time - 60 * 60 * 24 * 7, time }
print if /^(\d\d)(\d\d)(\d\d):/ && "$3$1$2" ge $start && "$3$1$2" le $end' abc.log
Use strftime "%y%m%d" to get the ends of the date range in big-endian format (which allows for lexicographical comparisons).
Use a regex to extract day/month/year from each line into separate variables.
Compare the rearranged date fields of the current line to the ends of the range to determine whether to print the line.
To get around the issue of looking for dates that may not be there, you could generate a pattern that matches any of the dates (since there are only 8 of them it doesn’t get too big, if you want to look for the last year it might not work as well):
for d in 7 6 5 4 3 2 1 0
do
pattern="${pattern:+${pattern}\\|}$(date --date "${d} days ago" +%m%d%y)"
done
grep "^\\(${pattern}\\)" abc.log

Filter Linux logs based on Unix Timestamp

I have a log on a linux server. The entries are in the format:
[timestamp (seconds since jan 1 1970)] log data entry
I need a bash script that will take the name of the log file and output only yesterdays entries (from 12:00 to 23:59:59 of previous day) and output those lines to a new file.
I've seen various scripts that filter logs based on dates but all of them so far deal with date stamps in more human readable formats, or are not dynamic. They rely on hard coded dates. I want a script that is going to run in a cron job daily so it has to be aware of what the current date is each time it runs.
Thanks.
Update: This is what I have so far. It just never seems to do the evaluation of the date. It prints 00 for the date so everything gets through.
head -5 logfile.log | awk '{
if($1 >= (date -d "today 00:00:00" +"%s"))
print $1 (date -d "today 00:00:00" +"%s");
}'
I'm confused though, even if the date evaluates properly, $1 is going to have numbers inside square brackets, and my date will be just numbers. Will it do the comparison properly if the strings are formatted differently like that? I haven't figured out how to shove the date number returned by date into a string with brackets yet.
Well, maybe using the dates as Dale said. But using a little trick to extract the "[" and "]", and after compare the dates. Something like this:
YESTERDAY=$(date -d "yesterday 00:00:00" +"%s")
TODAY=$(date -d "today 00:00:00" +"%s")
# Combine the processing in awk
awk -v MIN=${YESTERDAY} -v MAX=${TODAY} -F["]""["] '{ if ( $2 >= MIN && $2 <= MAX) print $0}' logfile.log
Combining tips and tricks from Glenn, Dale, and Davison:
awk -v today=$(date -d "today 00:00:00" +"%s") -v yesterday=$(date -d "yesterday 00:00:00" +"%s") -F'[\\[\\] ]' '{ if($2 >= yesterday && $2 < today) print }' logfile.log
Uses the shell's $() command substitution to feed variables to awk's -v argument parser
-F'[\\[\\] ]' sets the field separator to be either [, ], or
input data:
[1300000000 log1 data1 entry1]
[1444370000 log2 data2 entry2]
[1444374000 log3 data3 entry3]
[1444460399 log4 data4 entry4]
[1500000000 log5 data5 entry5]
output:
[1444370000 log2 data2 entry2]
You might try something like this:
YESTERDAY=$(date -d "yesterday 00:00:00" +"%s")
TODAY=$(date -d "today 00:00:00" +"%s")
cat your_log.log | \
awk -v MIN=${YESTERDAY} -v MAX=${TODAY} \
'{if($1 >= MIN && $1 < MAX) print}'
:)
Dale

Extract month and day from linux "date" command

I would like to modify the following statement to extract the Month, as well as day:
testdate=$(date -d "2 days" +%d| sed 's/0([1-9])/ \1/')
right now it only extracts the day...
I'm currently googling how sed works. looks like a regular expression of sorts that's being used but... I thought I'd check with the forum for certain.
Thanks.
EDIT 1
now I have:
testdate=$(date -d "2 days" "+%b %_d %Y")
Which is great... except when the leading zeros in the day are stripped out. then i end up with two spaces between the Month and day
for example, for April 29, 2014, I get:
Apr 29 2014
which is great. But for May 01, 2014 I get:
May 1, 2014
where there's a double space between "May" and "1". I guess i can do it in two steps... by doing the date command, followed by sed to find and replace double spaces.
Just wondering if there's a way to do it all in one command.
Thanks.
date accepts a FORMAT arg which should be used here. sed is not necessary:
date +%d.%m.
Outputs:
03.04.
in European time format. If you want the month's name printed use
date +'%d. %B'
Output:
03. April
To read the month and day into shell variables (assuming bash/ksh/zsh)
read month day < <(date -d "2 days" "+%m %d")
If you're planning to do arithmetic on these values, be of numbers that would be treated as octal but are invalid octal numbers 08 and 09. If you want to strip off the leading zero, use "+%_m %_d"
Using read, the shell takes care of the excess whitespace for you:
read mon day year < <(date -d "2 days" "+%b %_d %Y")
testdate="$mon $day $year"
If you don't want to use the temp vars:
testdate="Apr 5 2014" # $(date -d "2 days" "+%b %_d %Y")
testdate=${testdate// / } # globally replace 2 spaces with 1 space
echo "$testdate"
For date format %d would print only the day. It's unlikely to extract month unless you specify that in the format.
Specify the format for obtaining the month too.
%b locale's abbreviated month name (e.g., Jan)
%B locale's full month name (e.g., January)
%m month (01..12)
For example:
$ date -d "2 days" +%d/%b
05/Apr
$ date -d "2 days" +%d/%B
05/April
$ date -d "2 days" +%d/%m
05/04
Seems there is a simpler way to extract [the] month and day from [the] linux “date” command, then all the previous answers.
From $ date --help:
By default, date pads numeric fields with zeroes.
The following optional flags may follow `%':
- (hyphen) do not pad the field
Which gives us the below, without the use of sed or read to remove any kind of padding:
testdate=$(date -d "3 days" "+%b %-d %Y") (days adjusted to show single-digit day number, from today's date)
And outputs the below:
$ echo $testdate
Sep 1 2014

How to search a log file for two different dates in Linux

I'm using an RPM-based distro and I want to dynamically search a log file for today's date and yesterday's date to output a report. The string has to be dynamic ( no egrep "\b2012-10-[20-30]\b" ) meaning that I can take the same one-liner or script and search a file for today's date and yesterday's date and print some output. Basically searching log files for specific entries.
Here's what I got, but I want to replace the egrep with something dynamic:
grep "No Such User Here" /var/log/maillog | egrep "\b2012-10-2[3-4]\b" | cut -d "<" -f 3 | egrep -o '\b[a-zA-Z0-9._%-]+#[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}\b' | cut -d "#" -f 2 | sort -d |uniq -ci | awk -F" " '{ print "Domain: " $2 " has been sent " $1 " messages that got a No Such User Here error." }'
Any help is appreciated. I'm looking for something that very likely uses the date command
date "+%Y-%m-%d"
but I need to take the %d and search for both the current day, and yesterday. Can this be done?
Any insight is much appreciated.
If you have GNU date:
$ x=$(date "+%Y-%m-%d")
$ y=$(date "+%Y-%m-%d" -d "-1 day")
$ egrep "($x|$y)" file
x contains current date and y contains the yesterday's date.
With GNU awks time functions:
gawk 'BEGIN{
today = strftime("%Y-%m-%d")
yesterday = strftime("%Y-%m-%d",systime()-24*60*60)
}
$0 ~ "(" today "|" yesterday ")"
' file

Resources