I have a log file that includes lines that are formatted like the following below. I am trying to create a script in Linux that will remove the lines older then x days from the current date.
Wed Jan 26 10:44:35 2022 : Auth: (72448) Login incorrect (mschap: MS-CHAP2-Response is incorrect): [martin.zeus] (from client CoreNetwork port 0 via TLS tunnel)
Wed Jan 16 10:45:32 2022 : Auth: (72482) Login OK: [george.kye] (from client CoreNetwork port 5 cli CA-93-F0-6C-7E-77)
I think you should take a look at logrotate and Kibana & Elastic search to parse and filter the logs.
Nevertheless, I made a simple script that prints only the entries from the day that you pass as an argument until the current date,
E.g. This will print only the logs since the last 5 days. bash filter.sh log.txt 5
#!/usr/bin/env bash
file="${1}"
days="${2:-1}"
epoch_days=$(date -d "now -${days} days" +%s)
OFS=$IFS
IFS=$'\n'
while read line; do
epoch_log=$(date --date="$(echo $line | cut -d':' -f1,2,3)" +%s)
if [ ${epoch_log} -ge ${epoch_days} ]; then
echo ${line}
fi
done < ${file}
IFS=$OFS
Related
I'm creating a bash script in which whenever my apache exceeds a fixed amount of numbers then it restart itself on next cronjob. I've created something but it is not running properly.
#!/bin/bash
RESTART="systemctl restart httpd"
COUNT="ps aux | grep httpd | wc -l"
SERVICE="httpd"
LOGFILE="/opt/httpd/autostart-apache2.log"
if COUNT > 45
then
echo "starting apache at $(date)" >> $LOGFILE
$RESTART >> $LOGFILE
else
echo "apache is running at $(date)"
fi
The error it is throwing is
line 10: COUNT: command not found
apache is running at Sat Sep 10 23:34:30 IST 2022
Can anyone help me on how to store the value of the output of ps aux | grep httpd | wc -l as a number and compare it with another number.
RESTART="systemctl restart httpd" is a variable assignment, although it might work for your use case, you should really use a function instead. See the Complex page for more info.
COUNT="ps aux | grep httpd | wc -l" is a variable assignment as well use Command Substitution, like what you did with $(date), on a side note grep has the -c option, there is no need for wc.
COUNT > 45 You're executing the COUNT word/string but since there is nothing in your script/path/function/executable with the same name (COUNT is a variable name in your script), thus you get the error: line 10: COUNT: command not found
> is a form of redirection if not used in Arithmetic Expression., it truncates a file or create it if it does not exists. Like in your code > 45 created a file named 45. Open the file for writing is the proper term word for it according to the bash manual.
Here is how I would write it.
#!/usr/bin/env bash
service="httpd"
count="$(ps aux | grep -c "$service")"
logfile="/opt/httpd/autostart-apache2.log"
restart(){ systemctl restart "$service" ; }
if (( count > 45 )); then
echo "starting apache at $(date)" >> "$logfile"
restart >> "$logfile" 2>&1
else
echo "apache is running at $(date)"
fi
(( count > 45 )) is a form of test in bash for comparing/testing numbers. The $ can be omitted inside it.
I have a bunch of log files which are named according to their creation dates. For example; if my log file is created on 12 March 2018, the name of the logfile is log-2018-03-12.log
Here is what I want to do: From today's date, I want to check the name of my log files and zip the log files which are created in last 10 days.
Here is my code that zip all log files in a specific directory:
#!/bin/bash
# current time
now=$(date +"%F")
backupfile="backup-$now"
scripthome=/opt/MyStore/action_scripts/deneme/
tendaysbefore= date -d "$now - 10 days" '+%F'
for file in $scripthome;
do
find "$(basename "$file")" | zip -R $backupfile.zip "log-2*.log"
done
But I want to zip last 10 days log file, not all log files, and also I want to continue doing it for every 10 days after this. Also, after having zip file, I want to delete old log files.
In other words, I am trying to write a log-backup script. Can you help me please?
Thank you very much!
#!/bin/bash
END=10
for ((i=1;i<=END;i++)); do
file=log-`date -d "$i days ago" +"%F"`.log
echo $file
done
With the above script you have file names for last 10 days. Later(inside loop) you can do whatever you want like adding it to existing zip or searching for its existence.
Edit:
Following code may be useful according to your requirement
#!/bin/bash
# current time
now=$(date +"%F")
backupfile="backup-$now"
scripthome=/home/bhanu/opt/MyStore/action_scripts/deneme/
tendaysbefore=`date -d "$now - 10 days" '+%F'`
for file in $scripthome;
do
zip -r -tt $now -t $tendaysbefore "$backupfile.zip" $scripthome/log-*.log > add.log 2>&1
zip "$backupfile.zip" -d "*" -tt $tendaysbefore > delete.log 2>&1
done
We have a custom script ( in ksh) which was developed in RHEL Linux.
The functionality is
1) Read the input ASCII file
2) Replace "\" with "\\" using sed -i inplace the files
3) Load the history file into memory
4) Compare the data with current day
5) Generate the net change records
During a platform upgrade, we had to migrate this script on AIX 7.1 and
replaced the ksh with bash since, typeset -A is not available on ksh AIX and sed -i command with perl -pi -e and the rest of the script is almost the same.
We observe that the script processes for 1 hour ( 691 files) in Linux but, in AIX it is taking 7+ hours for the same.
We observe for one input file the below snippet is having a performance difference, Linux code completes within 1-2 seconds whereas, in AIX it takes 13-15 seconds. Due to this performance difference for each file , for 691 files, the script is taking 7 hours to complete.
Could you please help me understand if we can tune this script for a better performance on AIX. Any pointers will be very helpful.
Thank you in advance for your help!
Adding test results below for more precise issue
Linux Test script:
#!/bin/sh
export LANG="C"
echo `date`
typeset -A Archive_Lines
if [ -f "8249cii1.ASC" ]
then
echo `date` Starting sed
sed -i 's/\\/\\\\/g' 1577cii1.ASC
echo `date` Ending sed
while read line; do
if [[ "${#line}" == "401" ]]
then
Archive_Lines["${line:0:19}""${line:27}"]="${line:27:10}"
else
echo ${#line}
fi
done < 1577cii1.ASC
echo `date` Starting sed
sed -i 's/\\\\/\\/g' 1577cii1.ASC
echo `date` Ending sed
fi
echo `date`
Linux execution:
ksh read4.sh
Sun Nov 12 15:03:18 CST 2017
Sun Nov 12 15:03:18 CST 2017 Starting sed
Sun Nov 12 15:03:19 CST 2017 Ending sed
402
405
403
339
403
403
Sun Nov 12 15:03:22 CST 2017 Starting sed
Sun Nov 12 15:03:23 CST 2017 Ending sed
Sun Nov 12 15:03:23 CST 2017
AIX Test Script:
#!/usr/bin/bash
export LANG="C"
echo `date`
typeset -A Archive_Lines
if [ -f "1577cii1.ASC" ]
then
echo `date` Starting perl
perl -pi -e 's/\\/\\\\/g' 1577cii1.ASC
echo `date` Ending perl
while read line; do
if [[ "${#line}" == "401" ]]
then
Archive_Lines["${line:0:19}""${line:27}"]="${line:27:10}"
else
echo ${#line}
fi
done < 1577cii1.ASC
echo `date` Starting perl
perl -pi -e 's/\\\\/\\/g' 1577cii1.ASC
echo `date` Ending perl
fi
echo `date`
AIX Test execution:
bash read_test.sh
Sun Nov 12 15:00:17 CST 2017
Sun Nov 12 15:00:17 CST 2017 Starting perl
Sun Nov 12 15:00:18 CST 2017 Ending perl
402
405
313
403
337
403
403
Sun Nov 12 15:01:29 CST 2017 Starting perl
Sun Nov 12 15:01:29 CST 2017 Ending perl
Sun Nov 12 15:01:29 CST 2017
Replacing Archive_Lines["${line:0:19}""${line:27}"]="${line:27:10}" with echo"."
bash read_test.sh
Sun Nov 12 16:56:27 CST 2017
Sun Nov 12 16:56:27 CST 2017 Starting perl
Sun Nov 12 16:56:27 CST 2017 Ending perl
.
.
.
.
.
Sun Nov 12 16:56:42 CST 2017 Starting perl
Sun Nov 12 16:56:42 CST 2017 Ending perl
Sun Nov 12 16:56:42 CST 2017
With Archive_Lines["${line:0:19}""${line:27}"]="${line:27:10}"
bash read_test.sh
Sun Nov 12 16:59:52 CST 2017
Sun Nov 12 16:59:52 CST 2017 Starting perl
Sun Nov 12 16:59:52 CST 2017 Ending perl
402
405
313
403
337
403
403
Sun Nov 12 17:01:11 CST 2017 Starting perl
Sun Nov 12 17:01:11 CST 2017 Ending perl
Sun Nov 12 17:01:11 CST 2017
Thanks,
Vamsi
As Walter had suggested, it looks like there are some performance hits in bash for the substring processing (and possibly the length test).
It might be of interest to see what kind of timings you get with other solutions.
Here's a simplistic awk solution that should do the same thing as the original bash/substring logic (using your current sample data file; sans the output of line lengths != 401):
awk 'length($0)==401 { print substr($0,1,20)substr($0,28)"|"substr($0,28,10) }' 1577cii1.ASC | \
while IFS="|" read idx val
do
Archive_Lines["${idx}"]="${val}"
done
length($0)==401 : if line length is 401 then ...
print ...."|" ... : print 2 sections of output/fields separated by a pipe (|), where the fields are ...
substr($0,1,20)substr($0,28) : equivalent to your ${line:0:19}${line:27}
substr($0,28,10) : equivalent to your ${line:27:10}
at this point every line of length 401 is generating output like string1|string2
while IFS="|" read idx val : split the input back out into 2 variables ...
Archive_Lines["${idx}"]="${val}" : use the 2 variables as the array index/value pairs
NOTE: The addition of the pipe (|) as a field separator was added in case your substrings could include spaces; and of course if your substrings could include the pipe (|) then replace with some other character that won't show up in your substrings and which you can use as a field delimiter.
The objective is to see if awk's built-in length/substring processing is faster than bash's length/substring processing ...
This solved my problem
#!/usr/bin/ksh93
export LANG="C"
echo `date`
typeset -A Archive_Lines
if [ -f "1577cii1.ASC" ]
then
echo `date` Starting perl
perl -pi -e 's/\\/\\\\/g' 1577cii1.ASC
echo `date` Ending perl
while read line; do
if [[ "${#line}" == "401" ]]
then
Archive_Lines[${line:0:19}${line:27}]="${line:27:10}"
else
echo ${#line}
fi
done < 1577cii1.ASC
echo `date` Starting perl
perl -pi -e 's/\\\\/\\/g' 1577cii1.ASC
echo `date` Ending perl
fi
echo `date`
ksh93 read_test3.sh
Sun Nov 12 19:19:34 CST 2017
Sun Nov 12 19:19:34 CST 2017 Starting perl
Sun Nov 12 19:19:34 CST 2017 Ending perl
402
405
403
339
403
403
Sun Nov 12 19:19:38 CST 2017 Starting perl
Sun Nov 12 19:19:39 CST 2017 Ending perl
Sun Nov 12 19:19:39 CST 2017
Background Info:
I'm trying to follow the example posted here: http://www.cyberciti.biz/faq/bash-for-loop/
I would like loop 9 times using a control variable called "i".
Problem Description
My code looks like this:
for i in {0..8..1}
do
echo "i is $i"
tmpdate=$(date -d "$i days" "+%b %d")
echo $tmpdate
done
When I run this code, the debug prints show me:
"i is {0..8..1}"
instead of being a value between 0 and 8.
What I've Checked So Far:
I've tried to check my version of bash to make sure it supports this type of syntax. I'm running version 4,2,25(1)
I also tried using C like syntax where you do for (i=0;i<=8;i++) but that doesn't work either.
Any suggestions would be appreciated.
Thanks.
EDIT 1
I've also tried the following code:
for i in {0..8};
do
echo "i is $i"
tmpdate=$(date -d "$i days" "+%b %d")
echo $tmpdate
done
And...
for i in {0..8}
do
echo "i is $i"
tmpdate=$(date -d "$i days" "+%b %d")
echo $tmpdate
done
They all fail with the same results.
I also tried:
#!/bin/bash
for ((i=0;i<9;i++));
do
echo "i is $i"
tmpdate=$(date -d "$i days" "+%b %d")
echo $tmpdate
done
And that gives me the error:
test.sh: 4: test.sh: Syntax error: Bad for loop variable
FYI. I'm running on ubuntu 12
EDIT 2
Ok... so i think Weberick tipped me off to the issue...
To execute the script, I was running "sh test.sh"
when in the code I had defined it as a BASH script! My bad!
But here's the thing. Ultimately, I need it to work in both bash and sh.
so now that I'm being careful to make sure that I invoke the script the right way... I've noticed the following results:
when defined as a bash script and i execute using bash, the C-style version works!
when defined as an sh script and i execute using sh, the C-style version fails
me#devbox:~/tmp/test$ sh test.sh
test.sh: 5: test.sh: Syntax error: Bad for loop variable
when defined as an sh script and i execute using sh the NON c style version ( aka for i in {n ..x}), I get the "i is {0..8}" output.
PS. The ";" doesn't make a difference if you have the do on the next line...just FYI.
Ubuntu's default shell is dash, which doesn't recognise either of the bashisms (brace expansion, C-style for loop) you tried. Try running your script using bash explicitly:
bash myscript.sh
or by setting the shebang to #!/bin/bash. Make sure NOT to run the script with sh myscript.sh.
dash should work if you use seq:
for i in $(seq 0 1 8); do
echo "$i"
done
Just {0..8} should work in bash, the default increment is 1. If you want to use a C-style for loop in bash:
for ((i=0;i<9;i++)); do
echo "$i"
done
I'm confident that
#!/bin/bash
for ((i=0;i<9;i++))
do
echo "i is $i"
tmpdate=$(date -d "$i days" "+%b %d")
echo $tmpdate
done
work on Ubuntu 12.04
If you still have an error, can you please try running
chmod +x test.sh
then
./test.sh
And the result is
i is 0
Apr 04
i is 1
Apr 05
i is 2
Apr 06
i is 3
Apr 07
i is 4
Apr 08
i is 5
Apr 09
i is 6
Apr 10
i is 7
Apr 11
i is 8
Apr 12
I'm no expert at bash but according to tdlp you need a ; after the for statement. There are many ways to to a range. This is one of them.
#!/bin/bash
for i in `seq 1 8`; do
echo $i
done
The site you quote says
Bash v4.0+ has inbuilt support for setting up a step value using {START..END..INCREMENT} syntax:
So you can just use {0..8..1} when you have a bash version greater than 4.0, which I guess is not the case (try bash --version in your terminal). Instead of {0..8..1} you can also use {0..8}.
If you have an older version you can use instead of {START..END..INCREMENT} the command $(seq START INCREMENT END) in the for loop.
I can get a last modification date of the remote file using
curl --head http://url 2>/dev/null | grep -Po '(?<=^Last-Modified:\s).*$'
This gets me date/time such as
Wed, 04 Sep 2013 19:53:18 GMT
For local file I can use
find /path/file -exec stat \{} --printf="%y\n" \;
and it gets me date/time such as
2012-01-09 09:50:30.000000000 -0500
How can I compare this dat/time with last modification date of the local file? Please note that time zone may be different for remote and local file.
You can actually use date -d to parse the string, as #fedorqui says. Try running the below commands:
$ date -d "$(<your curl command grepped>)" +%s #+%s gets you timestamp.
$ date -d "$(<your find command>)" +%s
To actually compare, you can subtract the timestamps, something like:
$ echo $(( $(date -d "$(<curl cmd>)" +%s) - $(date -d "$(<find cmd>)" +%s) ))