I have to extract particular date's log from any application log. what we generally do
vi app.log
/date (searching)
.= (position where date found)
G(last line)
first_line,Last_line w filname.log (to generate log file with particular date)
I made small script but I think My logic is not correct:
#!/bin/bash
read -p " enter the App name" : app
read =p " enter the Date in this Format 10 Jan 20:01:00" : date
FILE=/logs/$app.log
# Check if $app exists or not
if test ! -f "$app" /logs
then
echo "Error - $app not found."
else
First_line=$(cat $FILE | grep $date | .=)
Second_line=$(tail -1 $FILE | .=)
vi $FILE | $First_line, $Second_line w $app.txt
fi
There is an error in the line
if test ! -f "$app" /logs
You seperated "$app" and /logs with a space; test -f expects one parameter, so it should be:
if test ! -f "$app"/logs
To get the log starting from the date given to the current end of the file, You should use sed (meaning "stream editor"), which is designed for noninteractive editing of files:
sed -n '/'$date'/,$ p' $app.log >$app.txt
The parameter -n is to only output, what matched and not the whole file.
Here I am telling sed to extract everything between two regular expressions, the first one, being the $date parameter given, the second one meaning "end of file". You could change this easily to extract all logs for one specific day or whatever.
As something seems to be unclear I am adding a complete example:
I have an apache log file, where the date logged looks like this [01/Apr/2015:22:31:21 +0200].
So I can do:
export date="01\/Apr\/2015"
It is important to note that I am escaping the slashes here, as they are going into a regex. This has to be done with everything that has some special meaning inside regexes too.
Then I can do:
sed -n '/'$date'/,$ p' myaccess.log
and I get everything logged for the 1st of April until the end of the file.
You could pass commands to vi from your script with a here document, if your task is as mechanical as you describe.
I'm talking about something like this
#!/bin/bash
read -p " enter the App name" : app
read =p " enter the Date in this Format 10 Jan 20:01:00" : date
FILE=/logs/$app.log
# Check if $app exists or not
if test ! -f "$app" /logs
then
echo "Error - $app not found."
else
vi $FILE <<-_EOF_
/date
.=
G
first_line,Last_line w filname.log
etc, etc
_EOF_
fi
I just wrote this as an idea that you mighr find fun to explore. I know the code will not work.
I'd use awk:
awk -v date="$date" '$0 ~ date {p=1} p' file.in > file.out
That will start printing lines when the date is first seen, until the end of the line.
Related
I've been trying to get a script working to backup some files from one machine to another but have been running into an issue.
Basically what I want to do is copy two files, one .log and one (or more) .dmp. Their format is always as follows:
something_2022_01_24.log
something_2022_01_24.dmp
I want to do three things with these files:
find the second to last one .log file (i.e. something_2022_01_24.log is the latest,I want to find the one before that say something_2022_01_22.log)
get a substring with just the date (2022_01_22)
copy every .dmp that matches the date (i.e something_2022_01_24.dmp, something01_2022_01_24.dmp)
For the first one from what I could find the best way is to do: ls -t *.log | head-2 as it displays the second to last file created.
As for the second one I'm more at a loss because I'm not sure how to parse the output of the first command.
The third one I think I could manage with something of the sort:
[ -f "/var/www/my_folder/*$capturedate.dmp" ] && cp "/var/www/my_folder/*$capturedate.dmp" /tmp/
What do you guys think is there any way to do this? How can I compare the substring?
Thanks!
Would you please try the following:
#!/bin/bash
dir="/var/www/my_folder"
second=$(ls -t "$dir/"*.log | head -n 2 | tail -n 1)
if [[ $second =~ .*_([0-9]{4}_[0-9]{2}_[0-9]{2})\.log ]]; then
capturedate=${BASH_REMATCH[1]}
cp -p "$dir/"*"$capturedate".dmp /tmp
fi
second=$(ls -t "$dir"/*.log | head -n 2 | tail -n 1) will pick the
second to last log file. Please note it assumes that the timestamp
of the file is not modified since it is created and the filename
does not contain special characters such as a newline. This is an easy
solution and we may need more improvement for the robustness.
The regex .*_([0-9]{4}_[0-9]{2}_[0-9]{2})\.log will match the log
filename. It extracts the date substring (enclosed with the parentheses) and assigns the bash variable
${BASH_REMATCH[1]} to it.
Then the next cp command will do the job. Please be cateful
not to include the widlcard * within the double quotes so that
the wildcard is properly expanded.
FYI here are some alternatives to extract the date string.
With sed:
capturedate=$(sed -E 's/.*_([0-9]{4}_[0-9]{2}_[0-9]{2})\.log/\1/' <<< "$second")
With parameter expansion of bash (if something does not include underscores):
capturedate=${second%.log}
capturedate=${capturedate#*_}
With cut command (if something does not include underscores):
capturedate=$(cut -d_ -f2,3,4 <<< "${second%.log}")
Lets say i have multiple filesnames e.g. R014-20171109-1159.log.20171109_1159.
I want to create a shell script which creates for every given date a folder and moves the files matching the date to it.
Is this possible?
For the example a folder "20171109" should be created and has the file "R014-20171109-1159.log.20171109_1159" on it.
Thanks
This is a typical application of a for-loop in bash to iterate thru files.
At the same time, this solution utilizes GNU [ shell param substitution ].
for file in /path/to/files/*\.log\.*
do
foldername=${file#*-}
foldername=${foldername%%-*}
mkdir -p "${foldername}" # -p suppress errors if folder already exists
[ $? -eq 0 ] && mv "${file}" "${foldername}" # check last cmd status and move
done
Since you want to write a shell script, use commands. To get date, use cut cmd like ex:
cat 1.txt
R014-20171109-1159.log.20171109_1159
cat 1.txt | cut -d "-" -f2
Output
20171109
is your date and create folder. This way you can loop and create as many folders as you want
Its actually quite easy(my Bash syntax might be a bit off) -
for f in /path/to/your/files*; do
## Check if the glob gets expanded to existing files.
## If not, f here will be exactly the pattern above
## and the exists test will evaluate to false.
[ -e "$f" ] && echo $f > #grep the file name for "*.log."
#and extract 8 charecters after "*.log." .
#Next check if a folder exists already with the name of 8 charecters.
#If not { create}
#else just move the file to that folder path
break
done
Main idea is from this post link. Sorry for not providing the actual code as i havent worked anytime recently on Bash
Below commands can be put in script to achieve this,
Assign a variable with current date as below ( use --date='n day ago' option if need to have an older date).
if need to get it from File name itself, get files in a loop then use cut command to get the date string,
dirVar=$(date +%Y%m%d) --> for current day,
dirVar=$(date +%Y%m%d --date='1 day ago') --> for yesterday,
dirVar=$(echo $fileName | cut -c6-13) or
dirVar=$(echo $fileName | cut -d- -f2) --> to get from $fileName
Create directory with the variable value as below, (-p : create directory if doesn't exist.)
mkdir -p ${dirVar}
Move files to directory to the directory with below line,
mv *log.${dirVar}* ${dirVar}/
I have a folder with lots of files having a pattern, which is some string followed by a date and time:
BOS_CRM_SUS_20130101_10-00-10.csv (3 strings before date)
SEL_DMD_20141224_10-00-11.csv (2 strings before date)
SEL_DMD_SOUS_20141224_10-00-10.csv (3 strings before date)
I want to loop through the folder and extract only the part before the date and output into a file.
Output
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
This is my script but it is not working
#!/bin/bash
# script variables
FOLDER=/app/list/l088app5304d1/socles/Data/LEMREC/infa_shared/Shell/Check_Header_T24/
LOG_FILE=/app/list/l088app5304d1/socles/Data/LEMREC/infa_shared/Shell/Check_Header_T24/log
echo "Starting the programme at: $(date)" >> $LOG_FILE
# Getting part of the file name from FOLDER
for file in `ls $FOLDER/*.csv`
do
mv "${file}" "${file/date +%Y%m%d HH:MM:SS}" 2>&1 | tee -a $LOG_FILE
done #> $LOG_FILE
Use sed with extended-regex and groups to achieve this.
cat filelist | sed -r 's/(.*)[0-9]{8}_[0-9][0-9]-[0-9][0-9].[0-9][0-9].csv/\1/'
where filelist is a file with all the names you care about. Of course, this is just a placeholder because I don't know how you are going to list all eligible files. If a glob will do, for example, you can do
ls mydir/*.csv | sed -r 's/(.*)[0-9]{8}_[0-9][0-9]-[0-9][0-9].[0-9][0-9].csv/\1/'
Assuming you wont have numbers in the first part, you could use:
$ for i in *csv;do str=$(echo $i|sed -r 's/[0-9]+.*//'); echo $str; done
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
Or with parameter substitution:
$ for i in *csv;do echo ${i%_*_*}_; done
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
When you use ${var/pattern/replace}, the pattern must be a filename glob, not command to execute.
Instead of using the substitution operator, use the pattern removal operator
mv "${file}" "${file%_*-*-*.csv}.csv"
% finds the shortest match of the pattern at the end of the variable, so this pattern will just match the date and time part of the filename.
The substitution:
"${file/date +%Y%m%d HH:MM:SS}"
is unlikely to do anything, because it doesn't execute date +%Y%m%d HH:MM:SS. It just treats it as a pattern to search for, and it's not going to be found.
If you did execute the command, though, you would get the current date and time, which is also (apparently) not what you find in the filename.
If that pattern is precise, then you can do the following:
echo "${file%????????_??-??-??.csv}" >> "$LOG_FILE"
using grep:
ls *.csv | grep -Po "\K^([A-Za-z]+_)+"
output:
BOS_CRM_SUS_
SEL_DMD_
SEL_DMD_SOUS_
In my log file the time format is 2014-10-10 HH:MM:SS:sss.
I am reading the time and date from the file and converting it into seconds for further processing. But it's giving error the date is invalid. May be it's because time is not in ..SS.sss format.
while read line;
do
d1=$(echo $line | cut -d, -f2);
case "$line" in \s*) continue ;; esac
t1=$(echo $line | cut -d, -f3);
d1t1="${d1} ${t1}";
echo "$d1t1";
ds1=$(date -d"$d1t1" "+%s");
echo "$ds1";
done < error.log
I want to replace ":" by "." which is the way to solve the problem?
Why not to simply replace the milliseconds separator in your log file?
perl -p -i -e 's/:(\d{3}),/.\1/g' error.log
A ':' separator for milliseconds is wrong, and date command will complain aboult it. With a '.' separator date command will succeed.
UPDATE:
The command is a "perl inliner": it applies text changes to a file ("error.log", in this case), globally (for every row).
The change command here is a regular expression substitution: s/:(\d{3}),/.\1/g, which means: "substitute every colon followed by exaclty 3 digits and a comma with a dot and the digits".
I want add the date next to a filename ("somefile.txt"). For example: somefile_25-11-2009.txt or somefile_25Nov2009.txt or anything to that effect
Maybe a script will do or some command in the terminal window. I'm using Linux(Ubuntu).
The script or command should update the filename to a new date everytime you want to save the file into a specific folder but still keeping the previous files. So there would be files like this in the folder eventually: filename_18Oct2009.txt , filename_9Nov2009.txt , filename_23Nov2009.txt
Info/Summary
With bash scripting you can enclose commands in back ticks or parantheses. This works great for labling files, the following wil create a file name with the date appended to it.
Methods
Backticks -
$ echo myfilename-"`date +"%d-%m-%Y"`"
$(parantheses) - :
$ echo myfilename-$(date +"%d-%m-%Y")
Example Usage:
echo "Hello World" > "/tmp/hello-$(date +"%d-%m-%Y").txt"
(creates text file '/tmp/hello-28-09-2022.txt' with text inside of it)
Note, in Linux quotes are your friend, best practice to enclose the file name to prevent issues with spaces and such in variables.
There's two problems here.
1. Get the date as a string
This is pretty easy. Just use the date command with the + option. We can use backticks to capture the value in a variable.
$ DATE=`date +%d-%m-%y`
You can change the date format by using different % options as detailed on the date man page.
2. Split a file into name and extension.
This is a bit trickier. If we think they'll be only one . in the filename we can use cut with . as the delimiter.
$ NAME=`echo $FILE | cut -d. -f1
$ EXT=`echo $FILE | cut -d. -f2`
However, this won't work with multiple . in the file name. If we're using bash - which you probably are - we can use some bash magic that allows us to match patterns when we do variable expansion:
$ NAME=${FILE%.*}
$ EXT=${FILE#*.}
Putting them together we get:
$ FILE=somefile.txt
$ NAME=${FILE%.*}
$ EXT=${FILE#*.}
$ DATE=`date +%d-%m-%y`
$ NEWFILE=${NAME}_${DATE}.${EXT}
$ echo $NEWFILE
somefile_25-11-09.txt
And if we're less worried about readability we do all the work on one line (with a different date format):
$ FILE=somefile.txt
$ FILE=${FILE%.*}_`date +%d%b%y`.${FILE#*.}
$ echo $FILE
somefile_25Nov09.txt
cp somefile somefile_`date +%d%b%Y`
You can add date next to a filename invoking date command in subshell.
date command with required formatting options invoked the braces of $() or between the backticks (`…`) is executed in a subshell and the output is then placed in the original command.
The $(...) is more preferred since in can be nested. So you can use command substitution inside another substitution.
Solutions for requests in questions
$ echo somefile_$(date +%d-%m-%Y).txt
somefile_28-10-2021.txt
$ echo somefile_$(date +%d%b%Y).txt
somefile_28Oct2021.txt
The date command comes with many formatting options that allow you to customize the date output according to the requirement.
%D – Display date in the format mm/dd/yy (e.g. : 10/28/21)
%Y – Year (e.g. : 2021)
%m – Month (e.g. : 10)
%B – Month name in the full string format (e.g. : October)
%b – Month name in the shortened string format (e.g. : Oct)
%d – Day of month (e.g. : 28)
%j – Day of year (e.g. : 301)
%u – Day of the week (e.g. : 4)
%A – Weekday in full string format (e.g. : Thursday)
%a – Weekday in shortened format (e.g. : Thu)
I use this script in bash:
#!/bin/bash
now=$(date +"%b%d-%Y-%H%M%S")
FILE="$1"
name="${FILE%.*}"
ext="${FILE##*.}"
cp -v $FILE $name-$now.$ext
This script copies filename.ext to filename-date.ext, there is another that moves filename.ext to filename-date.ext, you can download them from here.
Hope you find them useful!!
I use it in raspberry pi, and the first answer doesn't work for me, maybe because I typed wrong or something? I don't know. So I combined the above answers and came up with this:
now=$(date +'%Y-%m-%d')
geany "OptionalName-${now}.txt"
That if you want to use geany or anything else
a bit more convoluted solution that fully matches your spec
echo `expr $FILENAME : '\(.*\)\.[^.]*'`_`date +%d-%m-%y`.`expr $FILENAME : '.*\.\([^.]*\)'`
where first 'expr' extracts file name without extension, second 'expr' extracts extension