I'm trying to write a script which help to follows the logs of my application.
The logs of my application are written to "var/log/MyLogs/" with the following pattern:
runningNumber_XXX.txt , for example:
0_XXX.txt
37_xxx.txt
99_xxx.txt
101_xxx.txt
103_xxx.txt
I'm trying to write a bash script (without a success for now) which will print last 20 rows of the last log file (the last log file is the file with has the biggest prefix number).
I know I need to go over the files in the folder (for file in /var/log/MyLogs/*) and check which file name has the biggest prefix, and after it print the last 20 rows from the selected file.
please help me....
Thanks...
find /var/log/MyLogs -iname '*_xxx.txt' | sort -n | tail -1 | xargs tail -20
Get correct files
Sort numerically
Get last log file
Get last 20 rows
tail -20 $(ls -1 /var/log/MyLogs/*_*.txt | sort -n -t _ -k 1 -r | head -1)
ls -1 [0-9]*_XXX.txt | sort -rn | head -1 | xargs tail -20
Usually is the bad practice using ls in shell scripts, but if you can ensure than the logfiles doesn't contains spaces and other strange characters, you can use a simple:
tail -20 $(ls -t1 /var/log/[0-9]*_XXX.txt | head -1)
The:
ls -t sorts the files my modification time newest comes first
head the the 1st
tail print the last lines
AGAIN, this is usually a bad practice, you can use it only when you knows what you're doing.
Related
I am trying to tail the latest file in a directory that matches a name.
Below example doesn't work
tail -f | ls -t /var/log/impala/impalad.demo.local.impala.log.INFO.* | head -1
tail: warning: following standard input indefinitely is ineffective
/var/log/impala/impalad.demo.local.impala.log.INFO.20180322-104843.43442
What is the best way to tail impalad.demo.local.impala.log.INFO.* that has latest time on it?
Use
tail -f $(ls -t /var/log/impala/impalad.demo.local.impala.log.INFO.* | head -1)
instead. tail expects a file to read from, which you obtain with ls -t /var/log/impala/impalad.demo.local.impala.log.INFO.* | head -1. Piping follows an input | output pattern, though, so your current scheme won't get you far. Piping the filename into tail won't work either because tail doesn't expect filenames from standard input. To pass a filename to tail you must pass it as an argument.
I'm a new player in bash scripting. There's something that I want to know about capture logfile using bash script.
Let's say there is a server which store logfile every hour with format file below.
file[20160509_130000].log
The logfile has detailed information like this.
13:00:00 INFO [file.chdev130] Event: ADIEVN_PLAY_DONE, Finished , nbytes=16360
13:00:00 INFO [file.chdev39] adiCollectDigits() success
My question is how can i read and store the running log or one hour before to get specific parameter (e.g. "Event") to new file using bash scripting?
Can someone teach me how to do it? Thanks.
Update
Here the flow that I want (for this time I want to know how the first point works):
Get the running log or one hour before.
Read the 15 minute interval (13:00:00 - 13:15:00).
grep the parameter (e.g. "Event) in that interval.
Count the parameter.
Store it to another file.
SOLVED
Here the solution in case someone need it.
List all the content based on time stamp using ls -t then pipe it
Use grep -v ^d (i still doesn't know the exact explanation for ^d), pipe again
Display first few lines with head
So the result is,
ls -t *.log | grep -v ^d | head -1 (for display the running log)
ls -t *.log | grep -v ^d | head -2 | tail -1 (for display the one log before the running log)
Hope it'll help. Thanks
== Case Closed ==
tail -f /path-to-logfile will allow you to read a file continuously until you cancel tail execution.
grep "^13:" /path-to-logfile will show you all strings in the file, which starts from "13:", in our case you'll get every record for 13-th hour.
grep "Event" /path-to-logfile will show you all strings with "Event" in them, i think, you got the idea about grep already. )
You can figure out the current or the previous log filename using date, using your logfile name convention:
fname="file[`date -v-1H '+%Y%m%d_%H0000'`].log"
will give you the previous filename. Omit -v-1H to get the current one.
For the 15-minute intervals, you may use regexps like '^\d\d:(0\d|1[0-4])' for 00:00-14:59 interval, '^\d\d:(1[5-9]|2\d)' for 15:00-29:59, etc.
For example, in the first regex, ^ matches the beginning of the line, \d\d: matches "two digits and a colon", and (0\d|1[0-4]) matches either 0 with any adjacent digit, or 1 with adjacent digit from 0 to 4. In the second regex, (1[5-9]|2\d) matches 1 with digit from 5 to 9, or 2 with any digit.
Then you grep -Po '(?<=Event: ).+(?=,)' the type of events in your log, assuming that the type of event always ends with a ,. That regexp will greedily match any symbols, as many as it can, starting from one symbol, if they are between strings Event: and , (Event: and , themselves are not matched, that's what lookbehind/lookaheads are for). Then use sort | uniq -c to count number of different events entries .
So the resulting script would look something like
fname="file[`date -v-1H '+%Y%m%d_%H0000'`].log"
grep -P '^\d\d:(0\d|1[0-4])' "$fname" | grep -Po '(?<=Event: ).+(?=,)' | sort | uniq -c > "/tmp/$fname_00-14" # entries for first 15 minutes
grep -P '^\d\d:(1[5-9]|2\d)' "$fname" | grep -Po '(?<=Event: ).+(?=,)' | sort | uniq -c > "/tmp/$fname_15-29" # entries for second 15 minutes
grep -P '^\d\d:(3\d|4[0-4])' "$fname" | grep -Po '(?<=Event: ).+(?=,)' | sort | uniq -c > "/tmp/$fname_30-44" # entries for third 15 minutes
grep -P '^\d\d:(4[5-9]|5\d)' "$fname" | grep -Po '(?<=Event: ).+(?=,)' | sort | uniq -c > "/tmp/$fname_45-59" # entries for fourth 15 minutes
for the last hour log.
Another approach is to use logtail with cron entries to get last log entries, instead of grepping. You can set up your script to be run by cron at 00, 15, 30 and 45 minutes each hour, determine the log filename inside it and logtail the file like logtail -f "$fname". The catch here would be that when running at 00 minutes, you'd need to use the -v-1H switch for date, and this approach is not as accurate as grepping out the times.
Found several posts like this one to tell how to find the latest file inside of a folder.
My question is one step forward, how to find the second latest file inside the same folder? The purpose is that I am looking for a way to diff the latest log with a previous log so as to know what have been changed. The log was generated in a daily basis.
Building on the linked solutions, you can just make tail keep the last two files, and then pass the result through head to keep the first one of those:
ls -Art | tail -n 2 | head -n 1
To do diff of the last (lately modified) two files:
ls -t | head -n 2 | xargs diff
Here's a stat-based solution (tested on linux)
for x in ./*;
do
if [[ -f "$x" ]]; then
stat --printf="%n %Y\n" "$x"; fi;
done |
sort -k2,2 -n -r |
sed -n '2{p;q}'
ls -dt {{your file pattern}} | head -n 2 | tail -n 1
Will provide second latest file in the pattern you search.
Here's the command returns you latest second file in the folder
ls -lt | tail -n 1 | head -n 2
enjoy...!
I want to store the result of a command to a variable in my shell script. I cant seem to get it to work. I want the most recently dated file in the directory.
PRODUCT= 'ls -t /some/dir/file* | head -1 | xargs -n1 basename'
it wont work
you have two options, either $ or backsticks`.
1) x=$(ls -t /some/dir/file* | head -1 | xargs -n1 basename)
or
2) x=`ls -t /some/dir/file* | head -1 | xargs -n1 basename`
echo $x
Edit: removing unnecessary bracket for (2).
The problem that you're having is that the command needs to be surrounded by back-ticks rather than single quotes. This is known as 'Command Substitution'.
Bash allows you to use $() for command substitution, but this is not available in all shells. I don't know if it's available in KSH; if it is, it's probably not available in all versions.
If the $() syntax is available in your version of ksh, you should definitely use it; it's easier to read (back ticks are too easy to confuse with single quotes); back-ticks are also hard to nest.
This only addresses one of the problems with your command, however: ls returns directories as well as files, so if the most recent thing modified in the specified directory is a sub-directory, that is what you will see.
If you only want to see files, I suggest using some version of the following (I'm using Bash, which supports default variables, you'll probably have to play around with the syntax of $1)
lastfile ()
{
find ${1:-.} -maxdepth 1 -type f -printf "%T+ %p\n" | sort -n | tail -1 | sed 's/[^[:space:]]\+ //'
}
This runs find on the directory, and only pulls files from that directory. It formats all of the files like this:
2012-08-29+16:21:40.0000000000 ./.sqlite_history
2013-01-14+08:52:14.0000000000 ./.davmail.properties
2012-04-04+16:16:40.0000000000 ./.DS_Store
2010-04-21+15:49:00.0000000000 ./.joe_state
2008-09-05+17:15:28.0000000000 ./.hplip.conf
2012-01-31+13:12:28.0000000000 ./.oneclick
sorts the list, takes the last line, and chops off everything before the first space.
You want $() (preferred) or backticks (``) (older style), rather than single quotes:
PRODUCT=$(ls -t /some/dir/file* | head -1 | xargs -n1 basename)
or
PRODUCT=`ls -t /some/dir/file* | head -1 | xargs -n1 basename`
You need both quotes to ensure you keep the name even if it contains spaces, and also in case you later want more than 1 file, and "$(..)" to run commands in background
I believe you also need the '-1' option to ls, otherwise you could have several names per lines (you only keep 1 line, but it could be several files)
PRODUCT="$(ls -1t /some/dir/file* | head -1 | xargs -n1 basename)"
Please do not put space around the "=" variable assignments (as I saw on other solutions here) , as it's not very compatible as well.
I would do something like:
Your version corrected:
PRODUCT=$(ls -t /some/dir/file* | head -1 | xargs -n1 basename)
Or simpler:
PRODUCT=$(cd /some/dir && ls -1t file* | head -1)
change to the directory
list one filename per line and sort by time/date
grab the first line
I am using ls -l -t to get a list of files in a directory ordered by time.
I would like to limit the search result to the top 2 files in the list.
Is this possible?
I've tried with grep and I struggled.
You can pipe it into head:
ls -l -t | head -3
Will give you top 3 lines (2 files and the total).
This will just give you the first 2 lines of files, skipping the size line:
ls -l -t | tail -n +2 | head -2
tail strips the first line, then head outputs the next 2 lines.
To avoid dealing with the top output line you can reverse the sort and get the last two lines
ls -ltr | tail -2
This is pretty safe, but depending what you'll do with those two file entries after you find them, you should read Parsing ls on the problems with using ls to get files and file information.
Or you could try just this
ls -1 -t | head -2
The -1 switch skips the title line.
You can use the head command to grab only the first two lines of output:
ls -l -t | head -2
You have to pipe through head.
ls -l -t | head -n 3
will output the two first results.
Try this:
ls -td -- * | head -n 2