I have a program that during it's operation it writes to a text file. In this text file each line consists of 4 parts.
Thread ID (a number)
A date in the format yyyy-mm-dd
A timestamp in the format 12:34:56.123456
A function name
Some useful comments printed out by the programs
An example of what a log line would look like would be something like this:
127894 2020-07-30 22:04:30.234124 foobar caught an unknown exception
127895 2020-07-30 22:05:30.424134 foobar clearing the programs cache
127896 2020-07-30 22:06:30.424134 foobar recalibrating dankness
The logs are printed in chronological order and I would like to know how I to get the highest frequency of these logs. For example I wanted to know at what minute or second of the day the program has the highest congestion.
Ideally I'd like an output that could tell me for example, "The highest logging frequency is between 22:04:00 and 22:05:00 with 10 log lines printed in this timeframe".
Let's consider this test file:
$ cat file.log
127894 2020-07-30 22:04:30.234124 foobar caught an unknown exception
127895 2020-07-30 22:05:20.424134 foobar clearing the programs cache
127895 2020-07-30 22:05:30.424134 foobar clearing the programs cache
127895 2020-07-30 22:05:40.424134 foobar clearing the programs cache
127896 2020-07-30 22:06:30.424134 foobar recalibrating dankness
127896 2020-07-30 22:06:40.424134 foobar recalibrating dankness
To get the most congested minutes, ranked in order:
$ awk '{sub(/:[^:]*$/, "", $3); a[$2" "$3]++} END{for (d in a)print a[d], d}' file.log | sort -nr
3 2020-07-30 22:05
2 2020-07-30 22:06
1 2020-07-30 22:04
22:05 appeared three times in the log file and is, thus, the most congested, followed by 22:06.
To get only the top most congested minutes, add head. For example:
$ awk '{sub(/:[^:]*$/, "", $3); a[$2" "$3]++} END{for (d in a)print a[d], d}' file.log | sort -nr | head -1
3 2020-07-30 22:05
Note that we select here based on the second and third fields. The presense of dates or times in the texts of log messages will not confuse this code.
How it works
sub(/:[^:]*$/, "", $3) removes everything after minutes in the third field.
a[$2" "$3]++ counts the number of times that date and time (up to minutes) appeared.
After the whole file has been read, for (d in a)print a[d], d prints out the count and date for every date observed.
sort -nr sorts the output with the highest count at the top. (Alternatively, we could have awk do the sorting but sort -nr is simple and portable.)
To sort down to the second
Instead of minutes resolution, we can get seconds resolution:
$ awk '{sub(/\.[^.]*$/, "", $3); a[$2" "$3]++} END{for (d in a)print a[d], d}' file.log | sort -nr
1 2020-07-30 22:06:40
1 2020-07-30 22:06:30
1 2020-07-30 22:05:40
1 2020-07-30 22:05:30
1 2020-07-30 22:05:20
1 2020-07-30 22:04:30
With GNU utilities:
grep -o ' [0-9][0-9]:[0-9][0-9]' file.log | sort | uniq -c | sort -nr | head -n 1
Prints
frequency HH:MM
HH:MM is the hour and minute the highest frequency occurs and frequency is the highest frequency. If you drop the | head -n 1 then you will see the list of frequencies and minutes ordered by frequencies.
I'm using uptime in bash in order to get the current runtime of the machine. I need to grab the time and display a format like 2 days, 12 hours, 23 minutes.
My uptime produces output that looks like:
$ uptime
12:49:10 up 25 days, 21:30, 28 users, load average: 0.50, 0.66, 0.52
To convert that to your format:
$ uptime | awk -F'( |,|:)+' '{print $6,$7",",$8,"hours,",$9,"minutes."}'
25 days, 21 hours, 34 minutes.
How it works
-F'( |,|:)+'
awk divides its input up into fields. This tells awk to use any combination of one or more of space, comma, or colon as the field separator.
print $6,$7",",$8,"hours,",$9,"minutes."
This tells awk to print the sixth field and seventh fields (separated by a space) followed by a comma, the 8th field, the string hours, the ninth field, and, lastly, the string minutes..
Handling computers with short uptimes using sed
Starting from a reboot, my uptime produces output like:
03:14:20 up 1 min, 2 users, load average: 2.28, 1.29, 0.50
04:12:29 up 59 min, 5 users, load average: 0.06, 0.08, 0.48
05:14:09 up 2:01, 5 users, load average: 0.13, 0.10, 0.45
03:13:19 up 1 day, 0 min, 8 users, load average: 0.01, 0.04, 0.05
04:13:19 up 1 day, 1:00, 8 users, load average: 0.02, 0.05, 0.21
12:49:10 up 25 days, 21:30, 28 users, load average: 0.50, 0.66, 0.52
The following sed command handles these formats:
uptime | sed -E 's/^[^,]*up *//; s/, *[[:digit:]]* users.*//; s/min/minutes/; s/([[:digit:]]+):0?([[:digit:]]+)/\1 hours, \2 minutes/'
With the above times, this produces:
1 minutes
59 minutes
2 hours, 1 minutes
1 day, 0 minutes
1 day, 1 hours, 0 minutes
25 days, 21 hours, 30 minutes
How it works
-E turns on extended regular expression syntax. (On older GNU seds, use -r in place of -E)
s/^[^,]*up *//
This substitutes command removes all text up to up.
s/, *[[:digit:]]* users.*//
This substitute command removes the user count and all text which follows it.
s/min/minutes/
This replaces min with minutes.
s/([[:digit:]]+):0?([[:digit:]]+)/\1 hours, \2 minutes/'
If the line contains a time in the hh:mm format, this separates the hours from the minutes and replaces it with hh hours, mm minutes.
Handling computers with short uptimes using awk
uptime | awk -F'( |,|:)+' '{d=h=m=0; if ($7=="min") m=$6; else {if ($7~/^day/) {d=$6;h=$8;m=$9} else {h=$6;m=$7}}} {print d+0,"days,",h+0,"hours,",m+0,"minutes."}'
On the same test cases as above, this produces:
0 days, 0 hours, 1 minutes.
0 days, 0 hours, 59 minutes.
0 days, 2 hours, 1 minutes.
1 days, 0 hours, 0 minutes.
1 days, 1 hours, 0 minutes.
25 days, 21 hours, 30 minutes.
For those who prefer awk code spread out over multiple lines:
uptime | awk -F'( |,|:)+' '{
d=h=m=0;
if ($7=="min")
m=$6;
else {
if ($7~/^day/) { d=$6; h=$8; m=$9}
else {h=$6;m=$7}
}
}
{
print d+0,"days,",h+0,"hours,",m+0,"minutes."
}'
Just vor completeness... what's about:
$ uptime -p
up 2 weeks, 3 days, 14 hours, 27 minutes
Solution: In order to get the linux uptime in seconds, Go to bash and type cat /proc/uptime.Parse the first number and convert it according to your requirement.
From RedHat documentation:
This file contains information detailing how long the system has been on since its last restart. The output of /proc/uptime is quite minimal:
350735.47 234388.90
The First number is the total number of seconds the system has been
up.
The Second number is how much of that time the machine has spent
idle, in
seconds.
I made a universal shell script, for systems which support uptime -p like newer linux and for those that don't, like Mac OS X.
#!/bin/sh
uptime -p >/dev/null 2>&1
if [ "$?" -eq 0 ]; then
# Supports most Linux distro
# when the machine is up for less than '0' minutes then
# 'uptime -p' returns ONLY 'up', so we need to set a default value
UP_SET_OR_EMPTY=$(uptime -p | awk -F 'up ' '{print $2}')
UP=${UP_SET_OR_EMPTY:-'less than a minute'}
else
# Supports Mac OS X, Debian 7, etc
UP=$(uptime | sed -E 's/^[^,]*up *//; s/mins/minutes/; s/hrs?/hours/;
s/([[:digit:]]+):0?([[:digit:]]+)/\1 hours, \2 minutes/;
s/^1 hours/1 hour/; s/ 1 hours/ 1 hour/;
s/min,/minutes,/; s/ 0 minutes,/ less than a minute,/; s/ 1 minutes/ 1 minute/;
s/ / /; s/, *[[:digit:]]* users?.*//')
fi
echo "up $UP"
Gist
Referenced John1024 answer with my own customizations.
For this:
0 days, 0 hours, 1 minutes.
0 days, 0 hours, 59 minutes.
0 days, 2 hours, 1 minutes.
1 days, 0 hours, 0 minutes.
1 days, 1 hours, 0 minutes.
25 days, 21 hours, 30 minutes
More simple is:
uptime -p | cut -d " " -f2-
For the sake of variety, here's an example with sed:
My raw output:
$ uptime
15:44:56 up 3 days, 22:58, 7 users, load average: 0.48, 0.40, 0.31
Converted output:
$uptime|sed 's/.*\([0-9]\+ days\), \([0-9]\+\):\([0-9]\+\).*/\1, \2 hours, \3 minutes./'
3 days, 22 hours, 58 minutes.
This answer is pretty specific for the uptime shipped in OS X, but takes into account any case of output.
#!/bin/bash
INFO=`uptime`
echo $INFO | awk -F'[ ,:\t\n]+' '{
msg = "↑ "
if ($5 == "day" || $5 == "days") { # up for a day or more
msg = msg $4 " " $5 ", "
n = $6
o = $7
} else {
n = $4
o = $5
}
if (int(o) == 0) { # words evaluate to zero
msg = msg int(n)" "o
} else { # hh:mm format
msg = msg int(n)" hr"
if (n > 1) { msg = msg "s" }
msg = msg ", " int(o) " min"
if (o > 1) { msg = msg "s" }
}
print "[", msg, "]"
}'
Some example possible outputs:
22:49 up 24 secs, 2 users, load averages: 8.37 2.09 0.76
[ ↑ 24 secs ]
22:50 up 1 min, 2 users, load averages: 5.59 2.39 0.95
[ ↑ 1 min ]
23:39 up 51 mins, 3 users, load averages: 2.18 1.94 1.74
[ ↑ 51 mins ]
23:54 up 1:06, 3 users, load averages: 3.67 2.57 2.07
[ ↑ 1 hr, 6 mins ]
16:20 up 120 days, 10:46, 3 users, load averages: 1.21 2.88 0.80
[ ↑ 120 days, 10 hrs, 46 mins ]
uptime_minutes() {
set `uptime -p`
local minutes=0
shift
while [ -n "$1" ]; do
case $2 in
day*)
((minutes+=$1*1440))
;;
hour*)
((minutes+=$1*60))
;;
minute*)
((minutes+=$1))
;;
esac
shift
shift
done
echo $minutes
}
This sed command is described as follows
Delete the cars that are $10,000 or more. Pipe the output of the sort into a sed to do this, by quitting as soon as we match a regular expression representing 5 (or more) digits at the end of a record (DO NOT use repetition for this):
So far the command is:
$ grep -iv chevy cars | sort -nk 5
I have to add another pipe at the end of that command I think which "quits as soon as we match a regular expression representing 5 or more digits at the end of a record"
I tried things like
$ grep -iv chevy cars | sort -nk 5 | sed "/[0-9][0-9][0-9][0-9][0-9]/ q"
and other variations within the // but nothing works! What is the command which matches a regular expression representing 5 or more digits and quits according to this question?
Nominally, you should add a $ before the second / to match 5 digits at the end of the record. If you omit the $, then any sequence of 5 digits will cause sed to quit, so if there is another number (a VIN, perhaps) before the price, it might match when you didn't intend it to.
grep -iv chevy cars | sort -nk 5 | sed '/[0-9][0-9][0-9][0-9][0-9]$/q'
On the whole, it's safer to use single quotes around the regex, unless you need to substitute a shell variable into it (or unless the regex contains single quotes itself). You can also specify the repetition:
grep -iv chevy cars | sort -nk 5 | sed '/[0-9]\{5,\}$/q'
The \{5,\} part matches 5 or more digits. If for any reason that doesn't work, you might find you're using GNU sed and you need to do something like sed --posix to get it working in the normal mode. Or you might be able to just remove the backslashes. There certainly are options to GNU sed to change the regex mechanism it uses (as there are with GNU grep too).
Another way.
As you don't post a file sample, a did it as a guess.
Here I'm looking for lines with the word "chevy" where the field 5 is less than 10000.
awk '/chevy/ {if ( $5 < 10000 ) print $0} ' cars
I forgot the flag -i from grep ... so the correct is:
awk 'BEGIN{IGNORECASE=1} /chevy/ {if ( $5 < 10000 ) print $0} ' cars
$ cat > cars
Chevy 2 3 4 10000
Chevy 2 3 4 5000
chEvy 2 3 4 1000
CHEVY 2 3 4 10000
CHEVY 2 3 4 2000
Prevy 2 3 4 1000
Prevy 2 3 4 10000
$ awk 'BEGIN{IGNORECASE=1} /chevy/ {if ( $5 < 10000 ) print $0} ' cars
Chevy 2 3 4 5000
chEvy 2 3 4 1000
CHEVY 2 3 4 2000
grep -iv chevy cars | sort -nk 5 | sed '/[0-9][0-9][0-9][0-9][0-9]$/d'
I need some help with parsing a string in Linux.
I have a string:
[INFO] Total time: 2 minutes 8 seconds
and want to get only
2 minutes 8 seconds
Using grep:
$ echo '[INFO] Total time: 2 minutes 8 seconds' | grep -o '[[:digit:]].*$'
2 minutes 8 seconds
Or sed:
$ echo '[INFO] Total time: 2 minutes 8 seconds' | sed 's/.*: //'
2 minutes 8 seconds
Or awk:
$ echo '[INFO] Total time: 2 minutes 8 seconds' | awk -F': ' '{print $2}'
2 minutes 8 seconds
Or cut:
$ echo '[INFO] Total time: 2 minutes 8 seconds' | cut -d: -f2
2 minutes 8 seconds
And then read sed & awk, Second Edition.
The sed and perl options do work, but in this trivial case, I'd prefer
echo "[INFO] Total time: 2 minutes 8 seconds" | cut -d: -f2
If you have something against spaces, you can just use
echo "[INFO] Total time: 2 minutes 8 seconds" | cut -d: -f2 | xargs
or even...
echo "[INFO] Total time: 2 minutes 8 seconds" | cut -d: -f2 | cut -c2-
PS. Trivia: you could do this with grep only if grep implemented positive lookbehind like this egrep -o '(?<=: ).*'; Unfortunately neither POSIX extended regex nor GNU extended regex implement lookbehind (http://www.regular-expressions.info/refflavors.html)
If the line prefix is always the same, simply use sed and replace the prefix with an empty string:
sed 's/\[INFO\] Total Time: //'
Assuming that the time is always the last thing in a line after a colon, use the following regex (replace each line with everything after the colon):
sed 's/^.*: \(.*\)$/\1/'
If you prefer AWK then it is quite simple
echo "[INFO] Total time: 2 minutes 8 seconds" | awk -F": " '{ print $2 }'
Use sed or perl:
echo "[INFO] Total time: 2 minutes 8 seconds" | sed -e 's/^\[INFO\] Total time:\s*//'
echo "[INFO] Total time: 2 minutes 8 seconds" | perl -pe "s/^\[INFO\] Total time:\s*//;"
If you are getting the info from the terminal then you can grep out the info and use cut with the delimiter to remove everything before the info you want.
grep INFO | cut -f2 -d:
If you want the info out of a file then you can grep the file
grep INFO somefilename | cut -f2 -d: