While doing system administration I often write one-liners to get the status of a process, or some value like used disk-space, number of files processed or seconds left (for example replication). I use tools like watch or echo in a loop with date to assess the status in real-time.
Often I know the outcome of the value I'm watching: it will go up to a defined number, or zero. To calculate the expected time it will be done (some processes take hours) I would put some timestamps and values in a spreadsheet, calculate some rates of increment or decrement of the value between the timestamps, and average the rate and extrapolate to estimate the expected end-time of the process.
I'm looking for a way to automate this like pv does this for a pipe. I would expect it to work something like this:
$ rate --expected-value=0 --interval=10 "mysql -e 'show slave status' -E | grep Seconds_Behind_Master | awk '{print $2}'"
Fri May 31 10:31:48 CEST 2019 | value: 52952
Fri May 31 10:31:58 CEST 2019 | value: 52918 | rate: 3.4/s | ETA: 10:57:27
Fri May 31 10:32:08 CEST 2019 | value: 52886 | rate: 3.2/s | ETA: 10:58:29
or for another example:
$ rate --unit byte --interval=1 "stat / -f -t | awk '{print $9}'"
Fri May 31 10:58:03 CEST 2019 | value: 11908091
Fri May 31 10:58:04 CEST 2019 | value: 11829190 | rate: 78900 bytes/s
These are examples of course, and the fictive rate utility does not exist. I could build it myself but I wonder if there is an existing utility (which I have not found yet) that can do this, maybe a library, or a simple one-liner that would do something similar.
Related
I am trying to convert dates from different timezones with UNIX date (I am on Ubuntu 20.04).
With current date, it works well:
$ date
dim. 12 déc. 2021 11:59:16 CET
$ TZ=Pacific/Tahiti date
dim. 12 déc. 2021 00:59:32 -10
But when I am working with a string, it fails:
$ export testdate="2021/10/28 17:47:26"
$ date -d "$test"
jeu. 28 oct. 2021 17:47:26 CEST
$ TZ=Pacific/Tahiti date -d "$test"
jeu. 28 oct. 2021 17:47:26 -10
as I am expecting:
jeu 28 oct. 2021 05:47:26 -10
I don't understand why I don't get the proper shift. And of course if I try with a date and time where the day should also change, it doesn't work either:
$ export test="2021/10/28 7:47:26"
$ date -d "$test"
jeu. 28 oct. 2021 07:47:26 CEST
$ TZ=Pacific/Tahiti date -d "$test" "+%F %T %Z"
2021-10-28 07:47:26 -10
while I am expecting:
mer 27 oct. 2021 19:47:26 -10
why I don't get the proper shift
test="2021/10/28 17:47:26"
Is a date in unknown timezone. No one knows what timezone it is in, what the daylight is. GNU date tries to "guess" what offset you meant, it generally traverses TZ database for current timezone and just picks the first offset that matches. Also, specifying timezone is not enough to know what daylight it is, you have to be specific.
Also, because of the daylight time you can "go back" in time, it's now known what the offset to UTC is even when you know the timezone.
Also, you don't have to export it - date does not care about test environment variable.
Converting date in a different timezone with date
If the input is in UTC, tell date that.
$ LC_ALL=C TZ=Pacific/Tahiti date -d "2021/10/28 17:47:26 UTC"
Thu Oct 28 07:47:26 -10 2021
If the input is with any other offset, tell date that.
$ LC_ALL=C TZ=Pacific/Tahiti date -d "2021/10/28 17:47:26 CEST"
Thu Oct 28 05:47:26 -10 2021
Te parsing of GNU date of input format is generally a mystery. The documentation lists 2004-02-29 16:21:42 format as an example input, so I recommend that format. If you want to be exact, I recommend strptime from dateutils (or a real programming language).
One simple way is to convert time first to epoch time :
test="2021/10/28 17:47:26"
TZ=Pacific/Tahiti date -d #$(date -d "$test" +%s)
date -d "$test" +%s converts local time to epoch time.
TZ=Pacific/Tahiti date -d #$(date -d "$test" +%s) prints Tahiti time from epoch time.
I'm not fully confident, but i get the impression you're looking the following syntax:
LC_TIME="es_ES.UTF8" TZ="America/New_York" date --date='TZ="Europe/Amsterdam" 2021/10/28 17:47:26' "+%A %F %T %B"
That takes a predefined datetime (interpreted as being local to Amsterdam), adjusts the datetime (based on the time difference) to New York-time at that same moment; Then it prints that result with Spanish names for the months/weekdays (provided that language' locale is present on your system).
Imagine i have a log file or a text file like below
16 Dec 2014 11:20:00 [INFO] com.example.Test.java PlaneName: JetAirways360 This flight just cleaned up
16 Dec 2014 11:22:01 [INFO] com.example.Test.java PlaneName: JetAirways360 This flight is in queue
16 Dec 2014 11:23:02 [INFO] com.example.Test.java PlaneName: JetAirways360 This flight passengers loaded
16 Dec 2014 11:24:03 [INFO] com.example.Test.State.java PlaneName: JetAirways360 This flight ready to take off
16 Dec 2014 11:25:00 [INFO] com.example.Test.State.java PlaneName: JetAirways360 This flight took off
17 Dec 2014 11:25:00 [INFO] com.example.Test.java PlaneName: JetAirways360 This flight returned back
Now imagine i this log file is fully filled off lot of flight information. it is heard to find a particular flight information. Now i would like to grep to see following details
16 Dec 2014 11:20:00 This flight just cleaned up
16 Dec 2014 11:22:01 This flight is in queue
16 Dec 2014 11:23:02 This flight passengers loaded
16 Dec 2014 11:24:03 This flight ready to take off
16 Dec 2014 11:25:00 This flight took off
17 Dec 2014 11:25:00 This flight returned back
How to do this using grep and sed command?.
This might work for you (GNU sed):
sed -rn '/JetAirWays360/s/(.{21}).{54}/\1/p' file
This saves the first part of the file in a back reference and replaces a portion of the remaining file with it.
This is simple to do with awk file
awk -F" [[]|JetAirways360 " '{print $1,$3}' file
16 Dec 2014 11:20:00 This flight just cleaned up
16 Dec 2014 11:22:01 This flight is in queue
16 Dec 2014 11:23:02 This flight passengers loaded
16 Dec 2014 11:24:03 This flight ready to take off
16 Dec 2014 11:25:00 This flight took off
17 Dec 2014 11:25:00 This flight returned back
sed -n '/\[.*JetAirways360/ s///p' YourFile
should do your job based on your sample
perl -ne 'print if s!.INFO.*Name: JetAirways360!!' file
If you prefer a new command flygrep:
#!/usr/bin/perl -n
BEGIN{ $fl = shift;}
print if s!.INFO.*Name: $fl !!
Usage: flygrep JetAirways360 file
There are many ways to do this with sed the command below also works.
sed 's#\([[:digit:]]\{2,2\}.*[[:digit:]]\{2,2\}:[[:digit:]]\{2,2\}:[[:digit:]]\{2,2\}\).*PlaneName.*JetAirways360 \(.*\)#\1 \2#g' 'my_log_file'
It marks the first two digits in the line up to the up to the flight time and remembers it. sed then searches the line for JetAirways360 and after this name it marks the remainder of the line. The two patterns are then combined.
Using grep and cut
Another easier trick to do this without using grep or sed is using "cut" (even though this might no pertain to answering your question)
cut -d ' ' -f1-4,9- 'my_log.txt'
from field (column in this case) 1 to 4 is printed and then field 9 onwards
results using "cut"
16 Dec 2014 11:20:00 This flight just cleaned up
16 Dec 2014 11:22:01 This flight is in queue
16 Dec 2014 11:23:02 This flight passengers loaded
16 Dec 2014 11:24:03 This flight ready to take off
16 Dec 2014 11:25:00 This flight took off
17 Dec 2014 11:25:00 This flight returned back
Note: this just cuts the columns you need and output no grepping nor "sedding" is involved.
You can also use grep for a Plane name which will print all lines with the Plane name and then use "cut"
example
grep "JetAirways36" 'my_log.txt'|cut -d ' ' -f1-4,9-
grep 'JetAirways360' lofFile | sed -e 's/\(.*\) \[INFO.*JetAirways360\(.*\)/\1 \2/g'
To explain this.
First i am grepping all the log entries which has JetAirways360.
I am using sed. each bracket (.) is what i am trying to get from log. \1 is for first (.) and \2 for second (.*). Its like print statements for string value i cut from original line.
Details:
sed regular expression should contain between this s/ /g
\1 and \2 are the print statements for (.*).
(.) is represented as (.). Escape sequence
s/(.*) [INFO This Explains Put all characters into the parenthesis till you see [INFO (i have put escape character \ for [ and hence [INFO). That's how i got Date selected and printed using \1
[INFO.JetAirways360(.) This explains, After seeing [INFO start ignoring till you see JetAirways360. After which put all remaining character into parenthesis. That's how i got log message selected and printed using \2
I have an output from a shell script like this:
aaa.sh output
Tue Mar 04 01:00:53 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 07:21:52 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 13:17:16 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 16:56:01 2014
SQL> ALTER DISKGROUP fra ADD DISK '/dev/rhdisk20'
Wed Mar 05 00:03:42 2014
Time drift detected. Please check VKTM trace file for more details.
Wed Mar 05 04:13:39 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 05 05:56:07 2014
GMON querying group 3 at 10 for pid 18, osid 27590856
GMON querying group 3 at 11 for pid 18, osid 27590856
I need to get the part, beginning from today's date:
Wed Mar 05 00:03:42 2014
Time drift detected. Please check VKTM trace file for more details.
Wed Mar 05 04:13:39 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 05 05:56:07 2014
GMON querying group 3 at 10 for pid 18, osid 27590856
GMON querying group 3 at 11 for pid 18, osid 27590856
You can get the date in the correct format like this:
today=$(date +'%a %b %d')
and then search for it like this:
grep "$today" aaa.sh
If there are lines from today without a date, such as your GMON lines, you could add -A to say how many lines after the match you want and use a big number:
grep -A 999999 "$today" aaa.sh
If you are on AIX and there is no -A option, use sed like this:
today=$(date +'%a %b %d')
sed -n "/${today}/,$ p" aaa.sh
Explanation:
That says store today's date in the variable today in the format "Wed Mar 05". Then search, without printing anything (-n) till you find that date, From that point on, till the end of file ($) print all lines (p).
I think I have an easy solution:
Get date to output the date in a format that would match the date in the file (check man date on formatting options). Since we don't want to match the hours/minutes/seconds we have to call date twice: once for the weekday/month/day half and once for the year half on the end of the full date. Between these two halves we match the horus/minutes/seconds with .* regex.
Then do:
aaa.sh | grep -E '`date --only-weekday-month-day`.*`date --only-year`' -A 999999
though I am using answer by NewWorld it can be modified as,
convert output of date similar to your file format
suppose in variable 'D'you get that output
sed '1,/${D}/d' aaa.sh
that will output all lines after match date match.
example: suppose you get D="Wed Mar 05 00:03:42 2014"
output will be as expected.
You can use
tail -n 7 filename
for getting the desired output . It will basically give you the last seven lines of the text file named filename .
For getting solution from today's date you can use :
k=$(date +"%a %b %d")
g=$(grep -nr "$k" in|cut -f1 -d:|head -1)
total=$(wc -l<in)
l=`expr $total - $g + 1
tail -n$l in
Try
sed -n '/Wed Mar 05/,$p' aaa.sh
Here -n means "don't print anything unless specified to".
First appearance of a line that matches the expression /Wed\ Mar\ 05/ till the end of the file, will be printed(p)"
I want to do some calendar manipulations in bash - specifically, I want to figure out the last date of a given month (including leap-year, and a preparing a table for a lookup is not a valid solution for me).
Supposedly I have the following code:
$year=2009
$start_month=2
$end_month=10
for $month in $(seq $start_month $end_month); do
echo "Last date of "$(date +"%B" -d "${year}-${month}-01")" is: " ???
done
I can't figure out how to do something like this. I though date -d would work like POSIX mktime and fold invalid dates to their valid equivalents, so I could say something like date -d "2009-03-00" and get '2009-02-28', but no such luck.
Is there anyway to do it using only what is available on bash in a standard GNU environment?
Try: date -d 'yesterday 2009-03-01'
Intuitive I know. Earlier versions of date used to work the POSIX way.
date(1)'s -d is GNU specific; so using that will only work on GNU Linux.
A more portable solution (this should even work in sh AFAIK), is this:
: $(cal 4 2009); echo $_
If you don't mind playing with grep/awk/perl, you can take a look at cal.
$ cal 4 2009
April 2009
Su Mo Tu We Th Fr Sa
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30
Edit (MarkusQ): To atone for my joke solution below I'll contribute to yours:
cal 4 2009 | tr ' ' '\n' | grep -v ^$ | tail -n 1
Well, one way would be to watch the current date in a loop until the month component changes, saving the day component for one round. That would give you both the first and last day of the month, but it might be too slow.
Posted 1 April 2009
I want to view the ORA errors in alertlogfile of past 7 (monday-sunday)days,
by writting in shell scripts.
Can anybody help me.
Thanks
Something like:
sed -n -e '/start_time/,/end_time/ {/ORA/ p}' logfile
or with awk
$ start="Fri Feb 27 08:00:00 2009"
$ end="Fri Mar 6 08:00:00 2009"
$ awk -v prev="$start" -v last="$end" '$0 ~ prev,$0 ~ last' logfile
A more sophisticated script looking for last date entries in ORA file is available here, but also at dba-oracle.com
This does not answer exactly your request but might give you some clues to start your own script.
I want the scripts which give output as follows (one week errors) and it should be mail to my id.
Sat Mar 14 10:30:51 IST 2009
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
Sat Mar 12 12:35:06 IST 2009
ORA-01110: data file 2: '/u02/oradata/Globe/undotbs01.dbf'
Sat Mar 10 09:54:05 IST 2009
ORA-27037: unable to obtain file status
Sat Mar 08 :15:02 IST 2009
ORA-1157 signalled during: ALTER DATABASE OPEN...
Sat Mar 07 12:35:51 IST 2009
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file