I got a log from remote linux computer. It looks like:
2013-10-23T08:19:05+0300 Last login: Wed Oct 23 08:17:38 EEST 2013 from 10.9.167.55 on pts/0
2013-10-23T08:19:05+0300 Last login: Wed Oct 23 08:19:05 2013 from 10.9.167.55^M
2013-10-23T08:19:07+0300 ^[[?1034h-bash-4.1$ date
2013-10-23T08:19:07+0300 Wed Oct 23 08:19:07 EEST 2013
2013-10-23T08:19:08+0300 -bash-4.1$ ls
2013-10-23T08:19:08+0300 ^[[0m^[[01;34m99^[[0m #avail.info ^[[01;34mgmoTemp^[[0m raml21.dtd SNMP4JTestAgentBC.cfg
2013-10-23T08:19:08+0300 an_mainHost_localhost_20131023081654000136.xml #avail.info~ gsh.txt ^[[01;34mresults^[[0m
2013-10-23T08:19:09+0300 ^[[m-bash-4.1$ exit
2013-10-23T08:19:09+0300 logout
But it should be:
Last login: Wed Oct 23 08:17:38 EEST 2013 from 10.9.167.55 on pts/0
Last login: Wed Oct 23 08:19:05 2013 from 10.9.167.55
-bash-4.1$ date
Wed Oct 23 08:19:07 EEST 2013
-bash-4.1$ ls
99 #avail.info gmoTemp raml21.dtd SNMP4JTestAgentBC.cfg
an_mainHost_localhost_20131023081654000136.xml #avail.info~ gsh.txt results
-bash-4.1$ exit
logout
The messy codes are terminal control escape sequences, you can use command "infocmp xterm" and "man terminfo" to get more details.
My question is how can I remove these terminal control escape sequences in the file?
Thanks a lot!
Simple way to remove most parts of the control character is using the command below in vim:
:%s/<escape-key>\[[0-9;]*m/ /g
Press Ctrl+V followed by esc-key for the <escape-key> character above. Everything else is the same literal key as in your keyboard.
i use a pipe or direct sed like this
sed 's/[^[:print:]]\[[^a-zA-Z]*[a-zA-Z]//g' YourFile
I solved this issue using lots of regular expressions according to http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
Related
I have an output from a shell script like this:
aaa.sh output
Tue Mar 04 01:00:53 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 07:21:52 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 13:17:16 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 04 16:56:01 2014
SQL> ALTER DISKGROUP fra ADD DISK '/dev/rhdisk20'
Wed Mar 05 00:03:42 2014
Time drift detected. Please check VKTM trace file for more details.
Wed Mar 05 04:13:39 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 05 05:56:07 2014
GMON querying group 3 at 10 for pid 18, osid 27590856
GMON querying group 3 at 11 for pid 18, osid 27590856
I need to get the part, beginning from today's date:
Wed Mar 05 00:03:42 2014
Time drift detected. Please check VKTM trace file for more details.
Wed Mar 05 04:13:39 2014
Time drift detected. Please check VKTM trace file for more details.
Tue Mar 05 05:56:07 2014
GMON querying group 3 at 10 for pid 18, osid 27590856
GMON querying group 3 at 11 for pid 18, osid 27590856
You can get the date in the correct format like this:
today=$(date +'%a %b %d')
and then search for it like this:
grep "$today" aaa.sh
If there are lines from today without a date, such as your GMON lines, you could add -A to say how many lines after the match you want and use a big number:
grep -A 999999 "$today" aaa.sh
If you are on AIX and there is no -A option, use sed like this:
today=$(date +'%a %b %d')
sed -n "/${today}/,$ p" aaa.sh
Explanation:
That says store today's date in the variable today in the format "Wed Mar 05". Then search, without printing anything (-n) till you find that date, From that point on, till the end of file ($) print all lines (p).
I think I have an easy solution:
Get date to output the date in a format that would match the date in the file (check man date on formatting options). Since we don't want to match the hours/minutes/seconds we have to call date twice: once for the weekday/month/day half and once for the year half on the end of the full date. Between these two halves we match the horus/minutes/seconds with .* regex.
Then do:
aaa.sh | grep -E '`date --only-weekday-month-day`.*`date --only-year`' -A 999999
though I am using answer by NewWorld it can be modified as,
convert output of date similar to your file format
suppose in variable 'D'you get that output
sed '1,/${D}/d' aaa.sh
that will output all lines after match date match.
example: suppose you get D="Wed Mar 05 00:03:42 2014"
output will be as expected.
You can use
tail -n 7 filename
for getting the desired output . It will basically give you the last seven lines of the text file named filename .
For getting solution from today's date you can use :
k=$(date +"%a %b %d")
g=$(grep -nr "$k" in|cut -f1 -d:|head -1)
total=$(wc -l<in)
l=`expr $total - $g + 1
tail -n$l in
Try
sed -n '/Wed Mar 05/,$p' aaa.sh
Here -n means "don't print anything unless specified to".
First appearance of a line that matches the expression /Wed\ Mar\ 05/ till the end of the file, will be printed(p)"
I have a script that basically runs tmux ls:
session1: 3 windows (created Fri Sep 20 13:16:13 2013) [157x56]
stuff: 3 windows (created Fri Sep 20 13:25:21 2013) [157x56]
asdf: 2 windows (created Sun Sep 29 23:06:33 2013) [77x17] (attached)
website: 1 windows (created Tue Sep 24 17:22:14 2013) [157x26]
I want the output of this script to be aligned at the colon, for readability. I know to use column -t but it doesn't quite do what I want (note the double spacing, and that the colons aren't actually aligned):
session1: 3 windows (created Fri Sep 20 13:16:13 2013) [157x56]
stuff: 3 windows (created Fri Sep 20 13:25:21 2013) [157x56]
asdf: 2 windows (created Sun Sep 29 23:06:33 2013) [77x17] (attached)
website: 1 windows (created Tue Sep 24 17:22:14 2013) [157x26]
Here's the output I really want:
session1 : 3 windows (created Fri Sep 20 13:16:13 2013) [157x56]
stuff : 3 windows (created Fri Sep 20 13:25:21 2013) [157x56]
asdf : 2 windows (created Sun Sep 29 23:06:33 2013) [77x17] (attached)
website : 1 windows (created Tue Sep 24 17:22:14 2013) [157x26]
What's the easiest/best way to achieve this in the Linux shell?
Edit: If you want to test your answer, you can use curl -s nylen.tv/tmux.txt instead of tmux ls.
With GNU sed you can specify the occurrence of the match so where s/ +/ /g would single space the whole file s/ +/ /2g single spaces the whole file but after the second match on each line:
$ column -t file | sed -re 's/: ( +)/\1: /' -e 's/ +/ /2g'
session1 : 3 windows (created Fri Sep 20 13:16:13 2013) [157x56]
stuff : 3 windows (created Fri Sep 20 13:25:21 2013) [157x56]
asdf : 2 windows (created Sun Sep 29 23:06:33 2013) [77x17] (attached)
website : 1 windows (created Tue Sep 24 17:22:14 2013) [157x26]
To sort out the first columns spacing we use s/: ( +)/\1: / to match all the spaces from the : and we store n-1 spaces in the first capture group. We then replace the : and all the spaces with n-1 spaces followed by a : followed by a space (making n total space again).
How about this?
cat tmux.txt | sed 's/:/ : /' | column -t -o' ' | sed 's/ \+/ /2g'
session1 : 3 windows (created Fri Sep 20 13:16:13 2013) [157x56]
stuff : 3 windows (created Fri Sep 20 13:25:21 2013) [157x56]
asdf : 2 windows (created Sun Sep 29 23:06:33 2013) [77x17] (attached)
website : 1 windows (created Tue Sep 24 17:22:14 2013) [157x26]
I have a timestamped log file that I am watching. Every so often the log writes a message, "OK"
How could I capture the number of "OK"s written per second? E.g. output like
10.5 OK/sec
9 OK/sec
20 OK/sec
The log is pretty big, its not practical to do a grep on the entire file. I started with
tail -f my.log | grep OK
which is helpful but doesn't quite get me the rate.
Thanks!
First solution below is great (and other is good too), I altered it a bit and this is what I came up with
tail -f my.log | perl -lne 'if(/(\d+:\d+:\d+).*OK/) { print $1; }'| uniq -c
When your log is in form like:
Wed Jul 10 22:44:00 CEST 2013 OK
Wed Jul 10 22:44:00 CEST 2013 OK
Wed Jul 10 22:44:00 CEST 2013 OK
Wed Jul 10 22:44:00 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
Wed Jul 10 22:44:01 CEST 2013 OK
so, in form timestamp OK, you can try the simple:
tail -f logfile | uniq -c
and you will get the next result:
4 Wed Jul 10 22:44:00 CEST 2013 OK
8 Wed Jul 10 22:44:01 CEST 2013 OK
^
+---- number of OK in the same timestamp
Here's a quick script to do it:
#! /bin/bash
log="/var/log/messages"
seconds="1"
filter="OK"
while true; do
echo "$(timeout $seconds tail -n0 -f $log|grep $filter|wc -l) OK/sec"
done
There's a bit of a race condition where it might lose one or two lines during the loop, but it is unlikely. Also, not every Linux distro has timeout installed by default, so you might want to check.
I'm running rdiff-backup to backup some folders on a remote system.
rdiff-backup root#<REMOTESERVER>::/apps/myapp/shared/system /home/backups/system
echo "$(date): Completed... removing backup data older than 4 weeks"
rdiff-backup --remove-older-than 4W /home/backups/system
echo "$(date): Completed..."
Running this leads to errors though during the --remove-older-than command:
Fatal Error: Found 81 relevant increments, dated:
Wed Aug 29 00:41:47 2012
Thu Aug 30 00:57:49 2012
Fri Aug 31 01:04:03 2012
Sat Sep 1 00:44:59 2012
Sun Sep 2 00:41:49 2012
Mon Sep 3 00:41:39 2012
If you want to delete multiple increments in this way, use the --force.
So yeah, I can probably just put --force on it, but I'd like to understand what's going on here and haven't been able to find much on this problem.
Any ideas?
It's just rdiff-backup making sure you really mean it. The usual workflow for --remove-older-than is to run regularly, removing just the oldest increment. In order to avoid data-loss, rdiff-backup requires you to be really sure before letting you remove multiple increments at once.
I want to view the ORA errors in alertlogfile of past 7 (monday-sunday)days,
by writting in shell scripts.
Can anybody help me.
Thanks
Something like:
sed -n -e '/start_time/,/end_time/ {/ORA/ p}' logfile
or with awk
$ start="Fri Feb 27 08:00:00 2009"
$ end="Fri Mar 6 08:00:00 2009"
$ awk -v prev="$start" -v last="$end" '$0 ~ prev,$0 ~ last' logfile
A more sophisticated script looking for last date entries in ORA file is available here, but also at dba-oracle.com
This does not answer exactly your request but might give you some clues to start your own script.
I want the scripts which give output as follows (one week errors) and it should be mail to my id.
Sat Mar 14 10:30:51 IST 2009
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
Sat Mar 12 12:35:06 IST 2009
ORA-01110: data file 2: '/u02/oradata/Globe/undotbs01.dbf'
Sat Mar 10 09:54:05 IST 2009
ORA-27037: unable to obtain file status
Sat Mar 08 :15:02 IST 2009
ORA-1157 signalled during: ALTER DATABASE OPEN...
Sat Mar 07 12:35:51 IST 2009
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file