Would it be possible to give the input to a very long command in the end of the command ? Below example would explain my query more clearly.
currently while grepping I have to do like below:
zgrep -i "A VERY LONG TEXT" file |awk '{print $1}'
Every time I have to move the cursor back to the "A VERY LONG TEXT" to change the pattern.I wanted to alter the command in such a way that "A VERY LONG TEXT" comes in the end of the command so I can quickly change it.
command1 |command2 |some_magic "A VERY LONG TEXT"
I know I can achieve this result by doing CAT and then grepping ,wondering if there is any alternate way to do this. May be like assigning it to a temp variable?
EXAMPLE 2:
I need to get real time time stamp of all the commands and their output in my session files.So I have to use below command. But before executing any command I have to move my cursor till unbuffer and change the commands. Is there is any way I can alter the below command such that I can enter my commands in the end of the line ?
/yumm 194>unbuffer ls -lrt | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }'
2014-10-01 10:38:19 total 0
2014-10-01 10:38:19 -rw-rw-r-- 1 user bcam 0 Oct 1 10:37 1
2014-10-01 10:38:19 -rw-rw-r-- 1 user bcam 0 Oct 1 10:38 test1
2014-10-01 10:38:19 -rw-rw-r-- 1 user bcam 0 Oct 1 10:38 test2
2014-10-01 10:38:19 -rw-rw-r-- 1 user bcam 0 Oct 1 10:38 test3
2014-10-01 10:38:19 -rw-rw-r-- 1 user bcam 0 Oct 1 10:38 test4
yumm 195>
In short, I need some command to get time stamp of all the commands and their output I execute.
What if you just set this text to a variable?
mystring="A VERY LONG TEXT"
zgrep -i "$mystring" file | awk '{print $1}'
^ ^
note you need double quotes to make it work
Based on your edit, you can also do:
awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; }' <<< "$(unbuffer ls -ltr)"
^^^^^^^^^^^^^^^^^^^^^^^^^
When editing and re-submitting a command, use:
Ctrl-A to move the cursor back to the start of the line quickly
Ctrl-E to move to the end of the line quickly
Alt-F to move forwards one word
Alt-B to move backwards one word
Or use fc command to open the last command in the editor and allow you to edit - say with vi commands, and then when you save it, it gets re-submitted for execution.
These shortcut key maybe help you.
Ctrl-a: move to beginning of line
Ctrl-e: move to end of line
Ctrl-b: move to previous character
Ctrl-f: move to next character
Ctrl-p: previous command (same as "UP" key in bash history)
Ctrl-n: next command (same as "DOWN" key in bash history)
Ctrl-h: Delete backward character
Ctrl-d: Delete current cursored character
Ctrl-k: Delete characters after cursor
You can easily image if you know Emacs editor.
These shortcut is same key bind as same as emacs.
You can define a function, e.g., in your ~/.bash_profile:
some_magic() {
zgrep "$1" file | awk '{print $1}'
}
And use it the following way:
some_magic "A VERY LONG TEXT"
As for your second example, as soon as a command output is piped, it gets buffered by the pipe. Therefore the timestamp acquired at the other side of the pipe is wrong. Anyway, if you don't mind having a wrong timestamp, you can use this function:
some_other_magic() {
$1 | awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0}'
}
And use it the following way:
some_other_magic "ls -lrt"
Related
I'm working now on a project. In this project for some reasons I need to exclude first string from the output (or file) that matches the pattern. The difficulty is in that I need to exclude just one string, just first string from the stream.
For example, if I have:
1 abc
2 qwerty
3 open
4 abc
5 talk
After some script working I should have this:
2 qwerty
3 open
4 abc
5 talk
NOTE: I don't know anything about digits before words, so I can't filter the output using knowledge about them.
I've written small script with grep, but it cuts out every string, that matches the pattern:
'some program' | grep -v "abc"
Read info about awk, sed, etc. but didn't understand if I can solve my problem.
Anything helps, Thank you.
Using awk:
some program | awk '{ if (/abc/ && !seen) { seen = 1 } else print }'
Alternatively, using only filters:
some program | awk '!/abc/ || seen { print } /abc/ && !seen { seen = 1 }'
You can use Ex editor. For example to remove the first pattern from the file:
ex +"/abc/d" -scwq file.txt
From the input (replace cat with your program):
ex +"/abc/d" +%p -scq! <(cat file.txt)
You can also read from stdin by replacing cat with /dev/stdin.
Explanation:
+cmd - execute Ex/Vim command
/pattern/d - find the pattern and delete,
%p - print the current buffer
-s - silent mode
-cq! - execute quite without saving (!)
<(cmd) - shell process substitution
give line numbers using sed which you want to delete
sed 1,2d
instead of 1 2 use line numbers that you want to delete
otherwise you can use
sed '/pattrent to match/d'
here we can have
sed '0,/abc/{//d;}'
You can also use a list of commands { list; } to read the first line and print the rest:
command | { read first_line; cat -; }
Simple example:
$ cat file
1 abc
2 qwerty
3 open
4 abc
5 talk
$ cat file | { read first_line; cat -; }
2 qwerty
3 open
4 abc
5 talk
awk '!/1/' file
2 qwerty
3 open
4 abc
5 talk
Thats all!
How to cut a specific field from a line?
The problem is I can't use cut -d ' ' -f 1,2,3,4,5,9,10,11,12,13,14, since the field changes.
Let's say I have a file called /var/log/test, and one of the lines inside the file looks like this :
Apr 12 07:48:11 172.89.92.41 %ASA-5-713120: Group = People, Username = james.robert, IP = 219.89.259.32, PHASE 2 COMPLETED (msgid=9a4ce822)
I only need to get the Username and Time/Date ( please note columns keep changing, that's why I need to match the Username = james.robert and Apr 12 07:48:11
When I use :
grep "james" /var/log/tes | cut -d ' ' -f 1,2,3,4,5,9,10,11,12,13,14
Doesn't work for me. So it has to match the username and prints only username and data/time. Any suggestions?
Ok so when I use this :
awk -F'[ ,]' '$12~/username/{print $1,$2,$3,$12}' /var/log/test
but it works for some users, but not the others, because fields keep moving.
The sample output of this command is :
Apr 12 06:00:39 james.robert
But when I try this command on this username, it doesn't work. See below :
here is another example that with the above command doesn't show anything:
Apr 8 12:16:13 172.24.32.1 %ASA-6-713228: Group = people, Username = marry.tarin, IP = 209.157.190.11, Assigned private IP address 192.168.237.38 to remote user
if your file is structured consistently
awk -F'[ ,]' '{print $1,$2,$3,$12}' file
Apr 12 07:48:11 james.robert
if you need to match the username, using your sample input
$ awk -F'[ ,]' '$12~/james/{print $1,$2,$3,$12}' file
Apr 12 07:48:11 james.robert
UPDATE
OK, your spaces are not consistent, to fix change the -F
$ awk -F' +|,' '{print $1,$2,$3,$12}' file
Apr 12 07:48:11 james.robert
Apr 8 12:16:13 marry.tarin
you can add the /pattern/ to restrict the match to users as above. Note the change in -F option.
-F' +|,' sets the field separator to spaces (one or more) or comma,
the rest is counting the fields and picking up the right one to print.
/pattern/ will filter the lines that matches the regex pattern, which can > be constrained to certain field only (e.g. 12) by $12~/pattern/
if your text may contain mixed case and you want to be case insensitive, use tolower() function, for example
$ awk -F' +|,' 'tolower($12)~/patterninlowercase/{print $1,$2,$3,$12}' file
With sed:
sed -r 's/^([A-Za-z]{3} [0-9]{1,2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*(Username = [^,]*).*/\1 \2/g' file
You could use awk to delimit by comma and then use substr() and length() to get at the pieces you care about:
awk -F"," '{print substr($1,1,15), substring($3, 13, length($3)-12)}' /var/log/test
With gawk
awk '{u=gensub(/.*(Username = [^,]*).*/,"\\1","g",$0);if ( u ~ "james") {print u,$1,$2,$3}}' file
The following perl will print the date and username delimited by a tab. Add additional valid username characters to [\w.]:
perl -ne '
print $+{date}, "\t", $+{user}, "\n" if
/^(?<date>([^\s]+\s+){2}[^\s]+).*\bUsername\s*=\s*(?<user>[\w.]+)/
'
Varying amounts a tabs and spaces are allowed.
I got a text file file.txt (12 MB) containing:
something1
something2
something3
something4
(...)
Is there a way to split file.txt into 12 *.txt files, let’s say file2.txt, file3.txt, file4.txt, etc.?
You can use the Linux Bash core utility split:
split -b 1M -d file.txt file
Note that M or MB both are OK but size is different. MB is 1000 * 1000, M is 1024^2
If you want to separate by lines you can use -l parameter.
UPDATE
a=(`wc -l yourfile`) ; lines=`echo $(($a/12)) | bc -l` ; split -l $lines -d file.txt file
Another solution as suggested by Kirill, you can do something like the following
split -n l/12 file.txt
Note that is l not one, split -n has a few options, like N, k/N, l/k/N, r/N, r/k/N.
$ split -l 100 input_file output_file
where -l is the number of lines in each files. This will create:
output_fileaa
output_fileab
output_fileac
output_filead
....
CS Pei's answer won't produce .txt files as the OP wants. Use:
split -b=1M -d file.txt file --additional-suffix=.txt
Using Bash:
readarray -t lines < file.txt
count=${#lines[#]}
for i in "${!lines[#]}"; do
index=$(( (i * 12 - 1) / count + 1 ))
echo "${lines[i]}" >> "file${index}.txt"
done
Using AWK:
awk '{
a[NR] = $0
}
END {
for (i = 1; i in a; ++i) {
x = (i * 12 - 1) / NR + 1
sub(/\..*$/, "", x)
print a[i] > "file" x ".txt"
}
}' file.txt
Unlike split, this one makes sure that the number of lines are most even.
Regardless to what was said in previous answers, on my Ubuntu 16.04 (Xenial Xerus) I had to do:
split -b 10M -d system.log system_split.log
Please note the space between -b and the value.
I agree with #CS Pei, however this didn't work for me:
split -b=1M -d file.txt file
...as the = after -b threw it off. Instead, I simply deleted it and left no space between it and the variable, and used lowercase "m":
split -b1m -d file.txt file
And to append ".txt", we use what #schoon said:
split -b=1m -d file.txt file --additional-suffix=.txt
I had a 188.5MB txt file and I used this command [but with -b5m for 5.2MB files], and it returned 35 split files all of which were txt files and 5.2MB except the last which was 5.0MB. Now, since I wanted my lines to stay whole, I wanted to split the main file every 1 million lines, but the split command didn't allow me to even do -100000 let alone "-1000000, so large numbers of lines to split will not work.
Try something like this:
awk -vc=1 'NR%1000000==0{++c}{print $0 > c".txt"}' Datafile.txt
for filename in *.txt; do mv "$filename" "Prefix_$filename"; done;
On my Linux system (Red Hat Enterprise 6.9), the split command does not have the command-line options for either -n or --additional-suffix.
Instead, I've used this:
split -d -l NUM_LINES really_big_file.txt split_files.txt.
where -d is to add a numeric suffix to the end of the split_files.txt. and -l specifies the number of lines per file.
For example, suppose I have a really big file like this:
$ ls -laF
total 1391952
drwxr-xr-x 2 user.name group 40 Sep 14 15:43 ./
drwxr-xr-x 3 user.name group 4096 Sep 14 15:39 ../
-rw-r--r-- 1 user.name group 1425352817 Sep 14 14:01 really_big_file.txt
This file has 100,000 lines, and I want to split it into files with at most 30,000 lines. This command will run the split and append an integer at the end of the output file pattern split_files.txt..
$ split -d -l 30000 really_big_file.txt split_files.txt.
The resulting files are split correctly with at most 30,000 lines per file.
$ ls -laF
total 2783904
drwxr-xr-x 2 user.name group 156 Sep 14 15:43 ./
drwxr-xr-x 3 user.name group 4096 Sep 14 15:39 ../
-rw-r--r-- 1 user.name group 1425352817 Sep 14 14:01 really_big_file.txt
-rw-r--r-- 1 user.name group 428604626 Sep 14 15:43 split_files.txt.00
-rw-r--r-- 1 user.name group 427152423 Sep 14 15:43 split_files.txt.01
-rw-r--r-- 1 user.name group 427141443 Sep 14 15:43 split_files.txt.02
-rw-r--r-- 1 user.name group 142454325 Sep 14 15:43 split_files.txt.03
$ wc -l *.txt*
100000 really_big_file.txt
30000 split_files.txt.00
30000 split_files.txt.01
30000 split_files.txt.02
10000 split_files.txt.03
200000 total
If each part has the same number of lines, for example 22, here is my solution:
split --numeric-suffixes=2 --additional-suffix=.txt -l 22 file.txt file
And you obtain file2.txt with the first 22 lines, file3.txt the 22 next line, etc.
Thank #hamruta-takawale, #dror-s and #stackoverflowuser2010
My search of how to do this led me here, so I'm posting this here for others too:
To get all of the contents of the file, split is the right answer! But, for those looking to just extract a piece of a file, as a sample of the file, use head or tail:
# extract just the **first** 100000 lines of /var/log/syslog into
# ~/syslog_sample.txt
head -n 100000 /var/log/syslog > ~/syslog_sample.txt
# extract just the **last** 100000 lines of /var/log/syslog into
# ~/syslog_sample.txt
tail -n 100000 /var/log/syslog > ~/syslog_sample.txt
I'm trying to write a script, In this script i'm passing a shell variable into an awk command, But when i run it nothing happens, i tried to run that line only in the shell, i found that no variable expansion happened like i expected. Here's the code :
1 #!/bin/bash
2
3 # Created By Rafael Adel
4
5 # This script is to start dwm with customizations needed
6
7
8 while true;do
9 datestr=`date +"%r %d/%m/%Y"`
10 batterystr=`acpi | grep -oP "([a-zA-Z]*), ([0-9]*)%"`
11 batterystate=`echo $batterystr | grep -oP "[a-zA-Z]*"`
12 batterypercent=`echo $batterystr | grep -oP "[0-9]*"`
13
14 for nic in `ls /sys/class/net`
15 do
16 if [ -e "/sys/class/net/${nic}/operstate" ]
17 then
18 NicUp=`cat /sys/class/net/${nic}/operstate`
19 if [ "$NicUp" == "up" ]
20 then
21 netstr=`ifstat | awk -v interface=${nic} '$1 ~ /interface/ {printf("D: %2.1fKiB, U: %2.1fKiB",$6/1000, $8/1000)}'`
22 break
23 fi
24 fi
25 done
26
27
28 finalstr="$netstr | $batterystr | $datestr"
29
30 xsetroot -name "$finalstr"
31 sleep 1
32 done &
33
34 xbindkeys -f /etc/xbindkeysrc
35
36 numlockx on
37
38 exec dwm
This line :
netstr=`ifstat | awk -v interface=${nic} '$1 ~ /interface/ {printf("D: %2.1fKiB, U: %2.1fKiB",$6/1000, $8/1000)}'`
Is what causes netstr variable not to get assigned at all. That's because interface is not replaced with ${nic} i guess.
So could you tell me what's wrong here? Thanks.
If you want to /grep/ with your variable, you have 2 choices :
interface=eth0
awk "/$interface/{print}"
or
awk -v interface=eth0 '$0 ~ interface{print}'
See http://www.gnu.org/software/gawk/manual/gawk.html#Using-Shell-Variables
it's like I thought, awk substitutes variables properly, but between //, inside regex ( or awk regex, depending on some awk parameter AFAIR), awk variable cannot be used for substitution
I had no issue grepping with variable inside an awk program (for simple regexp cases):
sawk1='repo\s+module2'
sawk2='#project2\s+=\s+module2$'
awk "/${sawk1}/,/${sawk2}/"'{print}' aFile
(Here the /xxx/,/yyy/ displays everything between xxx and yyy)
(Note the double-quoted "/${sawk1}/,/${sawk2}/", followed by the single-quoted '{print}')
This works just fine, and comes from "awk: Using Shell Variables in Programs":
A common method is to use shell quoting to substitute the variable’s value into the program inside the script.
For example, consider the following program:
printf "Enter search pattern: "
read pattern
awk "/$pattern/ "'{ nmatches++ }
END { print nmatches, "found" }' /path/to/data
The awk program consists of two pieces of quoted text that are concatenated together to form the program.
The first part is double-quoted, which allows substitution of the pattern shell variable inside the quotes.
The second part is single-quoted.
It does add the caveat though:
Variable substitution via quoting works, but can potentially be messy.
It requires a good understanding of the shell’s quoting rules (see Quoting), and it’s often difficult to correctly match up the quotes when reading the program.
I need to get a row based on column value just like querying a database. I have a command output like this,
Name ID Mem VCPUs State
Time(s)
Domain-0 0 15485 16 r-----
1779042.1
prime95-01 512 1
-b---- 61.9
Here I need to list only those rows where state is "r". Something like this,
Domain-0 0 15485 16
r----- 1779042.1
I have tried using "grep" and "awk" but still I am not able to succeed.
Any help me is much appreciated
Regards,
Raaj
There is a variaty of tools available for filtering.
If you only want lines with "r-----" grep is more than enough:
command | grep "r-----"
Or
cat filename | grep "r-----"
grep can handle this for you:
yourcommand | grep -- 'r-----'
It's often useful to save the (full) output to a file to analyse later. For this I use tee.
yourcommand | tee somefile | grep 'r-----'
If you want to find the line containing "-b----" a little later on without re-running yourcommand, you can just use:
grep -- '-b----' somefile
No need for cat here!
I recommend putting -- after your call to grep since your patterns contain minus-signs and if the minus-sign is at the beginning of the pattern, this would look like an option argument to grep rather than a part of the pattern.
try:
awk '$5 ~ /^r.*/ { print }'
Like this:
cat file | awk '$5 ~ /^r.*/ { print }'
grep solution:
command | grep -E "^([^ ]+ ){4}r"
What this does (-E switches on extended regexp):
The first caret (^) matches the beginning of the line.
[^ ] matches exactly one occurence of a non-space character, the following modifier (+) allows it to also match more occurences.
Grouped together with the trailing space in ([^ ]+ ), it matches any sequence of non-space characters followed by a single space. The modifyer {4} requires this construct to be matched exactly four times.
The single "r" is then the literal character you are searching for.
In plain words this could be written like "If the line starts <^> with four strings that are followed by a space <([^ ]+ ){4}> and the next character is , then the line matches."
A very good introduction into regular expressions has been written by Jan Goyvaerts (http://www.regular-expressions.info/quickstart.html).
Filtering by awk cmd in linux:-
Firstly find the column for this cmd and store file2 :-
awk '/Domain-0 0 15485 /' file1 >file2
Output:-
Domain-0 0 15485 16
r----- 1779042.1
after that awk cmd in file2:-
awk '{print $1,$2,$3,$4,"\n",$5,$6}' file2
Final Output:-
Domain-0 0 15485 16
r----- 1779042.1