I have a list like the following:
Name_JR_1
1.1.1.1
Name_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3
If I want to chose all associated name with following numerical syntax I can look for a pattern and print the matching line plus the after context or next line using the -A1 option, as follows:
grep "JR" -A1 file_name
and this will print what I want:
Name_JR_1
1.1.1.1
I need a way to invert this however, where I can REMOVE all entries which match the search pattern. However using the -v option with this syntax doesen't give me the results I want:
grep -v "JR" -A1 file_name
What I want the output to be like after this command is as follows:
Names_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3.
Try:
$ awk '/JR/{getline;next} 1' file
Name_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3
How it works
/JR/{getline;next}
This selects the lines containing JR. For those lines, this instructs awk to get the next line (getline) and then to skip the rest of the commands and start over on the following line (next).
1
For any lines that don't have JR in them, this prints the line.
Another awk:
$ awk '/JR/||f==1{f=!f;next}1' file
Name_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3
If we see JR or flag is up, reverse the flag and skip to next line.
You can also use sed
$ sed '/JR/{N;d;}' ip.txt
Name_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3
N will add next line to pattern space and then d will delete it
use N;N for two more lines, N;N;N for three more lines and so on
For a generic solution with awk
$ awk '/JR/{c=2} !(c && c--)' ip.txt
Name_SR_1
2.2.2.2
Name_NONE_1
3.3.3.3
here 2 is count of matching line and one line after
so for -A2 equivalent, you'd need c=3
See Printing with sed or awk a line following a matching pattern for plenty of cases related to this
Related
I am trying to number the lines of a txt file and hide the empty ones . I use this code :
cat -n file.txt | grep . file.txt
But it doesnt work . It ignores the cat command . I want to display all the non-empty lines and number them ( the txt file is not a static one , like a list that a user can type in ).
edit : Given the great solutions below , i would also add that grep . file.txt | cat -n also worked .
I assume you want to number the lines that remain after the empty lines are removed.
Solution #1
Use sed '/^$/d' to delete the empty lines then pipe its output to cat -n to number them:
sed '/^$/d' file.txt | cat -n
The sed program contains only one command: d (delete the line). The sed commands can be prefixed by zero, one or two addresses that tell what lines the command applies to.
In this case there is only one address /^$/. It is a regex (enclosed in /) that selects the empty lines; the lines where start of the line (^) is followed by the end of the line ($).
Solution #2
You can also use grep -v '^$' to filter out the empty lines:
grep -v '^$' file.txt | cat -n
Again, ^$ is a regular expression that matches the empty lines. -v reverses the condition and tells grep to display the lines that do not match the regex.
The commands above do not modify the file. They read the content of file.txt, process it and display the result on screen.
Update
As #robc suggests in a comment, nl is even better than cat -n to number the lines. Thank you #robc, I didn't know about nl until now (I didn't know about cat -n either). It is never too late to learn new things.
This could be easily done with awk. This will print line with line numbers and ignore empty lines.
awk 'NF{print FNR,$0}' file.txt
Explanation: Adding detailed explanation for above code.
awk ' ##Starting awk program from here.
NF{ ##Checking condition if NF(number of fields) is NOT NULL in current line then do following.
print FNR,$0 ##Printing current line number by mentioning FNR and then current line value.
}
' file.txt ##Mentioning Input_file name which we are passing to awk program here.
FILE1 (/var/widenet.jcml) holds the LAN's server entries while FILE2 (hosts.out) contains a list of IPs. My idea is to use FILE2 to search for IPs on FILE1 and update the entries based on matched IPs.
This is how FILE1 looks
[romolo#remo11g ~]$ grep -F -f hosts.out /var/widenet.jcml |head -2
2548,0,00:1D:55:00:D4:D1,10.0.209.76,wd18-209-76-man 91.widenet.lan,10.0.101.2,255.255.0.0,NULL,NULL,NULL,NULL,NULL,NULL,NAS,ALL
2549,0,00:1D:55:00:D4:D2,10.0.209.77,wd18-209-77-man 91.widenet.lan,10.0.101.2,255.255.0.0,NULL,NULL,NULL,NULL,NULL,NULL,NAS,ALL
While FILE2 is essentially a list of IPs, one IP per line
cat hosts.out
10.0.209.76
10.0.209.77
10.0.209.158
10.0.209.105
10.0.209.161
10.0.209.169
10.0.209.228
Basically FILE2 contains 160 IPs which entries in /var/widenet.jcml are needed to be updated. In specific the word NAS on column 14 of /var/widenet.jcml needs to be replaced with SAS.
I came up with the following syntax, however instead of just replacing the word NAS for the matched IPs, it will instead replace every entries in FILE1 which does contain the word NAS, therefore ignoring the list of IPs from FILE2.
grep -F -f hosts.out /var/widenet.jcml |awk -F"," '{print$4,$14}' |xargs -I '{}' sed -i 's/NAS/SAS/g' /var/widenet.jcml
I spent hours googling for an answer but I couldn't find any examples that cover search and replace between two text files. Thanks
Assuming file2 doesn't really have leading blanks (if it does it's an easy tweak to fix):
$ awk 'BEGIN{FS=OFS=","} NR==FNR{ips[$1];next} $4 in ips{$14="SAS"} 1' file2 file1
2548,0,00:1D:55:00:D4:D1,10.0.209.76,wd18-209-76-man 91.widenet.lan,10.0.101.2,255.255.0.0,NULL,NULL,NULL,NULL,NULL,NULL,SAS,ALL
2549,0,00:1D:55:00:D4:D2,10.0.209.77,wd18-209-77-man 91.widenet.lan,10.0.101.2,255.255.0.0,NULL,NULL,NULL,NULL,NULL,NULL,SAS,ALL
If I understand the question correctly, you only want to change NAS to SAS per IP address found in hosts.out?
while read line
do
grep $line file1 | sed 's/NAS/SAS/g' >> results
done < hosts.out
I have this file:
this is line 1 192.168.1.1
this is line 2 192.168.1.2
this is line 3 192.168.1.2
this is line 4 192.168.1.1
this is line 5 192.168.1.2
I would like to get, in a bash script, all lines (with tabs) which contains, for example, the pattern "192.168.1.1". By this way, I would get:
this is line 1 192.168.1.1
this is line 4 192.168.1.1
But i don't want this result:
this is line 1 192.168.1.1 this is line 4 192.168.1.1
I tried it with sed without success:
var='192.168.1.1'
body=`sed -n -e '/$var/p' $file`
echo $body
Thanks beforehand!
awk to the rescue!
$ awk -v var='192.168.1.1' '$NF==var' file
this is line 1 192.168.1.1
this is line 4 192.168.1.1
Following sed may help you on same.
var="your_ip"
sed -n "/$var/p" Input_file
Or in awk following will help on same.
var="your_ip"
awk -v var="$var" 'var==$NF' Input_file
A simple grep command could do that:
grep 192.168.1.1 filename
Or a more "complex" grep command could do that and write it to another file:
grep 192.168.1.1 filename > new_filename
Hope this helps!
Assuming the data format is well defined like you said:
"this is line <seq> <ip>"
you could do simply this:
cat file | egrep "192.168.0.1$"
I have created a simple script that prints the contents of a text file using cat command. Now I want to print a line along with its number, but at the same time I need to ignore blank lines. The following format is desired:
1 George Jones Berlin 2564536877
2 Mike Dixon Paris 2794321976
I tried using
cat -n catalog.txt | grep -v '^$' catalog.txt
But I get the following results:
George Jones Berlin 2564536877
Mike Dixon Paris 2794321976
I have managed to get rid of the blank lines, but line's number is not printed. What am I doing wrong?
Here are the contents of catalog.txt:
George Jones Berlin 2564536877
Mike Dixon Paris 2794321976
Your solution doesn't work because cat -n catalog.txt is already giving you non-blank lines.
You can pipe grep's output to cat -n:
grep -v '^$' yourFile | cat -n
Example:
test.txt:
Hello
how
are
you
?
$ grep -v '^$' test | cat -n
1 Hello
2 how
3 are
4 you
5 ?
At first glance, you should drop the file name in the command line to grep to make grep read from stdin:
cat -n catalog.txt | grep -v '^$'
^^^
In your code, you supplied catalog.txt to grep, which made it read from the file and ignore its standard input. So you're basically grepping from the file instead of the output of cat piped to its stdin.
To correctly ignore blank lines the prepend line numbers, switch the order of grep and cat:
grep -v '^$' catalog.txt | cat -n
Another awk
$ awk 'NF{$0=FNR " " $0}NF' 48488182
1 George Jones Berlin 2564536877
3 Mike Dixon Paris 2794321976
The second line was blank in this case.
single, simple, basic awk solution could help you here.
Solution 1st:
awk 'NF{print FNR,$0}' Input_file
Solution 2nd: Above will print line number including the line number of NULL lines, in case you want to leave empty lines line number then following may help you in same.
awk '!NF{FNR--;next} NF{print FNR,$0}' Input_file
Solution 3rd: Using only grep, though output will have a colon in between line number and the line.
grep -v '^$' Input_file | grep -n '.*'
Explanation of Solution 1st:
NF: Checking condition here if NF(Number of fields in current line, it is awk's out of the box variable which has the value of number of fields in a line) is NOT NULL, if this condition is TRUE then following the actions mentioned next to it.
{print FNR,$0}: Using print function of awk here to print FNR(Line number, which will have the line's number in it, it is awk's out of box variable) then print $0 which means current line.
By this we satisfy OP's both the conditions of leaving empty lines and print the line numbers along with lines too. I hope this helps you.
I have two files like below:
file1
"Connect" CONNECT_ID="12"
"Connect" CONNECT_ID="11"
"Connect" CONNECT_ID="122"
"Connect" CONNECT_ID="109"
file2
"Quit" CONNECT_ID="12"
"Quit" CONNECT_ID="11"
The file contents are not exactly same but similar to above and the number of records are minimum 100,000.
Now i want to get the result as show below into file1 (means the final result should be there in file1)
"Connect" CONNECT_ID="122"
"Connect" CONNECT_ID="109"
I have used a while loop something like below:
awk {'print $2'} file2 | sed "s/CONNECTION_ID=//g" > sample.txt
while read actual; do
grep -w -v $actual file1 > file1_tmp
mv -f file1_tmp file1
done < sample.txt
Here I have adjusted my code according to example. So it may or may not work.
My problem is the loop is repeating for more than 1 hour to complete the process.
So can any one suggest me how to achieve the same with any other ways like using diff or comm or sed or awk or any other linux command which will run faster?
Here mainly I want to eliminate this big typical while loop.
Most UNIX tools are line based and as you don't have whole line matches that means grep, comm and diff are out the window. To extract field based information like you want awk is perfect:
$ awk 'NR==FNR{a[$2];next}!($2 in a)' file2 file1
"Connect" CONNECT_ID="122"
"Connect" CONNECT_ID="109"
To store the results back to file1 you'll need to redict the output to a temporary file and then move the file into file1 like so:
$ awk 'NR==FNR{a[$2];next}!($2 in a)' file2 file1 > tmp && mv tmp file1
Explanation:
The awk variable NR increments for every record read, that is each line in every file. The FNR variable increments for every record but gets reset for every file.
NR==FNR # This condition is only true when reading file1
a[$2] # Add the second field in file1 into array as a lookup table
next # Get the next line in file1 (skips any following blocks)
!($2 in a) # We are now looking at file2 if the second field not in the look up
# array execute the default block i.e print the line
To modify this command you just need to change the fields that matched. In your real case if you want to match field 1 from file1 with field 4 from file2 then you would do:
$ awk 'NR==FNR{a[$1];next}!($4 in a)' file2 file1
This might work for you (GNU sed):
sed -r 's|\S+\s+(\S+)|/\1/d|' file2 | sed -f - -i file1
The tool best suited to this job is join(1). It joins two files based on values in a given column of each file. Normally it just outputs the lines that match across the two files, but it also has a mode to output the lines from one of the files that do not match the other file.
join requires that the files be sorted on the field(s) you are joining on, so either pre-sort the files, or use process substitution (a bash feature - as in the example below) to do it on the one command line:
$ join -j 2 -v 1 -o "1.1 1.2" <(sort -k2,2 file1) <(sort -k2,2 file2)
"Connect" CONNECT_ID="122"
"Connect" CONNECT_ID="109"
-j 2 says to join the files on the second field for both files.
-v 1 says to only output fields from file 1 that do not match any in file 2
-o "1.1 1.2" says to order the output with the first field of file 1 (1.1) followed by the second field of file 1 (1.2). Without this, join will output the join column first followed by the remaining columns.
You may need to analyze file2 at fist, and append all ID which have appered to a cache(eg. memory)
Than scan file1 line by line to adjust whether the ID in the cache.
python code like this:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
p = re.compile(r'CONNECT_ID="(.*)"')
quit_ids = set([])
for line in open('file2'):
m = p.search(line)
if m:
quit_ids.add(m.group(1))
output = open('output_file', 'w')
for line in open('file1'):
m = p.search(line)
if m and m.group(1) not in quit_ids:
output.write(line)
output.close()
The main bottleneck is not really the while loop, but the fact that you rewrite the output file thousands of times.
In your particular case, you might be able to get away with just this:
cut -f2 file2 | grep -Fwvf - file1 >tmp
mv tmp file1
(I don't think the -w option to grep is useful here, but since you had it in your example, I retained it.)
This presupposes that file2 is tab-delimited; if not, the awk '{ print $2 }' file2 you had there is fine.