so I have a project for uni, and I can't get through the first exercise. Here is my problem:
I have a file, and I want to select some data inside of it and 'display' it in another file. But the data I'm looking for is a little bit scattered in the file, so I need several awk commands in my script to get them.
Query= fig|1240086.14.peg.1
Length=76
Score E
Sequences producing significant alignments: (Bits) Value
fig|198628.19.peg.2053 140 3e-42
> fig|198628.19.peg.2053
Length=553
Here on the picture, you can see that there are 2 types of 'Length=', and I only want to 'catch' the "Length=" that are just after a "Query=".
I have to use awk so I tried this :
awk '{if(/^$/ && $(NR+1)/^Length=/) {split($(NR+1), b, "="); print b[2]}}'
but it doesn't work... does anyone have an idea?
You need to understand how Awk works. It reads a line, evaluates the script, then starts over, reading one line at a time. So there is no way to say "the next line contains this". What you can do is "if this line contains, then remember this until ..."
awk '/Query=/ { q=1; next } /Length/ && q { print } /./ { q=0 }' file
This sets the flag q to 1 (true) when we see Query= and then skips to the next line. If we see Length and we recently saw Query= then q will be 1, and so we print. In other cases, set q back to "not recently seen" on any non-empty line. (I put in the non-empty condition to allow for empty lines anywhere without affecting the overall logic.)
awk solution:
awk '/^Length=/ && r~/^Query/{ sub(/^[^=]+=/,""); printf "%s ",$0 }
NF{ r=$0 }END{ print "" }' file
NF{ r=$0 } - capture the whole non-empty line
/^Length=/ && r~/^Query/ - on encountering Length line having previous line started with Query(ensured by r~/^Query/)
It sounds like this is what you want for the first part of your question:
$ awk -F'=' '!NF{next} f && ($1=="Length"){print $2} {f=($1=="Query")}' file
76
but idk what the second part is about since there's no "data" lines in your input and only 1 valid output from your sample input best I can tell.
Related
I read all of answers for similar problems but they are not working for me because my files are not uniformal, they contain several control headers and in such case is safer to create script than one-liner and all the answers focused on one-liners. In theory one-liners commands should be convertible to script but I am struggling to achieve:
printing the control headers
print only the records started with 16 in <file 1> where value of column 2 NOT EXISTS in column 2 of the <file 2>
I end up with this:
BEGIN {
FS="\x01";
OFS="\x01";
RS="\x02\n";
ORS="\x02\n";
file1=ARGV[1];
file2=ARGV[2];
count=0;
}
/^#/ {
print;
count++;
}
# reset counters after control headers
NR=1;
FNR=1;
# Below gives syntax error
/^16/ AND NR==FNR {
a[$2];next; 'FNR==1 || !$2 in a' file1 file2
}
END {
}
Googling only gives me results for command line processing and documentation is also silent in that regard. Does it mean it cannot be done?
Perhaps try:
script.awk:
BEGIN {
OFS = FS = "\x01"
ORS = RS = "\x02\n"
}
NR==FNR {
if (/^16/) a[$2]
next
}
/^16/ && !($2 in a) || /^#/
Note the parentheses: !$2 in a would be parsed as (!$2) in a
Invoke with:
awk -f script.awk FILE2 FILE1
Note order of FILE1 / FILE2 is reversed; FILE2 must be read first to pre-populate the lookup table.
First of all, short answer to my question should be "NOT POSSIBLE", if anyone read question carefully and knew AWK in full that is obvious answer, I wish I knew it sooner instead of wasting few days trying to write script.
Also, there is no such thing as minimal reproducible example (this was always constant pain on TeX groups) - I need full example working, if it works on 1 row there is no guarantee if it works on 2 rows and my number of rows is ~ 127 mln.
If you read code carefully than you would know what is not working - I put in comment section what is giving syntax error. Anyway, as #Daweo suggested there is no way to use logic operator in pattern section. So because we don't need printing in first file the whole trick is to do conditional in second brackets:
awk -F, 'BEGIN{} NR==FNR{a[$1];next} !($1 in a) { if (/^16/) print $0} ' set1.txt set2.txt
assuming in above example that separator is comma. I don't know where assumption about multiple RS support only in gnu awk came from. On MacOS BSD awk it works exactly the same, but in fact RS="\x02\n" is single separator not two separators.
I have a bash script which gets a text file as input and takes two parameters (Line N° one and line N° two), then changes both lines with each other in the text. Here is the code:
#!/bin/bash
awk -v var="$1" -v var1="$2" 'NR==var {
s=$0
for(i=var+1; i < var1 ; i++) {
getline; s1=s1?s1 "\n" $0:$0
}
getline; print; print s1 s
next
}1' Ham > newHam_changed.txt
It works fine for every two lines which are not consecutive. but for lines which follows after each other (for ex line 5 , 6) it works but creates a blank line between them. How can I fix that?
I think your actual script is not what you posted in the question. I think the line with all the prints contains:
print s1 "\n" s
The problem is that when the lines are consecutive, s1 will be empty (the for loop is skipped), but it will still print a newline before s, producing a blank line.
So you need to make that newline conditional.
awk -v var="4" -v var1="6" 'NR==var {
s=$0
for(i=var+1; i < var1 ; i++) {
getline; s1=s1?s1 "\n" $0:$0
}
getline; print; print (s1 ? s1 "\n" : "") s
next
}1' Ham > newHam_changed.txt
Using getline makes awk scripts always a bit complicated. It is better to prevent the use of getline and just make use of the awk pattern { action } syntax. This will make perfectly readable scripts. In any other language you would just do a loop and get the next line, but in awk I think it is best to make good use of this feature.
awk -v var="$1" -v var1="$2" '
NR==var {s=$0; collect=1; next;}
NR==var1 {collect=0; print; printf inbetween; print s}
collect {inbetween=inbetween""$0"\n"; next;}
1' Ham
Here I capture the first line in s when I found it and set the collect flag. This will trigger the collect block on the next iteration which collects all lines in between. Whenever the second line is found it sets the collect back to zero and prints first the current line, than the inbetween lines and then s. If the lines are consecutive inbetween is empty and printf will than do nothing.
Too complex for my taste, here is something quite simple that achieves the same task:
#!/bin/bash
ORIGFILE='original.txt' # original text file
PROCFILE='processed.txt' # copy of the original file to be proccesed
CHGL1=`sed "$1q;d" $ORIGFILE` # get original $1 line
CHGL2=`sed "$2q;d" $ORIGFILE` # get original $2 line
`cat $ORIGFILE > $PROCFILE`
sed -i "$2s/^.*/$CHGL1/" $PROCFILE # replace
sed -i "$1s/^.*/$CHGL2/" $PROCFILE # replace
More code doesn't mean more useful, keep it simple. This code do not use for and instead goes directly to the specific lines.
EDIT:
A simple way on one line to do this task:
printf '%s\n' 14m26 26-m14- w q | ed -s file
Found in this answer.
I have a log that returns thousands of lines of data, I want to extract a few values from that.
In the log there is only one line containing the unquie unit reference so I can grep for that using:
grep "unit=Central-C152" logfile.txt
That produces a line of output similar to the following:
a3cd23e,85d58f5,53f534abef7e7,unit=Central-C152,locale=32325687-8595-9856-1236-12546975,11="School",1="Mr Green",2="Qual",3="SWE",8="report",5="channel",7="reset",6="velum"
The format of the line may change in that the order of the values won't always be in the same position.
I'm trying to work out how to get the value of 2 and 7 in to separate variables.
I had thought about cut on , or = but as the values aren't in a set order I couldn't work out that best way to do it.
I' trying to get:
var state=value of 2 without quotes
var mode=value of 7 without quotes
Can anyone advise on the best way to do this ?
Thanks
Could you please try following to create variable's values.
state=$(awk '/unit=Central-C152/ && match($0,/2=\"[^"]*/){print substr($0,RSTART+3,RLENGTH-3)}' Input_file)
mode=$(awk '/unit=Central-C152/ && match($0,/7=\"[^"]*/){print substr($0,RSTART+3,RLENGTH-3)}' Input_file)
You could print them too by doing following.
echo "$state"
echo "$mode"
Explanation: Adding explanation of command too now.
awk ' ##Starting awk program here.
/unit=Central-C152/ && match($0,/2=\"[^"]*/){ ##Checking condition if a line has string (unit=Central-C152) and using match using REGEX to check from 2 to till "
print substr($0,RSTART+3,RLENGTH-3) ##Printing substring starting from RSTART+3 till RLENGTH-3 characters.
}
' Input_file ##Mentioning Input_file name here.
You are probably better off doing all of the processing in Awk.
awk -F, '/unit=Central-C152/ {
for(i=1;i<=NF;++i)
if($i ~ /^[27]="/) {
b[++k] = $i
sub(/^[27]="/, "", b[k])
sub(/"$/, "", b[k])
gsub(/\\/, "", b[k])
}
print "state " b[1] ", mode " b[2]
}' logfile.txt
This presupposes that the fields always occur in the same order (2 before 7). Maybe you need to change or disable the gsub to remove backslashes in the values.
If you want to do more than print the values, refactoring whatever Bash code you have into Awk is often a better approach than doing this processing in Bash.
Assuming you already have the line in a variable such as with:
line="$(grep 'unit=Central-C152' logfile.txt | head -1)"
You can then simply use the built-in parameter substitution features of bash:
f2=${line#*2=\"} ; f2=${f2%%\"*} ; echo ${f2}
f7=${line#*7=\"} ; f7=${f7%%\"*} ; echo ${f7}
The first command on each line strips off the first part of the line up to and including the <field-number>=". The second command then strips everything off that beyond (and including) the first quote. The third, of course, simply echos the value.
When I run those commands against your input line, I see:
Qual
reset
which is, from what I can see, what you were after.
I have a file with almost 5*(10^6) lines of integer numbers. So, my file is big enough.
The question is all about extract specific lines, filtering them by a condition.
For example, I'd like to:
Extract the N first lines without read entire file.
Extract the lines with the numbers less or equal X (or >=, <=, <, >)
Extract the lines with a condition related a number (math predicate)
Is there a cleaver way to perform these tasks? (using sed or awk or cat or head)
Thanks in advance.
To extract the first $NUMBER lines,
head -n $NUMBER filename
Assuming every line contains just a number (although it will also work if the first token is one), 2 can be solved like this:
awk '$1 >= 1234 && $1 < 5678' filename
And keeping in spirit with that, 3 is just the extension
awk 'condition' filename
It would have helped if you had specified what condition is supposed to be, though. This way, you'll have to read the awk documentation to find out how to code it. Again, the number will be represented by $1.
I don't think I can explain anything about the head call, it's really just what it says on the tin. As for the awk lines: awk, like sed, works linewise. awk fetches lines in a loop and applies your code to each line. This code takes the form
condition1 { action1 }
condition2 { action2 }
# and so forth
For every line awk fetches, the conditions are checked in the order they appear, and the associated action to each condition is performed if the condition is true. It would, for example, have been possible to extract the first $NUMBER lines of a file with awk like this:
awk -v number="$NUMBER" '1 { print } NR == number { exit }' filename
where 1 is synonymous with true (like in C) and NR is the line number. The -v command line option initializes the awk variable number to $NUMBER. If no action is specified, the default action is { print }, which prints the whole line. So
awk 'condition' filename
is shorthand for
awk 'condition { print }' filename
...which prints every line where the condition holds.
I have a file like this file, and I am trying to verify one field of each line, and add some wording if that field has a duplicate earlier in the file.
\\FILE04\BUET-PCO;\\SERVER24\DFS\SHARED\CORP\ET\PROJECT CONTROL OFFICE;/FS7_150a/FILE04/BU-D/PROJECT CONTROL OFFICE;10000bytes;9888;;;
\\FILE12\BUAG-GOLDMINE$;\\SERVER24\DFS\SHARED\CAN\AGENCY\GOLDMINE;/FS3_150a/FILE12/BU/AGENCY/GOLDMINE;90000bytes;98834;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;other stuff;;
In my example Line #3 and #4 have the same physical path.
I would to have a script that could compare third field for example /FS3_150a/FILE12/BU/GB/BUSINTEG against the same file,
and if it found the exact match to print something like "same physical path as Line #" for both cases,
\\FILE04\BUET-PCO;\\SERVER24\DFS\SHARED\CORP\ET\PROJECT CONTROL OFFICE;/FS7_150a/FILE04/BU-D/PROJECT CONTROL OFFICE;10000bytes;9888;;;
\\FILE12\BUAG-GOLDMINE$;\\SERVER24\DFS\SHARED\CAN\AGENCY\GOLDMINE;/FS3_150a/FILE12/BU/AGENCY/GOLDMINE;90000bytes;98834;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;;;Same Physical Path as Line #4
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;other stuff;; Same Physical Path as Line #3
This code tackles a simplified version of your problem. It identifies each line that contains a duplicate value compared to a previous line in field 3. It doesn't handle the tagging of a line that has subsequent duplicates.
awk -F';' '{ tag = ""
if (field3[$3] != 0) tag = " Same physical path as line " field3[$3]
else field3[$3] = NR
printf "%s%s\n", $0, tag
}' "$#"
There are probably other ways to organize it, but the key point is to use the associative array field3 to keep track of which names are seen in field 3 and the line number at which a given name was first seen. This assumes you're processing a single file of input. Lookup FNR etc if you must process multiple files (but you have to decide whether the same name can appear in different files or not).
It works almost as desired on the data given:
\\FILE04\BUET-PCO;\\SERVER24\DFS\SHARED\CORP\ET\PROJECT CONTROL OFFICE;/FS7_150a/FILE04/BU-D/PROJECT CONTROL OFFICE;10000bytes;9888;;;
\\FILE12\BUAG-GOLDMINE$;\\SERVER24\DFS\SHARED\CAN\AGENCY\GOLDMINE;/FS3_150a/FILE12/BU/AGENCY/GOLDMINE;90000bytes;98834;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;other stuff;; Same physical path as line 3
The difficulty with producing the 'tag' on line 3 is predicting the future; it is hard. To do that, you'd have to slurp the entire file into memory, keeping tabs on the line numbers where a given value in field 3 appears (in general, that could be an extensive list of line numbers), and then iterating through the data and tagging appropriately. Very, very much harder to do; I'd prefer to use Perl to awk for that job, though it is probably feasible to organize the data correctly in awk too.
Were it me, I'd be OK with the 90% of the job done; lines with duplicates are identified. If you want the last 10% done, expect it to take the other 90% of the time planned for the first phase.
Here's one way using GNU awk. It is a little hackish, YMMV. Run like:
awk -f script.awk file.txt{,}
Contents of script.awk:
BEGIN {
FS = ";"
}
FNR==NR {
array[$3]=array[$3] "#" NR
next
}
{
if ($3 in array && array[$3] ~ /#.#/) {
copy = array[$3]
sub("#"FNR, "", copy)
printf "%s Same Physical Path as Line as %s\n", $0, copy
}
else {
print
}
}
Results:
\\FILE04\BUET-PCO;\\SERVER24\DFS\SHARED\CORP\ET\PROJECT CONTROL OFFICE;/FS7_150a/FILE04/BU-D/PROJECT CONTROL OFFICE;10000bytes;9888;;;
\\FILE12\BUAG-GOLDMINE$;\\SERVER24\DFS\SHARED\CAN\AGENCY\GOLDMINE;/FS3_150a/FILE12/BU/AGENCY/GOLDMINE;90000bytes;98834;;;
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;;; Same Physical Path as Line as #4
\\FILE12\BUGB-BUSINTEG$;\\SERVER24\DFS\SHARED\CAN\GB\BUSINTEG;/FS3_150a/FILE12/BU/GB/BUSINTEG;50000bytes;988822;other stuff;; Same Physical Path as Line as #3