I'm quite new to AWK so apologies for the basic question. I've found many references for removing windows end-line characters from files but none that match a regular expression and subsequently remove the windows end line characters.
I have a file named infile.txt that contains a line like so:
...
DATAFILE data5v.dat
...
Within a shell script I want to capture the filename argument data5v.dat from this infile.txt and remove any carriage return character, \r, IF present. The carriage return may not always be present. So I have to match a word and then remove the \r subsequently.
I have tried the following but it is not working how I expect:
FILENAME=$(awk '/DATAFILE/ { print gsub("\r", "", $2) }' $INFILE)
Can I store the string returned from matching my regex /DATAFILE/ in a variable within my AWK statement to subsequently apply gsub?
File names can contain spaces, including \rs, blanks and tabs, so to do this robustly you can't remove all \rs with gsub() and you can't rely on there being any field, e.g. $2, that contains the whole file name.
If your input fields are tab-separated you need:
awk '/DATAFILE/ { sub(/[^\t]+\t/,""); sub(/\r$/,""); print }' file
or this otherwise:
awk '/DATAFILE/ { sub(/[^[:space:]]+[[:space:]]+/,""); sub(/\r$/,""); print }' file
The above assumes your file names don't start with spaces and don't contain newlines.
To test any solution for robustness try:
printf 'DATAFILE\tfoo \r bar\r\n' | awk '...' | cat -TEv
and make sure that the output looks like it does below:
$ printf 'DATAFILE\tfoo \r\tbar\r\n' | awk '/DATAFILE/ { sub(/[^\t]+\t/,""); sub(/\r$/,""); print }' | cat -TEv
foo ^M^Ibar$
$ printf 'DATAFILE\tfoo \r\tbar\r\n' | awk '/DATAFILE/ { sub(/[^[:space:]]+[[:space:]]+/,""); sub(/\r$/,""); print }' | cat -TEv
foo ^M^Ibar$
Note the blank, ^M (CR), and ^I (tab) in the middle of the file name as they should be but no ^M at the end of the line.
If your version of cat doesn't support -T or -E then do whatever you normally do to look for non-printing chars, e.g. od -c or vi the output.
With GNU awk, would you please try the following:
FILENAME=$(awk -v RS='\r?\n' '/DATAFILE/ {print $2}' "$INFILE")
echo "$FILENAME"
It assigns the record separator RS to a sequence of zero or one \r followed by \n.
As a side note, it is not recommended to use uppercases for user's variable names because it may conflict with system reserved variable names.
Awk simply applies each line of script to each input line. You can easily remove the carriage return and then apply some other logic to the input line. For example,
FILENAME=$(awk '/\r/ { sub(/\r/, "") }
/DATAFILE/ { print $2 }' "$INFILE")
Notice also When to wrap quotes around a shell variable.
who says you need gnu-awk :
gecho -ne "test\r\nabc\n\rdef\n" \
\
| mawk NF=NF FS='\r' OFS='' | odview
0000000 1953719668 1667391754 1717920778 10
t e s t \n a b c \n d e f \n
164 145 163 164 012 141 142 143 012 144 145 146 012
t e s t nl a b c nl d e f nl
116 101 115 116 10 97 98 99 10 100 101 102 10
74 65 73 74 0a 61 62 63 0a 64 65 66 0a
0000015
gawk -P posix mode is also fine with it :
gecho -ne "test\r\nabc\n\rdef\n" \
\
| gawk -Pe NF=NF FS='\r' OFS='' | odview
0000000 1953719668 1667391754 1717920778 10
t e s t \n a b c \n d e f \n
164 145 163 164 012 141 142 143 012 144 145 146 012
t e s t nl a b c nl d e f nl
116 101 115 116 10 97 98 99 10 100 101 102 10
74 65 73 74 0a 61 62 63 0a 64 65 66 0a
0000015
Related
I am trying to display the second result of duplicate lines in a text file.
Data:
60 60 61 64 63 78 78
Duplicate Lines:
60 60 78 78
Attempted Code:
echo "60 60 61 64 63 78 78" | sed 's/ /\n/g' | uniq -D | tail -1
Current Result:
78
Expected Result:
60 78
You may try this gnu awk solution:
s='60 60 61 64 63 78 78'
awk -v RS='[[:space:]]+' '++fq[$0] == 2' <<< "$s"
60
78
To avoid getting line breaks after each line:
awk -v RS='[[:space:]]+' '++fq[$0] == 2 {printf "%s", $0 RT}' <<< "$s"
60 78
Considering that you could have multiple lines in your Input_file then you could try following.
awk '
{
delete value
num=split($0,arr," ")
for(i=1;i<=num;i++){
value[arr[i]]++
}
for(i=1;i<=num;i++){
if(value[arr[i]]>1){
print arr[i]
delete value[arr[i]]
}
}
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
{
delete value ##Deleting value array here.
num=split($0,arr," ") ##Splitting current line to array arr here.
for(i=1;i<=num;i++){ ##Traversing through all fields here.
value[arr[i]]++ ##Creating value array with value of arr array.
}
for(i=1;i<=num;i++){ ##Traversing through all fields here.
if(value[arr[i]]>1){ ##Checking condition if value array value is coming more than 1 times then do following.
print arr[i] ##printing array value here(which is value which comes more than 1 time).
delete value[arr[i]] ##Deleting value array value to avoid duplicate printing here.
}
}
}
' Input_file ##Mentioning Input_file name here.
So if you do not want to then do not use tail -1. And instead of printing all, print duplicates once per duplicate.
echo "60 60 61 64 63 78 78" | sed 's/ /\n/g' | sort | uniq -d
Note - if the input is not sorted (or duplicate lines are not adjacent), you need to sort it first.
$ printf '%s\n' 60 60 61 64 63 78 78 | uniq -d
60
78
The above is assuming a hard-coded list of numbers works for you since, per the example in your question, you thought echo "60 60 61 64 63 78 78" | sed 's/ /\n/g' | uniq -D | tail -1 would work for you. If you need to store them in a variable first then make it an array variable:
$ vals=(60 60 60 61 64 63 78 78)
$ printf '%s\n' "${vals[#]}" | uniq -d
60
78
Reference file: http://snap.stanford.edu/data/wiki-Vote.txt.gz
(It is a tape archive that contains a file called Wiki-Vote.txt)
The first few lines in the file that contains the following, head -n 10 Wiki-Vote.txt
# Directed graph (each unordered pair of nodes is saved once): Wiki-Vote.txt
# Wikipedia voting on promotion to administratorship (till January 2008).
# Directed edge A->B means user A voted on B becoming Wikipedia administrator.
# Nodes: 7115 Edges: 103689
# FromNodeId ToNodeId
30 1412
30 3352
30 5254
30 5543
30 7478
3 28
I want to find the number of nodes in the graph, (although it's already given in line 3). I ran the following command,
awk '!/^#/ { print $1; print $2; }' Wiki-Vote.txt | sort | uniq | wc -l
Explanation:
/^#/ matches all the lines that start with #. And !/^#/ matches that doesn't.
awk '!/^#/ { print $1; print $2; }' Wiki-Vote.txt prints the first and second column of all those matched lines in new lines.
| sort pipes the output to sort them.
| uniq should display all those unique values, but it doesn't.
| wc -l counts the previous lines and it is wrong.
The result of the above command is, 8491, which is not 7115 (as mentioned in the line 3). I don't know why uniq repeats the values. I can tell that since awk '!/^#/ { print $1; print $2; }' Wiki-Vote.txt | sort -i | uniq | tail returns,
992
993
993
994
994
995
996
998
999
999
Which contains the repeated values. Someone please run the code and tell me that I am not the only one getting the wrong answer and please help me figure out why I'm getting what I am getting.
The file has dos line endings - each line is ending with \r CR character.
You can inspect your tail output for example with hexdump -C, lines starting with # added by me:
$ awk '!/^#/ { print $1; print $2; }' ./wiki-Vote.txt | sort | uniq | tail | hexdump -C
00000000 39 39 32 0a 39 39 33 0a 39 39 33 0d 0a 39 39 34 |992.993.993..994|
# ^^ HERE
00000010 0a 39 39 34 0d 0a 39 39 35 0d 0a 39 39 36 0a 39 |.994..995..996.9|
# ^^ ^^
00000020 39 38 0a 39 39 39 0a 39 39 39 0d 0a |98.999.999..|
# ^^
0000002c
Because uniq sees unique lines, one with CR and one not, they are not removed. Remove the CR character before pipeing. Note that sort | uniq is better to sort -u.
$ awk '!/^#/ { print $1; print $2; }' ./wiki-Vote.txt | tr -d '\r' | sort -u | wc -l
7115
I have a command that generates 3 lines of output such as
$ ./mycommand
1
asdf
qwer zxcv
I'd like to assign those 3 lines to 3 different variables ($a, $b, $c) such that
$ echo $a
1
$ echo $b
asdf
$ echo $c
qwer zxcv
I'm familiar with the while loop method that would normally be used for reading 3 lines at a time from output that contains sets of 3 lines. But that seems less than elegant considering my command will only ever output 3 lines.
I tried playing around with various combinations of values for IFS= and options for read -r a b c, sending the command output as stdin, but I could only ever get it to set the first line to the first variable. Some examples:
IFS= read -r a b c < <(./mycommand)
IFS=$'\n' read -r a b c < <(./mycommand)
IFS= read -r -d $'\n' < <(./mycommand)
If I modify my command so that the 3 lines are separated by spaces instead of newlines, I can successfully just use this variation as long as each former line is properly quoted:
read -r a b c < <(./mycommand)
And while that is working for the purposes of my current project, it's still bugging me that I couldn't get it to work the other way. So I'm wondering if anyone can see and explain what I was missing in my original attempt with the 3 lines of output.
If you want to read data from three lines, use three reads:
{ read -r a; read -r b; read -r c; } < <(./mycommand)
read reads a chunk of data and then splits it up. You couldn't get it to work because your chunks were always single lines.
Newer BASH versions support mapfile command. Using that you can read all the lines into an array:
mapfile -t ary < <(./command)
Check the array content:
declare -p ary
declare -a ary='([0]="1" [1]="asdf" [2]="qwer zxcv")'
Perhaps this explanation will be useful to you.
... it's still bugging me that I couldn't get it to work the other way. So I'm wondering if anyone can see and explain what I was missing in my original attempt with the 3 lines of output.
Simple: read works only with one line (by default). This:
#!/bin/bash
mycommand(){ echo -e "1\nasdf\nqwer zxcv"; }
read a b c < <(mycommand)
printf 'first : %s\nsecond : %s\nthird : %s\n' "$a" "$b" "$c"
Will print:
first : 1
second :
third :
However, using a null character will capture the whole string in (replace this line above):
read -d '' a b c < <(mycommand)
Will print:
first : 1
second : asdf
third : qwer zxcv
The read command absorbed the whole output of the command and was broken into parts with the default value of IFS: SpaceTabEnter.
In this specific example, that worked correctly because the last value is the one with more than one "part".
But this kind of processing is very brittle. For example: this other possible output of the command, the assignment to variables will break:
mycommand(){ echo -e "1 and 2\nasdf and dfgh\nqwer zxcv"; }
Will output (incorrectly):
first : 1
second : and
third : 2
asdf and dfgh
qwer zxcv
The processing is brittle. To make it robust we need to use a loop. But you say that that is something you already know:
#!/bin/bash
mycommand(){ echo -e "1 and 2\nasdf and dfgh\nqwer zxcv"; }
i=0; while read arr[i]; do ((i++)); done < <(mycommand)
printf 'first : %s\nsecond : %s\nthird : %s\n' "${arr[0]}" "${arr[1]}" "${arr[2]}"
Which will (correctly) print:
first : 1 and 2
second : asdf and dfgh
third : qwer zxcv
However, the loop could be made simpler using bash command readarray:
#!/bin/bash
mycommand(){ echo -e "1 and 2\nasdf and dfgh\nqwer zxcv"; }
readarray -t arr < <(mycommand)
printf 'first : %s\nsecond : %s\nthird : %s\n' "${arr[0]}" "${arr[1]}" "${arr[2]}"
And using a printf "loop" will make the structure work for any count of input lines:
#!/bin/bash
mycommand(){ echo -e "1 and 2\nasdf and dfgh\n*\nqwer zxcv"; }
readarray -t arr < <(mycommand)
printf 'value : %s\n' "${arr[#]}"
Hope that this helped.
EDIT
About nulls (in simple read):
In bash, the use of nulls is almost never practical. In specific, nulls are erased silently in most condidions. This solution does suffer of that limitation.
Including a null in the input:
mycommand(){ echo -e "1 and 2\nasdf and dfgh\n\000\n*\nqwer zxcv"; }
will make a simple read -r -d '' get the input up to the first null (understanding such null as the character with octal 000).
echo "test one:"; echo
echo "input"; echo
mycommand | od -tcx1
echo "output"; echo
read -r -d '' arr < <(mycommand)
echo "$arr" | od -tcx1
Gives this as output:
test one:
input
0000000 1 a n d 2 \n a s d f a n d
31 20 61 6e 64 20 32 0a 61 73 64 66 20 61 6e 64
0000020 d f g h \n \0 \n * \n q w e r z
20 64 66 67 68 0a 00 0a 2a 0a 71 77 65 72 20 7a
0000040 x c v \n
78 63 76 0a
0000044
output
0000000 1 a n d 2 \n a s d f a n d
31 20 61 6e 64 20 32 0a 61 73 64 66 20 61 6e 64
0000020 d f g h \n
20 64 66 67 68 0a
0000026
It is clear that the value captured by read stops at the first octal 000.
Which, frankly, is to be expected.
About nulls (in readarray):
I have to report, however, that readarray does not stop at the octal 000 but just silently removes it (an usual shell trait).
Running this code:
#!/bin/bash
mycommand(){ echo -e "1 and 2\nasdf and dfgh\n\000\n*\nqwer zxcv"; }
echo "test two:"; echo
echo "input"; echo
mycommand | od -tcx1
echo "output"; echo
readarray -t arr < <(mycommand)
printf 'value : %s\n' "${arr[#]}"
echo
printf 'value : %s\n' "${arr[#]}"|od -tcx1
Renders this output:
test two:
input
0000000 1 a n d 2 \n a s d f a n d
31 20 61 6e 64 20 32 0a 61 73 64 66 20 61 6e 64
0000020 d f g h \n \0 \n * \n q w e r z
20 64 66 67 68 0a 00 0a 2a 0a 71 77 65 72 20 7a
0000040 x c v \n
78 63 76 0a
0000044
output
value : 1 and 2
value : asdf and dfgh
value :
value : *
value : qwer zxcv
0000000 v a l u e : 1 a n d 2 \n
76 61 6c 75 65 20 3a 20 31 20 61 6e 64 20 32 0a
0000020 v a l u e : a s d f a n d
76 61 6c 75 65 20 3a 20 61 73 64 66 20 61 6e 64
0000040 d f g h \n v a l u e : \n v
20 64 66 67 68 0a 76 61 6c 75 65 20 3a 20 0a 76
0000060 a l u e : * \n v a l u e :
61 6c 75 65 20 3a 20 2a 0a 76 61 6c 75 65 20 3a
0000100 q w e r z x c v \n
20 71 77 65 72 20 7a 78 63 76 0a
0000113
That is, the null 000 or just \0 gets silently removed.
Say I have a file with 100 lines (not including header). I want to cut that file down, only keeping the content between line 51 and 70 (inclusive), as well as the header so that the resulting file is 20+1 lines.
So far, I have this code:
awk 'NR==1 {h=$0; next} (NR-1)>50 && (NR-1)<71 {filename = "file20.csv"; print h >> filename} {print >> filename}' file100.csv
But it's giving me an error:
fatal: expression for `>>' redirection has null string value
Can somebody help me understand where my syntax is wrong?
You can directly use:
awk 'NR==1 || (NR>=51 && NR<=70)'
Note that this evaluates the condition of NR. In case it is true, it performs awk's default action: {print $0}. Hence, you do not have to explicit it.
Then you can redirect to another file:
awk 'NR==1 || (NR>=51 && NR<=70)' file > new_file
Test
$ seq 100 | awk 'NR==1 || (NR>=51 && NR<=70)'
1
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
It returns 21 lines:
$ seq 100 | awk 'NR==1 || (NR>=51 && NR<=70)' | wc -l
21
Can someone please help! I'm trying to delete the last portion (following "_cO") in the second column of the following list in the bash shell. E.G where it says "_seq1" in this particular list. I do not want to change any other info in the remaining columns.
Thanks!
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
XP_003962102 comp1000054_c0_seq1 24.07 54 41 0 164 3
Here you go, a simple substitution using sed:
sed -e s/_seq1//
Using sed:
sed -i.bak 's/^\(.*_c0\)[^ ]*\( .*\)$/\1\2/' file
OR using awk:
awk '{sub(/_c0[^ ]*/, "_c0", $2)} 1' file