690070 690070 A
690451 690451 B
690571 690571 C
690578 690578 D
690637 690637 F
How can I replace the first column values with a sequential number, starting from 1...n. So it becomes:
1 690070 A
2 690451 B
3 690571 C
4 690578 D
5 690637 F
Can this be done in Vim or some linux command?
You can use awk or vim macro.
awk is really great for such text manipulation
awk '{count++; print count " " $2 " "$3;}' data.stat > /tmp/data.stat && mv /tmp/data.stat data.stat
in Vim:
:let i=1 | g/^[^/\t]*\t/s//\= i. "\t"/ | let i=i+1
Reference
Update
For splitting the first two columns and saving into another file,
I recommend using awk as in Tomáš Šíma's answer, specifically:
awk '{print $1 "\t" $2;}' data.stat > newfilename.txt
If you want to to do everything in Vim:
Copy the current file to a new one
:w newfilename.txt
Open the newly copied file:
:o newfilename.txt
Split the first two columns of the rest of the line:
:%s/^\([^\t]*\)\t\([^\t]*\).*$/\1\t\2/g
Save your edits of course
:w newfilename.txt
Related
My text file should be of two columns separated by a tab-space (represented by \t) as shown below. However, there are a few corrupted values where column 1 has two values separated by a space (represented by \s).
A\t1
B\t2
C\sx\t3
D\t4
E\sy\t5
My objective is to create a table as follows:
A\t1
B\t2
C\t3
D\t4
E\t5
i.e. discard the 2nd value that is present after the space in column 1 for eg. in C\sx\t3 I can discard the x that is present after space and store the columns as C\t3.
I have tried a couple of things but with no luck.
I tried to cut the cols based on \t into independent columns and then cut the first column based on \s and join them again. However, it did not work.
Here is the snippet:
col1=(cut -d$'\t' -f1 $file | cut -d' ' -f1)
col2=(cut -d$'\t' -f1 $file)
myArr=()
for((idx=0;idx<${#col1[#]};idx++));do
echo "#{col1[$idx]} #{col2[$idx]}"
# I will append to myArr here
done
The output is appending the list of col2 to the col1 as A B C D E 1 2 3 4 5. And on top of this, my file is very huge i.e. 5,300,000 rows so I would like to avoid looping over all the records and appending them one by one.
Any advice is very much appreciated.
Thank you. :)
And another sed solution:
Search and replace any literal space followed by any number of non-TAB-characters with nothing.
sed -E 's/ [^\t]+//' file
A 1
B 2
C 3
D 4
E 5
If there could be more than one actual space in there just make it 's/ +[^\t]+//' ...
Assuming that when you say a space you mean a blank character then using any awk:
awk 'BEGIN{FS=OFS="\t"} {sub(/ .*/,"",$1)} 1' file
Solution using Perl regular expressions (for me they are easier than seds, and more portable as there are few versions of sed)
$ cat ls
A 1
B 2
C x 3
D 4
E y 5
$ cat ls |perl -pe 's/^(\S+).*\t(\S+)/$1 $2/g'
A 1
B 2
C 3
D 4
E 5
This code gets all non-empty characters from the front and all non-empty characters from after \t
Try
sed $'s/^\\([^ \t]*\\) [^\t]*/\\1/' file
The ANSI-C Quoting ($'...') feature of Bash is used to make tab characters visible as \t.
take advantage of FS and OFS and let them do all the hard work for you
{m,g}awk NF=NF FS='[ \t].*[ \t]' OFS='\t'
A 1
B 2
C 3
D 4
E 5
if there's a chance of leading edge or trailing edge spaces and tabs, then perhaps
mawk 'NF=gsub("^[ \t]+|[ \t]+$",_)^_+!_' OFS='\t' RS='[\r]?\n'
I am trying to filter a text file with columns based on two conditions. Due to the size of the file, I cannot use the column numbers (as there are thousands and are unnumbered) but need to use the column names. I have searched and tried to come up with multiple ways to do this but nothing is returned to the command line.
Here are a few things I have tried:
awk '($colname1==2 && $colname2==1) { count++ } END { print count }' file.txt
to filter out the columns based on their conditions
and
head -1 file.txt | tr '\t' | cat -n | grep "COLNAME
to try and return the possible column number related to the column.
An example file would be:
ID ad bd
1 a fire
2 b air
3 c water
4 c water
5 d water
6 c earth
Output would be:
2 (count of ad=c and bd=water)
with your input file and the implied conditions this should work
$ awk -v c1='ad' -v c2='bd' 'NR==1{n=split($0,h); for(i=1;i<=n;i++) col[h[i]]=i}
$col[c1]=="c" && $col[c2]=="water"{count++} END{print count+0}' file
2
or you can replace c1 and c2 with the values in the script as well.
to find the column indices you can run
$ awk -v cols='ad bd' 'BEGIN{n=split(cols,c); for(i=1;i<=n;i++) colmap[c[i]]}
NR==1{for(i=1;i<=NF;i++) if($i in colmap) print $i,i; exit}' file
ad 2
bd 3
or perhaps with this chain
$ sed 1q file | tr -s ' ' \\n | nl | grep -E 'ad|bd'
2 ad
3 bd
although may have false positives due to regex match...
You can rewrite the awk to be more succinct
$ awk -v cols='ad bd' '{while(++i<=NF) if(FS cols FS ~ FS $i FS) print $i,i;
exit}' file
ad 2
bd 3
As I mentioned in an earlier comment, the answer at https://unix.stackexchange.com/a/359699/133219 shows how to do this:
awk -F'\t' '
NR==1 {
for (i=1; i<=NF; i++) {
f[$i] = i
}
}
($(f["ad"]) == "c") && ($(f["bd"]) == "water") { cnt++ }
END { print cnt+0 }
' file
2
I'm assuming your input is tab-separated due to the tr '\t' in the command in your question that looks like you're trying to convert tabs to newlines to convert column names to numbers. If I'm wrong and they're just separated by any chains of white space then remove -F'\t' from the above.
Use miller toolkit to manipulate tab-delimited files using column names. Below is a one-liner that filters a tab-delimited file (delimiter is specified using --tsv) and writes the results to STDOUT together with the header. The header is removed using tail and the lines are counted with wc.
mlr --tsv filter '$ad == "c" && $bd == "water"' file.txt | tail -n +2 | wc -l
Prints:
2
SEE ALSO:
miller manual
Note that miller can be easily installed, for example, using conda, like so:
conda create --name miller miller
For years it bugged me there is no succinct way in Unix to do this sort of thing, although miller is a pretty good tool for this. Recently I wrote pick to choose columns by name, and additionally modify, combine and add them by name, as well as filtering rows by clauses using column names. The solution to the above with pick is
pick -h #ad=c #bd=water < data.txt | wc -l
By default pick prints the header of the selected columns, -h is to omit it. To print columns you simply name them on the command line, e.g.
pick ad water < data.txt | wc -l
Pick has many modes, all of them focused on manipulating columns and selecting/filtering rows with a minimal amount of syntax.
I have a tab delimited file with 3 rows and 7 columns. I want to use the number at the end of the file to rename another file.
Example of tab delimited file:
a b c d e f g
a b c d e f g
a b c d e f 1235
So, I want to extract the number from tab delimited file and then rename "file1" to the number extracted (mv file1 1235)
I can print the column, but I cannot seem to extract just the number from the file. Even if I can extract the number I can't seem to figure out how to store that number to use as the new file name.
You can use this awk
name=$(awk 'END {print $NF}' file)
mv file $name
something along these lines perhaps?
num=$(tail -1 file1 | rev | awk '{print $1}' | rev)
mv file1 $num
Using a perl one-liner
perl -ne 'BEGIN{($f) = #ARGV} ($n) = /(\d+)$/; END{rename($f, $n)}' file1
I have a file that contains a matrix. For example, I have:
1 a 2 b
2 b 5 b
3 d 4 b
4 b 7 b
I know it's easy to use sed command to delete specific lines with specific strings. But what if I only want to delete those lines where the second field's value is b (i.e., second line and fourth line)?
You can use regex in sed.
sed -i 's/^[0-9]\s+b.*//g' xxx_file
or
sed -i '/^[0-9]\s+b.*/d' xxx_file
The "-i" argument will modify the file's content directly, you can remove "-i" and output the result to other files as you want.
Awk just work fine, just use code as below:
awk '{if ($2 != "b") print $0;}' file
if you want get more usage about awk, just man it!
awk:
cat yourfile.txt | awk '{if($2!="b"){print;}}'
I have several text files whose lines are tab-delimited.
The second column contains incorrect data.
How do I change everything in the second column to a specific text string?
awk ' { $2="<STRING>"; print } ' <FILENAME>
cat INFILE | perl -ne '$ln=$_;#x=split(/","/); #a=split(/","/, $ln,8);#b=splice(#a,0,7); $l=join("\",\"", #b); $r=join("\",\"", splice(#x,8)); print "$l\",\"10\",\"$r"'
This is an example that changes the 10th column to "10". I prefer this as I don't have to count the matching parenthesis like in the sed technique.
A simple and cheap hack:
cat INFILE | sed 's/\(.*\)\t\(.*\)\t\(.*\)/\1\tREPLACEMENT\t\3/' > OUTFILE
testing it:
echo -e 'one\ttwo\tthree\none\ttwo\tthree' | sed 's/\(.*\)\t\(.*\)\t\(.*\)/\1\tREPLACEMENT\t\3/'
takes in
one two three
one two three
and produces
one REPLACEMENT three
one REPLACEMENT three