I have a directory with files that looks like this:
CCG02-215-WGS.format.flt.txt
CCG05-707-WGS.format.flt.txt
CCG06-203-WGS.format.flt.txt
CCG04-967-WGS.format.flt.txt
CCG05-710-WGS.format.flt.txt
CCG06-215-WGS.format.flt.txt
Contents of each files look like this
1 9061390 14 93246140
1 58631131 2 31823410
1 108952511 3 110694548
1 168056494 19 23850376
etc...
Ideal output would be a file, let's call it all-samples.format.flt.txt, that would contain the concatenation of all files, but an additional column that displays which sample/file the row came from ( some minor formatting involved to remove the .format.flt.txt ):
1 9061390 14 93246140 CCG02-215-WGS
...
1 58631131 2 31823410 CCG05-707-WGS
...
1 108952511 3 110694548 CCG06-203-WGS
...
1 168056494 19 23850376 CCG04-967-WGS
Currently, I have the following code which works for individual files.
awk 'BEGIN{OFS="\t"; split(ARGV[1],f,".")}{print $1,$2,$3,$4,f[1]}' CCG05-707-WGS.format.flt.txt
#OUTPUT
1 58631131 2 31823410 CCG05-707-WGS
...
However, when I try to apply it to all files, using the star, it adds the first filename it finds to all the files as the 4th column.
awk 'BEGIN{OFS="\t"; split(ARGV[1],f,".")}{print $1,$2,$3,$4,f[1]}' *
#OUTPUT, 4th column should be as seen in previous code block
1 9061390 14 93246140 CCG02-215-WGS
...
1 58631131 2 31823410 CCG02-215-WGS
...
1 108952511 3 110694548 CCG02-215-WGS
...
1 168056494 19 23850376 CCG02-215-WGS
I feel like the solution may just lie in adding an additional parameter to awk... but I'm not sure where to start.
Thanks!
UPDATE
Using OOTB awk var FILENAME solved the issue, plus some elegant formatting logic for the file names.
Thank #RavinderSingh13!
awk 'BEGIN{OFS="\t"} FNR==1{file=FILENAME;sub(/..*/,"",file)} {print $0,file}' *.txt
With your shown samples please try following awk code. We need to use FILENAME OOTB variable here of awk. Then whenever there is first line of any txt file(all txt files passed to this program) then remove everything from . to till last of value and in main program printing current line followed by file(file's name as per requirement)
awk '
BEGIN { OFS="\t" }
FNR==1{
file=FILENAME
sub(/\..*/,"",file)
}
{
print $0,file
}
' *.txt
OR in a one-liner form try following awk code:
awk 'BEGIN{OFS="\t"} FNR==1{file=FILENAME;sub(/\..*/,"",file)} {print $0,file}' *.txt
You may use:
Any version awk:
awk -v OFS='\t' 'FNR==1{split(FILENAME, a, /\./)} {print $0, a[1]}' *.txt
Or in gnu-awk:
awk -v OFS='\t' 'BEGINFILE{split(FILENAME, a, /\./)} {print $0, a[1]}' *.txt
I want to run a Perl script multiple times using the command line for a folder containing .coordinates.txt files, doing multiples "actions," and as the last step, I want to make a sort based on first-line value.
I wrote this:
for i in ./*gb.coordinates.txt; do perl myscript $i |
awk 'NR==1 {print $2,"\t***here"; next } 1'|sed '2d'| #the output has an empty line in the second row so I remove it and I add "\t***here" to have and idea about the first line value after my final sorting
if [[awk 'FNR == 1 && $1>0']] then {sort -k1nr} else {sort -k1n} fi
> $i.allvalues.txt;
done
Until here:
for i in ./*gb.coordinates.txt; do perl myscript $i | awk 'NR==1 {print $2,"\t***here"; next } 1'|sed '2d' > $i.allvalues.txt; done
Everything works properly.
So as l wrote above my final step I want to obtain is a sort like this:
if the first line of my output >=0 then sort -k1n else sort -k1nr
The output before the if condition is :
XXXX eiter positive number or negative \t***here
32
4455
-2333
23
-123
And I want my output be like:
if xxxx= positive
xxxx (going in the correct order) \t***here
4455
32
23
-123
-2333
if xxxx= negative
xxxx (going in the correct order) \t***here
-2333
-123
23
32
4455
So my problem is that I don't know who to connect if statement along with sort.
There's no need to use awk. Pipe the output of the perl script to a shell block that reads the first line, tests whether it's positive or negative, and then calls the appropriate sort.
for i in ./*gb.coordinates.txt; do
perl myscript $i | {
read _ firstline __
if (( firstline > 0 ))
then sort -k1nr
else sort -k1n
fi
} > $i.allvalues.txt
done
I have a file in Linux OS containing some random numbers:
1
22
333
4444
55555
666666
7777777
88888888
Now, I have two conditions:
1. Remove last 3 digit from every entry and put / in between rest.
2. For the numbers <=3, just add/replace with / symbol.
command I am trying which fulfilling only 1st requirement is:
sed -e 's|\(.\)|\1/|g;s|\(.*\)/\(.\/\)\{3\}|\1|g'
Desired out required:
/
/
/
4
5/5
6/6/6
7/7/7/7
8/8/8/8/8
Please help.
Something like this might work for you:
% sed 's/.\{1,3\}$//;s/./\/&/g;s/.//;s/^$/\//' file
/
/
/
4
5/5
6/6/6
7/7/7/7
8/8/8/8/8
No smart moves here:
s/.\{1,3\}$//; # Remove last 3 character
s/./\/&/g; # Insert / before each character
s/.//; # Remove first character (it's now a /)
s/^$/\// # Insert slash on all empty lines
Alternative solution with gawk:
awk -v FS='' -v OFS='/' '{if (NF > 3) NF=(NF-3); else $0 = OFS}1' file
This might work for you (GNU sed):
sed -r 's/.{1,3}$//;s#\B#/#g' file
Remove the last three (or less) characters from the end of the line. Replace the void between characters with /'s.
Can I use a step after finding a line by regex?
for example,
For a file
A
B
C
D
E
F
This works,
sed '2~2p' -n
B
D
F
but this doesn't work
sed '/B/~2p' -n
I am blamed by unknown command ~
Thanks
if you read sed (GNU sed)'s info/man page, there is NO address format like this:
/REGEX/~step
there are:
FIRST~STEP, which needs two numbers, and
ADDR1, ~N which matches the range between ADDR1 and the 1st line number(no), that no%N==0
I think you were either confused by the two above, or didn't check man page, just thought there "should be" an addr like that.
I guess, you want to do this:
kent$ seq 10|sed -n '/2/,${p;n}'
2
4
6
8
10
awk can do what you're asking, in a more verbose way:
seq 20 |
awk -v pattern="4" -v step=3 '
!start && $0 ~ pattern {start = NR}
start && (NR-start)%step == 0
'
4
7
10
13
16
19
Here's a way you could achieve the same thing in awk, in case you're interested:
awk '/B/,0 {i++}i%2' file
The range operator evaluates to true when the pattern is matched and remains true until the end of the file. While it is true, i is incremented and when it is odd, the line is printed.
The version above works well for printing every other line but not for the more general case of every N lines. In order to do that, you can modify it slightly like this:
awk '/B/,0 {i++} i%N==1' file
where N is your step.
I have a file like this called Sample:-
206,,,206,14.9,0,2012/04/24 00:00:05
206,,,206,14.9,0,2012/04/24 00:00:21
205,,,205,14.9,0,2012/04/24 00:00:23
205,,,205,14.9,0,2012/04/24 00:00:29
207,,,207,14.9,0,2012/04/24 00:00:34
205,,,205,14.9,0,2012/04/24 00:00:40
204,,,204,14.9,0,2012/04/24 00:00:46
202,,,202,14.9,0,2012/04/24 00:00:52
201,,,201,14.9,0,2012/04/24 00:01:00
202,,,202,14.9,0,2012/04/24 00:01:04
And the following AWK command:-
awk -F, '{ gsub("/"," ",$7); gsub(":"," ",$7); t+=(mktime($7)-mktime(p)); printf ("%s,%s,%s\n",mktime($7),mktime(p),t); p=$7 }' Sample
Giving the following output:-
1335222005,-1,1335222006
1335222021,1335222005,1335222022
1335222023,1335222021,1335222024
1335222029,1335222023,1335222030
1335222034,1335222029,1335222035
1335222040,1335222034,1335222041
1335222046,1335222040,1335222047
1335222052,1335222046,1335222053
1335222060,1335222052,1335222061
1335222064,1335222060,1335222065
for each line, the 7th column is converted to an epoch date and the difference between the epoch date on the previous line is calculated and added to t.
On the first line being processed, because p is currently not a date, mktime returns -1 throwing out my figures.
What I want to do is, tell the AWK script, if line 1 is being processed then assume the difference is 6. At the moment it is subtracting -1 from 1335222005 resulting in 1335222006.
I want to say, start t at 6, then on the second line, work out the difference in epoch seconds to the previous line and increment t by that amount.
You just need to do something special for line 1.
awk -F, '
{gsub(/[\/:]/," ",$7); this_time = mktime($7)}
NR != 1 {t += this_time - prev_time; print this_time, prev_time, t}
{prev_time = this_time}
' << END
Given your input data, this prints
1335240021 1335240005 16
1335240023 1335240021 18
1335240029 1335240023 24
1335240034 1335240029 29
1335240040 1335240034 35
1335240046 1335240040 41
1335240052 1335240046 47
1335240060 1335240052 55
1335240064 1335240060 59
Alternately, a convenient way to initialize a variable is with awk'f -v option
awk -v t=6 '... same as before ...'
In awk you can initialize a variable in a BEGIN block, and exist two variables to get line number, both are useful for your case, FNR and NR:
BEGIN { t = 6 }
or
FNR == 1 { t = 6 }
Would using BEGIN (see here) help?
That will allow initialization of t variable to whatever you want. Something like
awk -F, 'BEGIN {t=6} { gsub("/"," ",$7); gsub(":"," ",$7); t+=(mktime($7)-mktime(p)); printf ("%s,%s,%s\n",mktime($7),mktime(p),t); p=$7 }' Sample