Remove lines containing non-numeric entries in bash - linux

I have a sample data file (sample.log) that has entries
0.0262
0.0262
0.7634
5.7262
0.abc02
I need to filter out the lines that contain non-numeric data, in the above lines, the last entry.
I tried this
sed 's/[^0-9]//g' sample.log
It removes the non-numeric line but also removes the decimal values, the output I get is
00262
00262
07634
57262
How can I get the original values retained after eliminating the non-numeric lines. Can I use tr or awk

You can't do this job robustly with sed or grep or any other tool that doesn't understand numbers, you need awk instead:
$ cat file
1e3
1f3
0.1.2.3
0.123
$ awk '$0==($0+0)' file
1e3
0.123
The best you could do with a sed solution would be:
$ sed '/[^0-9.]/d; /\..*\./d' file
0.123
which removes all lines that contains anything other than a digit or period then all those that contain 2 or more periods (e.g. an IP address) but that still can't recognize the exponent notation as a number.
If you have hex input data and GNU awk (see #dawg's comment below):
$ echo "0x123" | awk --non-decimal-data '$0==($0+0){printf "%s => %f\n", $0, ($0+0)}'
0x123 => 291.000000

In awk:
awk '/^[[:digit:].]+$/{print $0}' file
Or, you negate that (and add potential + or - if that is in your strings):
awk '/[^[:digit:].+-]/{next} 1' file
Or, same logic with sed:
sed '/[^[:digit:].+-]/d' file
Ed Morton's solution is robust. Given:
$ cat nums.txt
1e6
.1e6
1E6
.001
.
0.001
.1.2
1abc2
0.0
-0
-0.0
0x123
0223
011
NaN
inf
abc
$ awk '$0==($0+0) {printf "%s => %f\n", $0, ($0+0)}
$0!=($0+0) {notf[$0]++;}
END {for (e in notf) print "\""e"\""" not a float"}' /tmp/nums.txt
1e6 => 1000000.000000
.1e6 => 100000.000000
1E6 => 1000000.000000
.001 => 0.001000
0.001 => 0.001000
0.0 => 0.000000
-0 => 0.000000
-0.0 => 0.000000
0x123 => 291.000000
0223 => 223.000000
011 => 11.000000
NaN => nan
inf => inf
".1.2" not a float
"1abc2" not a float
"abc" not a float
"." not a float

You can do it easily with grep if you discard any line that contains any letter:
grep -v [a-z] test

Use:
$ sed -i '/.*[a-z].*/d' sample.log

This might work for you (GNU sed):
sed '/[^0-9.]/d' file
However this may give a false positive on say an IP address i.e. allowing more than one ..
Using your test data:
sed '/^[0-9]\.[0-9]\{4\}$/!d' file
Would only match a digit, followed by a . followed by 4 digits.

Related

Extract the uptime value from "w" command output

How can I get the value of up from below command on linux?
# w
01:16:08 up 20:29, 1 user, load average: 0.50, 0.34, 0.30
USER TTY LOGIN# IDLE JCPU PCPU WHAT
root pts/0 00:57 0.00s 0.11s 0.02s w
# w | grep up
01:16:17 up 20:29, 1 user, load average: 0.42, 0.33, 0.29
On Linux, the easiest way to get the uptime in (fractional) seconds is via the 1st field of /proc/uptime (see man proc):
$ cut -d ' ' -f1 /proc/uptime
350735.47
To format that number the same way that w and uptime do, using awk:
$ awk '{s=int($1);d=int(s/86400);h=int(s % 86400/3600);m=int(s % 3600 / 60);
printf "%d days, %02d:%02d\n", d, h, m}' /proc/uptime
4 days, 01:25 # 4 days, 1 hour, and 25 minutes
To answer the question as asked - parsing the output of w (or uptime, whose output is the same as w's 1st output line, which contains all the information of interest), which also works on macOS/BSD, with a granularity of integral seconds:
A perl solution:
<(uptime) is a Bash process substitution that provides uptime's output as input to the perl command - see bottom.
$ perl -nle 'print for / up +((?:\d+ days?, +)?[^,]+)/' <(uptime)
4 days, 01:25
This assumes that days is the largest unit every displayed.
perl -nle tells Perl to process the input line by line, without printing any output by default (-n), automatically stripping the trailing newline from each input line on input, and automatically appending one on output (-l); -e tells Perl to treat the next argument as the script (expression) to process.
print for /.../ tells Perl to output what each capture group (...) inside regex /.../ captures.
up + matches literal up, preceded by (at least) one space and followed by 1 or more spaces (+)
(?:\d+ days?, +)? is a non-capturing subexpression - due to ?: - that matches:
1 or more digits (\d+)
followed by a single space
followed by literal day, optionally followed by a literal s (s?)
the trailing ? makes the entire subexpression optional, given that a number-of-days part may or may not be present.
[^,]+ matches 1 or more (+) subsequent characters up to, but not including a literal , ([^,]) - this is the hh:mm part.
The overall capture group - the outer (...) therefore captures the entire up-time expression - whether composed of hh:mm only, or preceded by <n> day/s> - and prints that.
<(uptime) is a Bash process substitution (<(...))
that, loosely speaking, presents uptime's output as a (temporary, self-deleting) file that perl can read via stdin.
Something like this with gnu sed:
$ w |head -n1
02:06:19 up 3:42, 1 user, load average: 0.01, 0.05, 0.13
$ w |sed -r '1 s/.*up *(.*),.*user.*/\1/g;q'
3:42
$ echo "18:35:23 up 18 days, 9:08, 6 users, load average: 0.09, 0.31, 0.41" \
|sed -r '1 s/.*up *(.*),.*user.*/\1/g;q'
18 days, 9:08
Given that the format of the uptime depends on whether it is less or more than 24 hours, the best I could come up with is a double awk:
$ w
18:35:23 up 18 days, 9:08, 6 users,...
$ w | awk -F 'user|up ' 'NF > 1 {print $2}' \
| awk -F ',' '{for(i = 1; i < NF; i++) {printf("%s ",$i)}} END{print ""}'
18 days 9:08

awk after grep: print value when grep returns nothing

I have a question when I use awk and grep to parse log files.
The log file contains some strings with figures, e.g.
Amount: 20
Amount: 30.1
And I use grep to parse the lines with keyword "Amount", and then use awk to get the amount and do a sum:
the command is like:
cat mylog.log | grep Amount | awk -F 'Amount: ' '{sum+=$2}END{print sum}'
It works fine for me. However, sometimes the mylog.log file does not contains the keyword 'Amount'. In this case, I want to print 0, but the above awk command will print nothing. How can I make awk print something when grep returns nothing?
You can use this:
awk '/^Amount/ {amount+=$2} END {print amount+0}' file
With the +0 trick you make it print 0 in case the value is not set.
Explanation
There is no need to grep + awk. awk alone can grep (and many more things!):
/^Amount/ {} on lines starting with "Amount", perform what is in {}.
amount+=$2 add field 2's value to the counter "amount".
END {print amount+0} after processing the whole file, print the value of amount. Doing +0 makes it print 0 if it wasn't set before.
Note also there is no need to set 'Amount' as the field separator. It suffices with the default one (the space).
Test
$ cat a
Amount: 20
Amount: 30.1
$ awk '/^Amount/ {amount+=$2} END {print amount+0}' a
50.1
$ cat b
hello
$ awk '/^Amount/ {amount+=$2} END {print amount+0}' b
0
If your line only contains "Amount: 20" then use #fedorqui's solution, but if it's more like "The quick brown fox had Amount: 20 bananas" then use:
awk -F'Amount:' 'NF==2{sum+=$2} END{print sum+0}' file
Awk one-liner,
awk -F 'Amount: ' '/Amount:/{print "1";sum+=$2}!/Amount:/{print "0"}END{print sum}' file
The above awk command would print the number 1 for the lines which has the string Amount and it prints 0 for the lines which don't have the string Amount. And also if the string Amount is found on a line then it stores the value(column 2) to the sum variable and adds it with any further values. Finally the value of the variable sum is printed at the last.
Example:
$ cat file
Amount: 20
Amount: 30.1
foo bar
adbcksjc
sbcskcbks
cnskncsnc
$ awk -F 'Amount: ' '/Amount:/{print "1";sum+=$2}!/Amount:/{print "0"}END{print sum}' file
1
1
0
0
0
0
50.1

Is there any way that i can increment some number in sed after pattern matching

Is there any way that i can increment some number in sed after pattern matching
suppose i have this file
201 AD BBH NN
376 AD HGH JU
I want to match the starting integers and then add the number 5 to it in sed
Is this possible
You're probably better off using a slightly more advanced tool, such as awk:
pax$ cat qq.in
201 AD BBH NN
376 AD HGH JU
pax$ awk '{ print $0 " " $1+5 }' qq.in
201 AD BBH NN 206
376 AD HGH JU 381
If you really must do it in sed then, yes, it can be done. But it's butt-ugly. See here for a way of doing it:
#!/usr/bin/sed -f
/[^0-9]/ d
# replace all leading 9s by _ (any other character except digits, could
# be used)
:d
s/9\(_*\)$/_\1/
td
# incr last digit only. The first line adds a most-significant
# digit of 1 if we have to add a digit.
#
# The tn commands are not necessary, but make the thing
# faster
s/^\(_*\)$/1\1/; tn
s/8\(_*\)$/9\1/; tn
s/7\(_*\)$/8\1/; tn
s/6\(_*\)$/7\1/; tn
s/5\(_*\)$/6\1/; tn
s/4\(_*\)$/5\1/; tn
s/3\(_*\)$/4\1/; tn
s/2\(_*\)$/3\1/; tn
s/1\(_*\)$/2\1/; tn
s/0\(_*\)$/1\1/; tn
:n
y/_/0/
This particular script adds 1 to a number and you can now (hopefully) understand why I called it butt-ugly. Trying to do this with sed is akin to trying to cut down a Karri tree with a goldfish.
You should use the right tool for the job.
with awk you can try
cat fileName | awk '{num = 0; if ($1 ~ /[0-9][0-9][0-9]/) num = $1 + 5; print num $1 $2 $3;}'

How do I use sed to remove decimal numbers from a string?

I have the following string as an example:
ex. "Abandoned 16 1.10 2.62 3.50"
I would like to pipe this result to sed and remove all decimal numbers to leave me with the following:
ex. "Abandoned 16"
I was using the following command: sed 's/.//g'
which apparently doesn't work.
Can someone let me know how to use the wildcard character with sed to remove anything matching ".".
Thanks
You haven't said what you want to do with the whitespace, but how about
sed -e 's/[0-9]*\.[0-9]*//g' -e 's/ *$//'
this would be easier with awk, at least for me
echo "Abandoned 16 1.10 2.62 3.50" | awk '{print $1FS$2}'
but is the list of numbers random afterwards?
if so, this works too
echo "Abandoned 16 1.10 2.62 3.50" | sed -r 's/\s([0-9]+)\.([0-9]+)//g'
note that \s catches the white space, and that the numbers before and after the decimal are saved, so if you want to retain them and do something with them you can access them with \1 and \2 respectfully
Why catch the white sapce? well imagine if 16 came after 3.50 in your example you would then return
Abandoned [5spaces*] 16
*I can only insert one space in this <textarea>
Trowing in a awk solution that loops ovewr the input and skips entries with a period in them
{
printf("%s ", $1)
for(i=2;i<NF;i++) {
if ($i !~ /\./) {
printf( " %s ", $i)
}
}
}
$ echo Abandoned 16 1.10 2.62 3.50 | awk -f f.awk
Abandoned 16

Using awk to print all columns from the nth to the last

This line worked until I had whitespace in the second field.
svn status | grep '\!' | gawk '{print $2;}' > removedProjs
is there a way to have awk print everything in $2 or greater? ($3, $4.. until we don't have anymore columns?)
I suppose I should add that I'm doing this in a Windows environment with Cygwin.
Print all columns:
awk '{print $0}' somefile
Print all but the first column:
awk '{$1=""; print $0}' somefile
Print all but the first two columns:
awk '{$1=$2=""; print $0}' somefile
There's a duplicate question with a simpler answer using cut:
svn status | grep '\!' | cut -d\ -f2-
-d specifies the delimeter (space), -f specifies the list of columns (all starting with the 2nd)
You could use a for-loop to loop through printing fields $2 through $NF (built-in variable that represents the number of fields on the line).
Edit:
Since "print" appends a newline, you'll want to buffer the results:
awk '{out = ""; for (i = 2; i <= NF; i++) {out = out " " $i}; print out}'
Alternatively, use printf:
awk '{for (i = 2; i <= NF; i++) {printf "%s ", $i}; printf "\n"}'
awk '{out=$2; for(i=3;i<=NF;i++){out=out" "$i}; print out}'
My answer is based on the one of VeeArr, but I noticed it started with a white space before it would print the second column (and the rest). As I only have 1 reputation point, I can't comment on it, so here it goes as a new answer:
start with "out" as the second column and then add all the other columns (if they exist). This goes well as long as there is a second column.
Most solutions with awk leave an space. The options here avoid that problem.
Option 1
A simple cut solution (works only with single delimiters):
command | cut -d' ' -f3-
Option 2
Forcing an awk re-calc sometimes remove the added leading space (OFS) left by removing the first fields (works with some versions of awk):
command | awk '{ $1=$2="";$0=$0;} NF=NF'
Option 3
Printing each field formatted with printf will give more control:
$ in=' 1 2 3 4 5 6 7 8 '
$ echo "$in"|awk -v n=2 '{ for(i=n+1;i<=NF;i++) printf("%s%s",$i,i==NF?RS:OFS);}'
3 4 5 6 7 8
However, all previous answers change all repeated FS between fields to OFS. Let's build a couple of option that do not do that.
Option 4 (recommended)
A loop with sub to remove fields and delimiters at the front.
And using the value of FS instead of space (which could be changed).
Is more portable, and doesn't trigger a change of FS to OFS:
NOTE: The ^[FS]* is to accept an input with leading spaces.
$ in=' 1 2 3 4 5 6 7 8 '
$ echo "$in" | awk '{ n=2; a="^["FS"]*[^"FS"]+["FS"]+";
for(i=1;i<=n;i++) sub( a , "" , $0 ) } 1 '
3 4 5 6 7 8
Option 5
It is quite possible to build a solution that does not add extra (leading or trailing) whitespace, and preserve existing whitespace(s) using the function gensub from GNU awk, as this:
$ echo ' 1 2 3 4 5 6 7 8 ' |
awk -v n=2 'BEGIN{ a="^["FS"]*"; b="([^"FS"]+["FS"]+)"; c="{"n"}"; }
{ print(gensub(a""b""c,"",1)); }'
3 4 5 6 7 8
It also may be used to swap a group of fields given a count n:
$ echo ' 1 2 3 4 5 6 7 8 ' |
awk -v n=2 'BEGIN{ a="^["FS"]*"; b="([^"FS"]+["FS"]+)"; c="{"n"}"; }
{
d=gensub(a""b""c,"",1);
e=gensub("^(.*)"d,"\\1",1,$0);
print("|"d"|","!"e"!");
}'
|3 4 5 6 7 8 | ! 1 2 !
Of course, in such case, the OFS is used to separate both parts of the line, and the trailing white space of the fields is still printed.
NOTE: [FS]* is used to allow leading spaces in the input line.
I personally tried all the answers mentioned above, but most of them were a bit complex or just not right. The easiest way to do it from my point of view is:
awk -F" " '{ for (i=4; i<=NF; i++) print $i }'
Where -F" " defines the delimiter for awk to use. In my case is the whitespace, which is also the default delimiter for awk. This means that -F" " can be ignored.
Where NF defines the total number of fields/columns. Therefore the loop will begin from the 4th field up to the last field/column.
Where $N retrieves the value of the Nth field. Therefore print $i will print the current field/column based based on the loop count.
awk '{ for(i=3; i<=NF; ++i) printf $i""FS; print "" }'
lauhub proposed this correct, simple and fast solution here
This was irritating me so much, I sat down and wrote a cut-like field specification parser, tested with GNU Awk 3.1.7.
First, create a new Awk library script called pfcut, with e.g.
sudo nano /usr/share/awk/pfcut
Then, paste in the script below, and save. After that, this is how the usage looks like:
$ echo "t1 t2 t3 t4 t5 t6 t7" | awk -f pfcut --source '/^/ { pfcut("-4"); }'
t1 t2 t3 t4
$ echo "t1 t2 t3 t4 t5 t6 t7" | awk -f pfcut --source '/^/ { pfcut("2-"); }'
t2 t3 t4 t5 t6 t7
$ echo "t1 t2 t3 t4 t5 t6 t7" | awk -f pfcut --source '/^/ { pfcut("-2,4,6-"); }'
t1 t2 t4 t6 t7
To avoid typing all that, I guess the best one can do (see otherwise Automatically load a user function at startup with awk? - Unix & Linux Stack Exchange) is add an alias to ~/.bashrc; e.g. with:
$ echo "alias awk-pfcut='awk -f pfcut --source'" >> ~/.bashrc
$ source ~/.bashrc # refresh bash aliases
... then you can just call:
$ echo "t1 t2 t3 t4 t5 t6 t7" | awk-pfcut '/^/ { pfcut("-2,4,6-"); }'
t1 t2 t4 t6 t7
Here is the source of the pfcut script:
# pfcut - print fields like cut
#
# sdaau, GNU GPL
# Nov, 2013
function spfcut(formatstring)
{
# parse format string
numsplitscomma = split(formatstring, fsa, ",");
numspecparts = 0;
split("", parts); # clear/initialize array (for e.g. `tail` piping into `awk`)
for(i=1;i<=numsplitscomma;i++) {
commapart=fsa[i];
numsplitsminus = split(fsa[i], cpa, "-");
# assume here a range is always just two parts: "a-b"
# also assume user has already sorted the ranges
#print numsplitsminus, cpa[1], cpa[2]; # debug
if(numsplitsminus==2) {
if ((cpa[1]) == "") cpa[1] = 1;
if ((cpa[2]) == "") cpa[2] = NF;
for(j=cpa[1];j<=cpa[2];j++) {
parts[numspecparts++] = j;
}
} else parts[numspecparts++] = commapart;
}
n=asort(parts); outs="";
for(i=1;i<=n;i++) {
outs = outs sprintf("%s%s", $parts[i], (i==n)?"":OFS);
#print(i, parts[i]); # debug
}
return outs;
}
function pfcut(formatstring) {
print spfcut(formatstring);
}
Would this work?
awk '{print substr($0,length($1)+1);}' < file
It leaves some whitespace in front though.
Printing out columns starting from #2 (the output will have no trailing space in the beginning):
ls -l | awk '{sub(/[^ ]+ /, ""); print $0}'
echo "1 2 3 4 5 6" | awk '{ $NF = ""; print $0}'
this one uses awk to print all except the last field
This is what I preferred from all the recommendations:
Printing from the 6th to last column.
ls -lthr | awk '{out=$6; for(i=7;i<=NF;i++){out=out" "$i}; print out}'
or
ls -lthr | awk '{ORS=" "; for(i=6;i<=NF;i++) print $i;print "\n"}'
If you need specific columns printed with arbitrary delimeter:
awk '{print $3 " " $4}'
col#3 col#4
awk '{print $3 "anything" $4}'
col#3anythingcol#4
So if you have whitespace in a column it will be two columns, but you can connect it with any delimiter or without it.
Perl solution:
perl -lane 'splice #F,0,1; print join " ",#F' file
These command-line options are used:
-n loop around every line of the input file, do not automatically print every line
-l removes newlines before processing, and adds them back in afterwards
-a autosplit mode – split input lines into the #F array. Defaults to splitting on whitespace
-e execute the perl code
splice #F,0,1 cleanly removes column 0 from the #F array
join " ",#F joins the elements of the #F array, using a space in-between each element
Python solution:
python -c "import sys;[sys.stdout.write(' '.join(line.split()[1:]) + '\n') for line in sys.stdin]" < file
I want to extend the proposed answers to the situation where fields are delimited by possibly several whitespaces –the reason why the OP is not using cut I suppose.
I know the OP asked about awk, but a sed approach would work here (example with printing columns from the 5th to the last):
pure sed approach
sed -r 's/^\s*(\S+\s+){4}//' somefile
Explanation:
s/// is the standard command to perform substitution
^\s* matches any consecutive whitespace at the beginning of the line
\S+\s+ means a column of data (non-whitespace chars followed by whitespace chars)
(){4} means the pattern is repeated 4 times.
sed and cut
sed -r 's/^\s+//; s/\s+/\t/g' somefile | cut -f5-
by just replacing consecutive whitespaces by a single tab;
tr and cut:
tr can also be used to squeeze consecutive characters with the -s option.
tr -s [:blank:] <somefile | cut -d' ' -f5-
If you don't want to reformat the part of the line that you don't chop off, the best solution I can think of is written in my answer in:
How to print all the columns after a particular number using awk?
It chops what is before the given field number N, and prints all the rest of the line, including field number N and maintaining the original spacing (it does not reformat). It doesn't mater if the string of the field appears also somewhere else in the line.
Define a function:
fromField () {
awk -v m="\x01" -v N="$1" '{$N=m$N; print substr($0,index($0,m)+1)}'
}
And use it like this:
$ echo " bat bi iru lau bost " | fromField 3
iru lau bost
$ echo " bat bi iru lau bost " | fromField 2
bi iru lau bost
Output maintains everything, including trailing spaces
In you particular case:
svn status | grep '\!' | fromField 2 > removedProjs
If your file/stream does not contain new-line characters in the middle of the lines (you could be using a different Record Separator), you can use:
awk -v m="\x0a" -v N="3" '{$N=m$N ;print substr($0, index($0,m)+1)}'
The first case will fail only in files/streams that contain the rare hexadecimal char number 1
This awk function returns substring of $0 that includes fields from begin to end:
function fields(begin, end, b, e, p, i) {
b = 0; e = 0; p = 0;
for (i = 1; i <= NF; ++i) {
if (begin == i) { b = p; }
p += length($i);
e = p;
if (end == i) { break; }
p += length(FS);
}
return substr($0, b + 1, e - b);
}
To get everything starting from field 3:
tail = fields(3);
To get section of $0 that covers fields 3 to 5:
middle = fields(3, 5);
b, e, p, i nonsense in function parameter list is just an awk way of declaring local variables.
All of the other answers given here and in linked questions fail in various ways given various possible FS values. Some leave leading and/or trailing white space, some convert every FS to the OFS, some rely on semantics that only apply when FS is the default value, some rely on negating FS in a bracket expression which will fail given a multi-char FS, etc.
To do this robustly for any FS, use GNU awk for the 4th arg to split():
$ cat tst.awk
{
split($0,flds,FS,seps)
for ( i=n; i<=NF; i++ ) {
printf "%s%s", flds[i], seps[i]
}
print ""
}
$ printf 'a b c d\n' | awk -v n=3 -f tst.awk
c d
$ printf ' a b c d\n' | awk -v n=3 -f tst.awk
c d
$ printf ' a b c d\n' | awk -v n=3 -F'[ ]' -f tst.awk
b c d
$ printf ' a b c d\n' | awk -v n=3 -F'[ ]+' -f tst.awk
b c d
$ printf 'a###b###c###d\n' | awk -v n=3 -F'###' -f tst.awk
c###d
$ printf '###a###b###c###d\n' | awk -v n=3 -F'###' -f tst.awk
b###c###d
Note that I'm using split() above because it's 3rg arg is a field separator, not just a regexp like the 2nd arg to match(). The difference is that field separators have additional semantics to regexps such as skipping leading and/or trailing blanks when the separator is a single blank char - if you wanted to use a while(match()) loop or any form of *sub() to emulate the above then you'd need to write code to implement those semantics whereas split() already implements them for you.
Awk examples looks complex here, here is simple Bash shell syntax:
command | while read -a cols; do echo ${cols[#]:1}; done
Where 1 is your nth column counting from 0.
Example
Given this content of file (in.txt):
c1
c1 c2
c1 c2 c3
c1 c2 c3 c4
c1 c2 c3 c4 c5
here is the output:
$ while read -a cols; do echo ${cols[#]:1}; done < in.txt
c2
c2 c3
c2 c3 c4
c2 c3 c4 c5
This would work if you are using Bash and you could use as many 'x ' as elements you wish to discard and it ignores multiple spaces if they are not escaped.
while read x b; do echo "$b"; done < filename
Perl:
#m=`ls -ltr dir | grep ^d | awk '{print \$6,\$7,\$8,\$9}'`;
foreach $i (#m)
{
print "$i\n";
}
UPDATE :
if you wanna use no function calls at all while preserving the spaces and tabs in between the remaining fields, then do :
echo " 1 2 33 4444 555555 \t6666666 " |
{m,g}awk ++NF FS='^[ \t]*[^ \t]*[ \t]+|[ \t]+$' OFS=
=
2 33 4444 555555 6666666
===================
You can make it a lot more straight forward :
svn status | [m/g]awk '/!/*sub("^[^ \t]*[ \t]+",_)'
svn status | [n]awk '(/!/)*sub("^[^ \t]*[ \t]+",_)'
Automatically takes care of the grep earlier in the pipe, as well as trimming out extra FS after blanking out $1, with the added bonus of leaving rest of the original input untouched instead of having tabs overwritten with spaces (unless that's the desired effect)
If you're very certain $1 does not contain special characters that need regex escaping, then it's even easier :
mawk '/!/*sub($!_"[ \t]+",_)'
gawk -c/P/e '/!/*sub($!_"""[ \t]+",_)'
Or if you prefer customizing FS+OFS to handle it all :
mawk 'NF*=/!/' FS='^[^ \t]*[ \t]+' OFS='' # this version uses OFS
This should be a reasonably comprehensive awk-field-sub-string-extraction function that
returns substring of $0 based on input ranges, inclusive
clamp in out of range values,
handle variable length field SEPs
has speedup treatments for ::
completely no inputs, returning $0 directly
input values resulting in guaranteed empty string ("")
FROM-field == 1
FS = "" that has split $0 out by individual chars
(so the FROM <(_)> and TO <(__)> fields behave like cut -c rather than cut -f)
original $0 restored, w/o overwriting FS seps with OFS
|
{m,g}awk '{
2 print "\n|---BEFORE-------------------------\n"
3 ($0) "\n|----------------------------\n\n ["
4 fld2(2, 5) "]\n [" fld2(3) "]\n [" fld2(4, 2)
5 "]<----------------------------------------------should be
6 empty\n [" fld2(3, 11) "]<------------------------should be
7 capped by NF\n [" fld2() "]\n [" fld2((OFS=FS="")*($0=$0)+11,
8 23) "]<-------------------FS=\"\", split by chars
9 \n\n|---AFTER-------------------------\n" ($0)
10 "\n|----------------------------"
11 }
12 function fld2(_,__,___,____,_____)
13 {
if (+__==(_=-_<+_ ?+_:_<_) || (___=____="")==__ || !NF) {
return $_
16 } else if (NF<_ || (__=NF<+__?NF:+__)<(_=+_?_:!_)) {
return ___
18 } else if (___==FS || _==!___) {
19 return ___<FS \
? substr("",$!_=$!_ substr("",__=$!(NF=__)))__
20 : substr($(_<_),_,__)
21 }
22 _____=$+(____=___="\37\36\35\32\31\30\27\26\25"\
"\24\23\21\20\17\16\6\5\4\3\2\1")
23 NF=__
24 if ($(!_)~("["(___)"]")) {
25 gsub("..","\\&&",___) + gsub(".",___,____)
27 ___=____
28 }
29 __=(_) substr("",_+=_^=_<_)
30 while(___!="") {
31 if ($(!_)!~(____=substr(___,--_,++_))) {
32 ___=____
33 break }
35 ___=substr(___,_+_^(!_))
36 }
37 return \
substr("",($__=___ $__)==(__=substr($!_,
_+index($!_,___))),_*($!_=_____))(__)
}'
those <TAB> are actual \t \011 but relabeled for display clarity
|---BEFORE-------------------------
1 2 33 4444 555555 <TAB>6666666
|----------------------------
[2 33 4444 555555]
[33]
[]<---------------------------------------------- should be empty
[33 4444 555555 6666666]<------------------------ should be capped by NF
[ 1 2 33 4444 555555 <TAB>6666666 ]
[ 2 33 4444 555555 <TAB>66]<------------------- FS="", split by chars
|---AFTER-------------------------
1 2 33 4444 555555 <TAB>6666666
|----------------------------
I wasn't happy with any of the awk solutions presented here because I wanted to extract the first few columns and then print the rest, so I turned to perl instead. The following code extracts the first two columns, and displays the rest as is:
echo -e "a b c d\te\t\tf g" | \
perl -ne 'my #f = split /\s+/, $_, 3; printf "first: %s second: %s rest: %s", #f;'
The advantage compared to the perl solution from Chris Koknat is that really only the first n elements are split off from the input string; the rest of the string isn't split at all and therefor stays completely intact. My example demonstrates this with a mix of spaces and tabs.
To change the amount of columns that should be extracted, replace the 3 in the example with n+1.
ls -la | awk '{o=$1" "$3; for (i=5; i<=NF; i++) o=o" "$i; print o }'
from this answer is not bad but the natural spacing is gone.
Please then compare it to this one:
ls -la | cut -d\ -f4-
Then you'd see the difference.
Even ls -la | awk '{$1=$2=""; print}' which is based on the answer voted best thus far is not preserve the formatting.
Thus I would use the following, and it also allows explicit selective columns in the beginning:
ls -la | cut -d\ -f1,4-
Note that every space counts for columns too, so for instance in the below, columns 1 and 3 are empty, 2 is INFO and 4 is:
$ echo " INFO 2014-10-11 10:16:19 main " | cut -d\ -f1,3
$ echo " INFO 2014-10-11 10:16:19 main " | cut -d\ -f2,4
INFO 2014-10-11
$
If you want formatted text, chain your commands with echo and use $0 to print the last field.
Example:
for i in {8..11}; do
s1="$i"
s2="str$i"
s3="str with spaces $i"
echo -n "$s1 $s2" | awk '{printf "|%3d|%6s",$1,$2}'
echo -en "$s3" | awk '{printf "|%-19s|\n", $0}'
done
Prints:
| 8| str8|str with spaces 8 |
| 9| str9|str with spaces 9 |
| 10| str10|str with spaces 10 |
| 11| str11|str with spaces 11 |
The top-voted answer by zed_0xff did not work for me.
I have a log where after $5 with an IP address can be more text or no text. I need everything from the IP address to the end of the line should there be anything after $5. In my case, this is actually within an awk program, not an awk one-liner so awk must solve the problem. When I try to remove the first 4 fields using the solution proposed by zed_0xff:
echo " 7 27.10.16. Thu 11:57:18 37.244.182.218" | awk '{$1=$2=$3=$4=""; printf "[%s]\n", $0}'
it spits out wrong and useless response (I added [..] to demonstrate):
[ 37.244.182.218 one two three]
There are even some suggestions to combine substr with this wrong answer, but that only complicates things. It offers no improvement.
Instead, if columns are fixed width until the cut point and awk is needed, the correct answer is:
echo " 7 27.10.16. Thu 11:57:18 37.244.182.218" | awk '{printf "[%s]\n", substr($0,28)}'
which produces the desired output:
[37.244.182.218 one two three]

Resources