I'm trying to loop an awk command using bash script and I'm having a hard time including a variable within the single quotes for the awk command. I'm thinking I should be doing this completely in awk, but I feel more comfortable with bash right now.
#!/bin/bash
index="1"
while [ $index -le 13 ]
do
awk "'"/^$index/ {print}"'" text.txt
done
Use the standard approach -- -v option of awk to set/pass the variable:
awk -v idx="$index" '$0 ~ "^"idx' text.txt
Here i have set the variable idx as having the value of shell variable $index. Inside awk, i have simply used idx as an awk variable.
$0 ~ "^"idx matches if the record starts with (^) whatever the variable idx contains; if so, print the record.
awk '/'"$index"'/' text.txt
# A lil play with the script part where you split the awk command
# and sandwich the bash variable in between using double quotes
# Note awk prints by default, so idiomatic awk omits the '{print}' too.
should do, alternatively use grep like
grep "$index" text.txt # Mind the double quotes
Note : -le is used for comparing numerals, so you may change index="1" to index=1.
I have this line:
102:20620453:d=2017021012:UGRD:10 m above ground:15 hour fcst::lon=79.500000,lat=9.000000,val=-5.35
Now I want to just print the value -5.35 from this line and nothing else.
I also want this command to be able to extract the -7.04 from this line and nothing else.
102:20620453:d=2017021012:UGRD:10 m above ground:15 hour fcst::lon=280.500000,lat=11.000000,val=-7.04
I have read the other stack overflow questions and they did not seem to quite get at what I was looking for. I noticed that they did you awk or sed. What types of things should I do to be able to extract just the part of the above lines after val=?
There's no need for awk, sed, or any other external tool: bash has its own built-in regular expression support, via the =~ operator to [[ ]], and the BASH_REMATCH array (populated with matched contents).
val_re='[, ]val=([^ ]+)'
line='102:20620453:d=2017021012:UGRD:10 m above ground:15 hour fcst::lon=79.500000,lat=9.000000,val=-5.35'
[[ $line =~ $val_re ]] && echo "${BASH_REMATCH[1]}"
That said, if you really want to remove everything up to and including the string val= (and thus to have your code break if other values were added to the format in the future), you could also do so like this:
val=${line##*val=} # assign everything from $line after the last instance of "val=" to val
The syntax here is parameter expansion. See also BashFAQ #100: How do I do string manipulations in bash?
You can use awk with field separator as = and print last field:
awk -F'=' '{print $NF}' <<< "$str"
-5.35
this will search the string val= from the end and give anything after that
str='102:20620453:d=2017021012:UGRD:10 m above ground:15 hour fcst::lon=79.500000,lat=9.000000,val=-5.35'
echo "$str" | grep -Po '(?<=val=).*'
answer works on GNU grep only
I have some logs called ts.log that look like
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Parameters: []
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Types: []
[957670][DEBUG:2016-11-30 16:50:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} ResultSet
[957670][DEBUG:2016-11-30 16:51:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Header: [LAST_INSERT_ID()]
[957670][DEBUG:2016-11-30 16:52:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Result: [731747]
[065417][DEBUG:2016-11-30 16:53:17,986:sdk.protocol.process.InitProcessor.process(InitProcessor.java:61)]query String=requestid=10547
I have a script in which there's sth like
#!/bin/bash
begin=$1
cat ts.log | awk -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="16:50:17"){print $0}}'
instead of inputting the time like 16:50:17 I want to just pass $1 of shell to awk so that all I need to do is ./script time:hh:mm:ss The script will look like
#!/bin/bash
begin=$1
cat ts.log | awk -v var=$begin -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="var"){print $0}}'
But the double quotes need to be there OR it won't work.
I tried 2>"\""var"\""
but it doesn't work.
so is there a way to keep the double quotes there?
preferred result ./script
then extract the log from the time specified as $1.
There's many ways to do what you want.
Option 1: Using double quotes enclosing awk program
#!/bin/bash
begin=$1
awk -F '[ ,]' "\$2 ~ /^..:..:../ && \$2 >= \"${begin}\" " ts.log
Inside double quotes strings, bash does variable substitution. So $begin or ${begin} will be replaced with the shell variable value (whatever sent by the user)
Undesired effect: awk special variables starting with $ must be escaped with '\' or bash will try to replace them before execute awk.
To get a double quote char (") in bash double quote strings, it has to be escaped with '\', so in bash " \"16:50\" " will be replaced with "16:50". (This won't work with single quote strings, that don't have expansion of variables nor escaped chars at all in bash).
To see what variable substitutions are made when bash executes the script, you can execute it with debug option (it's very enlightening):
$ bash -x yourscript.sh 16:50
Option 2: Using awk variables
#!/bin/bash
begin=$1
awk -F '[ ,]' -v begin=$begin '$2 ~ /^..:..:../ && $2 >= begin' ts.log
Here an awk variable begin is created with option -v varname=value.
Awk variables can be used in any place of awk program as any other awk variable (don't need double quotes nor $).
There are other options, but I think you can work with these two.
In both options I've changed a bit your script:
It doesn't need cat to send data to awk, because awk can execute your program in one or more data files sent as parameters after your program.
Your awk program doesn't need include print at all (as #fedorqui said), because a basic awk program is composed by pairs of pattern {code}, where pattern is the same as you used in the if sentence, and the default code is {print $0}.
I've also changed the time pattern, primarly to clarify the script, but in a log file there's almost no chance that exists some 8 char length string that has 2 colons inside (regexp: . repaces any char)
i am trying to replace the following string for ex:
from
['55',2,1,10,30,23],
to
['55',2,555,10,30,23],
OR
['55',2,1,10,30,23],
to
['55',2,1,10,9999,23],
i search around and find this :
$ echo "[55,2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="[55"){$2=10}{print}}'
[55,10,1,10,30,23],
but it's not working in my case since there is " ' " around the value of $1 in my if condition :
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['55'"){$2=10}{print}}'
['55',2,1,10,30,23],
The problem is not in the awk code, it's the shell expansion. You cannot have single quotes in a singly-quoted shell string. This is the same problem you run into when you try to put the input string into single quotes:
$ echo '['55',2,1,10,30,23],'
[55,2,1,10,30,23],
-- the single quotes are gone! And this makes sense, because they did their job of quoting the [ and the ,2,1,10,30,23], (the 55 is unquoted here), but it is not what we wanted.
A solution is to quote the sections between them individually and squeeze them in manually:
$ echo '['\''55'\'',2,1,10,30,23],'
['55',2,1,10,30,23],
Or, in this particular case, where nothing nefarious is between where the single quotes should be,
echo '['\'55\'',2,1,10,30,23],' # the 55 is now unquoted.
Applied to your awk code, that looks like this:
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['\'55\''"){$2=10}{print}}'
['55',10,1,10,30,23],
Alternatively, since this doesn't look very nice if you have many single quotes in your code, you can write the awk code into a file, say foo.awk, and use
echo "['55',2,1,10,30,23]," | awk -F, -f foo.awk
Then you don't have to worry about shell quoting mishaps in the awk code because the awk code is not subject to shell expansion anymore.
I think how to match and replace is not the problem for you. The problem you were facing is, how to match a single quote ' in field.
To avoid to escape each ' in your codes, and to make your codes more readable, you can assigen the quote to a variable, and use the variable in your codes, for example like this:
echo "['55' 1
['56' 1"|awk -v q="'" '$1=="["q"55"q{$2++}7'
['55' 2
['56' 1
In the above example, only in line with ['55', the 2nd field got incremented.
In following shell script I want to perform two different tasks depending on file type,
but it is giving an error: "[==c]: command not found"
echo "enter file name"
read num
var_check= echo $str |awk -F . '{if (NF>1) {print $NF}}'
if ["$var_check"=="c"];then
echo "Some task for c"
elif ["$var_check"=="cpp"];then
echo "Some task for cpp"
else
echo "Wrong file extension"
fi
You wrote:
if ["$var_check"=="c"];then
The [ command is a command; its name must be surrounded by spaces (put simplistically).
if [ "$var_check" == "c" ]; then
The last argument, ], must also be preceded by a space. The operands within must also be space separated; they need to be separate arguments. The rules for the [[ ... ]] operator are a bit different, but using spaces helps people read the code even there. What you wrote is a bit like expecting:
ls"-l"/dev/tty
to work; it won't.
You also need to double check whether your test or [ operator supports ==; the normal form is =.
The line:
var_check= echo $str |awk -F . '{if (NF>1) {print $NF}}'
This runs the echo command with var_check set as an environment variable, which is unlikely to be what you wanted. You almost certainly intended to write:
var_check=$(echo $str |awk -F . '{if (NF>1) {print $NF}}')
This runs the echo and awk commands and captures the output in var_check. Use the $(...) notation in preference to the older but more complex to use `...` notation. In simple cases, they look the same; when you nest them, the $(...) notation is far, far simpler to understand and use.
Also, looking on the larger scale (3 lines instead of just 1 line):
echo "enter file name"
read num
var_check=$(echo $str |awk -F . '{if (NF>1) {print $NF}}')
You read the file name into variable num; you then echo $str instead of $num. If you've already got $str set somewhere earlier in the script (in unshown code), what you've got may be fine. Taken as a standalone fragment, it isn't right.
You could also simplify the awk a little:
var_check=$(echo $str |awk -F . 'NF > 1 {print $NF}')
This would work the same as what you wrote, but uses fewer parentheses and braces.