This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 5 years ago.
I am trying to display which apps is deployed in each host. Using a bash script I had some success except I want to replace the following "hostname" part with the actual host name stored under the ${i} variable. I can't just substitute it because I cannot put a curly bracket inside another. EVEN if I can do it, I am still having trouble as ${i} will be replaced by the result of ls. How do I fix this?
hosts=(usa1 london2)
for i in ${hosts[#]}; do
echo ---${i}---
ssh ttoleung#${i} ls /apps | awk '{ printf("%s:%s\n", "hostname", $0) }'
done
Current output, based on code fragment above:
---usa01---
hostname:E2.gui
hostname:E1.server
---london2---
hostname:E1.gui
Desired output:
---usa01---
usa01:E2.gui
usa01:E1.server
---london2---
london2:E1.gui
Replace:
awk '{ printf("%s:%s\n", "hostname", $0) }'
With:
awk -v h="$i" '{ printf("%s:%s\n", h, $0) }'
-v h="$i" tells awk to create an awk variable h and assign to it the value of the shell variable $i.
Aside: we used h="$i" rather than h=$i because it is good practice to put shell variables inside double-quotes unless you want the shell to perform word-splitting and pathname expansion.
As an intro note, it is not a safe expand array names without double quotes unless for obvious reasons because doing so would split quoted strings in array that themselves have spaces
So change
for i in ${hosts[#]};
to
for i in "${hosts[#]}"; # Note the quoted array
Now, coming to your problem, you can pass bash variables to awk using its -v parameter. So change
ssh ttoleung#${i} ls /apps | awk '{ printf("%s:%s\n", "hostname", $0) }'
to
ssh ttoleung#${i} ls /apps | awk -v hname="${i}" '{ printf("%s:%s\n", hname, $0) }'
Here we pass shell parameter ${i} to awk variable hname.
Side Note: Don't parse ls output for the reasons mentioned [ here ]. In your case though, it doesn't make much of a difference.
I have some logs called ts.log that look like
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Parameters: []
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Types: []
[957670][DEBUG:2016-11-30 16:50:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} ResultSet
[957670][DEBUG:2016-11-30 16:51:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Header: [LAST_INSERT_ID()]
[957670][DEBUG:2016-11-30 16:52:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Result: [731747]
[065417][DEBUG:2016-11-30 16:53:17,986:sdk.protocol.process.InitProcessor.process(InitProcessor.java:61)]query String=requestid=10547
I have a script in which there's sth like
#!/bin/bash
begin=$1
cat ts.log | awk -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="16:50:17"){print $0}}'
instead of inputting the time like 16:50:17 I want to just pass $1 of shell to awk so that all I need to do is ./script time:hh:mm:ss The script will look like
#!/bin/bash
begin=$1
cat ts.log | awk -v var=$begin -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="var"){print $0}}'
But the double quotes need to be there OR it won't work.
I tried 2>"\""var"\""
but it doesn't work.
so is there a way to keep the double quotes there?
preferred result ./script
then extract the log from the time specified as $1.
There's many ways to do what you want.
Option 1: Using double quotes enclosing awk program
#!/bin/bash
begin=$1
awk -F '[ ,]' "\$2 ~ /^..:..:../ && \$2 >= \"${begin}\" " ts.log
Inside double quotes strings, bash does variable substitution. So $begin or ${begin} will be replaced with the shell variable value (whatever sent by the user)
Undesired effect: awk special variables starting with $ must be escaped with '\' or bash will try to replace them before execute awk.
To get a double quote char (") in bash double quote strings, it has to be escaped with '\', so in bash " \"16:50\" " will be replaced with "16:50". (This won't work with single quote strings, that don't have expansion of variables nor escaped chars at all in bash).
To see what variable substitutions are made when bash executes the script, you can execute it with debug option (it's very enlightening):
$ bash -x yourscript.sh 16:50
Option 2: Using awk variables
#!/bin/bash
begin=$1
awk -F '[ ,]' -v begin=$begin '$2 ~ /^..:..:../ && $2 >= begin' ts.log
Here an awk variable begin is created with option -v varname=value.
Awk variables can be used in any place of awk program as any other awk variable (don't need double quotes nor $).
There are other options, but I think you can work with these two.
In both options I've changed a bit your script:
It doesn't need cat to send data to awk, because awk can execute your program in one or more data files sent as parameters after your program.
Your awk program doesn't need include print at all (as #fedorqui said), because a basic awk program is composed by pairs of pattern {code}, where pattern is the same as you used in the if sentence, and the default code is {print $0}.
I've also changed the time pattern, primarly to clarify the script, but in a log file there's almost no chance that exists some 8 char length string that has 2 colons inside (regexp: . repaces any char)
I would like to calculate percentage in shell. But I can't do it. My script is
#!/bin/bash
#n1=$(wc -l < input.txt) #input.txt is a text file with 10000 lines
n1=10000
n2=$(awk '{printf "%.2f", $n1*0.05/100}')
echo 0.05% of $n1 is $n2
It is neither showing any value nor terminating when executing this script.
awk will give you an n1 illegal field name if you do that, as it's inside single quotes.
Also, to avoid awk keep reading stdin you should pass /dev/null as file. Then:
n2=$(awk -v n1="$n1" 'BEGIN {printf "%.2f", n1*0.05/100}' /dev/null)
Rather than start a new process to count the records in the file and then passing that to awk, I would suggest you let awk count the records itself which it does anyway in the variable NR. So, your entire script would become:
percentage=$(awk 'END{print NR*0.05/100}' input.txt)
i am trying to replace the following string for ex:
from
['55',2,1,10,30,23],
to
['55',2,555,10,30,23],
OR
['55',2,1,10,30,23],
to
['55',2,1,10,9999,23],
i search around and find this :
$ echo "[55,2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="[55"){$2=10}{print}}'
[55,10,1,10,30,23],
but it's not working in my case since there is " ' " around the value of $1 in my if condition :
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['55'"){$2=10}{print}}'
['55',2,1,10,30,23],
The problem is not in the awk code, it's the shell expansion. You cannot have single quotes in a singly-quoted shell string. This is the same problem you run into when you try to put the input string into single quotes:
$ echo '['55',2,1,10,30,23],'
[55,2,1,10,30,23],
-- the single quotes are gone! And this makes sense, because they did their job of quoting the [ and the ,2,1,10,30,23], (the 55 is unquoted here), but it is not what we wanted.
A solution is to quote the sections between them individually and squeeze them in manually:
$ echo '['\''55'\'',2,1,10,30,23],'
['55',2,1,10,30,23],
Or, in this particular case, where nothing nefarious is between where the single quotes should be,
echo '['\'55\'',2,1,10,30,23],' # the 55 is now unquoted.
Applied to your awk code, that looks like this:
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['\'55\''"){$2=10}{print}}'
['55',10,1,10,30,23],
Alternatively, since this doesn't look very nice if you have many single quotes in your code, you can write the awk code into a file, say foo.awk, and use
echo "['55',2,1,10,30,23]," | awk -F, -f foo.awk
Then you don't have to worry about shell quoting mishaps in the awk code because the awk code is not subject to shell expansion anymore.
I think how to match and replace is not the problem for you. The problem you were facing is, how to match a single quote ' in field.
To avoid to escape each ' in your codes, and to make your codes more readable, you can assigen the quote to a variable, and use the variable in your codes, for example like this:
echo "['55' 1
['56' 1"|awk -v q="'" '$1=="["q"55"q{$2++}7'
['55' 2
['56' 1
In the above example, only in line with ['55', the 2nd field got incremented.
Description: The 1-line awk cmd is used to print all lines after the matched line in my shell script as below.
#!/bin/bash
...
awk "f;/${PATTERN}/{f=1}" ${FILE}
Since the ${PATTERN} may contains special character, the cmd will fail in this case.
Q1. How should I handle such kind of situation if regex is used in awk?
Q2. Is it possible to just use the raw string in this cmd instead of regex e.g. /$PATTERN/ to avoid the special character problem?
Close. It's better to pass shell variables in to awk with -v than to place them in the awk script directly.
awk -v pat="${PATTERN}" 'f; $0 ~ pat {f=1}' "${FILE}"
If ${PATTERN} is not a regex, you can use a different operator:
awk -v pat="${PATTERN}" 'f; $0 == pat {f=1}' "${FILE}"
or you can even handle non-regex substrings:
awk -v pat="${PATTERN}" 'f; index($0, pat) {f=1}' "${FILE}"