This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 5 years ago.
I am trying to display which apps is deployed in each host. Using a bash script I had some success except I want to replace the following "hostname" part with the actual host name stored under the ${i} variable. I can't just substitute it because I cannot put a curly bracket inside another. EVEN if I can do it, I am still having trouble as ${i} will be replaced by the result of ls. How do I fix this?
hosts=(usa1 london2)
for i in ${hosts[#]}; do
echo ---${i}---
ssh ttoleung#${i} ls /apps | awk '{ printf("%s:%s\n", "hostname", $0) }'
done
Current output, based on code fragment above:
---usa01---
hostname:E2.gui
hostname:E1.server
---london2---
hostname:E1.gui
Desired output:
---usa01---
usa01:E2.gui
usa01:E1.server
---london2---
london2:E1.gui
Replace:
awk '{ printf("%s:%s\n", "hostname", $0) }'
With:
awk -v h="$i" '{ printf("%s:%s\n", h, $0) }'
-v h="$i" tells awk to create an awk variable h and assign to it the value of the shell variable $i.
Aside: we used h="$i" rather than h=$i because it is good practice to put shell variables inside double-quotes unless you want the shell to perform word-splitting and pathname expansion.
As an intro note, it is not a safe expand array names without double quotes unless for obvious reasons because doing so would split quoted strings in array that themselves have spaces
So change
for i in ${hosts[#]};
to
for i in "${hosts[#]}"; # Note the quoted array
Now, coming to your problem, you can pass bash variables to awk using its -v parameter. So change
ssh ttoleung#${i} ls /apps | awk '{ printf("%s:%s\n", "hostname", $0) }'
to
ssh ttoleung#${i} ls /apps | awk -v hname="${i}" '{ printf("%s:%s\n", hname, $0) }'
Here we pass shell parameter ${i} to awk variable hname.
Side Note: Don't parse ls output for the reasons mentioned [ here ]. In your case though, it doesn't make much of a difference.
Related
I'm trying to loop an awk command using bash script and I'm having a hard time including a variable within the single quotes for the awk command. I'm thinking I should be doing this completely in awk, but I feel more comfortable with bash right now.
#!/bin/bash
index="1"
while [ $index -le 13 ]
do
awk "'"/^$index/ {print}"'" text.txt
done
Use the standard approach -- -v option of awk to set/pass the variable:
awk -v idx="$index" '$0 ~ "^"idx' text.txt
Here i have set the variable idx as having the value of shell variable $index. Inside awk, i have simply used idx as an awk variable.
$0 ~ "^"idx matches if the record starts with (^) whatever the variable idx contains; if so, print the record.
awk '/'"$index"'/' text.txt
# A lil play with the script part where you split the awk command
# and sandwich the bash variable in between using double quotes
# Note awk prints by default, so idiomatic awk omits the '{print}' too.
should do, alternatively use grep like
grep "$index" text.txt # Mind the double quotes
Note : -le is used for comparing numerals, so you may change index="1" to index=1.
I created my basic variable from a read command (I've done this manually and using a script):
read NAME
I then want to use that NAME variable to search a file and create another variable:
STUDENT=$(grep $NAME <students.dat | awk -F: '/$NAME/ {print $1}')
If I run the command manually with an actual name from that students.dat file (and not $NAME), it executes and displays what I want. However, when I run this command (manually or from the script using $NAME), it returns blank, and I'm not sure why.
#user1615415: Try:
cat script.ksh
echo "Enter name.."
read NAME
STUDENT=$(awk -vname="$NAME" -F: '($0 ~ name){print $3}' student.dat)
Shell variables aren't interpolated in single quotes, only double quotes.
STUDENT=$(grep $NAME <students.dat | awk -F: "/$NAME/ {print \$1}")
$1 needs to be escaped to ensure it's not expanded by the shell, but by awk.
I have some logs called ts.log that look like
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Parameters: []
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Types: []
[957670][DEBUG:2016-11-30 16:50:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} ResultSet
[957670][DEBUG:2016-11-30 16:51:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Header: [LAST_INSERT_ID()]
[957670][DEBUG:2016-11-30 16:52:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Result: [731747]
[065417][DEBUG:2016-11-30 16:53:17,986:sdk.protocol.process.InitProcessor.process(InitProcessor.java:61)]query String=requestid=10547
I have a script in which there's sth like
#!/bin/bash
begin=$1
cat ts.log | awk -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="16:50:17"){print $0}}'
instead of inputting the time like 16:50:17 I want to just pass $1 of shell to awk so that all I need to do is ./script time:hh:mm:ss The script will look like
#!/bin/bash
begin=$1
cat ts.log | awk -v var=$begin -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="var"){print $0}}'
But the double quotes need to be there OR it won't work.
I tried 2>"\""var"\""
but it doesn't work.
so is there a way to keep the double quotes there?
preferred result ./script
then extract the log from the time specified as $1.
There's many ways to do what you want.
Option 1: Using double quotes enclosing awk program
#!/bin/bash
begin=$1
awk -F '[ ,]' "\$2 ~ /^..:..:../ && \$2 >= \"${begin}\" " ts.log
Inside double quotes strings, bash does variable substitution. So $begin or ${begin} will be replaced with the shell variable value (whatever sent by the user)
Undesired effect: awk special variables starting with $ must be escaped with '\' or bash will try to replace them before execute awk.
To get a double quote char (") in bash double quote strings, it has to be escaped with '\', so in bash " \"16:50\" " will be replaced with "16:50". (This won't work with single quote strings, that don't have expansion of variables nor escaped chars at all in bash).
To see what variable substitutions are made when bash executes the script, you can execute it with debug option (it's very enlightening):
$ bash -x yourscript.sh 16:50
Option 2: Using awk variables
#!/bin/bash
begin=$1
awk -F '[ ,]' -v begin=$begin '$2 ~ /^..:..:../ && $2 >= begin' ts.log
Here an awk variable begin is created with option -v varname=value.
Awk variables can be used in any place of awk program as any other awk variable (don't need double quotes nor $).
There are other options, but I think you can work with these two.
In both options I've changed a bit your script:
It doesn't need cat to send data to awk, because awk can execute your program in one or more data files sent as parameters after your program.
Your awk program doesn't need include print at all (as #fedorqui said), because a basic awk program is composed by pairs of pattern {code}, where pattern is the same as you used in the if sentence, and the default code is {print $0}.
I've also changed the time pattern, primarly to clarify the script, but in a log file there's almost no chance that exists some 8 char length string that has 2 colons inside (regexp: . repaces any char)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
Im trying to write an awk line in my shell script that will search for the passwd files information belonging to the user that was inputted. Since awk only searches in specific columns and the fact that the first column will vary based on whoever username is used, i think i have to use the script input $username inside the awk line.
So far the line in the script is:
awk -f awkpass.awk /etc/passwd
And the line in the awkpass.awk file is:
/anyone/{print "Username\t\t\t" $1
I think i need to insert $username instead of "anyone" since $username is the variable that i used to recieve input from the user but not quite sure. Help is very appreciated. Thanks!
You can probably use either of these:
awk -F: "\$1 ~ /$username/ { print \"Username:\t\t\t\", \$1}"
awk -F: "\$1 == \"$username\" { print \"Username:\t\t\t\", \$1}"
where the backslash before $1 protects the $ from the shell, and the backslash before the double quotes protects them from the shell. There are times when it is worth using -f awk.script; it is not clear that this is one of them.
The difference between the ~ and the == notation is that the ~ variant matches using a regex. The version using the == goes for simple equality, of course. Using a match means you could have username='^(root|daemon|bin|sys)$' and you'd get the entries for the four named users in a single invocation of the script. The downside is that if you specify just root, you might get multiple entries (for example, on my Mac, I get both root and _cmvsroot listed).
These scripts use double quotes so that the username can be embedded into the script from the command line. However, as correctly pointed out by Ed Morton in his comment, this leads to a surfeit of backslashes in the script. I generally use single quotes around scripts (despite not doing so in the first edition of this answer).
awk -F: '$1 ~ /'"$username"'/ { print "Username:\t\t\t", $1}'
awk -F: '$1 == "'"$username"'" { print "Username:\t\t\t", $1}'
Those quote sequences are subtle; it is better to avoid them. You can do that by specifying initial values for variables on the command line with -v varname="value" notation. Therefore, you could also use:
awk -F: -v username="$username" \
"\$1 == username { print \"Username:\t\t\t\", \$1}" /etc/passwd
or, better, since it uses single quotes around the script:
awk -F: -v username="$username" \
'$1 == username { print "Username:\t\t\t", $1}' /etc/passwd
This sets the awk variable username from the shell variable $username. This variation can be used with a file:
awk -F: -v username="$username" -f awk.script /etc/passwd
with awk.script containing:
$1 == username { print "Username\t\t\t", $1 }
Alternatively, you can use getent:
printf "Username: \t\t\t%s\n" "$(getent passwd "$username" | cut -d: -f1)"
I have a large text file that contains multiple columns of data. I'm trying to write a script that accepts a column number and keyword from the command line and searches for any hits before displaying the entire row of any matches.
I've been trying something along the lines of:
grep $fileName | awk '{if ($'$columnNumber' == '$searchTerm') print $0;}'
But this doesn't work at all. Am I on the right lines? Thanks for any help!
The -v option can be used to pass shell variables to awk command.
The following may be what you're looking for:
awk -v s=$SEARCH -v c=$COLUMN '$c == s { print $0 }' file.txt
EDIT:
I am always trying to write more elegant and tighter code. So here's what Dennis means:
awk -v s="$search" -v c="$column" '$c == s { print $0 }' file.txt
Looks reasonable enough. Try using set -x to look at exactly what's being passed to awk. You can also use different and/or more awk things, including getting rid of the separate grep:
awk -v colnum=$columnNumber -v require="$searchTerm"
"/$fileName/ { if (\$colnum == require) print }"
which works by setting awk variables (colnum and require, in this case) and then using the literal string $colnum to get the desired field, and the variable require to get the required-string.
Note that in all cases (with or without the grep command), any regular expression meta-characters in $fileName will be meta-y, e.g., this.that will match the file named this.that but also the file named thisXthat.