I'm trying to loop an awk command using bash script and I'm having a hard time including a variable within the single quotes for the awk command. I'm thinking I should be doing this completely in awk, but I feel more comfortable with bash right now.
#!/bin/bash
index="1"
while [ $index -le 13 ]
do
awk "'"/^$index/ {print}"'" text.txt
done
Use the standard approach -- -v option of awk to set/pass the variable:
awk -v idx="$index" '$0 ~ "^"idx' text.txt
Here i have set the variable idx as having the value of shell variable $index. Inside awk, i have simply used idx as an awk variable.
$0 ~ "^"idx matches if the record starts with (^) whatever the variable idx contains; if so, print the record.
awk '/'"$index"'/' text.txt
# A lil play with the script part where you split the awk command
# and sandwich the bash variable in between using double quotes
# Note awk prints by default, so idiomatic awk omits the '{print}' too.
should do, alternatively use grep like
grep "$index" text.txt # Mind the double quotes
Note : -le is used for comparing numerals, so you may change index="1" to index=1.
I have some logs called ts.log that look like
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Parameters: []
[957670][DEBUG:2016-11-30 16:49:17,968:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{pstm-9805256} Types: []
[957670][DEBUG:2016-11-30 16:50:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} ResultSet
[957670][DEBUG:2016-11-30 16:51:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Header: [LAST_INSERT_ID()]
[957670][DEBUG:2016-11-30 16:52:17,969:com.ibatis.common.logging.log4j.Log4jImpl.debug(Log4jImpl.java:26)]{rset-9805257} Result: [731747]
[065417][DEBUG:2016-11-30 16:53:17,986:sdk.protocol.process.InitProcessor.process(InitProcessor.java:61)]query String=requestid=10547
I have a script in which there's sth like
#!/bin/bash
begin=$1
cat ts.log | awk -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="16:50:17"){print $0}}'
instead of inputting the time like 16:50:17 I want to just pass $1 of shell to awk so that all I need to do is ./script time:hh:mm:ss The script will look like
#!/bin/bash
begin=$1
cat ts.log | awk -v var=$begin -F '[ ,]' '{if($2 ~/^[0-2][0-9]:[0-6][0-9]:[0-6][0-9]&& $2>="var"){print $0}}'
But the double quotes need to be there OR it won't work.
I tried 2>"\""var"\""
but it doesn't work.
so is there a way to keep the double quotes there?
preferred result ./script
then extract the log from the time specified as $1.
There's many ways to do what you want.
Option 1: Using double quotes enclosing awk program
#!/bin/bash
begin=$1
awk -F '[ ,]' "\$2 ~ /^..:..:../ && \$2 >= \"${begin}\" " ts.log
Inside double quotes strings, bash does variable substitution. So $begin or ${begin} will be replaced with the shell variable value (whatever sent by the user)
Undesired effect: awk special variables starting with $ must be escaped with '\' or bash will try to replace them before execute awk.
To get a double quote char (") in bash double quote strings, it has to be escaped with '\', so in bash " \"16:50\" " will be replaced with "16:50". (This won't work with single quote strings, that don't have expansion of variables nor escaped chars at all in bash).
To see what variable substitutions are made when bash executes the script, you can execute it with debug option (it's very enlightening):
$ bash -x yourscript.sh 16:50
Option 2: Using awk variables
#!/bin/bash
begin=$1
awk -F '[ ,]' -v begin=$begin '$2 ~ /^..:..:../ && $2 >= begin' ts.log
Here an awk variable begin is created with option -v varname=value.
Awk variables can be used in any place of awk program as any other awk variable (don't need double quotes nor $).
There are other options, but I think you can work with these two.
In both options I've changed a bit your script:
It doesn't need cat to send data to awk, because awk can execute your program in one or more data files sent as parameters after your program.
Your awk program doesn't need include print at all (as #fedorqui said), because a basic awk program is composed by pairs of pattern {code}, where pattern is the same as you used in the if sentence, and the default code is {print $0}.
I've also changed the time pattern, primarly to clarify the script, but in a log file there's almost no chance that exists some 8 char length string that has 2 colons inside (regexp: . repaces any char)
i am trying to replace the following string for ex:
from
['55',2,1,10,30,23],
to
['55',2,555,10,30,23],
OR
['55',2,1,10,30,23],
to
['55',2,1,10,9999,23],
i search around and find this :
$ echo "[55,2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="[55"){$2=10}{print}}'
[55,10,1,10,30,23],
but it's not working in my case since there is " ' " around the value of $1 in my if condition :
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['55'"){$2=10}{print}}'
['55',2,1,10,30,23],
The problem is not in the awk code, it's the shell expansion. You cannot have single quotes in a singly-quoted shell string. This is the same problem you run into when you try to put the input string into single quotes:
$ echo '['55',2,1,10,30,23],'
[55,2,1,10,30,23],
-- the single quotes are gone! And this makes sense, because they did their job of quoting the [ and the ,2,1,10,30,23], (the 55 is unquoted here), but it is not what we wanted.
A solution is to quote the sections between them individually and squeeze them in manually:
$ echo '['\''55'\'',2,1,10,30,23],'
['55',2,1,10,30,23],
Or, in this particular case, where nothing nefarious is between where the single quotes should be,
echo '['\'55\'',2,1,10,30,23],' # the 55 is now unquoted.
Applied to your awk code, that looks like this:
$ echo "['55',2,1,10,30,23]," | awk -F',' 'BEGIN{OFS=","}{if($1=="['\'55\''"){$2=10}{print}}'
['55',10,1,10,30,23],
Alternatively, since this doesn't look very nice if you have many single quotes in your code, you can write the awk code into a file, say foo.awk, and use
echo "['55',2,1,10,30,23]," | awk -F, -f foo.awk
Then you don't have to worry about shell quoting mishaps in the awk code because the awk code is not subject to shell expansion anymore.
I think how to match and replace is not the problem for you. The problem you were facing is, how to match a single quote ' in field.
To avoid to escape each ' in your codes, and to make your codes more readable, you can assigen the quote to a variable, and use the variable in your codes, for example like this:
echo "['55' 1
['56' 1"|awk -v q="'" '$1=="["q"55"q{$2++}7'
['55' 2
['56' 1
In the above example, only in line with ['55', the 2nd field got incremented.
I have a wierd situation. I am trying to cut some info from a file and everything works fine when I run the command straight into the terminal, but as soon as I make it a variable in a script it returns a mixture of what it should cut and a list of the files in the current directory.
cat query.sql | cut -d':' -f3,4
works but...
QUERY_SQL="query.sql"
MYSQL_COMMAND=`cat $QUERY_SQL | cut -d':' -f3,4`
echo $MYSQL_COMMAND
returns the wierd output mentioned above.
What am I doing wrong?
EDIT:
The query file looks something like this...
email#somehwhere.com:3:SQL CODE
I suspect something in the contents of MYSQL_COMMAND is being interpreted as a filename glob pattern. Try changing
MYSQL_COMMAND=`cat $QUERY_SQL | cut -d':' -f3,4`
echo $MYSQL_COMMAND
to
MYSQL_COMMAND="$(cut -d: -f3,4 < "$QUERY_SQL")"
printf '%s\n' "$MYSQL_COMMAND"
Best defensive coding practice for shell is to put double quotes around every variable substitution, unless you know for a fact that you need word splitting and glob expansion to happen after a particular substitution. Changing echo to printf '%s\n' avoids a related set of problems. I can never remember whether you actually need double quotes around $(...) in a variable assignment, so I put them in just to be safe.
I'm having issues implementing this awk statement that I need for my script:
rsh fooDNS '
...
BROADCAST_IP_ADDRESS=$(/usr/sbin/ifconfig $IF_NAME | grep broadcast | awk '{print \$6}')
...
'
The issue here is that the statement above is contained within an rsh command surrounded by single quotations. Consequently, bash cannot interpret the single quotations around {print $6}, which is giving me a lot of problems. So far, I haven't been able to determine how to get around this issue.
You can't nest single quotes, but you can end the single quoted string, include an escaped single quote, and then re-enter the quotes. Try this:
rsh fooDNS '
...
BROADCAST_IP_ADDRESS=$(/usr/sbin/ifconfig $IF_NAME | grep broadcast | awk '\''{print $6}'\'')
...
'
Nonetheless, this kind of quoting madness gets ugly very quickly. If at all possible, I recommend using scp/rcp/ftp to copy a normal bash script over to the remote and then run that. Failing that, I think you can use a trick like this if you don't need to feed anything to the script's stdin:
cat script_file | rsh fooDNS bash
(Use rsh fooDNS /bin/sh if your script is plain-sh-compatible and the remote side doesn't have bash, of course.)
As yet another alternative, if your snippet is short you can use here docs:
rsh fooDNS sh <<'EOF'
...
BROADCAST_IP_ADDRESS=$(/usr/sbin/ifconfig $IF_NAME | grep broadcast | awk '{print $6}')
...
EOF
Replace the embedded single quotes by the sequence '\'' each time. In theory, you can just do:
rsh fooDNS '
...
BROADCAST_IP_ADDRESS=$(/usr/sbin/ifconfig $IF_NAME | grep broadcast |
awk '\''{print $6}'\'')
...
'
The first ' ends the current single quoted string; the \' sequence adds a single quote; the final ' restarts a new (but contiguous) single-quoted string. So, this introduces no spaces or anything.
On the other hand, it is best to avoid needing to do that. It is vulnerable to causing problems when the string is reinterpreted. And this is doubly the case when dealing with remote shells.
I usually use double quotes for deep nesting when doing it manually, or a mix of double and single quotes automatically.
If you do it manually, it's probably simpler to just escape your nested double quotes, apostrophes and dollar signs with backslashes, and avoid doing more than one or two levels deep.
Sometimes, instead of nesting the quotes manually, I'll use:
http://stromberg.dnsalias.org/~strombrg/bashquote.html
It can wrap up a string with double quotes automatically, exponentially. It uses a mix of double quotes and single quotes.
EG. from a minimal test of the code:
dstromberg#zareason-limbo6000a ~/src/home-svn/bashquote/trunk $ ./bashquote.py
unquoted: 'This is a test'
repetition 0: ''"'"'This is a test'"'"''
repetition 1: ''"'"''"'"'"'"'"'"'"'"'This is a test'"'"'"'"'"'"'"'"''"'"''
repetition 2: ''"'"''"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'This is a test'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"''"'"''
Final version: ''"'"''"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'This is a test'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"''"'"''
''"'"''"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'This is a test'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"''"'"''
That should be good for hopping through 3 ssh's and executing on the 4th.
For AWK scripts I set a variable to the octal equivalent.
But in a case like yours, I would use double quotes and make sure to escape everything that might be interpreted by the shell such as the dollar sign as you have done.
And I wouldn't use rsh since it's insecure.