This is my below shell script which I am using to query the hive tables and I saved this file as test4.sh
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT1
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT split(ckey, '\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1;"`
Problem Statement:-
I am running the above shell script as-
bash -x test4.sh
If you see this line in the above hive sql query:
**SELECT split(ckey, '\\|') AS t1**.
Do I need to escape slash sign to make it to work in shell script?
Inside backticks and double quotes, you basically need to double your backslashes, yes. If a backslash is not a known escape sequence, it will be preserved, though.
bash$ echo "foo\bar"
foo\bar
bash$ echo "foo\\bar"
foo\bar
bash$ echo "split(ckey, '\\|')"
split(ckey, '\|')
So if you want hive to see a double backslash there, you will need to put four backslashes in the Bash script.
See further e.g. http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_03.html
Related
This question already has answers here:
Batch file: How to replace "=" (equal signs) and a string variable?
(7 answers)
Closed 3 years ago.
First time post - sorry!
Yes, I have seen many posts on how to replace characters in a string in batch script, but I can't seem to make any of them work with "=". Here is what I would like to do:
This is a batch file, running in a bash shell:
set ss=param1= 3, pram2 = 27.3, param3 = 11,
echo %ss% | sed 's/=/ /g' | sed 's/,/ /g'
it beautifully writes to the screen:
param1 3 pram2 27.3 param3 11
That is exactly what I would like a variable to be full of, not look at it on a screen! I would like it to write instead to a variable - say, something like:
set sss=echo %ss% | sed 's/=/ /g' | sed 's/,/ /g'
echo %sss% returns ECHO is off.
echo $sss returns $sss
Thanks very much for your help!
set "ss=param1= 3, pram2 = 27.3, param3 = 11,"
for /f "delims=" %%A in (
'echo %ss%^|sed "s/=/ /g"^|sed "s/,/ /g"'
) do set "sss=%%A"
echo sss: %sss%
In Bash, you can assign the Stdout of a command directly to a variable.
In CMD, you can use a for /f loop to run a command and return each line of Stdout.
The command run needs the | escaped with ^.
The arguments of sed may require double quotes instead of single quotes that Bash may use.
The delims= option tells the for loop to not delimit each line into tokens.
View for /? for help about running commands in a for loop.
I have this shell script running in a Jenkins pipeline
def planResults = sh(returnStdout: true, script: "cd $it; PLAN=\$(terragrunt plan --terragrunt-source-update | landscape); echo "$PLAN"; CHANGES=$(echo "$PLAN" | tail -2); echo $CHANGES")
The issue is when I try to echo the "$PLAN" variables.
Here is the solution that groovy recommends, which works near where PLAN is set at \$(terragrunt, however it does not work for a $ inside double quotes. And I NEED double quotes for this command to work properly.
solution: either escape a literal dollar sign "\$5" or bracket the value
expression "${5}" # line 34, column 148.
ce-update | landscape); echo "$PLAN"; CH
Thank you!
For double quoted string, Groovy will do interpolation on the string firstly.
Because the it, PLAN and CHANGES are runtime variables of the shell, rather than the variables of Groovy runtime. Groovy can't find the responding value from Groovy variable stack to replace the $it/PLAN/CHANGS during interpolation.
So you need to escape all $ if you use double quote in your case:
script: "cd \$it; PLAN=\$(terragrunt plan --terragrunt-source-update | landscape);
echo \$PLAN; CHANGES=\$(echo \$PLAN | tail -2); echo \$CHANGES"
Or use single quote which not support interpolation:
script: 'cd $it; PLAN=$(terragrunt plan --terragrunt-source-update | landscape);
echo $PLAN; CHANGES=$(echo $PLAN | tail -2); echo $CHANGES'
More detail about Groovy string at here
I am trying to automate string replace thing for my project ...
I need output like this in file
insert into libraries values("Schema_name", "table_name", "table_name", "/data/Projects/Ope/ACT/Domain/Code/Files");
and what I am getting in the file is
`insert into libraries values(Schema_name, table_name, table_name, /data/Projects/Ope/ACT/Domain/Code/Files)`;
replace_script.sh
#!/bin/bash
while read line
do
param1=`echo $line | awk -F ' ' '{print $1}'`
param2=`echo $line | awk -F ' ' '{print $2}'`
echo "insert into libraries values(\"$param1\",\"$param2\",\"$param2\",\"/data/Projects/Ope/ACT/Domain/Code/Files\");" >> input_queries.hql
done <<EOF
schema_name table_name
schema_name table_name
EOF
The exact code given in your question emits as output:
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
This is, as I understand it, exactly what you claim to want.
However, SQL doesn't use double quotes for data -- it uses single quotes for that.
escape_sql() {
local val
val=${1//\\/\\\\}
val=${val//\'/\\\'}
val=${val//\"/\\\"}
printf '%s' "$val"
}
while read -r param1 param2; do
printf $'insert into libraries values(\'%s\', \'%s\', \'%s\', \'/data/Projects/Ope/ACT/Domain/Code/Files\');\\n' \
"$(escape_sql "$param1")" \
"$(escape_sql "$param2")" \
"$(escape_sql "$param2")"
done <<EOF
schema_name table_name
schema_name table_name
EOF
The above makes a rudimentary attempt to prevent malicious values from escaping their quotes -- though you should really use a language with native SQL bindings for your database for the purpose!
That said -- this is not safe escaping against malicious data (for instance, data containing literal quotes). For that, use a language built-to-purpose.
I can't figure out how to replace a comma followed by 0 or more spaces in a bash variable. here's what i have:
base="test00 test01 test02 test03"
options="test04,test05, test06"
for b in $(echo $options | sed "s/, \+/ /g")
do
base="${base} $b"
done
What i'm trying to do is append the "options" to the "base". Options is user input which can be empty or a csv list however that list can be
"test04, test05, test06" -> space after the comma
"test04,test05,test06" -> no spaces
"test04,test05, test06" -> mixture
what i need is my output "base" to be a space delimited list however no matter what i try my list keeps getting cut off after the first word.
My expected out is
"test00 test01 test02 test03 test04 test05 test06"
If your goal is to generate a command, this technique is wrong altogether: As described in BashFAQ #50, command arguments should be stored in an array, not a whitespace-delimited string.
base=( test00 test01 test02 test03 )
IFS=', ' read -r -a options_array <<<"$options"
# ...and, to execute the result:
"${base[#]}" "${options_array[#]}"
That said, even this isn't adequate to many legitimate use cases: Consider what happens if you want to pass an option that contains literal whitespace -- for instance, running ./your-base-command "base argument with spaces" "second base argument" "option with spaces" "option with spaces" "second option with spaces". For that, you need something like the following:
base=( ./your-base-command "base argument with spaces" "second base argument" )
options="option with spaces, second option with spaces"
# read options into an array, splitting on commas
IFS=, read -r -a options_array <<<"$options"
# trim leading and trailing spaces from array elements
options_array=( "${options_array[#]% }" )
options_array=( "${options_array[#]# }" )
# ...and, to execute the result:
"${base[#]}" "${options_array[#]}"
No need for sed, bash has built in pattern substitution parameter expansion. With bash 3.0 or later, extglob added support for more advanced regular expressions.
# Enables extended regular expressions for +(pattern)
shopt -s extglob
# Replaces all comma-space runs with just a single space
options="${options//,+( )/ }"
If you don't have bash 3.0+ available or don't like enabling extglob, simply strip all spaces which will work most of the time:
# Remove all spaces
options="${options// /}"
# Then replace commas with spaces
options="${options//,/ }"
Here we have 2 vars in bash shell
ID="ABC"
ID_STRING="Here is the [${ID}]"
Is there any approach that we could dynamic replace the var ${ID} to "ABC" in the ID_STRING and echo to concole?
If I understand the question correctly the ID_STRING variable contains the literal string ${ID} when it is echoed to the screen and isn't assigned the way you show in the question - assigning it the way you have it in your question using double quotes means the ID_STRING variable never actually contains the literal string ${ID} because with double quotes the variable replacement is done as it is assigned. So first: to get the literal string ${ID} into ID_STRING you need to use single quotes. And second: you need to reevaluate ID_STRING as it is echoed. I do it like this (the $ is the bash prompt):
$ ID=ABC
$ echo $ID
ABC
$ ID_STRING='Here is the [${ID}]'
$ echo $ID_STRING
Here is the [${ID}]
$ eval echo $ID_STRING
Here is the [ABC]
$ echo $ID, $ID_STRING, `eval echo $ID_STRING`
ABC, Here is the [${ID}], Here is the [ABC]
You you want to replace ${ID} from string:
ID="ABC"
sed 's/\${ID}/'$ID'/' filename
Output:
ID_STRING="Here is the "[ABC]"
If you want to use that value of ID while assigning the string to ID_STRING, you can use:
ID="ABC"
ID_STRING="Here is the [$ID]"
Now:
echo $ID_STRING
gives
Here is the [ABC]
sed -i -e 's/old/new/g' file
Is basically what you need, to output the changes have a look here.