In my Cassandra table, it had been created with all columns in Upper case. When we tried to select the columns in cqlsh terminal, we were able to select those columns, but when we tried to pull same query based on the cqlsh -e facing some issue with escaping character.
cqlsh:key1> select "PLAN_ID" ,"A_ADVERTISER_ID" ,"ADVERTISER_NAME" from key1.plan_advertiser where "PLAN_ID" = '382633' and "A_ADVERTISER_ID" = 15019;
PLAN_ID | A_ADVERTISER_ID | ADVERTISER_NAME
---------+-----------------+----------------------
382633 | 15019 | Hanesbrands, Updated
NMH206576286LM:sparklatest0802 KarthikeyanDurairaj$ cqlsh -e 'select "PLAN_ID" ,
"A_ADVERTISER_ID" ,"ADVERTISER_NAME" from key1.plan_advertiser
where "PLAN_ID" = '382633' and "A_ADVERTISER_ID" = 15019'
<stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query]
message="Invalid INTEGER constant (382633) for "PLAN_ID" of type text"
NMH206576286LM:sparklatest0802 KarthikeyanDurairaj$
cqlsh can be a little tricky in this regard. While it doesn't allow you to escape single quotes, it does allow you to escape double quotes. This works for me:
$ bin/cqlsh -u cassdba -p flynnLives -e "SELECT * FROM stackoverflow.plan_advertiser
where \"PLAN_ID\" = '382633' and \"A_ADVERTISER_ID\" = 15019"
PLAN_ID | A_ADVERTISER_ID | ADVERTISER_NAME
---------+-----------------+----------------------
382633 | 15019 | Hanesbrands, Updated
(1 rows)
In this way, we switch from single quotes to double quotes for the CQL statement, use single quotes for column values, and then escape out the double quotes around the column names.
Related
I'm trying to run several number of sql statement in bash script. I tried to run the following statement on psql, it works fine. However, when I run the exact statement on script, it says the column does not exist.
The statement
\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 100) to 'D:\$path\onetry1.csv' with csv HEADER;
I am aware in Postgres, it is required to use double quotes on uppercase column names. I did that.
This is my bash script:
#! /bin/sh
db="usm"
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
The error I'm getting:
ERROR: column h.idx does not exist
LINE 1: ...M table_a h JOIN table_b b ON h.IDX = b....
Your shell script doesn't send the double quotes. You have this:
"\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
it essentially works as a concatenation of the following strings
"\copy (SELECT * FROM table_a h JOIN table_b b ON h."
IDX
" = b."
IDX
" WHERE b."
XXX
" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
but PostgreSQL never gets to see the double quotes. Use backslash escape the nested double quotes:
"\copy (SELECT * FROM table_a h JOIN table_b b ON h.\"IDX\" = b.\"IDX\" WHERE b.\"XXX\" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
also, prefer to avoid forcing all upper case or any specific case for that matter. You can still type upper case if you want. I use quoted column names only to format final queries prettier.
For cases where programs takes input which may conflict with the shell process (e.g., quotes, double quotes, and various substitutions), it is usually better to use one of the other quoting options:
If the programs can process STDIN, here documents (<<word) can be used.
If the programs only works with command line parameters, here document can be inlined
Consider using variables to simplify quoting.
I believe psql also takes stdin as input, so here documents can work (removing -c). Making it easier to format the SQL for readibility.
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db <<SQL
\copy (
SELECT *
FROM table_a h
JOIN table_b b ON h."IDX" = b."IDX"
WHERE b."XXX" BETWEEN 0 AND 500)
to '$path\onetry2.csv' with csv HEADER;
SQL
If the programs takes only command line arguments, here strings can be used (note that this is bash only feature, not POSIX).
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "$(cat <<SQL
\copy (
SELECT *
FROM table_a h
JOIN table_b b ON h."IDX" = b."IDX"
WHERE b."XXX" BETWEEN 0 AND 500)
to '$path\onetry2.csv' with csv HEADER;
SQL
)
If the command is constructed from variables (or can be stored in variables), it can eliminate the complex quoting.
V=$(cat <<SQL
\copy (
SELECT *
...
to '$path\onetry2.csv' with csv HEADER;
SQL
)
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "$v"
In my shell script I am using this query to get the last_value of the column id.
last_val=`beeline -e "select nvl(max(id),0) from testing.1_test"`
The result is
+----------+--+
| _c0 |
+----------+--+
| 3380901 |
+----------+--+
Now I want to pass this value as variable ${last_val}
when I do echo ${last_val} I want to have 3380901 but I am receiving
+----------+--+
| _c0 |
+----------+--+
| 3380901 |
+----------+--+
How can I echo 3380901.
When I used hive option like below I got what I want
last_val=`hive -e "select nvl(max(id),0) from testing.1_test"`
echo ${last_val} gave me 3380901
Please let me know how can I do this?
last_val=`beeline --showHeader=false --outputformat=tsv2 -e "select nvl(max(id),0) from testing.1_test"`
I am trying to automate string replace thing for my project ...
I need output like this in file
insert into libraries values("Schema_name", "table_name", "table_name", "/data/Projects/Ope/ACT/Domain/Code/Files");
and what I am getting in the file is
`insert into libraries values(Schema_name, table_name, table_name, /data/Projects/Ope/ACT/Domain/Code/Files)`;
replace_script.sh
#!/bin/bash
while read line
do
param1=`echo $line | awk -F ' ' '{print $1}'`
param2=`echo $line | awk -F ' ' '{print $2}'`
echo "insert into libraries values(\"$param1\",\"$param2\",\"$param2\",\"/data/Projects/Ope/ACT/Domain/Code/Files\");" >> input_queries.hql
done <<EOF
schema_name table_name
schema_name table_name
EOF
The exact code given in your question emits as output:
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
insert into libraries values("schema_name","table_name","table_name","/data/Projects/Ope/ACT/Domain/Code/Files");
This is, as I understand it, exactly what you claim to want.
However, SQL doesn't use double quotes for data -- it uses single quotes for that.
escape_sql() {
local val
val=${1//\\/\\\\}
val=${val//\'/\\\'}
val=${val//\"/\\\"}
printf '%s' "$val"
}
while read -r param1 param2; do
printf $'insert into libraries values(\'%s\', \'%s\', \'%s\', \'/data/Projects/Ope/ACT/Domain/Code/Files\');\\n' \
"$(escape_sql "$param1")" \
"$(escape_sql "$param2")" \
"$(escape_sql "$param2")"
done <<EOF
schema_name table_name
schema_name table_name
EOF
The above makes a rudimentary attempt to prevent malicious values from escaping their quotes -- though you should really use a language with native SQL bindings for your database for the purpose!
That said -- this is not safe escaping against malicious data (for instance, data containing literal quotes). For that, use a language built-to-purpose.
This is my below shell script which I am using to query the hive tables and I saved this file as test4.sh
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT1
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT split(ckey, '\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1;"`
Problem Statement:-
I am running the above shell script as-
bash -x test4.sh
If you see this line in the above hive sql query:
**SELECT split(ckey, '\\|') AS t1**.
Do I need to escape slash sign to make it to work in shell script?
Inside backticks and double quotes, you basically need to double your backslashes, yes. If a backslash is not a known escape sequence, it will be preserved, though.
bash$ echo "foo\bar"
foo\bar
bash$ echo "foo\\bar"
foo\bar
bash$ echo "split(ckey, '\\|')"
split(ckey, '\|')
So if you want hive to see a double backslash there, you will need to put four backslashes in the Bash script.
See further e.g. http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_03.html
I'm trying to get psql to format nicely and am following the docs here. Right now, whenever I do a query on tables with lots of columns, no matter how big I make my screen each line overflows into the next line and producing a whole screen of unreadable junk.
The docs (link is above) say there's a way to align columns nicely for more readable output.
Normally, to start psql, I just type:
psql
and hit Enter. Now I'm trying:
psql \pset format aligned
And getting an error:
could not change directory to "/root"
psql: warning: extra command-line argument "aligned" ingored
psql: FATAL: Indent authentication failed for user "format"
Any ideas as to how I could get these command-line args to work for me?
These are not command line args. Run psql. Manage to log into database (so pass the hostname, port, user and database if needed). And then write it in the psql program.
Example (below are two commands, write the first one, press enter, wait for psql to login, write the second):
psql -h host -p 5900 -U username database
\pset format aligned
Use \x
Example from postgres manual:
postgres=# \x
postgres=# SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 3;
-[ RECORD 1 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE branches SET bbalance = bbalance + $1 WHERE bid = $2;
calls | 3000
total_time | 20.716706
rows | 3000
-[ RECORD 2 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE tellers SET tbalance = tbalance + $1 WHERE tid = $2;
calls | 3000
total_time | 17.1107649999999
rows | 3000
-[ RECORD 3 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE accounts SET abalance = abalance + $1 WHERE aid = $2;
calls | 3000
total_time | 0.645601
rows | 3000
psql --pset=format=FORMAT
Great for executing queries from command line, e.g.
psql --pset=format=unaligned -c "select bandanavalue from bandana where bandanakey = 'atlassian.confluence.settings';"