I'm trying to run several number of sql statement in bash script. I tried to run the following statement on psql, it works fine. However, when I run the exact statement on script, it says the column does not exist.
The statement
\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 100) to 'D:\$path\onetry1.csv' with csv HEADER;
I am aware in Postgres, it is required to use double quotes on uppercase column names. I did that.
This is my bash script:
#! /bin/sh
db="usm"
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
The error I'm getting:
ERROR: column h.idx does not exist
LINE 1: ...M table_a h JOIN table_b b ON h.IDX = b....
Your shell script doesn't send the double quotes. You have this:
"\copy (SELECT * FROM table_a h JOIN table_b b ON h."IDX" = b."IDX" WHERE b."XXX" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
it essentially works as a concatenation of the following strings
"\copy (SELECT * FROM table_a h JOIN table_b b ON h."
IDX
" = b."
IDX
" WHERE b."
XXX
" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
but PostgreSQL never gets to see the double quotes. Use backslash escape the nested double quotes:
"\copy (SELECT * FROM table_a h JOIN table_b b ON h.\"IDX\" = b.\"IDX\" WHERE b.\"XXX\" BETWEEN 0 AND 500) to '$path\onetry2.csv' with csv HEADER;"
also, prefer to avoid forcing all upper case or any specific case for that matter. You can still type upper case if you want. I use quoted column names only to format final queries prettier.
For cases where programs takes input which may conflict with the shell process (e.g., quotes, double quotes, and various substitutions), it is usually better to use one of the other quoting options:
If the programs can process STDIN, here documents (<<word) can be used.
If the programs only works with command line parameters, here document can be inlined
Consider using variables to simplify quoting.
I believe psql also takes stdin as input, so here documents can work (removing -c). Making it easier to format the SQL for readibility.
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db <<SQL
\copy (
SELECT *
FROM table_a h
JOIN table_b b ON h."IDX" = b."IDX"
WHERE b."XXX" BETWEEN 0 AND 500)
to '$path\onetry2.csv' with csv HEADER;
SQL
If the programs takes only command line arguments, here strings can be used (note that this is bash only feature, not POSIX).
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "$(cat <<SQL
\copy (
SELECT *
FROM table_a h
JOIN table_b b ON h."IDX" = b."IDX"
WHERE b."XXX" BETWEEN 0 AND 500)
to '$path\onetry2.csv' with csv HEADER;
SQL
)
If the command is constructed from variables (or can be stored in variables), it can eliminate the complex quoting.
V=$(cat <<SQL
\copy (
SELECT *
...
to '$path\onetry2.csv' with csv HEADER;
SQL
)
PGPASSWORD=XXX123 psql -h localhost -U postgres -d $db -c "$v"
Related
In my Cassandra table, it had been created with all columns in Upper case. When we tried to select the columns in cqlsh terminal, we were able to select those columns, but when we tried to pull same query based on the cqlsh -e facing some issue with escaping character.
cqlsh:key1> select "PLAN_ID" ,"A_ADVERTISER_ID" ,"ADVERTISER_NAME" from key1.plan_advertiser where "PLAN_ID" = '382633' and "A_ADVERTISER_ID" = 15019;
PLAN_ID | A_ADVERTISER_ID | ADVERTISER_NAME
---------+-----------------+----------------------
382633 | 15019 | Hanesbrands, Updated
NMH206576286LM:sparklatest0802 KarthikeyanDurairaj$ cqlsh -e 'select "PLAN_ID" ,
"A_ADVERTISER_ID" ,"ADVERTISER_NAME" from key1.plan_advertiser
where "PLAN_ID" = '382633' and "A_ADVERTISER_ID" = 15019'
<stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query]
message="Invalid INTEGER constant (382633) for "PLAN_ID" of type text"
NMH206576286LM:sparklatest0802 KarthikeyanDurairaj$
cqlsh can be a little tricky in this regard. While it doesn't allow you to escape single quotes, it does allow you to escape double quotes. This works for me:
$ bin/cqlsh -u cassdba -p flynnLives -e "SELECT * FROM stackoverflow.plan_advertiser
where \"PLAN_ID\" = '382633' and \"A_ADVERTISER_ID\" = 15019"
PLAN_ID | A_ADVERTISER_ID | ADVERTISER_NAME
---------+-----------------+----------------------
382633 | 15019 | Hanesbrands, Updated
(1 rows)
In this way, we switch from single quotes to double quotes for the CQL statement, use single quotes for column values, and then escape out the double quotes around the column names.
I Have a problem with a psql query in bash.
I really don't know why the PSQL understands the value HUB is a Column.
psql -q -A -h Some_host
-U User -d datashema -p 1111 -t -f query.txt -c 'SELECT id, text FROM great_201704 WHERE id = 10 and
text = 'HUB' ;'
ERROR: column "hub" does not exist in great_201704
You read your single quotes as if they nest:
-c 'SELECT id, text FROM great_201704 WHERE id = 10 and text = 'HUB' ;'
^---------------------------------1--------------------------------^
^-2-^
Bash reads them as two single quoted string with a literal between them:
-c 'SELECT id, text FROM great_201704 WHERE id = 10 and text = 'HUB' ;'
^------------------------------1----------------------------^
^2-^
This is equivalent to not having single quotes around HUB, which is why psql thinks its a column.
The easiest way to embed one set of quotes in another string is to just use two different types of quotes:
psql -q -A -h Some_host -U User -d datashema -p 1111 -t -f query.txt \
-c "SELECT id, text FROM great_201704 WHERE id = 10 and text = 'HUB' ;"
I can't seem to find a good way to do this using Unix/Linux commands (without writing a script). I'm thinking this can be done, though.
What I want to do is read the contents of a file, split them by some delimiter, and then discard any segments that don't match a RegEx (essentially, a map-filter-reduce problem).
something like
cat original-file.sql | split --separator=';' | pcregrep -M '(.|\n)*needed_schema(.|\n)*' | result-file.sql
so that the result file only contains SQL statements that include 'needed_schema' in the statement.
original-file.sql
SELECT *
FROM information_schema.tables
WHERE table_schema = 'public'
;
SELECT *
FROM needed_schema.some_table
WHERE some_col = 'some value'
;
SELECT *
FROM needed_schema.some_other_table
WHERE some_other_col = 'some other value'
;
result-file.sql
SELECT *
FROM needed_schema.some_table
WHERE some_col = 'some value'
;
SELECT *
FROM needed_schema.some_other_table
WHERE some_other_col = 'some other value'
;
Consider using awk and assign RS (Record Separator) and ORS (Output Record
Separator):
$ awk '/needed_schema/' RS=';' ORS=';' input
SELECT *
FROM needed_schema.some_table
WHERE some_col = 'some value'
;
SELECT *
FROM needed_schema.some_other_table
WHERE some_other_col = 'some other value'
;
The regex is not PCRE but Extended Regular Expression
This is my below shell script which I am using to query the hive tables and I saved this file as test4.sh
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*96);'`
echo $DATE_YEST_FORMAT1
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT split(ckey, '\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1;"`
Problem Statement:-
I am running the above shell script as-
bash -x test4.sh
If you see this line in the above hive sql query:
**SELECT split(ckey, '\\|') AS t1**.
Do I need to escape slash sign to make it to work in shell script?
Inside backticks and double quotes, you basically need to double your backslashes, yes. If a backslash is not a known escape sequence, it will be preserved, though.
bash$ echo "foo\bar"
foo\bar
bash$ echo "foo\\bar"
foo\bar
bash$ echo "split(ckey, '\\|')"
split(ckey, '\|')
So if you want hive to see a double backslash there, you will need to put four backslashes in the Bash script.
See further e.g. http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_03.html
I'm trying to get psql to format nicely and am following the docs here. Right now, whenever I do a query on tables with lots of columns, no matter how big I make my screen each line overflows into the next line and producing a whole screen of unreadable junk.
The docs (link is above) say there's a way to align columns nicely for more readable output.
Normally, to start psql, I just type:
psql
and hit Enter. Now I'm trying:
psql \pset format aligned
And getting an error:
could not change directory to "/root"
psql: warning: extra command-line argument "aligned" ingored
psql: FATAL: Indent authentication failed for user "format"
Any ideas as to how I could get these command-line args to work for me?
These are not command line args. Run psql. Manage to log into database (so pass the hostname, port, user and database if needed). And then write it in the psql program.
Example (below are two commands, write the first one, press enter, wait for psql to login, write the second):
psql -h host -p 5900 -U username database
\pset format aligned
Use \x
Example from postgres manual:
postgres=# \x
postgres=# SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 3;
-[ RECORD 1 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE branches SET bbalance = bbalance + $1 WHERE bid = $2;
calls | 3000
total_time | 20.716706
rows | 3000
-[ RECORD 2 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE tellers SET tbalance = tbalance + $1 WHERE tid = $2;
calls | 3000
total_time | 17.1107649999999
rows | 3000
-[ RECORD 3 ]------------------------------------------------------------
userid | 10
dbid | 63781
query | UPDATE accounts SET abalance = abalance + $1 WHERE aid = $2;
calls | 3000
total_time | 0.645601
rows | 3000
psql --pset=format=FORMAT
Great for executing queries from command line, e.g.
psql --pset=format=unaligned -c "select bandanavalue from bandana where bandanakey = 'atlassian.confluence.settings';"