I have the following script segment in a Linux script:
sqlplus /
<<QUERY_1
UPDATE BATCH_FILE SET BATCH_ID = 0 WHERE BATCH_ID = -1;
COMMIT;
exit
QUERY_1
I am expecting the update to occur and the script to exit sqlplus
What actually happens is the query is not executed, and the script exits leaving sqlplus logged into my database with a SQL> prompt. I can execute the statements from the prompt, but of course, that is not what I want to do.
My current version of Oracle is 12.2.0.1
The output of the HERE-document is intended for the std input of sqlplus, but for the shell a command should be on a single line. Adding a backslash will make the shell ignore the line-end, combining the two physical lines into one logical line:
sqlplus / \
<<QUERY_1
UPDATE BATCH_FILE SET BATCH_ID = 0 WHERE BATCH_ID = -1;
COMMIT;
exit
QUERY_1
Or just:
sqlplus / <<QUERY_1
UPDATE BATCH_FILE SET BATCH_ID = 0 WHERE BATCH_ID = -1;
COMMIT;
exit
QUERY_1
Related
I have an oracle db on my Linux machine.
A single sql query (1 connection) via bash is as follows:
su - oracle
sqlplus <dbuser>/<dbpass>
select * from cat;
exit
I'm trying to run parallel queries via bash, the following script is for running 10000 connections in parallel (Correct me if i'm wrong):
for i in $(seq 1 10000); do echo "select * from <tableName>;" | sqlplus <dbuser>/<dbpass>&done
I would like to make this code more robust and flexible, for the sake of example i want to add a sleep between each of the following command:
Create a connection
Create a table (Unique to this connection, i as index for example)
Select data from the table
Close the connection
The following code is my attempt of doing so: (Not working)
for i in $(seq 1 10000);
do
echo "CREATE TABLE test+i (id NUMBER NOT NULL);"
sleep 2
echo "select * from test+i"
sleep 2
echo "DROP TABLE test+i" | sqlplus <dbuser>/<dbpass>&
done
1) Syntactically, how should i write it?
2) How can i know how many queries/connections succeeded and how many failed?
3) How can i know how many connections actually ran in parallel
1) you can use ( and ) to group command into subshells, and send them background:
for i in $(seq 1 10000);
do
echo "CREATE TABLE test_$i (id NUMBER NOT NULL);
!sleep 2
select * from test_$i;
!sleep 2
DROP TABLE test_$i;" | sqlplus <dbuser>/<dbpass> &
done
2) you can set up error handling after each sqlplus call (examine output or exit value)
echo "CREATE TABLE test_$i (id NUMBER NOT NULL);" | sqlplus <dbuser>/<dbpass> 2>&1 | grep -i error
3) you can use the jobs command to examine how many job is running in the background:
> sleep 100 &
[1] 31642
> jobs
[1]+ Running sleep 100 &
10000 jobs in parallel will often cause overflow. By setting 'WHENEVER SQLERROR EXIT SQL.SQLCODE' sqlplus will return an error, if the SQL fails. GNU Parallel can then re-run the query.
my.log will show if the query failed after rerunning 3 times.
doit() {
i=$1
(echo "WHENEVER SQLERROR EXIT SQL.SQLCODE CREATE TABLE test$i (id NUMBER NOT NULL);"
sleep 2
echo "WHENEVER SQLERROR EXIT SQL.SQLCODE select * from test$i;"
sleep 2
echo "WHENEVER SQLERROR EXIT SQL.SQLCODE DROP TABLE test$i;") |
sqlplus <dbuser>/<dbpass>
}
export -f doit
seq 1 10000 | parallel --joblog my.log -j0 --retries 3 doit
i need to load data from flat file into mariaDB on linux environment.
i've plan to put mariaDB script on shell file. then call shell from cron.
mariadb script shown as follow:
set #path = (select path_file from param);
set #tbl = (select table_name from param);
set #x = concat(
'LOAD DATA LOCAL INFILE ',
#path,
' INTO TABLE ', #tbl,
' (#row) set id = trim(substr(#row,1,2)), name = trim(substr(#row,3,19)), address= trim(substr(#row,22,20))'
);
prepare y from #x;
execute y;
deallocate prepare y;
when i execute the script directly on heidisql,
error shown:
this command is not supported in the prepared statement protocol yet
does any one have better way to load data from flat file into MariaDB on linux environment regularly (scheduled) without using any ETL tools?
Thanks.
One option you can try is (adjust as needed):
File: load_data.sh
path=$(mysql -u ${mysql_user} -p${mysql_password} -s -N <<GET_PATH
SELECT '/path/to/file/data.csv';
GET_PATH
)
tbl=$(mysql -u ${mysql_user} -p${mysql_password} -s -N <<GET_TABLE
SELECT 'table';
GET_TABLE
)
# mysql -u ${mysql_user} -p${mysql_password} -s -N <<LOAD_DATA
# LOAD DATA LOCAL INFILE '${path}'
# INTO TABLE \`${tbl}\` ...
# LOAD_DATA
# TEST
cat <<LOAD_DATA
LOAD DATA LOCAL INFILE '${path}'
INTO TABLE \`${tbl}\` ...
LOAD_DATA
Command line:
$ ls -l
-r-x------ load_data.sh
$ ./load_data.sh
LOAD DATA LOCAL INFILE '/path/to/file/data.csv'
INTO TABLE `table` ...
For clarity, write as much of the SQL into a STORED PROCEDURE. Then use bash to call that SP.
New to sqlplus/bash scripting. I currently have a simple script that fetches some value in a table based on some ID values.
#do things. get login info. etc....
(cat<<HERE
set heading off;
select data_value from metadata where name='JOHN' and uniqueid in (1, 2, 3);
EOD
) | sqlplus -S $login
#do things.
What if instead of having to manually type of the ids (1, 2, 3, etc...), I do this:
#calls a script that gets the IDs from somewhere and outputs it in the correct format
./getIDscript > IDs
#do things. get login info. etc....
(cat<<HERE
set heading off;
select data_value from metadata where name='JOHN' and uniqueid in ($IDs);
EOD
) | sqlplus -S $login
#do things.
Would this work? I currently won't have access to the school lab for a few days so I can't test this out right now.
Is there a better and more efficient way of doing this?
Try this :
sqlplus -S /nolog <<_EOD1
WHENEVER SQLERROR EXIT SQL.SQLCODE
connect usr/xxx#db1
select 1 from dual
/
exit
_EOD1
I have written a small bash script to delete some rows from a table and drop some users using sqlplus. When I put the code in the function it is giving "syntax error: unexpected end of file" error message. Below is the code. Please let me know how to fix it.
function reset_db
{
sqlplus user1/password1#${input} << eof
set timing off
set serveroutput on size 10000
set feedback off
spool logfile_$input.out
delete from table1 where component = 'XYZ';
delete from table2 where component = 'XYZ';
commit;
exit
eof
sqlplus dba_usr/dba_password#${input} << eof
set timing off
set serveroutput on size 10000
set feedback off
spool logfile_$input.out
drop user ABC cascade;
drop user DEF cascade;
drop user HIG cascade;
commit;
exit;
}
You're missing the eof at the end of your second sqlplus command. Change this:
exit;
}
to this:
exit;
eof
}
Incidentally, you don't actually need to call sqlplus two separate times; you can use its connect command to drop one connection and open a new one:
function reset_db
{
sqlplus user1/password1#${input} << eof
set timing off
set serveroutput on size 10000
set feedback off
spool logfile_$input.out
delete from table1 where component = 'XYZ';
delete from table2 where component = 'XYZ';
commit;
connect dba_usr/dba_password#${input}
drop user ABC cascade;
drop user DEF cascade;
drop user HIG cascade;
eof
}
Below is my shell script from which I am trying to invoke few hive SQL queries which is working fine.
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*504);'`
echo $DATE_YEST_FORMAT1
hive -e "
SELECT t1 [0] AS buyer_id
,t1 [1] AS item_id
,created_time
FROM (
SELECT split(ckey, '\\\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1
) X
ORDER BY buyer_id
,created_time DESC;"
sleep 120
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
Problem Statement:-
If you see my first hive -e block after the echo $DATE_YEST_FORMAT1. Sometimes that query gets failed due to certain reasons. So currently what happens is that, if the first Hive SQL query gets failed, then it goes to second Hive SQL query after sleeping for 120 seconds. And that is the thing I don't want. So Is there any way if the first query gets failed dues to any reasons, it should get stopped automatically at that point. And it should start running automatically from the starting again after few minutes(should be configurable)
Update:-
As suggested by Stephen.
I tried something like this-
#!/bin/bash
hive -e " blaah blaah;"
RET_VAL=$?
echo $RET_VAL
if [ $RET_VAL -ne 0]; then
echo "HiveQL failed due to certain reason" | mailx -s "LIP Query Failed" -r rj#host.com rj#host.com
exit(1)
I got something like this below as an error and I didn't got any email too. Anything wrong with my syntax and approach?
syntax error at line 152: `exit' unexpected
Note:-
Zero is success here if the Hive Query is executed successfully.
Another Update after putting the space:-
After making changes like below
#!/bin/bash
hive -e " blaah blaah;"
RET_VAL=$?
echo $RET_VAL
if [ $RET_VAL -ne 0 ]; then
echo "HiveQL failed due to certain reason for LIP" | mailx -s "LIP Query Failed" -r rj#host.com rj#host.com
fi
exit
hive -e 'Another SQL Query;'
I got something like below-
RET_VAL=0
+ echo 0
0
+ [ 0 -ne 0 ]
+ exit
Status code was zero as my first query was successful but my program exited after that and it didn't went to execute my second query? Why? I am missing something here for sure again.
You may also find useful setting the exit immediately option:
set -e Exit immediately if a simple command (see SHELL GRAMMAR
above) exits with a non-zero status. The shell does not
exit if the command that fails is part of the command
list immediately following a while or until keyword,
part of the test in an if statement, part of a && or ||
list, or if the command's return value is being inverted
via !. A trap on ERR, if set, is executed before the
shell exits.
as in this example
#!/bin/bash
set -e
false
echo "Never reached"
Unless I'm misunderstanding the situation, it's very simple:
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*504);'`
echo $DATE_YEST_FORMAT1
QUERY0="
SELECT t1 [0] AS buyer_id
,t1 [1] AS item_id
,created_time
FROM (
SELECT split(ckey, '\\\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1
) X
ORDER BY buyer_id
,created_time DESC;"
if hive -e "$QUERY0"
then
sleep 120
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
# ...and whatever you do with $QUERY1...
fi
The string $QUERY0 is for convenience, not necessity. The key point is that you can test whether a command succeeded (returned status 0) with the if statement. The test command (better known as [) is just a command that returns 0 when the tested condition is met, and 1 (non-zero) when it is not met.
So, the if statement runs the first hive query; if it passes (exit status 0), then (and only then) does it move on to the actions in the then clause.
I've resisted the temptation to reformat your SQL; suffice to say, it is not the layout I would use in my own code.