Shell script to fetch sql query data in csv file - linux

Need to extract the below query data along with header in csv file using shell script.
Below is the query.
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode,NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UT)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj=0
UNION
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode, NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UTG)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj<>0
Could please let me know if below approach to fetch data using shell script is correct and script is correct.
#!/bin/bash
file="output.csv"
sqlplus -s username/password#Oracle_SID << EOF
SPOOL $file
select 'SourceIdentifier','SourceFileName','ProfitCentre2','PlantCode',
'tax_retur ReturnPeriod','document_number DocumentNumber','TO_CHAR(invoice_generation_date,'YYYY-MM-DD') Docume','Original','customer_name CustomerName','NVL(sns_pos,new_state_code)POS','PortCode','NEW_HSN_CODE HSNorSAC','(SGSATE+UTGSATE) Stat','(SGS+UT)StateUT','Userde' from dual
Union all
select 'TO_CHAR(SourceIdentifier)','TO_CHAR(SourceFileName)','TO_CHAR(ProfitCentre2)','TO_CHAR(PlantCode)',
'TO_CHAR(tax_retur ReturnPeriod)','TO_CHAR(document_number DocumentNumber)','TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume','TO_CHAR(Original)','TO_CHAR(customer_name CustomerName)','TO_CHAR(NVL(sns_pos,new_state_code)POS)','TO_CHAR(PortCode)','TO_CHAR(NEW_HSN_CODE HSNorSAC)','TO_CHAR((SGSATE+UTGSATE) Stat)','TO_CHAR((SGS+UT)StateUT)','TO_CHAR(Userde)' from
(SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode,NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UT)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj=0
UNION
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode, NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UTG)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj<>0)
SPOOL OFF
EXIT
EOF


In short: the ; is missing from the end of the select statement.
Some unrequested advice:
I think spool will put extra stuff into your file (at least some new lines), a redirect is better, further the first line is not db-related:
echo "SourceIdentifier;SourceFileName;ProfitCentre2..." > $file
I recommend to generate the csv format right in the select query, later it will be more headache (you can escape there what you want):
$query = "select SourceIdentifier || ';' || SourceFileName || ';' || ProfitCentre2 ... ;"
So querying the DB (I think capital -S is the right one) plus for the formatting of the records (and maybe you want to format your columns too):
sqlplus -S username/password#Oracle_SID >> $file << EOF
set linesize 32767 pagesize 0 heading off
$query
EOF


For me this one is working but one empty line before first query and second query is coming. Empty line remove using awk command
#!/bin/bash
FILE="A.csv"
$ORACLE_HOME/bin/sqlplus -s username/password#Oracle_SID<<EOF
SET PAGESIZE 50000 COLSEP "," LINESIZE 20000 FEEDBACK OFF HEADING off
SPOOL $FILE
select 'TYPE_OF_CALL_V','SWITCH_CALL_TYPE_V','RECORD_TYPE_V','TARF_TYPE_V' from dual;
SELECT TYPE_OF_CALL_V,SWITCH_CALL_TYPE_V,RECORD_TYPE_V,TARF_TYPE_V FROM TABLE;
SPOOL OFF
EXIT
EOF
awk 'NF > 0' $FILE > out.txt
mv out.txt $FILE

Related

Bash script with multiline heredoc doesn't output anything

I'm writing a script to send SQL output in mail, but it is not executing successfully and is not generating the output I want.
The query generates two columns with multiple rows. How can I generate the output in table format as below?
Below is my code:
#!/bin/bash
ORACLE_HOME= **PATH
export ORACLE_HOME
PATH=$PATH:$ORACLE_HOME/bin
export PATH
TNS_ADMIN= ** PATH
export TNS_ADMIN
today=$(date +%d-%m-%Y)
output=$(sqlplus -S user/pass#service <<EOF
set heading off;
SELECT distinct list_name ,max(captured_dttm) as Last_received FROM db.table1
group by list_name having max(captured_dttm) <= trunc(sysdate - interval '2' hour);
EOF)
if [ -z "$output" ];
then
echo"its fine"
exit
else
echo "
Dear All,
Kindly check we've not received the list for last 2 hour : $output
Regards,
Team" | mailx -S smtp=XX.XX.X.XX:XX -s "URGENT! Please check list FOR $today" user#abc.com
fi

When using a here document, the closing string can't be followed by anything but a newline. Move the closing parenthesis to the next line:
output=$(sqlplus -S user/pass#service <<EOF
...
EOF
)

Excel file generation using SQL query, SPOOL command in Shell script

I have to generate excel file from tables from Oracle Database. The code is working fine but however the column names are not coming completely, there are just coming as the length of there data length.
I want complete header/column names
The column names are coming like this
ST STAT_TYPE_DESC ST S NXT_STATME DELAY_DAYS ANN_PREM_LIM_LOW ANN_PREM_LIM_HI CONTRIB_HIST_LEN EVENT_DO C P
But I want is complete names of the columns, for example ST is STATEMENT_TYPE_ID
#!/bin/ksh
FILE="A.csv"
sqlplus -s lifelite/lifelite#dv10 <<EOF
SPOOL $FILE
SET HEADING ON
SET HEADSEP OFF
SET FEEDBACK OFF
SET LINESIZE 250
SET PAGESIZE 5000 embedded ON
SET COLSEP ","
SELECT * FROM TLLI_01150STATTYPE;
EOF
SPOOL OFF
EXIT 0

Before your select add a new one with the column names:
SELECT 'STATEMENT_TYPE_ID' || ',' || 'STAT_TYPE_DESC' || ',' || ... FROM dual;
And set heading off

shell script has been retrieving from edbplus sql results with echo outputs

I am trying to call edbplus to count a table from a command-line linux shell script, but I have been retrieving from edbplus the response number with others outputs in the same response, I am trying to retrieve from it only an integer response number.
#!/bin/sh
COUNT=`./edbplus.sh -silent user/password#localhost:5444/mydb<<-EOF
SET PAGESIZE 0 FEEDBACK OFF VERIFY OFF HEADING OFF ECHO OFF
SELECT COUNT(ID) FROM MYTABLE
EXIT;
EOF`
echo $COUNT
Response:
$ echo $COUNT
6-------------------d always takes 2 parameters: variable_name value
Do you know how get only the integer number?

If the 1st value is going to be integer. Please try the below commands
echo $COUNT | cut -d - -f 1
(or)
if only one int value if required, then please try
echo $COUNT | cut -c 1
To solve it from EDB perspective:
If the below flags are used in EDB in single line, then the above issue would have caused.
SET PAGESIZE 0
SET FEEDBACK OFF
SET VERIFY OFF
SET HEADING OFF
SET ECHO OFF
Kindly update it as above and provide it in individual lines.

split pipe separated string fetched using SQL

My shell script executes a SQL to fetch the data in the following format:
JOB_ID_001|[PROD] This is a mail subject one ${application_date}|a#example.com,b#example.com
JOB_ID_002|[PROD] This is a mail subject two ${application_date}|c#example.com,b#example.com
I want to split this pipe-separated string, but the output looks very odd as follows:
JOB_ID_001[0]
JOB_ID_001[1]
JOB_ID_001[2]
This[0]
This[1]
This[2]
is[0]
is[1]
is[2]
a[0]
a[1]
a[2]
mail[0]
mail[1]
mail[2]
subject[0]
subject[1]
subject[2]
one[0]
one[1]
one[2]
${application_date}[0]
${application_date}[1]
${application_date}[2]
example.com,b#example.com[0]
example.com,b#example.com[1]
example.com,b#example.com[2]
JOB_ID_002[0]
JOB_ID_002[1]
JOB_ID_002[2]
This[0]
This[1]
This[2]
is[0]
is[1]
is[2]
a[0]
a[1]
a[2]
mail[0]
mail[1]
mail[2]
subject[0]
subject[1]
subject[2]
two[0]
two[1]
two[2]
${application_date}[0]
${application_date}[1]
${application_date}[2]
ple.com,b#example.com[0]
ple.com,b#example.com[1]
ple.com,b#example.com[2]
My desired output is:
JOB_ID_001
[PROD] This is a mail subject one ${application_date}
a#example.com,b#example.com
JOB_ID_002
[PROD] This is a mail subject two ${application_date}
c#example.com,b#example.com
So that I can continue with those strings.
My shell script is as follows:
email_configs=(`sqlplus -silent $DB_CONN <<-EOF
whenever sqlerror exit 1 oserror exit oscode
set heading off feedback off echo off verify off pagesize 0
$sql_subject_of_mail;
exit;
EOF`)
for i in "${!email_configs[#]}"
do
email_config=${email_configs[i]}
IFS='|' read -r -a email_config_array <<< "$email_config"
job_id=$email_config_array[0]
subject_of_mail=$email_config_array[1]
to_mail_id=$email_config_array[2]
echo $job_id
echo $subject_of_mail
echo $to_mail_id
done
I checked some alternate solutions from this page, but in the output ${application_date} part is missing or there is some other problem.
Can anyone have an idea about my mistake?

The $email_configs array is not being set correctly. It's using spaces as the array delimiters, not newlines.
Rather than set an array, read the output of sqlplus in a loop.
while IFS='|' read -r job_id subject_of_mail to_mail_id
do
echo "$job_id"
echo "$subject_of_mail"
echo "$to_mail_id"
done < <(sqlplus -silent $DB_CONN <<-EOF
whenever sqlerror exit 1 oserror exit oscode
set heading off feedback off echo off verify off pagesize 0
$sql_subject_of_mail;
exit;
EOF
)

Stopping the shell script if any of the query gets failed

Below is my shell script from which I am trying to invoke few hive SQL queries which is working fine.
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*504);'`
echo $DATE_YEST_FORMAT1
hive -e "
SELECT t1 [0] AS buyer_id
,t1 [1] AS item_id
,created_time
FROM (
SELECT split(ckey, '\\\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1
) X
ORDER BY buyer_id
,created_time DESC;"
sleep 120
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
Problem Statement:-
If you see my first hive -e block after the echo $DATE_YEST_FORMAT1. Sometimes that query gets failed due to certain reasons. So currently what happens is that, if the first Hive SQL query gets failed, then it goes to second Hive SQL query after sleeping for 120 seconds. And that is the thing I don't want. So Is there any way if the first query gets failed dues to any reasons, it should get stopped automatically at that point. And it should start running automatically from the starting again after few minutes(should be configurable)
Update:-
As suggested by Stephen.
I tried something like this-
#!/bin/bash
hive -e " blaah blaah;"
RET_VAL=$?
echo $RET_VAL
if [ $RET_VAL -ne 0]; then
echo "HiveQL failed due to certain reason" | mailx -s "LIP Query Failed" -r rj#host.com rj#host.com
exit(1)
I got something like this below as an error and I didn't got any email too. Anything wrong with my syntax and approach?
syntax error at line 152: `exit' unexpected
Note:-
Zero is success here if the Hive Query is executed successfully.
Another Update after putting the space:-
After making changes like below
#!/bin/bash
hive -e " blaah blaah;"
RET_VAL=$?
echo $RET_VAL
if [ $RET_VAL -ne 0 ]; then
echo "HiveQL failed due to certain reason for LIP" | mailx -s "LIP Query Failed" -r rj#host.com rj#host.com
fi
exit
hive -e 'Another SQL Query;'
I got something like below-
RET_VAL=0
+ echo 0
0
+ [ 0 -ne 0 ]
+ exit
Status code was zero as my first query was successful but my program exited after that and it didn't went to execute my second query? Why? I am missing something here for sure again.

You may also find useful setting the exit immediately option:
set -e Exit immediately if a simple command (see SHELL GRAMMAR
above) exits with a non-zero status. The shell does not
exit if the command that fails is part of the command
list immediately following a while or until keyword,
part of the test in an if statement, part of a && or ||
list, or if the command's return value is being inverted
via !. A trap on ERR, if set, is executed before the
shell exits.
as in this example
#!/bin/bash
set -e
false
echo "Never reached"

Unless I'm misunderstanding the situation, it's very simple:
#!/bin/bash
DATE_YEST_FORMAT1=`perl -e 'use POSIX qw(strftime); print strftime "%Y-%m-%d",localtime(time()- 3600*504);'`
echo $DATE_YEST_FORMAT1
QUERY0="
SELECT t1 [0] AS buyer_id
,t1 [1] AS item_id
,created_time
FROM (
SELECT split(ckey, '\\\\|') AS t1
,created_time
FROM (
SELECT CONCAT (
buyer_id
,'|'
,item_id
) AS ckey
,created_time
FROM dw_checkout_trans
WHERE to_date(from_unixtime(cast(UNIX_TIMESTAMP(created_time) AS BIGINT))) = '$DATE_YEST_FORMAT1' distribute BY ckey sort BY ckey
,created_time DESC
) a
WHERE rank(ckey) < 1
) X
ORDER BY buyer_id
,created_time DESC;"
if hive -e "$QUERY0"
then
sleep 120
QUERY1=`hive -e "
set mapred.job.queue.name=hdmi-technology;
SELECT SUM(total_items_purchased), SUM(total_items_missingormismatch) from lip_data_quality where dt='$DATE_YEST_FORMAT2';"`
# ...and whatever you do with $QUERY1...
fi
The string $QUERY0 is for convenience, not necessity. The key point is that you can test whether a command succeeded (returned status 0) with the if statement. The test command (better known as [) is just a command that returns 0 when the tested condition is met, and 1 (non-zero) when it is not met.
So, the if statement runs the first hive query; if it passes (exit status 0), then (and only then) does it move on to the actions in the then clause.
I've resisted the temptation to reformat your SQL; suffice to say, it is not the layout I would use in my own code.

Resources