Oracle - How to save table as csv file from terminal with command - linux

I have this process to save a table to CSV file but is not perfect:
sqlplus / as sysdba
set heading off
SET TERMOUT OFF
spool MY_TABLE.csv
select column_id||','||column_name||','||data_type||','||nullable||','||data_length from user1.MY_TABLE;
spool off;
exit
Issue with that is that can not turn off printing to terminal (I have SET TERMOUT OFF but that does not suppress the output for some reason).
I would like to run a sqlplus with a query from a terminal (instead of opening sqlplus and running from there)..
How to do that?

Example 1 for linux create_csv.sh displays the table data on the monitor screen.
#!/bin/sh
sqlplus -s / as sysdba <<EOF
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set timing off
set echo off
set linesize 1000
set pagesize 0
SET COLSEP ','
spool test2.csv
select code,name,code_rail from alexs.all_station;
spool off
exit;
EOF
Example 2 for linux createv2_csv.sh does not display anything.
#!/bin/sh
$ORACLE_HOME/bin/sqlplus -s / as sysdba #csv2.sql
csv2.sql removed spaces and tabs between fields in the output file test2.csv
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set echo off
set linesize 1000
set pagesize 0
set wrap off
spool test2.csv
select code||','||name||','||code_rail from alexs.all_station;
spool off
exit;
Example for Windows test_csv.bat
sqlplus -s user/password#net_alias < csv.sql
csv.sql
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set timing off
set echo off
set linesize 1000
set pagesize 0
SET COLSEP ','
spool test.csv
select code,name,code_rail from alexs.all_station;
spool off
exit;
Example for Windows powershell script create_csv.ps1
# login DBA user
$username_dba = "system"
# PASSWORD DBA user !!!!
$password_dba = "manager"
# Connect without alias
$tnsalias_db = "(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = test.com)(PORT = 1521)))(CONNECT_DATA = (SID = test)))"
$sqlQuery =
#"
set heading off
set termout OFF
SET FEEDBACK OFF
SET TAB OFF
set pause off
set verify off
SET UNDERLINE OFF
set trimspool on
set timing off
set echo off
set linesize 1000
set pagesize 0
SET COLSEP ','
spool test.csv
select code,name,code_rail from alexs.all_station;
spool off
exit
"#
$sqlOutput = $sqlQuery | sqlplus -silent $username_dba/$password_dba#$tnsalias_db
Example output test.csv
966203,ГУРСКОЕ , 96
966307,УКТУР , 96
966400,КЕНАЙ , 96
966504,ВЫСОКОГОРНАЯ , 96
966608,КЕНАДА , 96
966805,ТУЛУЧИ , 96
966909,АКУР , 96
967009,ТУМНИН , 96
967102,ХУТУ , 96
967206,УСТЬ-ОРОЧИ , 96
967305,МОНГОХТО , 96
967403,ЛАНДЫШИ , 96
967526,ТОКИ , 96
967600,ВАНИНО , 96
967704,ВАНИНО-ПЕРЕВАЛКА , 96
967808,ВАНИНО-ЭКСПОРТ , 96
968001,ВИНЕВИТИНО , 96
Example output test2.csv
971907,ВЯЗЕМСКАЯ,96
972007,АВАН,96
972100,РОЗЕНГАРТОВКА,96
972204,БИКИН,96
972308,ЗВЕНЬЕВОЙ,96
972401,БУРЛИТ-ВОЛОЧАЕВСКИЙ,96
972505,ЛУЧЕГОРСК,96
972609,ЛАСТОЧКА,96
972702,ГУБЕРОВО,96
972721,ЭБЕРГАРД,96
972806,ДАЛЬНЕРЕЧЕНСК I,96
973207,ДАЛЬНЕРЕЧЕНСК II,96
973300,ЛАЗО,96
973404,ГРУШЕВОЕ,96
973508,ПРОХАСКО,96
973601,ФИЛАРЕТОВКА,96
973705,РУЖИНО,96
973809,ЛЕСОЗАВОДСК I,96
973813,1634 КМ,96
973902,ШМАКОВКА,96
974002,СУНГАЧ,96
974106,СВИЯГИНО,96
974229,ДРОЗДОВ,96

Related

Excel file generation using SQL query, SPOOL command in Shell script

I have to generate excel file from tables from Oracle Database. The code is working fine but however the column names are not coming completely, there are just coming as the length of there data length.
I want complete header/column names
The column names are coming like this
ST STAT_TYPE_DESC ST S NXT_STATME DELAY_DAYS ANN_PREM_LIM_LOW ANN_PREM_LIM_HI CONTRIB_HIST_LEN EVENT_DO C P
But I want is complete names of the columns, for example ST is STATEMENT_TYPE_ID
#!/bin/ksh
FILE="A.csv"
sqlplus -s lifelite/lifelite#dv10 <<EOF
SPOOL $FILE
SET HEADING ON
SET HEADSEP OFF
SET FEEDBACK OFF
SET LINESIZE 250
SET PAGESIZE 5000 embedded ON
SET COLSEP ","
SELECT * FROM TLLI_01150STATTYPE;
EOF
SPOOL OFF
EXIT 0
Before your select add a new one with the column names:
SELECT 'STATEMENT_TYPE_ID' || ',' || 'STAT_TYPE_DESC' || ',' || ... FROM dual;
And set heading off

Shell SQLPLUS prints CLOB data twice

I have the below code in my ksh shell script file and when i run this i get the output printed twice in the file. here the FILE_DATA is of type CLOB
sqlplus -s ${uidpwd} > ${DATA_FILE_NAME} << EOF
set head off
SET pagesize 0
SET LINESIZE 32767
SET LONGCHUNKSIZE 32767
SET LONG 320000000
SET RECSEP OFF
set SERVEROUTPUT off
SET ECHO off
SET TRIMOUT ON
SET TRIMSPOOL ON
set termout off
set feedback off verify off heading off newpage 0
whenever OSERROR EXIT 9
whenever SQLERROR EXIT sql.SQLCODE
SELECT FILE_DATA FROM FILES WHERE ID='208';
/
exit;
EOF
Data:
"Client_id,Client_id_type,Client_Segment
1,LOCAL_ID,Individual"

Shell script to fetch sql query data in csv file

Need to extract the below query data along with header in csv file using shell script.
Below is the query.
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode,NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UT)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj=0
UNION
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode, NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UTG)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj<>0
Could please let me know if below approach to fetch data using shell script is correct and script is correct.
#!/bin/bash
file="output.csv"
sqlplus -s username/password#Oracle_SID << EOF
SPOOL $file
select 'SourceIdentifier','SourceFileName','ProfitCentre2','PlantCode',
'tax_retur ReturnPeriod','document_number DocumentNumber','TO_CHAR(invoice_generation_date,'YYYY-MM-DD') Docume','Original','customer_name CustomerName','NVL(sns_pos,new_state_code)POS','PortCode','NEW_HSN_CODE HSNorSAC','(SGSATE+UTGSATE) Stat','(SGS+UT)StateUT','Userde' from dual
Union all
select 'TO_CHAR(SourceIdentifier)','TO_CHAR(SourceFileName)','TO_CHAR(ProfitCentre2)','TO_CHAR(PlantCode)',
'TO_CHAR(tax_retur ReturnPeriod)','TO_CHAR(document_number DocumentNumber)','TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume','TO_CHAR(Original)','TO_CHAR(customer_name CustomerName)','TO_CHAR(NVL(sns_pos,new_state_code)POS)','TO_CHAR(PortCode)','TO_CHAR(NEW_HSN_CODE HSNorSAC)','TO_CHAR((SGSATE+UTGSATE) Stat)','TO_CHAR((SGS+UT)StateUT)','TO_CHAR(Userde)' from
(SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode,NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UT)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj=0
UNION
SELECT SourceIdentifier,SourceFileName,ProfitCentre2,PlantCode,
tax_retur ReturnPeriod,document_number DocumentNumber,TO_CHAR(invoice_generation_date,'YYYY-MM-DD')
Docume,Original,customer_name CustomerName,NVL(sns_pos,new_state_code)POS,PortCode, NEW_HSN_CODE HSNorSAC,(SGSATE+UTGSATE) Stat,(SGS+UTG)StateUT,Userde FROM arbor.INV_REPO_FINA WHERE UPPER(document_type)='INV' AND UPPER(backout_flag)='VALID' AND new_gst_id_new IS NOT NULL AND new_charges<>0 AND taxable_adj<>0)
SPOOL OFF
EXIT
EOF
In short: the ; is missing from the end of the select statement.
Some unrequested advice:
I think spool will put extra stuff into your file (at least some new lines), a redirect is better, further the first line is not db-related:
echo "SourceIdentifier;SourceFileName;ProfitCentre2..." > $file
I recommend to generate the csv format right in the select query, later it will be more headache (you can escape there what you want):
$query = "select SourceIdentifier || ';' || SourceFileName || ';' || ProfitCentre2 ... ;"
So querying the DB (I think capital -S is the right one) plus for the formatting of the records (and maybe you want to format your columns too):
sqlplus -S username/password#Oracle_SID >> $file << EOF
set linesize 32767 pagesize 0 heading off
$query
EOF
For me this one is working but one empty line before first query and second query is coming. Empty line remove using awk command
#!/bin/bash
FILE="A.csv"
$ORACLE_HOME/bin/sqlplus -s username/password#Oracle_SID<<EOF
SET PAGESIZE 50000 COLSEP "," LINESIZE 20000 FEEDBACK OFF HEADING off
SPOOL $FILE
select 'TYPE_OF_CALL_V','SWITCH_CALL_TYPE_V','RECORD_TYPE_V','TARF_TYPE_V' from dual;
SELECT TYPE_OF_CALL_V,SWITCH_CALL_TYPE_V,RECORD_TYPE_V,TARF_TYPE_V FROM TABLE;
SPOOL OFF
EXIT
EOF
awk 'NF > 0' $FILE > out.txt
mv out.txt $FILE

SQLPLUS formatting output in excel

Below is the code to retrieve data from DB and output it in excel
sqlplus -s ${uidpwd} <<EOF
SET PAGESIZE 50000
SET FEEDBACK OFF
SET MARKUP HTML ON
SET NUM 24
SPOOL Extract.xls
whenever oserror exit 9
whenever sqlerror exit sql.sqlcode
#${SQL_SCRIPT}
SET SERVEROUTPUT ON
SET ECHO OFF
SET VERIFY OFF
SPOOL OFF
SET MARKUP HTML OFF
quit;
EOF
But what i see in excel report is all numbers in excel are with exponential like "5.30056E+15"
i tried below options, but none of them worked
SET NUMBERFORMAT '9999999999999999999999'
SET NUMWIDTH 20
in SQL i tried below
to_char(conf_num, '999999999999999999999999')
SELECT ' '||conf_num "Confirmation Number"
Also i have exec > ${LOGPATH}/log.``date +"%y%m%d"``.log 2>&1 in my script, Query out put is getting written in to this log file, how to avoid that? It should have only error logs.

Puzzling Batch String Slicing Error

So I am creating a program that separates numbers and letters into 2 different variables so it turns "word1234" into a variable containing "word" and a variable containing "1234", what I did was make a program that runs through a variable "info" letter by letter. "Is it a number?" "No", "Is It a letter?" "Yes". When the variable "toggle" was still on the value 0, it would carry on appending the letters to the "weatherd" variable and else append it to the "temperat" variable.
Note: I know my variable and label names are bad, I come up with them in about half a second :)
So it checks if it is a number, if it is the "toggle" variable changes to a 1 and it starts putting the rest of the text into a new variable ("temperat"). I was wondering, once it hits the numbers it gives me an error and crashes, why? Or is there a simpler way than what I'm doing?
Sorry for the massive post, I added some pauses and stuff for debugging, code here:
REM PREPARE FOR A MASSIVE AMOUNT OF CODE, all this stuff just gets the current weather, ignore it
#echo off
set "wherestay=%cd%"
cd /d C:
cd\
cd "C:/Users/%USERNAME%/Downloads"
del weather.txt
start chrome.exe teamhaxor.netau.net/getWeather.php
set times=0
:loading
set /a times=%times% + 1
if %times% == 2000 goto failed
if not exist weather.txt goto loading
cd /d C:
cd\
cd "C:/Users/%USERNAME%/Downloads"
< weather.txt (
set /p info=
)
REM PROBLEM STARTS HERE
setlocal enabledelayedexpansion
#echo on
set tempery=0
set "weatherd="
set "temperat="
set toggle=0
:loopy
set "char=!info:~%tempery%, 1!"
if "%char%" EQU "0" set toggle=1
if "%char%" EQU "1" set toggle=1
if "%char%" EQU "2" set toggle=1
if "%char%" EQU "3" set toggle=1
if "%char%" EQU "4" set toggle=1
if "%char%" EQU "5" set toggle=1
if "%char%" EQU "6" set toggle=1
if "%char%" EQU "7" set toggle=1
if "%char%" EQU "8" set toggle=1
if "%char%" EQU "9" set toggle=1
pause
if %toggle% EQU 0 (set weatherd=%weatherd%%char%) ELSE (set temperat=%temperat%%char%)
pause
set /a tempery=%tempery% + 1
pause
if %tempery% EQU 15 (goto out) ELSE (goto loopy)
:out
echo %weatherd%
echo %temperat%
setlocal disabledelayedexpansion
REM PROBLEM STOPS HERE
goto afterafterlol
:failed
set weatherd=Failed
set temperat=Failed
:afterafterlol
cd /d C:
cd\
cd %wherestay%
echo %weatherd%
echo %temperat%
pause >Nul
To answer your question directly, there's an easier way to split your value so that you end up with alpha in one variable and numeric in another.
Treat numbers as delimiters, thereby removing them without having to know specifically which numbers will occur. This will leave you with only the alphabetic portion of the value.
Remove the alphabetic portion from the whole, leaving you with the numeric porition.
#echo off & setlocal
set "var=Thunderstorms87"
for /f "delims=0123456789" %%I in ("%var%") do set "condition=%%~I"
call set "temperature=%%var:%condition%=%%"
echo %condition%
echo %temperature%
In a broader sense, your project seems like a very cumbersome way to scrape not a lot of useful data. It would be better to query a weather service that offers an API with a structured response that doesn't require splitting strings or other flat text hacks.
Here's an example script that uses Weather Underground's API, retrieving and parsing its JSON response. You'll need to create an account and sign up for a free API key, but you get so much more data and it's so much easier to parse. And there's no need to spawn a Chrome window to fetch your data this way.
#if (#CodeSection == #Batch) #then
#echo off & setlocal
rem // weather.bat
rem // https://stackoverflow.com/a/44015647/1683264
rem // go to https://www.wunderground.com/weather/api/ for an API key
set "APIkey=pasteyourAPIkeyhere"
if "%~1"=="" ( set "location=autoip" ) else set "location=%~1"
cscript /nologo /e:JScript "%~f0" "%APIkey%" "%location%"
rem // end main runtime
goto :EOF
#end // end batch / begin JScript chimera
var x = WSH.CreateObject("Microsoft.XMLHTTP"),
htmlfile = WSH.CreateObject('htmlfile'),
APIkey = WSH.Arguments(0),
loc = WSH.Arguments(1),
URL = 'http://api.wunderground.com/api/' + APIkey + '/conditions/q/' + loc + '.json',
JSON = obj = {}, pad = ' ';
htmlfile.write('<meta http-equiv="x-ua-compatible" content="IE=9" />');
htmlfile.close(JSON = htmlfile.parentWindow.JSON);
x.open("GET", URL, true);
x.setRequestHeader('User-Agent','XMLHTTP/1.0');
x.send('');
while (x.readyState!=4) {WSH.Sleep(50)};
response = JSON.parse(x.responseText).current_observation;
obj = {
"station": response.observation_location.full,
"timestamp": response.observation_time,
"conditions": response.weather,
"temperature": response.temperature_string,
"humidity": response.relative_humidity,
"wind": response.wind_string,
"heat index": response.heat_index_string,
"wind chill": response.windchill_string,
"forecast URL": response.forecast_url
};
for (var i in obj) while (i.length > pad.length) pad += ' '
for (var i in obj) {
var key = (i + pad).substring(0, pad.length);
WSH.Echo(key + ' : ' + obj[i]);
}
Output looks something like this:
You can use something like:
WSH.Echo(x.responseText);
... to view the raw JSON and see all the other data included in the feed. You can also modify the URL and replace conditions with forecast, hourly, and a bunch of other options (as well as replacing current_observation and the response object properties as appropriate).

Resources