I am exporting data from a SQL Server table to a .csv file, and then I use sp_send_email to email the file with data.
My problem is with this value:
Cantata Number 212 "Peasants Cantata", BWV 212
The value gets split into two columns in the .csv file that gets emailed. This value should be only in one column.
Some titles might contain a comma, which needs to be left in the string for those instances.
For example:
Cantata Number 212 Peasants Cantata" BWV 212"
I tried this method, but is not working:
Note: This SELECT statement resides inside a view vw_WeeklyReport
SELECT TOP 100 PERCENT
'"' + [p].[Title] + '"' [Title]
FROM
table
The code that exports the data and emails the .csv file:
BEGIN
SET NOCOUNT ON;
DECLARE #qry VARCHAR(8000);
-- Create the query, concatenating the column name as an alias
SET #Qry = 'SET NOCOUNT ON; SELECT Title FROM [vw_WeeklyReport] SET NOCOUNT OFF';
-- Send the e-mail with the query results in attachment.
EXEC [msdb].[dbo].[sp_send_dbmail]
#profile_name = 'default',
#recipients = '6lack#email.com',
#subject = 'Weekly Report',
#body = 'An attachment has been included in this email.',
#query_attachment_filename = 'WeeklyRep.csv',
#query = #qry,
#attach_query_result_as_file = 1,
#query_result_separator = ',',
#query_result_width = 32767,
#query_result_no_padding = 1;
END;
When there are comma's (or separators) in the field, that field should be enclosed with double quotes, and any double quotes within have to be escaped with another double quote:
"Cantata Number 212 ""Peasants Cantata"", BWV 212"
Once double quotes are used around fields, all fields containing double quotes should also be quoted and inside quotes escaped as well.
Maybe you could look for an option to export to csv using quoted fields.
Removing all the comma's could also be an option, but then you lose some information.
On the other hand, if there is only one column (as in your SELECT statement) there is no need at all to use csv. A plain text file can be used instead.
Change your query in the stored proc to something like this:
SET #Qry = 'SET NOCOUNT ON; SELECT replace(Title, ',', '') as Title FROM [vw_WeeklyReport] SET NOCOUNT OFF';
Note this is untested, but should give you what you're looking for. This is under the presumption that stripping out commas is acceptable, as was indicated in the initial post. If the commas need to remain intact, the answer isn't quite as simple.
Related
I have a query that looks like this:
INSERT INTO table VALUES ('47677;2019;2019;10T-1001-10010AS;A05;International;TieLineKoman-KosovoB;L_KOM-KOSB;2018;NULL;NULL;;NULL;Tieline;NULL;10XAL-KESH-----J;0;3')
that is produced by parsing a csv file.
The query is not in a valid form, I have to replace all semicolons with the string ',' (comma inside single quotes). What I want to get is:
('47677','2019','2019','10T-1001-10010AS','A05','International','TieLineKoman-KosovoB','L_KOM-KOSB','2018','NULL','NULL','','NULL','Tieline','NULL','10XAL-KESH-----J','0','3')
I have tried to do this in many different ways, but I end up with backshlashes added in my string. This is what I get:
"INSERT INTO AllocatedEICDetail VALUES ('47677\\',\\'2019\\',\\'2019\\',\\'10T-1001-10010AS\\',\\'A05\\',\\'International\\',\\'TieLineKoman-KosovoB\\',\\'L_KOM-KOSB\\',\\'2018\\',\\'NULL\\',\\'NULL\\',\\'\\',\\'NULL\\',\\'Tieline\\',\\'NULL\\',\\'10XAL-KESH-----J\\',\\'0\\',\\'3')"
Any ideas how to do this properly without having the backslashes added?
Thank you!
//the string you have
const string = '47677;2019;2019;10T-1001-10010AS;A05;International;TieLineKoman-KosovoB;L_KOM-KOSB;2018;NULL;NULL;;NULL;Tieline;NULL;10XAL-KESH-----J;0;3';
//the string you need:
const targetString = string.replace(/\;/g,',');
You specify a small regex between the forward slashes in replace which is a simple ';', give it a 'g' flag for global which will replace all instances, and in the second argument supply what you need it replaced with.
I have a CSV file with some integer column, now it 's saved as "" (empty string).
I want to COPY them to a table as NULL value.
With JAVA code, I have try these:
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', HEADER true)";
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', NULL '' HEADER true)";
I get: PSQLException: ERROR: invalid input syntax for type numeric: ""
String sql = "COPY " + tableName + " FROM STDIN (FORMAT csv,DELIMITER ',', NULL '\"\"' HEADER true)";
I get: PSQLException: ERROR: CSV quote character must not appear in the NULL specification
Any one has done this before ?
I assume you are aware that numeric data types have no concept of "empty string" ('') . It's either a number or NULL (or 'NaN' for numeric - but not for integer et al.)
Looks like you exported from a string data type like text and had some actual empty string in there - which are now represented as "" - " being the default QUOTE character in CSV format.
NULL would be represented by nothing, not even quotes. The manual:
NULL
Specifies the string that represents a null value. The default is \N
(backslash-N) in text format, and an unquoted empty string in CSV format.
You cannot define "" to generally represent NULL since that already represents an empty string. Would be ambiguous.
To fix, I see two options:
Edit the CSV file / stream before feeding to COPY and replace "" with nothing. Might be tricky if you have actual empty string in there as well - or "" escaping literal " inside strings.
(What I would do.) Import to an auxiliary temporary table with identical structure except for the integer column converted to text. Then INSERT (or UPSERT?) to the target table from there, converting the integer value properly on the fly:
-- empty temp table with identical structure
CREATE TEMP TABLE tbl_tmp AS TABLE tbl LIMIT 0;
-- ... except for the int / text column
ALTER TABLE tbl_tmp ALTER col_int TYPE text;
COPY tbl_tmp ...;
INSERT INTO tbl -- identical number and names of columns guaranteed
SELECT col1, col2, NULLIF(col_int, '')::int -- list all columns in order here
FROM tbl_tmp;
Temporary tables are dropped at the end of the session automatically. If you run this multiple times in the same session, either just truncate the existing temp table or drop it after each transaction.
Related:
How to update selected rows with values from a CSV file in Postgres?
Rails Migrations: tried to change the type of column from string to integer
postgresql thread safety for temporary tables
Since Postgres 9.4 you now have the ability to use FORCE_NULL. This causes the empty string to be converted into a NULL. Very handy, especially with CSV files (actually this is only allowed when using CSV format).
The syntax is as follow:
COPY table FROM '/path/to/file.csv'
WITH (FORMAT CSV, DELIMITER ';', FORCE_NULL (columnname));
Further details are explained in the documentation: https://www.postgresql.org/docs/current/sql-copy.html
If we want to replace all blank and empty rows with null then you just have to add emptyasnull blanksasnull in copy command
syntax :
copy Table_name (columns_list)
from 's3://{bucket}/{s3_bucket_directory_name + manifest_filename}'
iam_role '{REDSHIFT_COPY_COMMAND_ROLE}' emptyasnull blanksasnull
manifest DELIMITER ',' IGNOREHEADER 1 compupdate off csv gzip;
Note: It will apply for all the records which contains empty/blank values
when using %STR(amy.goodman#yaho.com;bill.goodman#yaho.com;jack.goodman#yaho.com);
This gives me the result: amy.goodman#yaho.com;bill.goodman#yaho.com;jack.goodman#yaho.com
which is what I want. and this is a macro variable called &email. I need this semicolon in between any two emails but no semicolon after the last email
But for someone who has an single quote in her email address like O'Connor,
%str will automatically match the single quote
using %STR(amy.goodman#yaho.com;bill.goodman#yaho.com;lily.O'Connor#yaho.com) will cause error.
How can I get the result amy.goodman#yaho.com;bill.goodman#yaho.com;lily.O'Connor#yaho.com
by using %str or any other macro quote function like %bquote or anything else to keep that single quote unmatched?
THE EMAIL LIST IS imported from excel list and I already put% before the single quote in the excel file: lily.O%'Connor#yaho.com
and HOW can I get the the macro variable &email. value inside the %str(), namely:
amy.goodman#yaho.com;bill.goodman#yaho.com;lily.O%'Connor#yaho.com
then I can put macro variable into %str()?
If you have the value already in a macro variable, say from a data step via CALL SYMPUTX() function call or from PROC SQL via INTO clause, then you can use %SUPERQ() macro function to macro quote the value.
data _null_;
call symputx('unquoted',"O'Connor#domain.com");
run;
%put Email list = %superq(unquoted);
If the value is already in a macro variable the the %BQUOTE() function would also work.
%put Email list = %bquote(&unquoted);
You could even use the RESOLVE() function to call %SUPERQ() to get the macro variable redefined with macro quoting immediately after creating it. So if you have a dataset MYTABLE with a variable EMAIL_LIST you could generate a series of separate macro variables (list1,list2, etc) from the data using code like this.
data _null_;
set mytable ;
length mvar rc $32 ;
mvar = cats('list',_n_);
call symputx(mvar,email_list);
rc = resolve(catx(' ','%let',mvar,'=%superq(',mvar,');'));
run;
The first example data step works since the string literal used double quotes on the outside and thus protected the unbalanced single quote. You could do the same thing in macro code by using the %QSYSFUNC() macro function to call the DEQUOTE() function. Note that you would have trouble if the value includes double quote characters. They would need to be converted to two adjacent double quotes.
%put Email list = %qsysfunc(dequote("O'Connor#domain.com"));
With %STR, you need to mark unmatched quotes with a % sign. So try:
%STR(amy.goodman#yaho.com;bill.goodman#yaho.com;lily.O%'Connor#yaho.com)
I need to fill a cell with a data, separated by 'new line' symbol.
I've tried:
data: l_con_sepa TYPE c VALUE cl_abap_char_utilities=>newline.
...
CONCATENATE <gf_aufk>-tplnr " 6000000159 Korchagin AS 02.02.2017
<gf_aufk>-pltxt
l_con_sepa
<gf_aufk>-aufnr
INTO lv_str
SEPARATED BY space.
Tried to use CL_ABAP_CHAR_UTILITIES=>CR_LF. Tried to use "&" and "#" symbols. Tried to wrap lv_str with quotes. Nothing.
I either got symbols as is, or just a blank space insted of 'alt+enter' equivalent.
A simple experiment with Excel, namely creating a cell with Alt+Enter characters and saving it as a CSV file, shows that such a new line symbol is LF and not CR_LF. Moreover it is put there in double quotes.
So just use double quotes and CL_ABAP_CHAR_UTILITIES=>NEWLINE.
It must work with CSV. You did not specify what API you use to export your data to XLS format, so I cannot test it. If you do not mind putting those details in the question, please do so.
Assuming you use FM SAP_CONVERT_TO_XLS_FORMAT, there is even no need for double quotes.
REPORT YYY.
TYPES: BEGIN OF gty_my_type,
col1 TYPE char255,
col2 TYPE char255,
END OF gty_my_type,
gtty_my_type TYPE STANDARD TABLE OF gty_my_type WITH EMPTY KEY.
START-OF-SELECTION.
DATA(gt_string_table) = VALUE gtty_my_type(
(
col1 = 'aaa'
&& cl_abap_char_utilities=>newline
&& 'bbb'
&& cl_abap_char_utilities=>newline
&& 'ccc'
col2 = 'ddd'
)
).
CALL FUNCTION 'SAP_CONVERT_TO_XLS_FORMAT'
EXPORTING
i_filename = 'D:\temp\abap.xlsx'
TABLES
i_tab_sap_data = gt_string_table
EXCEPTIONS
conversion_failed = 1
OTHERS = 2.
ASSERT sy-subrc = 0.
The result looks like follows
I thought that it might be caused by CONCATENATE .. INTO .. SEPARATED BY space but it is not. Please execute the following program in order to check it out.
REPORT YYY.
TYPES: BEGIN OF gty_my_type,
col1 TYPE char255,
col2 TYPE char255,
END OF gty_my_type,
gtty_my_type TYPE STANDARD TABLE OF gty_my_type WITH EMPTY KEY.
DATA: gs_string TYPE gty_my_type.
DATA: gt_string_table TYPE gtty_my_type.
START-OF-SELECTION.
CONCATENATE 'aaa' cl_abap_char_utilities=>newline 'bbb' cl_abap_char_utilities=>newline 'ccc'
INTO gs_string-col1 SEPARATED BY space.
gs_string-col2 = 'ddd'.
APPEND gs_string TO gt_string_table.
CALL FUNCTION 'SAP_CONVERT_TO_XLS_FORMAT'
EXPORTING
i_filename = 'D:\temp\abap.xlsx'
TABLES
i_tab_sap_data = gt_string_table
EXCEPTIONS
conversion_failed = 1
OTHERS = 2.
ASSERT sy-subrc = 0.
So the problem must be somewhere else. You are not showing us your whole code. Maybe you use some kind of a third party package to process your Excel files?
I don't remember if it's needed to add an "end of line" symbol.
Just append each line into a table and download the full table using FM SAP_CONVERT_TO_XLS_FORMAT.
I am trying to add changes data in a csv file:
This is the sample data:
DATE status code value value2
"2016-01-26","Subscription All","119432660","1315529431362550","0.0080099833517888"
"2016-01-26","Subscription All","119432664","5836995058433524","0.033825584764444"
"2016-01-26","Subscription All","119432664","8287300074499777","0.076913377834744"
"2016-01-26","Subscription All","119432664","14870697739968326","0.0074188355187426"
My code used to format the data:
CSVReader reader = new CSVReader(new FileReader(new File(fileToChange)), CSVParser.DEFAULT_SEPARATOR, CSVParser.NULL_CHARACTER, CSVParser.NULL_CHARACTER, 1)
info "Read all rows at once"
List<String[]> allRows = reader.readAll();
CSVWriter writer = new CSVWriter(new FileWriter(fileToChange), CSVWriter.DEFAULT_SEPARATOR, CSVWriter.NO_QUOTE_CHARACTER)
writer.writeAll(allRows)
writer.close()
The output i get is this, with extra quote added instead of removing it.
""2016-01-26"",""Subscription All"",""119432660"",""1315529431362550"",""0.0080099833517888""
""2016-01-26"",""Subscription All"",""119432664"",""5836995058433524"",""0.033825584764444""
""2016-01-26"",""Subscription All"",""119432664"",""8287300074499777"",""0.076913377834744""
""2016-01-26"",""Subscription All"",""119432664"",""14870697739968326"",""0.0074188355187426""
I want to remove the quotes.
Please can someone help.
Also, is it possible to change the date format to yyyymmdd instead of yyyy-mm-dd?
allRows.each { String[] theLine ->
String newDate = theLine[0].replaceAll('-', '')
String newline = theLine.eachWithIndex { String s, int i -> return i > 0 ? s : newDate}
writer.writeLine(newline)
}
Thanks
When you instantiated your CSVReader you told it to treat no characters as quotes, therefore it read the existing quotes as data and did not remove them.
When you told CSVWriter not to add any quotes it honored your request. However, the input data contained quote characters, and the convention for including quotes inside a string in CSV is to double the quotes. Thus the
string value
ABC"DEF
gets coded in CSV as
"ABC""DEF"
So the result you see is the combination of not removing the quotes on input (you told it not to) and then doubling the quotes on output.
To solve this change the input option from NULL_CHARACTER to DEFAULT_QUOTE_CHARACTER. However be aware that if any of your data actually contains embedded quotes or commas the resulting output will not be valid CSV.
Also I think this might be a valid bug report against OpenCSV. I believe that OpenCSV needs to inform you if it is about to generate invalid CSV when you told it to omit quotes, probably via a runtime exception. Although I suppose they might argue that you chose to work without a net and should accept whatever you get. Personally I go for the "principle of least surprise", which IMHO would be not to double quotes when the output is unquoted.
Because quotation in your CSVReader is set to CSVParser.NULL_CHARACTER " is treated as normal character which is part of read token. This causes your array to contain data in form:
["2016-01-26", "Subscription All", "119432660", "1315529431362550", "0.0080099833517888"]
rather than:
[2016-01-26, Subscription All, 119432660, 1315529431362550, 0.0080099833517888]
So try changing option from CSVParser.NULL_CHARACTER to either
'"'
CSVParser.DEFAULT_QUOTE_CHARACTER (it also stores '"').
CsvToBean csvToBean = new CsvToBeanBuilder(new StringReader(csv))
.withMappingStrategy(strategy)
.withIgnoreLeadingWhiteSpace(true)
.withSeparator(',')
.withIgnoreEmptyLine(true)
.withQuoteChar('\'')
.withQuoteChar('"')
.build();