My Java EE web app uses XSLT transformation to produce CSV reports.
If I specify "UTF-16" as the encoding on the XSLT (which seems to be the only way to handle accented characters in the XML) then I get a byte-order-mark character at the start of the CSV output
From what I understand, this is correct behavior. The problem is that this BOM character is visible if the CSV is opened in Excel (appears as: þÿ), but not if I open it in a proper text editor ;)
How can I hide or omit this character so it isn't visible when the file is opened in Excel?
You forgot to specify which version of XSLT you are using.
If you are using XSLT 2.0, you can suppress the BOM using the xsl:output element. For example...
<xsl:output method="xml" byte-order-mark="no" encoding="UTF-16" indent="yes" omit-xml-declaration="yes" />
What if you are using XSLT 1.0?
Here are two options:
Upgrade to XSLT 2.0
Upgrade your Excel to 2007 (See here for why)
if you are writing a CSV file the method should probably not be "xml" as Sean B. Durkin suggests.
i recommend this configuration to make a CSV that will correctly open in Excel
<xsl:output method="text" byte-order-mark="yes" encoding="UTF-16LE" indent="no"/>
Other tips when using XSLT to generate a CSV:
You can use TAB characters even though CSV means "comma separated value"
I find the data i separate is often void of tabs but has the occasional comma.
and <xsl:text>
</xsl:text> is a way to add a line-break.
you probably want to escape linebreaks that occur in the values you use, like this:
<xsl:value-of select="replace($foo,'\n','\\n')"/>
and of course, you will quickly notice that if you open the file in Excel, it becomes read-only which is a bother when you try to generate the file again while it is still open. unlike most text editors which simply tell you the file has changed, and offer to reload it.
-
I realize this question is dated and answered already, but I hope this helps a future someone.
Related
I'm little new to Sublime, wanted to know if there is a way to change display in sublime-3.0 from plain text to source colors automatically (i.e. Sublime should keep source colring from where code was copied and pasted like- HTML or XML or Java code) without I changing it manually.
Thanks in Advance!
Sublime Text already does this for some languages, like XML.
It works when you have a blank document that is set to Plain Text format (i.e. you open a new tab), and paste something in whose first line can be identified to be a specific language, using regular expressions.
For XML, it looks for an XML prolog or an XML element with a namespace. Regex
For HTML, it looks for a HTML doctype.
It currently doesn't support Java - I guess it's not easy to come up with a regex that would match only the first line of a Java file and not a C# file, for example. If you do have some ideas, you can use https://packagecontrol.io/packages/PackageResourceViewer to edit the relevant .sublime-syntax (YAML) file and add a first_line_match in.
You may also find the following packages helpful:
https://packagecontrol.io/packages/AutoSetSyntax
https://packagecontrol.io/packages/ApplySyntax
Extra note: these "first line matches" also apply when opening files that aren't automatically matched to a syntax by the file's name/extension.
There might be a better way, but I've done it with the Package control ctrl+shift+p (Win, Linux) or cmd+shift+p (OS X). Search for Package Control: install Package, press Enter and then search whatever package you need.
After installing the SCSS package, I'd get the HTML colouring as well.
I have a CSV that is encoded in Unicode, however lacks a byte order mark at the start. As such Excel (2013) opens without encoding correctly (i think it assumes ASCII if no BOM specified...), meaning that certain characters are displayed incorectly.
From reading around i have read that a BOM of "\uFEFF" should be entered at the start of the CSV file. I have tried opening in txt editor and adding the characters e.g.
\uFEFF
r1test 1, r1text2, r1text3
r2test 1, r2text2, r2text3
However, this does not solve the problem - the characters "\uFEFF" show up on the first row when I open in excel, rather than it beign interpreted as a BOM. I am not sure what I am doing wrong, and the format of how the text should be specified such that it is interpreted as a BOM, rather than text in the the first of the data
I have only very limited experience using CSV, and only just heard of a BOM... and thus I could be implementing this completely wrong!
(for reference, i know that I could specify the encoding if i use the import data option within excel... however I really want to work out how to get it correctly specified in advance such that I can just open the csv... I have several thousand of these files that I am creating and exporting - once I know how to do this 'manually' [i.e. by adding some text at start of a the file], I can configure to automatically do in Python).
Thanks in advance
For someone else wanting to tell Excel to add a BOM: See if you can "Save as Unicode Text".
source
I am writing Excel file through perl code. When I insert data in XML file and view in any browser, I see correct data with special characters, but when I write the same data in Excel file, it is showing garbage characters.
For eg.:
(word from XML file on browser) Gràcia - (word from Excel file) Grà cia
I am using 'Spreadsheet::XLSX' for reading excel and 'Excel::Writer::XLSX' for writing excel.
Also need help in finding the encoding format of excel fields.
Do you have any idea? Thanks in advance.
This seems very much like UTF-8 to iso-8859-1 conversion going wrong - seems like a string that contains UTF-8, but is not marked as being UTF-8, is being passed to $worksheet->write(). Since http://metacpan.org/pod/Excel::Writer::XLSX#UNICODE-IN-EXCEL claims to handle unicode correctly, it seems to be a problem with your input string, not the write method itself.
As you don't post any code, and don't tell us where your strings come from, i can't tell why the strings aren't marked correctly.
You can probably get away with
Encode::_utf8_on($str)
before passing your strings to $worksheet->write(), but this might just as well break other things, if not all of your strings are really utf-8. Basically the answer is "get the utf-8 flag on your strings right when you read them".
I'm currently developing CSV export with XSLT. And CSV file will be used %99 percent with Excel in my case, so I have to consider Excel behavior.
My first problem was German special characters in csv. Even fact that CSV encoding is UTF8, Excel cannot open properly CSV file with UTF8. The special characters are getting weird symbols. I found a solution for this problem. I just added 3 additional bytes(EF BB BF - a.k.a BOM Header) beginning of content bytes. Because UTF8 BOM is way to say that 'hey dude, it is UTF8, open it properly' to Excel. Problem solved!
And my second problem was about separator. The default separator could be comma or semicolon depending on region. I think it is semicolon in Germany and comma in UK. So, in order to prevent this problem, I had to add the line in below:
<xsl:text>sep=;</xsl:text>
or
<xsl:text>sep=,</xsl:text>
(This separator was not implemented as hard-coded)
But my problem which I cannot find any solution is that if you add "sep=;" or "sep=," beginning of the file while the CSV file is being generated with UT8-BOM, the BOM doesn't help for showing special characters properly anymore! And I'm sure that BOM bytes are always in the beginning of byte array. This screen shot is from MS Excel in Mac OS X:
First 3 symbols belong to BOM header.
Have you ever had like this problem or do you have any suggestions? Thank you.
Edit:
I share the printscreens.
a. With BOM and <xsl:text>sep=;</xsl:text>
b. Just with BOM
The Java code:
// Write the bytes
ServletOutputStream out = resp.getOutputStream();
if(contentType.toString().equals("CSV")) {
// The additional bytes in below is prefix indicates that the content is in UTF-8.
out.write(239);
out.write(187);
out.write(191);
}
out.write(bytes); // Content bytes, in this case XSL
The XSL code:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:template match="/">
<xsl:text>sep=;</xsl:text>
<table>
...
</table>
</xsl:template>
You are right, there is no way in Excel 2007 to get it load both the encoding and the seperator correctly across different locales when someone double clicks a CSV file.
It seems like when you specify sep= after the BOM it forgets the BOM has told it that it is UTF-8.
You have to specify the BOM because in certain locales Excel does not detect the seperator. For instance in danish, the default seperator is ;. If you output tab or comma seperated text then it does not detect the seperator and in other locales if you seperate with semi-colon it doesn't load. You can test this by changing the locae format in windows settings - excel then picks this up.
From this question:
Is it possible to force Excel recognize UTF-8 CSV files automatically?
and the answers it seems the only way is to use UTF16 le encoding with BOM.
Note also that as per http://wiki.scn.sap.com/wiki/display/ABAP/CSV+tests+of+encoding+and+column+separator?original_fqdn=wiki.sdn.sap.com
it seems that if you use utf16-le with tab seperators then it works.
I've wondered if excel reads sep=; and then re-calls the method to get the CSV text and loses the BOM - I've tried giving incorrect text and I can't find any work around that tells excel to take both the sep and the encoding.
This is the result of my testing with Excel 2013.
If you're stuck with UTF-8, there is a workaround which consists of BOM + data + sep=;
Input (written with UTF8 encoding)
\ufeffSome;Header;Columns
Wîth;Fàncÿ;Stûff
sep=;
Output
|Some|Header|Columns|
|Wîth|Fàncÿ |Stûff |
|sep=| | |
The issue with solution is that while Excel interprets sep=; properly, it displays sep= (yes, it swallows the ;) in the first column of the last row.
However, if you can write the file as UTF16-LE, then there is an actual solution. Use the \t delimiter without specifying sep and Excel will play ball.
Input (written with UTF16-LE encoding)
\ufeffSome;Header;Columns
Wîth;Fàncÿ;Stûff
Output
|Some|Header|Columns|
|Wîth|Fàncÿ |Stûff |
I can't write comments yet, but I'd like to address #Pier-Luc Gendreau's solution. While it is possible to open it in European Excel (which by default uses ;as delimiter) and have full utf-16LE support, it is apparently not possible to use this technique when you specify sep=,.
The issue with solution is that while Excel interprets sep=; properly, it displays sep= (yes, it swallows the ;) in the first column of the last row.
For me it did not work if I specified a delimiter which wasn't the default one (;in my case) so I assume Excel did not interpret the last line correctly and swallowed the last delimiter because this is the default behavior.
Please correct me if I'm wrong
I'm trying to convert this delimited PDF to an excel (or some other delimited format). Using Adobe Acrobat 9, I attempt to save it and copy it) as Excel but it gives the error message "BAD PDF; error in processing fonts. [348]".
I'm open to any solution that will create a delimited file, ranging from using Adobe Acrobat, to programming to using other apps. The only limitation is that I don't have a budget to buy anything (such as Able2Extract).
The way I was able to export my images and fonts without buying any extra software to do the conversion was this way. go to Advanced, PDG Optimizer, select all of the options you want on the LEFT COLUMN and where it says MAKE COMPATIBLE WITH select Acrobat 8.0 and later, OK....you are in your road to success
Note: not really an answer, but some suggestions.
Sounds to me that Crystal Reports is not following the PDF spec close enough.
I'd make sure CR is fully updated/patched and try genning another file making sure that "tagging" is enabled - tagging defines the layout structure. I don't have a copy of CR handy, but you may have to define a distiller template to use so when you print to PDF you can select that job option.
You can also tell its a bad PDF by using Preflight in Acrobat, it says there is no tag structure and you can do it manually (draw boxes around each item...). Also that there is no language set, and it is somehow compatible with Acrobat 1.3? which isn't supported anymore and should be 4 at the lowest?
Once you have a "good" pdf can export to xml/word and import that to excel. Also, with Acrobat 8+ you can highlight using the select tool, right-click and choose Open As SpreadSheet. You might be able to get away with just highlighting the whole document -- though I'd hope the xml approach would be best.
Able2Extract does some OCRing and tricky fuzzy logic not only to define tags/layout so it is exportable, but also avoids any font, encoding, etc issue - at least to my knowledge.
In the rare case that you can't get a new file, then exporting to plain text/accessible seems to generate a nice flat text file. You could write a vbscript to parse it (adding your delimiter) and import that into excel.