When I open a csv file in excel, it break up the lines longer than 32760 characters, put the rest on the next line, and delete at least 2 characters in the process.
There is no special characters at the break up place.
Is it a normal behaviour? Where does it come from? Can I change that in any way?
Thanks for help.
Resolve the problem of cell length using the right tool for the job.
Excel has limit for cell length as you described, but it is OK, because Excel has never supposed to be a CSV editor.
I strongly recommend to never use Excel for CSV files. Reason:
It will change CSV values like 0053 into 53 without warning you. This can have serious impact on your data, because especially in ERP systems, values like 0053 are common and leading zeros are often meaningful. Or just think about ZIP codes: You can find ZIP 040 12 converted by Excel into 4012 without warning you. Excel also introduces other unexpected behaviors (from CSV viewpoint) like the one you described in the question.
Use some dedicated CSV editors instead. Some of them are freeware. This will also resolve your problem with cell length and also other potential problems like problems with quotes, with line breaks inside the cells, with character encoding, with column headers etc.
Related
I have a medium-sized tab-delimited .txt file - about 40k lines. When I import to Openrefine, line 406 puts all the rest of the content - the whole 40,000 lines, into a single cell in column 13 of that line.
I've tried grep-serching the invisibles in two different text editors (Sublime Text 2 & TextWrangler), and everything looks like it should.
I've also tried using Excel to convert to csv, and that actually works, but:
it's an inelegant workaround,
it has trouble with diacriticals, and
I don't want to spend ay more time resolving it in Excel anyway
I tried excepting the offending line with 10 lines on either side, and that throws the same problem.
Here are those 21 lines, copied directly from TextWrangler. (I can copy from Terminal output if that makes any difference.)
Any help, as always, is very much appreciated!!
Solved It! Well, sort of. It turns out that Column 13 had text that included double-quotes within the text itself (In other words, not having to do with delimiters at all).
For now, I'm just going to remove those quotes in the entire file, which does work - I tested it. **I'd rather figure out how to keep the quotes as part of the text. Tried escaping them with /, but that didn't work.
Thanks SO Community. Especially #Ettore.
I see. The problem is related to the quotations marks. Try importing your file by unchecking "Quotation marks are used to enclose cells containing column separators".
The empty columns in my screenshot are due to the fact that your file sometimes has two or three tabs as a separator. You can delete them easily after import using "reorder / remove columns"
I am running Windows 7 and have MS Office installed. Any time I download a .csv file the "file type" line in the "save as..." dialog defaults to "Microsoft Office Excel comma separated values file". Is there actually a Microsoft specific format that is distinct from "plain" .csv?
Googling the relevant terms returns various incredibly uninformative pages such as this one. Is any information lost or gained, or anything encoded differently by using this format than by just treating a file as a .csv, conforming to the general standards?
Yes, there are almost certainly differences.
From the top of my head: English Excel uses "," as a seperator. German locale uses ";" as a seperator, requiring an additional importing step if you want to import a csv with a comma seperator. This is not unique to german locales, roughly 1/4 to 1/3 of the world uses ";".
Also, there might be differences in how complicated strings are escaped (; and " in texts) which are probably different from program to program.
This is not excels fault, since the csv "format" is not really standardised and there are uncountable numbers of programs which are rolling their own csv parser, which leads to all sorts of problems because they forgot to handle corner cases.
I once read the comment that csv is the plague of data exchange formats because it is so difficult to do right. I could not agree more, I have to deal with them on a daily basis and they are extremly annoying to work with.
Open source fans will hate me for this, but I think csv is a poor choice for data exchange, even xlsx is better because it has rules which are well defined.
There are two things going on. The abbreviation (and suffix) "CSV" can mean character-separated values or it can mean comma-separated values. "Microsoft Office Excel comma separated values file" is a disambiguation, and means that you have a number of values in a record, with the field values separated by a comma.
The values themselves, in comma-separated value files, may contain commas if they are properly stropped (quoted). Usually, the stropping is putting a double quote around some or all of the field.
MS Excel also supports newlines in the middle of fields, again being properly stropped.
The format of our member numbers has changed several times over the years, such that 00008, 9538, 746, 0746, 00746, 100125, and various other permutations are valid, unique and need to be retained. Exporting from our database into the custom Excel template needed for a mass update strips the leading zeros, such that 00746 and 0746 are all truncated to 746.
Inserting the apostrophe trick, or formatting as text, does not work in our case, since the data seems to be already altered by the time we open it in Excel. Formatting as zip won't work since we have valid numbers less than five digits in length that cannot have zeros added to them. And I am not having any luck with "custom" formatting as that seems to require either adding the same number of leading zeros to a number, or adding enough zeros to every number to make them all the same length.
Any clues? I wish there was some way to set Excel to just take what it's given and leave it alone, but that does not seem to be the case! I would appreciate any suggestions or advice. Thank you all very much in advance!
UPDATE - thanks everybody for your help! Here are some more specifics. We are using a 3rd party membership management app -- we cannot access the database directly, we need to use their "query builder" tool to get the data we want to mass update. Then we export using their "template" format, which is called XLSX but there must be something going on behind the scenes, because if we try to import a regular old Excel, we get an error. Only their template works.
The data is formatted okay in the database, because all of the numbers show correctly in the web-based management tool. Also, if I export to CSV, save it as a .txt and import it into Excel, the numbers show fine.
What I have done is similar to ooo's explanation below -- I exported the template with the incorrect numbers, then exported as CSV/txt, and copied / pasted THOSE numbers into the template and re-imported. I did not get an error, which is something I guess, but I will not be able to find out if it was successful until after midnight! :-(
Assuming the data is not corrupt in the database, then try and export from the database to a csv or text file.
The following can then be done to ensure the import is formatted correctly
Text file with comma delimiter:
In Excel Data/From text and selected Delimited, then next
In step 3 of the import wizard. For each column/field you want as text, highlight the column and select Text
The data should then be placed as text and retain leading zeros.
Again, all of this assumes the database contains non-corrupt data and you are able to export a simple text or csv file. It also assumes you have Excel 2010 but it can be done with minor variation across all versions.
Hopefully, #ooo's answer works for you. I'm providing another answer mainly for informational purposes, and don't feel like dealing with the constraints on comments.
One thing to understand is that Excel is very aggressive about treating "numeric-looking" data as actual numbers. If you were to open the CSV by double-clicking and letting Excel do its thing (rather than using ooo's careful procedure), those numbers would still have come up as numbers (no leading zeros). As you've found, one way to counteract this is to append clearly nonnumeric characters onto your data (before Excel gets its grubby hands on it), to really convince Excel that what it's dealing with is text.
Now, if the thing that uploads to their software is a file ending in .xlsx, then most likely it is the current Excel format (a compressed XML document, used by Excel 2007 and later). I suppose by "regular old Excel" you mean .xls (which still works with the newer Excels in "compatibility mode").
So in case what you've tried so far doesn't work, there are still avenues to explore before resorting to appending characters to the end of your data. (I'll update this answer as needed.)
You're on the right track with the apostrophe.
You'll need to store your numbers in excel as text at the time they are added to the file.
What are you using to create the original excel file / export from database?
This will likely be where your focus needs to be regarding your export.
For example one approach is that you could potentially modify the database export to include the ' symbol prefix before the numbers so that excel will know to display them as text.
I use the formula =text(cell,"# of zeros of the field") to add preceding zeros.
Example, Cell C2 has 12345 and I need it to be 10 characters long. I would put =text(c2,"0000000000").
The result will be 0000012345.
I'm working on a .NET application which exports CSV files to open in Excel and I'm having a problem with preserving leading zeros when the file is opened in Excel. I've used the method mentioned at http://creativyst.com/Doc/Articles/CSV/CSV01.htm#CSVAndExcel
This works great until the user decides to save the CSV file within Excel. If the file is opened again in Excel then the leading zeros are lost.
Is there anything I can do when generating the CSV file to prevent this from happening.
This is not a CSV issue.
This is Excel loving to play with CSV files.
Change the extension to something else.
As #GSerg mentions, this is not a CSV issue.
If your users must edit/save in Excel they need to select the entire worksheet, right-click and choose "Format Cells" and from the Category list select "Text" after opening the csv file. This will preserve the leading zeros since the numbers will be treated as simple text.
Alternatively, you could use Open XML SDK 2.0, or some other Excel library, to create an xlsx file from your csv data and programmaticaly set the Cell type to Text in order to take the end users out of the equation...
I found a nice way around this, if you add a space anywhere along the phone number, the cell is then not treated as number and is treated as a text cell in both Excel and Apple's iWork Numbers.
It's the only solution I've found so far that plays nice with Numbers.
Yes I realise the number then has a space, but this is easy to process out of large chunks of data, you just have to select a column and remove all spaces.
Also, if this is web related, most web type things are ok with users entering a space in the number field. E.g you can tap-to-call on mobiles.
The challenge is to get the space in there in the first place.
In use:
01202123456 = 1202123456
but
01202 123456 = 01202 123456
Ok, new discovery.
Using Quick Preview on Mac to view a CSV file the telephone column will display perfectly, but opening the file fully with Numbers or Excel will ruin that column.
On some level Mac OS X is capable of handling that column correctly with no user meddling.
I am now working on the best/easiest way to make a website output a universally accepted CSV with telephone numbers preserved.
But maybe with that info someone else has an idea on how to make Numbers handle the file in the same way that Quick Preview does?
I want to generate a CSV file for user to use Excel to open it.
If I want to escape the comma in values, I can write it as "640,480".
If I want to keep the leading zeros, I can use ="001234".
But if I want to keep both comma and leading zeros in the value, writing as ="001,002" will be splitted as two columns. It seems no solution to express the correct data.
Is there any way to express 001, 002 in CSV for Excel?
Kent Fredric's answer contains the solution:
"=""001,002"""
(I'm bothering to post this as a separate answer because it's not clear from Kent's answer that it is a valid Excel solution.)
Put a prefix String on your data:
"N001,002","N002,003"
( As long as that prefix is not an E )
That notation ( In OpenOffice at least) above parses as a total of 2 columns with the N001,002 bytes correctly stored.
CSV Specification says that , is permitted inside quote strings.
Also, A warning from experience: make sure you do this with phone numbers too. Excel will otherwise interpret phone numbers as a floating point number and save them in scientific notation :/ , and 1.800E10 is not a really good phone number.
In OpenOffice, this RawCSV chunk also decodes as expected:
"=""001,002""","=""002,004"""
ie:
$rawdata = '001,002';
$equation = "=\"$rawdata\"";
$escaped = str_replace('"','""',$equation);
$csv_chunk = "\"$escaped\"" ;
Do
"""001,002"""
I found this out by typing "001,002" and then doing save-as CSV in Excel. If this isn't exactly what you want (you don't want quotes), this might be a good way for you to find what you want.
Another option might be use tab-delimited text, if this is an option for you.
A reader of my blog found a solution, ="001" & CHAR(44) & "002", it seems workable on my machine!
Pretty old thread but why don't you just add whitespace after your value. It will be then treated as string and no leading zeros will be stripped.
"001,002"." "
Since no-one mentioned it already, figured it was worth mentioning it in this old post.
If you add a horizontal tab character \t before the number, then MS Excel will also show the leading zero's. And the tab character doesn't show in the excel sheet. Even if it's surrounded by double-quotes. (F.e. \"\t001,002\")
It also looks nicer in Notepad++, compared to putting a \0 aka NULL before such number.
Looking more at the Excel spreadsheet it looks what you want can't be done using CSV.
This site http://office.microsoft.com/en-us/excel/HP052002731033.aspx says "If cells display formulas instead of formula values, the formulas are converted as text. All formatting, graphics, objects, and other worksheet contents are lost. The euro symbol will be converted to a question mark."
However, you can change how you load it to get the result you want. See this web page:
Microsoft import a text file.
The key thing is to choose Import External Data-Import Data-Text Files, go Next, Next, and then tick "Text" under column data format. This will prevent it being interpreted as a number, and losing formatting.
I was fiddling around with CSV to Excel (i use PHP to create the CSV, but i guess this solution works for any language. When you spot that a leading characters (such as + , - or 0 are disappearing, create the CSV with chr(13) as a prefix. This is a non printable character and it works wonders for my Excel Office 2010 version. I tried other non printable characters, but with no luck.
so i use Chirp Internet solution but tweaked with my prefix:
if (preg_match("/^0/", $str) || preg_match("/^\+?\d{8,}$/", $str) || preg_match("/^\d{4}.\d{1,2}.\d{1,2}/", $str)) {
$str = chr(13)."$str";
}
If you are using "Content-Disposition" and exporting from asp to excel using HTML tags,then you have to add "style='mso-number-format:\#;'" to that tag and making it to accept only Text values ,thereby leading zeroes omission will be avoided,If Forward slash"\" is accepted use double forward slash "\"
All the suggested answers don't seem to work for me right now ("=""blahblah""" and others) in all current Excel versions or Numbers app on OS X.
The only solution I found to be working by fiddling around is to add an escaped null character at the beginning of the string (which is \0 in PHP or C based languages). Everything ends up treated as is without being calculated or processed by the software when opening the calc sheet.
echo "\0" . $data;
Excel uses a default formatting for CSV columns depending on the content. So if you have 001 in a csv, excel will automatically turn it to 1.
The only way to keep the leading zeros in excel from a csv file is by changing the extension of the csv file to .txt, then just open excel, click on open, select the txt file, and you'll see the Text Import Wizard. Select your csv format (separated by commas), then just make sure you select "Text" as the format.
And that's it, now you can export that previous csv data to any other while keeping the leading zeros.
This is straightforward using Excel's Power Query functionality that allows you to perform step-by-step transformations.
Original File:
Add a Custom Column: