NPOI sometimes only reading 10 lines from a spreadsheet - excel

PROBLEM: I've hit a troubleshooting wall and hoping for suggestions on what to check to get past an issue I'm having with an internet site I'm working on. When reading data from a spreadsheet using NPOI (C#), sometimes (not all the time) the row reading stops after just ten rows.
Sorry for the very long post but not sure what is/isn't useful. Primary reason for posting here is that I don't know the right question to ask the Great Google Machine.
I have an intranet site where I'm reading an XLSX file and pushing its contents into an Oracle table. As you can tell by the subject line, I'm using NPOI. For the most part, it's just working, but only sometimes...
In Oracle, I have a staging table, which is truncated and is supposed to be filled with data from the spreadsheet.
In my app (ASPX), users upload their spreadsheet to the server (this just works), then the app calls a WebMethod that truncates data from the Oracle staging table (this just works), then another WebMethod is called that is supposed to read data from the spreadsheet and load the staging table (this, kinda works).
It's this "kinda works" piece is what I need help with.
The spreadsheet has 170 data rows. When I run the app in VS, it reads/writes all 170 records most of the time but sometimes it reads just 10 records. When I run the app from the web server, the first time it fails (haven't been able to catch a specific error), the second and subsequent times, it reads just ten records from the spreadsheet and successfully loads all ten. I've checked the file uploaded to the server and it does have 170 data records.
Whether the process reads 10 records or 170 records, there are no error messages and no indication why it stopped reading after just ten. (I'll mention here that the file today has 170 but tomorrow could have 180 or 162, so it's not fixed).
So, I've described what it's supposed to do and what it's actually doing. I think it's time for code snippet.
/* snowSource below is the path/filename assembled separately */
/* SnowExcelFormat below is a class that basically maps row data with a specific data class */
IWorkbook workbook;
try
{
using (FileStream file = new FileStream(snowSource, FileMode.Open, FileAccess.Read, FileShare.Read))
{
workbook = WorkbookFactory.Create(file);
}
var importer = new Mapper(workbook);
var items = importer.Take<SnowExcelFormat>(0);
/* at this point, item should have 170 rows but sometimes it contains only 10 with no indication why */
/* I don't see anything in the workbook or importer objects that sheds any light on what's happening. */
Again, this works perfectly fine most of the time when running from VS. That tells me this is workable code. When running this on the web server, it fails the first time I try the process but subsequently it runs but only picking up that first 10 records, ignoring the rest. Also, all the data that's read (10 or 170) is successfully inserted into the staging table, which tells me that Oracle is perfectly okay with the data, its format, and this process. All I need is to figure out why my code doesn't read all the data from Excel.
I have verified numerous times that the local DLL and webserver DLL are the same. And I'm reading the same Excel file.
I'm hitting a serious wall here and have run out of ideas on how to troubleshoot where the code is failing, when it fails. I don't know if there's something limiting memory available to the FileStream object causing it to stop reading the file prematurely - and didn't run across anything that looked like a resource limiter. I don't know if there's something limiting the number of rows pulled by the importer.Take method. Any suggestions would be appreciated.

I faced the same issue on some files and after analyzing the problem, this is what worked for me.
importer.Take<SnowExcelFormat>(0) have three parameters and one of them is maxErrorRows. Its default value is 10.
Your parsing stop when facing more than 10 errors then the function stop reading.
What you have to do is to set maxErrorRows instead of taking default value that is 10.

Related

Excel VBA - Upload into MS Access - Error 3051

Background:
I have an Excel file that is a workflow tool for one of our teams. One of the key processes is importing data from csv files and then uploading into MS Access. This part of the process works fine every time, no errors.
I added a new process to the tool, similar to the one mentioned above, this process scans data from one sheet and if a row meets a criteria it is copied into another sheet. Once the analysis is complete it gets uploaded into MS Access using the following code:
objAccess.DoCmd.TransferSpreadsheet acImport, acSpreadsheetTypeExcel9, "tb_age_extract", Application.ActiveWorkbook.FullName, True, "AGEUPLOADDATA"
The strange thing is this will run ok for me but when the users try and run it they get a runtime error 3051, the full error message is:
The Microsoft Access database engine cannot open or write to the file '\Folder1\Folder 2\FileName.xlsm'. It is already opened exclusively by another user, or you need permission to view and write its data.
I ran a test using a 50 count loop, all 50 runs ran ok with no errors, then the user tries to run it and they get the error. I have also experienced the error as well on my machine.
What is puzzling is the error is sporadic.
Has anyone experienced something similar with a 3051 error?

64-bit Excel 365 crashes, 32-bit Excel 365 works fine

I have a question relating to the differences between the 32 bit and 64 bit versions of Excel and VBA.
I am working in Office 365, 64 bit. (I am using 64 bit because I ran into memory issues and as a test, to see if 64 bit works for our company). I am working on an Excel file which has a lot of VBA code used to gather product properties from SQL, calculate parameters from these properties, and output everything to another SQL database. In addition, the Excel file outputs PDF files.
The Excel file is stored as a template, so that users can open the template and save the new file to their respective workspace. At the moment, the users are using Office 365, 32 bit.
When I open the Excel template, Excel closes itself. It seems to crash before calculating the parameters on the new file. There is no error message; in the lower-right message box I can see that Excel doesn't get to the calculating stage.
However, when the other users open the file in 32-bit Excel, everything is fine.
I am not using pointers (no PtrSafe) and no LongLong variables.
I also have no compile errors.
I am currently rebuilding the Excel file by saving it as a file without macro's, and then copying the VBA modules - a different site mentioned the VBA code may be corrupt. I am halfway through adding macro's, and the error pops up again; the VBA code I have added last has been in use for about a year so I am tempted to exclude it as a culprit.
Where would you suggest I look to find the problem?
You're absolutely right of course. I have gone back to the last working version and began adding a text logfile, which would log the start and end of each procedure. I got lucky as the file crashed as I was doing this, and the logfile pointed me towards a procedure seemed to be working, until the right conditions were met for it to crash.
The procedure itself accesses global variables which were not initialised. I didn't catch errors that could occur properly. For some reason, this works fine in x86 Excel, doesn't trigger any problems in the compiler or during debugging but crashes in x64 Excel when enough memory is used.
Gekozen_Assen is the name of a global variable. It is a dictionary which contains information. I had tried to use .Exists as a method to test if the required item in the dictionary exists, but this doesn't work if the dictionary isn't initialized at all.
The errorous code:
GetFromGlobal = ""
If Global_Dictionary.Exists(Answer_Type) Then
GetFromGlobal = Global_Dictionary.Item(Answer_Type)
Else
GetFromGlobal = "Error!"
End If
Instead I should've been using "Is Nothing", the fixed code below:
If Global_Dictionary Is Nothing Then
GetFromGlobal = "Error: Dictionary not initialized"
Else
If Global_Dictionary.Exists(Answer_Type) Then
GetFromGlobal = Global_Dictionary.Item(Answer_Type)
Else
GetFromGlobal = "Error: Dictionary item not found"
End If
End If

Update ListObject in Excel --> Error Access Database Blocked

I have a Access Database running as a normal file on a shared drive. Some tables from this database are available to users to read through Excel Files with ListsObjects.
These ListObjects need to be refreshed regularly to be up-to-date. Currently I am doing that manually.
I want to do this automatically by running a VBA code in the access database to open each file and in each files all the ListObjects to refresh them.
However, if I run the ListObject.refresh() method I get an error message that the database is blocked. If I do the refresh manually it works perfectly. In both cases I have the database open, so that can't be the problem.
This is the error message:
The database has been placed in a state by an user ("here is my user ID") that
prevents it from being opened or locked. (Error 3733)
Thank you for any suggestions.
Here is the part of the code:
Set ExlFile = app.Workbooks.Add(myWB)
For Each mySheet In ExlFile.Worksheets
For Each myLO In mySheet.ListObjects
myLO.Refresh
Next
Next
UPDATE:
If I use the function compact and repair on the database, it works for 3-5 files and then it stops working again.
Does anyone know what Application.CompactRepair does? I have a very bad solution for my problem now, I added Application.CompactRepair to each iteration. Now it works but of course it takes very long. It seems to me Application.CompactRepair somehow resets some status in the DB and it would be a better solution to do that directly.

pyodbc fetchall() returns no results when a column returned by the query contains too much data

Setup:I am using Python 3.3 on a Windows 2012 client.
I have a select query running using pyodbc which is not returning any results via fetchall(). I know the query works fine because i can take it out and run it from Microsoft SQL Management Studio without any issues.
I can also remove one column from the select list and the query will return results. For the database row in question, this column contains a large amount of XML data (> 10,000 characters), so it seems as though there is some buffer overflow issue going on causing fetchall() to fail, though it doesn't throw any exceptions. I have tried googling around and i have seen rumors of a config option to raise the buffer size, but i haven't been able to nail down exactly how to do it, or what a workaround would be.
Is there a configuration option that I can use, or any alternative to pyodbc.
Disclaimer: I have only been using python for about 2 weeks now so i
am still quite the noob, though i have made every attempt to research
my problems thoroughly this one has proven to be elusive:
On a side note, i tried using odbc instead of pyodbc but the same query throws this oddball error which google isn't helping me solve either
[ERROR] An exception while executing the Select query: [][Negative size passed to PyBytes_FromStringAndSize]
It seems this issue was resolved by changing my SQL connection string
FROM:
DRIVER={SQL Server Native Client 11.0}
TO:
DRIVER={SQL Server}

Problems came up in the following areas during load: Table

I have generated an excel file from xml. But i can not open it with Excel. Excel gives the following error opening it:
Problems came up in the following areas during load:
Table
Then it shows a message that the log file corresponding the error can be found at : C:/Documents and Setting/myUserName/Local Settings/Temporary Internet Files/Content.MSO/xxxxx.log
But i can not find Content.MSO folder in my windows. I checked folder settings and made all folders visible but i still can not access this folder. So that i can not analise the log file.
how could i find the generated log file?
I found the problem without analising the log file. i stil can not access the log file in temporary internet files. But i realised that i put a string(non-number) characters on a number-styled cell in Excel xml. So if you having the similar issues about your Excel file generated from xml, then have a look at if your cell values are appopriate with your cell data type.
If you type or paste the path of the log file into Explorer or your text editor of choice, you may find that the folder does exist, despite being invisible.
In my case it was a <Row> with an incorrect ss:Index
I was using a template and the last row had a fixed Index=100. If the number of rows I added exceeded 100, this last row had a wrong index and excel threw the error without any other message or log (MacOSX, Excel 15.25.1). I wish they printed more informative error messages, what a waste of our time.
Excel 2016. My error message was "Worksheet Settings". Path was pointing to non-existing file.
My cause of the problem was ExpandedRowCount not big enough for number of rows in Worksheet. If you add rows in XML directly (i.e. on a machine where Excel is not installed), make sure to increment number of rows in ExpandedRowCount.
yes.Even i too faced the same problem and problem was with the data type of cells ofexcel generated using xslt
In addition to checking the data being used vs "Type" assigned, make sure that the list of characters that need to be encoded for XML are indeed encoded.
I had a system that appeared to be working, but then some user data including & and < was throwing this error.
If you're not sure what's going on with your file, try http://www.xmlvalidation.com/ - that helped be spot the issue in a large file immediately.
I used this function to fix it, modified from this post:
function xmlsafe($s) {
return str_replace(array('&','>','<','"'), array('&','>','<','"'), $s);
}
and then run echo xmlsafe($myvalue) where you were just echoing $myvalue in your script.
This seems to be more appropriate for XML than htmlentities() or other options built into PHP.
I had the same issue, and the answer was - type of Cell was Number and some values doesn't converts to this type on my backend.
I had the SAME problem,
and its because de file is TOO BIG.
I try an extract from SAP, more little than the one with that make the error) and save it in XML file. and it WORK, no more error.
so maybe if you can save in 2 Excel files XML instead of 1 it will be good ;)
ALicia

Resources