If i loop through 10 documents and fill an excel sheet with data from that documents then sometimes the export is slow and sometimes it is fast.
How is this possible, this is how I export data:
Set objExcel = CreateObject("Excel.Application")
objExcel.Application.Visible = True
Call objExcel.Application.Workbooks.Open(CorDos.CorBestandsnaam)
Set xlSheet = objExcel.Application.ActiveWorkbook.Activesheet
and then it fills the cells..
The documents are in a database which is on another server, this server also sometimes has some issues with I/O could that also be the problem?
This is NOT a problem of Lotus Notes but a typical excel- automation problem. Just search for "excel vba slow", and you will find tons of articles how to make this fast again. The fastest way to write something to excel is: create a 2 dimensional array in LotusScript and assign the document values to that array. THEN write the whole array at once. This looks like this:
...
Set dc = db.UnprocessedDocuments
...
Redim varArray( dc.Count - 1, NumberOfFields ) as String
Set doc = dc.getFirstDocument()
While not doc is Nothing
varArray(i , 0 ) = doc.GetitemValue( "FirstField" )(0)
varArray(i , 1 ) = doc.GetitemValue( "SecondField" )(0)
varArray(i , 2 ) = doc.GetitemValue( "ThirdField" )(0)
....
i = i + 1
Set doc = dc.GetNextDocument(doc)
Wend
...
xlSheet.Range( xlSheet.Cells(1,1), xlSheet.Cells(dc.Count,NumberOfFields) ) = varArray
This code is not tested and partly taken from a response to this excel- vba- question, but should show you the way to go.
I like #Torsten-Link's answer, but it might not be the bottleneck you're experiencing. You might want to consider a pattern where the building of the excel file, or at least collating the data to be inserted into it, is done on the actual machine that has the data and then passes the information on in one terse communication. Much of the issues involved with speed are related to local processing and sifting of remote data.
Also, you really should set Visible to false, then make sure it turns true even if it errors. Having that Visible property to True really slows things down.
Related
Excel crashes, VBA raises Error 3218 “Could Not Update” Record Locking Errors when multiple users try to update same table in a shared MS-Access database using DAO.
I have a special configuration like this: a MS-Access database located in shared network folder, multiple user connect to update that database using VBA DAO build on Excel file. The VBA code in each Excel file is the same. The problem happens when there are 2 users click on update button at the same time. User Excel file turn hanging, or showing error 3218 "Could not update".
Sub ExportToAccess()
Dim oSelect As Range, i As Long, j As Integer, sPath As String
'tblSuppliers.Active
Set oSelect = Application.InputBox("Range", , Range("A1").CurrentRegion.Address, , , , , 8)
Dim oDAO As DAO.DBEngine, oDB As DAO.Database, oRS As DAO.Recordset
sPath = "\\sharedfolder\Database.accdb"
Set oDAO = New DAO.DBEngine
Set oDB = oDAO.OpenDatabase(sPath)
Set oRS = oDB.OpenRecordset("tblSuppliers")
For i = 2 To oSelect.Rows.Count 'skip label row
oRS.AddNew
For j = 1 To oSelect.Columns.Count 'Field(0) is Auto#
oRS.Fields(j) = oSelect.Cells(i, j)
Next j
oRS.Update
Next i
oDB.Close
MsgBox ("Updated Done!")
End Sub
I know my configuration is not good for database application, however I have to stick with this for a while. Could you please advise any solutions to avoid error when multiple users update Access database in this case ? Is there a way to detect if database is being updating by others and script to wait until that process to finish first. Any technical solution for this issue is welcome!
Thank you!
You need some type of flag to tell if anyone is updating the table or not. Examples of what this flag can be:
An Excel file cell (that is probably the easiest in your case; if multiple excel files are used, just link to the one cell)
A field in an Access table (even a table with a single field and a single record dedicated just for that)
A (text) file in your shared drive (the flag can be the content of the file or even whether the file exists or not)
Then your update process would be:
- Check the flag, if set, loop until flag is cleared
- Set the flag
- Update the table
- Clear the flag
You will probably also need some way for the users (or just you) to clear the flag manually, in case something else goes wrong while updating the table and the flag gets stuck raised.
Well , this is not probably the most elegant solution
but you may create a field in a table , and ask for it before working with the table
something like this :
Set LockedStatus= oDB.OpenRecordset("mycontroltable")
if LockedStatus("lockedSuppiers")=False then
oDB.Execute"update mycontroltable set lockedSuppiers=true"
Set oRS = oDB.OpenRecordset("tblSuppliers")
For i = 2 To oSelect.Rows.Count 'skip label row
oRS.AddNew
For j = 1 To oSelect.Columns.Count 'Field(0) is Auto#
oRS.Fields(j) = oSelect.Cells(i, j)
........
......
oDB.Execute"update mycontroltable set lockedSuppiers=false"
end if
I need little help again.
I have an excel file and I have to do some formatting for further processing.
I stuck at one point:
find a word / or string
get back the number how often it is used in the excel
use this number to put it into a loop
I hope this is possible with VBscript, because I cannot use VBA.
This is how the code looks like, but I doesn't work
Dim objExcel
Set Excel = CreateObject("Excel.Application")
Excel.Visible = True
excel.workbooks.open "C:\Users\............."
excel.Rows("1:34").Select
excel.Selection.Delete
excel.Columns("A:A").Select
excel.Selection.Delete
excel.Range("A1").Select
Number = excel.countIf "A:A", "Ent.Date"
for i = 1 to Number
excel.Cells.Find("Ent.Date").Activate
excel.Activecell.Offset(-1,0).Select
excel.Activecell.resize(RowSize +3).EntireRow.Select
excel.Selection.Delete
next
Please help me.
Thanks
You are close. CountIf() is a method of the WorksheetFunction class in your Excel application. So:
Number = Excel.WorksheetFunction.CountIf("A:A", "Ent.Date")
Unrelated. All of this .Active .Select is unnecessary. Humans Select and Activate, but your script doesn't need to.
Instead:
Set Excel = CreateObject("Excel.Application")
Excel.Visible = True
excel.workbooks.open "C:\Users\............."
excel.Rows("1:34").Delete
excel.Columns("A:A").Delete
Number = excel.WorksheetFunction.CountIf(Excel.Range("A:A"), "Ent.Date")
for i = 1 to Number
excel.Cells.Find("Ent.Date").Offset(-1,0).resize(RowSize +3).EntireRow.Delete
next
Also (and this might be more preference) instead of looping with For you could instead us a While loop:
Set Excel = CreateObject("Excel.Application")
Excel.Visible = True
excel.workbooks.open "C:\Users\............."
excel.Rows("1:34").Delete
excel.Columns("A:A").Delete
Do While excel.WorksheetFunction.CountIf(Excel.Range("A:A"), "Ent.Date") >= 1
excel.Cells.Find("Ent.Date").Offset(-1,0).resize(RowSize +3).EntireRow.Delete
Loop
That causes a little extra overhead since it runs that countif on each loop, but it feels more succinct and less potential for error since you seem to be resizing the range returned by Find() to be more rows and then deleting them all... which means you might be picking off rows that also contain your search criteria, which means you are looping too much in your For Loop. But... perhaps your sheet is set up in such a way that this condition doesn't happen.
I need to import data from SAS to excel via VBA. The import needs to run eg. on workbookOpen or Worksheet_BeforeDoubleClick or it can be called in any macro. This is solved in the below code:
Sub GetSASdata()
Dim obConnection As ADODB.Connection
Dim obRecordset As ADODB.Recordset
Dim i As Integer
Set obConnection = New ADODB.Connection
' Do not get stuck on the choice of connection provider.
obConnection.Provider = "sas.LocalProvider"
obConnection.Properties("Data Source") = "C:\path\"
obConnection.Open
Set obRecordset = New ADODB.Recordset
obRecordset.Open "MySAStable", obConnection, adOpenDynamic, adLockReadOnly, ADODB.adCmdTableDirect
'add header row
Cells(1, 1).Select
For i = 0 To obRecordset.Fields.Count - 1
ActiveCell.Offset(0, i).Value = obRecordset.Fields(i).Name
Next i
obRecordset.MoveFirst
obRecordset.Filter = "Weight > 0"
Cells(2, 1).Select
ActiveCell.CopyFromRecordset obRecordset, 100
obRecordset.Close
Set obRecordset = Nothing
obConnection.Close
Set obConnection = Nothing
End Sub
In this example I have restricted the output to be only the first 100 rows. However, the original data set is 1.4 m rows and 150 columns, and I want to be able to restrict the data import to only take columns that I define and rows which meet certain criteria. In sql terms:
select col1, col2, col10, col11 from MySAStable where code = MyCode and Date > MyDate
But I cannot find a way to do it. The first criteria is that the code should run entirely from Excel.
I have experimented some with obRecordset.Filter but the performance is poor. It takes forever. So idealy I would like to import only the data that I need. Is there a way to do this?
The
obConnection.Provider = "sas.LocalProvider"
is arbitrary. I found an example online, tested it and it worked. If someone has an answer to my problem that involves a different connection type, i am still interested to know. Very idealy the code can also be run by users who do not have SAS installed on their computer (but have access to the folder where data is placed.)
Thank you for any help
I have used two methods to read SAS data from within Excel.
The first uses SAS Add-In to MS Office. Do you have this product?
You can define the source with filters, and when the user opens the workbook, it will automatically refresh agains the datasource. You can also automate the refresh task with VBA code.
Secondly, I have done it with a Stored Process. If you have a stored process server, you can set up a web query in Excel and read the Stored Process that way, using any filter you need.
I am currently working on a Outlook 2010 VBA macro to pull information from a email messages and place it into an Excel file. The idea is that each email has the same fields in tables embedded in the email message every time (name, order number, date, etc.) and that data is put into a spreadsheet. To do this, I have currently used the following code to get the table and move it into Excel:
'This code is inside a for each loop for each message
Set excelWorksheet2 = excelWorkbook.Worksheets.item(2)
Set excelWorksheet3 = excelWorkbook.Worksheets.item(3)
Set excelWorksheet4 = excelWorkbook.Worksheets.Add(After:=excelWorkbook.Sheets(excelWorkbook.Sheets.count))
Dim mailObj As Outlook.MailItem
Dim doc As Word.Document
Set doc = mailObj.GetInspector.WordEditor
Dim table1, table2, table3 As Object
Set table3 = doc.Tables(4).Range
Set table2 = doc.Tables(3).Range
Set table1 = doc.Tables(2).Range
table1.Copy
excelWorksheet2.Paste
table2.Copy
excelWorksheet4.Paste
table3.Copy
excelWorksheet3.Paste
Set table1 = Nothing
Set table2 = Nothing
Set table3 = Nothing
'I do much more of this to get the data from each table and put it into a master worksheet...
excelWorksheet.Cells(rows, cols + 1).Value = excelWorksheet2.Cells(4, 2).Value 'Contract Number
excelWorksheet.Cells(rows, cols + 2).Value = excelWorksheet2.Cells(4, 4).Value 'Contractor Name
Set doc = Nothing
Set excelWorksheet2 = Nothing
Set excelWorksheet3 = Nothing
Set excelWorksheet4 = Nothing
I get the following errors every so often, but it doesn't occur on the same data, it is sort of random and seems to occur on the Outlook/email side only:
"The requested member of the collection does not exist." (Error code
5941) at the .Range line
"Method 'Copy' of object 'Range' failed" at the .Copy
line
Sometimes both of these errors occur if I step through, if the copy fails, the macro will crash.
I have tried:
Putting in 2 second delays
Go through fewer emails (this code usually fails when I select > 10
emails to process)
Clearing the clipboard after every email
Close/deallocate objects through Nothing (not sure if
this is the best practice as I'm more of a C/C++/C#/Java guy)
None of these seemed to remotely fix this issue as both errors pop up frequently, but intermittently.
I'm truly at a loss as to what the next step would be in debugging this issue, any help would be much appreciated!
Based on research I have been doing on the issue of the WordEditor tables, it seems as the amount of copy/paste operations and the searching HTML tables just simply do not allow for reliable behavior. One possible solution to this could be to copy the entire email into Excel and parse the tables that way (this still requires copy/paste).
What I did to solve this problem (indirectly) is to save all the emails as HTML files (through Outlook.Application running in Excel: email.SaveAs folderPath + fileName + ".html", olHTML) in a temporary directory and have Excel open the HTML files in a workbook and work with the data that way. This has been much more reliable (added overhead though) and allows for large volumes of emails to be exported to Excel properly.
If anyone wants the exact code to my problem, message me (vindansam at hotmail.com, it's a tad long and has some proprietary information in it).
It's too bad that the WordEditor seems to not handle the information well, maybe Microsoft will beef things up in the next version of Office.
Currently I have an application that takes information from a SQLite database and puts it to Excel. However, I'm having to take each DataRow, iterate through each item, and put each value into it's own cell and determine highlighting. What this is causing is 20 minutes to export a 9000 record file into Excel. I'm sure it can be done quicker than that. My thoughts are that I could use a data source to fill the Excel Range and then use the column headers and row numbers to format only those rows that need to be formatted. However, when I look online, no matter what I seem to type, it always shows examples of using Excel as a database, nothing about importing into excel. Unless I'm forgetting a key word or to. Now, this function has to be done in code as it's part of a bigger application. Otherwise I would just have Excel connect to the DB and pull the information itself. Unfortunately that's not the case. Any information that could assist me in quick loading an excel sheet would be appreciated. Thanks.Additional Information:Another reason why the pulling of the information from the DB has to be done in code is that not every computer this is loaded on will have Excel on it. The person using the application may simply be told to export the data and email it to their supervisor. The setup app includes the needed dlls for the application to make the proper format.Example Code (Current):
For Each strTemp In strColumns
excelRange = worksheet.Cells(1, nCounter)
excelRange.Select()
excelRange.Value2 = strTemp
excelRange.Interior.Color = System.Drawing.Color.Gray.ToArgb()
excelRange.BorderAround(Excel.XlLineStyle.xlContinuous, Excel.XlBorderWeight.xlThin, Excel.XlColorIndex.xlColorIndexAutomatic, Type.Missing)
nCounter += 1
Next
Now, this is only example code in terms of the iteration I'm doing. Where I'm really processing the information from the database I'm iterating through a dataTable's Rows, then iterating through the items in the dataRow and doing essentially the same as above; value by value, selecting the range and putting the value in the cell, formatting the cell if it's part of a report (not always gray), and moving onto the next set of data. What I'd like to do is put all of the data in the excel sheet (A2:??, not a row, but multiple rows) then iterate through the reports and format each row then. That way, the only time I iterate through all of the records is when every record is part of a report.
Ideal Code
excelRange = worksheet.Cells("A2", "P9000")
excelRange.DataSource = ds 'ds would be a queried dataSet, and I know there is no excelRange.DataSource.
'Iteration code to format cells
Update:
I know my examples were in VB, but it's because I was also trying to write a VB version of the application since my boss prefers VB. However, here's my final code using a Recordset. The ConvertToRecordset function was obtained from here.
private void CreatePartSheet(Excel.Worksheet excelWorksheet)
{
_dataFactory.RevertDatabase();
excelWorksheet.Name = "Part Sheet";
string[] strColumns = Constants.strExcelPartHeaders;
CreateSheetHeader(excelWorksheet, strColumns);
System.Drawing.Color clrPink = System.Drawing.Color.FromArgb(203, 192, 255);
System.Drawing.Color clrGreen = System.Drawing.Color.FromArgb(100, 225, 137);
string[] strValuesAndTitles = {/*...Column Names...*/};
List<string> lstColumns = strValuesAndTitles.ToList<string>();
System.Data.DataSet ds = _dataFactory.GetDataSet(Queries.strExport);
ADODB.Recordset rs = ConvertToRecordset(ds.Tables[0]);
excelRange = excelWorksheet.get_Range("A2", "ZZ" + rs.RecordCount.ToString());
excelRange.Cells.CopyFromRecordset(rs, rs.RecordCount, rs.Fields.Count);
int nFieldCount = rs.Fields.Count;
for (int nCounter = 0; nCounter < rs.RecordCount; nCounter++)
{
int nRowCounter = nCounter + 2;
List<ReportRecord> rrPartReports = _lstReports.FindAll(rr => rr.PartID == nCounter).ToList<ReportRecord>();
excelRange = (Excel.Range)excelWorksheet.get_Range("A" + nRowCounter.ToString(), "K" + nRowCounter.ToString());
excelRange.Select();
excelRange.NumberFormat = "#";
if (rrPartReports.Count > 0)
{
excelRange.Interior.Color = System.Drawing.Color.FromArgb(230, 216, 173).ToArgb(); //Light Blue
foreach (ReportRecord rr in rrPartReports)
{
if (lstColumns.Contains(rr.Title))
{
excelRange = (Excel.Range)excelWorksheet.Cells[nRowCounter, lstColumns.IndexOf(rr.Title) + 1];
excelRange.Interior.Color = rr.Description.ToUpper().Contains("TAG") ? clrGreen.ToArgb() : clrPink.ToArgb();
if (rr.Description.ToUpper().Contains("TAG"))
{
rs.Find("PART_ID=" + (nCounter + 1).ToString(), 0, ADODB.SearchDirectionEnum.adSearchForward, "");
excelRange.AddComment(Environment.UserName + ": " + _dataFactory.GetTaggedPartPrevValue(rs.Fields["POSITION"].Value.ToString(), rr.Title));
}
}
}
}
if (nRowCounter++ % 500 == 0)
{
progress.ProgressComplete = ((double)nRowCounter / (double)rs.RecordCount) * (double)100;
Notify();
}
}
rs.Close();
excelWorksheet.Columns.AutoFit();
progress.Message = "Done Exporting to Excel";
Notify();
_dataFactory.RestoreDatabase();
}
Can you use ODBC?
''http://www.ch-werner.de/sqliteodbc/
dbName = "c:\docs\test"
scn = "DRIVER=SQLite3 ODBC Driver;Database=" & dbName _
& ";LongNames=0;Timeout=1000;NoTXN=0;SyncPragma=NORMAL;StepAPI=0;"
Set cn = CreateObject("ADODB.Connection")
cn.Open scn
Set rs = CreateObject("ADODB.Recordset")
rs.Open "select * from test", cn
Worksheets("Sheet3").Cells(2, 1).CopyFromRecordset rs
BTW, Excel is quite happy with HTML and internal style sheets.
I have used the Excel XML file format in the past to write directly to an output file or stream. It may not be appropriate for your application, but writing XML is much faster and bypasses the overhead of interacting with the Excel Application. Check out this Introduction to Excel XML post.
Update:
There are also a number of libraries (free and commercial) which can make creating excel document easier for example excellibrary which doesn't support the new format yet. There are others mentioned in the answers to Create Excel (.XLS and .XLSX) file from C#
Excel has the facility to write all the data from a ADO or DAO recordset in a single operation using the CopyFromRecordset method.
Code snippet:
Sheets("Sheet1").Range("A1").CopyFromRecordset rst
I'd normally recommend using Excel to pull in the data from SQLite. Use Excel's "Other Data Sources". You could then choose your OLE DB provider, use a connection string, what-have-you.
It sounds, however, that the real value of your code is the formatting of the cells, rather than the transfer of data.
Perhaps refactor the process to:
have Excel import the data
use your code to open the Excel spreadsheet, and apply formatting
I'm not sure if that is an appropriate set of processes for you, but perhaps something to consider?
Try this out:
http://office.microsoft.com/en-au/excel-help/use-microsoft-query-to-retrieve-external-data-HA010099664.aspx
Perhaps post some code, and we might be able to track down any issues.
I'd consider this chain of events:
query the SQLite database for your dataset.
move the data out of ADO.NET objects, and into POCO objects. Stop using DataTables/Rows.
use For Each to insert into Excel.