Writing to SQL server - too slow - excel

I have an Excel sheet (using Excel version 1902) where i use VBA and an ADODB.connection (using Microsoft ActiveX Data objects 2.8 library) to read/write to an sql server (2016).
At first both the read/write operations were very slow, taking approximately 15 seconds each. There will be multiple read/write sessions so 15 seconds is not acceptable.
The amount of data to be read/written varies but is approximately 500 rows in 15 columns in total (split between 7 tables, so the number of rows and columns varies per table). In other words, the amount of data to be transferred is not massive.
At first I thought the problems was in my VBA code (loops, searching for text etc). But by removing those steps, I narrowed the problem down to moving through the recordset (.movenext row by row to read or write to the database).
For the read operation I managed to get acceptable speeds by doing the following:
Changed the CursorType from adOpenKeyset to adOpenForwardOnly
Changed the LockType from adLockOptimistic to adLockReadOnly
This reduced read times from 15 seconds to 5 seconds, which is acceptable.
However, I have not managed to achieve any improved speeds for the write operations which are still at 15 seconds.
I first tried:
Changed the CursorType from adOpenKeyset to adOpenStatic
Changed the LockType from adLockOptimistic to adLockBatchOptimistic
And then altered the .update command to .updatebatch.
I thought maybe that updating all in a batch would speed things up, but that did nothing.
Then I tried changing the connection .open statement from:
cn.Open "Provider = sqloledb;" & _
"Data Source=datasourcename;" & _
"Initial Catalog=catalogname;" & _
"User ID=UserIdname;"
etc. to:
cn.Open "Driver={SQL Server Native Client 11.0};" & _
"Server=servername;" & _
"Database=databasename;" & _
"Uid=UserIDname;"
Again that did nothing. With the updatebatch the native sql server connection performed worse (28 seconds) than the oledb connection. But with .update (no batch update) both connections were similar in performance at about 15 seconds.
Does anyone have any tips on how to possibly speed up the write operation ?

Related

Using ADODB, SQL on recordset in VBA, strings are truncated [to 255]. How to resolve this in VBA?

I'm trying to import the data from CSV file using VBA. Here i'm trying retrieve the data without opening the .csv file
Connection used
con.Open ("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & short_path & ";" & _
"Extended Properties=""text; HDR=Yes;ReadOnly=true; FMT=Delimited; IMEX=1;ImportMixedTypes=Text;""")
and for that sql used
str_sql = "Select * from [" & str_filename & "]"
rst.Open str_sql, con, adOpenStatic, adLockOptimistic, adCmdText
OLEDB is trying to read the data. So after pasting data from recordset to excel, comments i.e. Long strings are getting truncated. I believe by default it is scanning first 8 rows to determine the datatype.
This is working perfectly when I have the comments/long strings which are more than 255 characters in first 8 rows.
However, the problem comes when more than 255 characters comment/long string is on 9th or '+ rows. so all later comments are truncated to 255.
I'm wandering around the web to find out the solution but got Nowhere.
Note. If I had the small size CSV file then I can do the open/copy/paste/filter/calculation very easily by using VBA to open file and then do the rest.
However, In my scenario the files size is 100+ MB. So i cant open. tried but failed because it goes into non responding mode and crashes later.
so how to achieve this in VBA and Excel.

File size of an Excel Workbook increases every time I create a new pivot table and save the file

I have a vba macro that creates a pivot table based on some data in an input sheet. Every time the macro runs, the old pivot is deleted and a new one is created.
The problem I'm facing is that every time I save the file after running the macro, the file size increases by roughly 14MB.
This is how I delete the old pivot table:
For Each pivot In reportSht.PivotTables
pivot.TableRange2.Clear
Next pivot
My theory is that some part of the pivot isn't being deleted but I can't seem to put my finger on what.
I have found the solution to my problem. When I create the pivot tables I also add connections, since I need to display the number of unique entries in the pivot table:
ActiveWorkbook.Connections.Add2 "WorksheetConnection_" & inputDataArea, "", _
"WORKSHEET;" & ActiveWorkbook.Path & "\[" & ActiveWorkbook.name & "]" _
& inputSht.name, inputDataArea, 7, True, False
Where inputDataArea is a String with the range used by for the pivot table. My problem was, that I was not deleting these connections when I was deleting the pivot table. So a new connection was being added every time the macro was executed.
I added this piece of code to also remove any connections that are no longer needed after removing the pivot table:
Dim connection As Object
For Each connection In ActiveWorkbook.Connections
If connection.name <> "ThisWorkbookDataModel" Then connection.Delete
Next connection
The file is still large but manageable and most importantly it's not growing anymore.
Thanks to Pᴇʜ for suggesting that I remove pivot caches and pointing out, that these are deleted along with the connections.

VBA Delete Visible Rows Very Slow w/ large database

I am not clear as to why this Delete command is so slow in VBA, when the database is large. The database has about 80,000 records and the delete command bellow takes about 5 minutes to run.
'Deletes all Card interactions
ActiveSheet.Range("A1:H1" & LastRowEx).AutoFilter Field:=3, Criteria1:="*card*", Operator:=xlFilterValues
ActiveSheet.Range("A1:H1" & LastRowEx).Offset(1, 0).SpecialCells _(xlCellTypeVisible).EntireRow.Delete
ActiveSheet.ShowAllData
Any idea why this is so slow and how I can speed it up? I have to do this a few times for other fields in the database and it is extending the run time quit a bit. Thanks for the help!!

Excel Microsoft.ACE.OLEDB.12.0 The connection for viewing your linked Microsoft Excel worksheet was lost

I have a VBscript that connects to a excel file and queries the data using Microsoft.ACE.OLEDB.12.0. I'm just displaying the recordcount right now but when I have the document open, the count is wrong. When the excel document is closed, it is correct. I'm getting an error now "The connection for viewing your linked Microsoft Excel worksheet was lost" CODE: 80004005 Source:Microsoft Access Database Engine. I've used VBScript to pull this data many times before but never got this error.
Any ideas on what is causing this?
The connection string is this:
cnnExcel.Open "Provider=Microsoft.ACE.OLEDB.12.0;" & "Data Source=" & strExcelFilePath & ";" & "Extended Properties=" & Chr(34) & "Excel 12.0 Xml;Readonly=1;HDR=Yes;IMEX=1" & Chr(34) & ";"
Recordset is this:
rsExcel.open "SELECT * FROM [col$] WHERE [name] <>''", cnnExcel, adOpenStatic, adLockReadOnly
I solved this when it happened to me. My SQL was failing because source data had a record ID of 10054748 in a cell which was formatted as a date causing an overflow error which caused another non-erroring column to return a NULL, even though the error was not in one of the columns queried. The field's data was type text in any case, and the errant formatting caused by pasting a copied recordset from Toad and XL TextToColumns automatically parsing it based on a different data set parsed earlier in the day! The solution was changing that column's formatting back to General which eliminated the overflow errors. Summary: probably any errors in source data to be selected using SQL and Microsoft.ACE.OLEDB.12.0 can cause null values to be returned and unexpected failure of a where clause that references those null values, source data should be clean, formatting should be minimized and well understood!

Using ADO to query text files - terrible performance

I'm writing some code in VBA behind Excel to pull some summary numbers out of potentially huge text files (10M+ rows) out on a network drive. In the past, these numbers have been pulled using greps in linux, but I was hoping to implement something that could be done with a click of a button in Excel for ease of use.
My solution works, but it's like 25 times slower than a linux grep - takes 4 minutes to query 10M records, while the grep can do it in 10 seconds. Should I not be using ADO for this? Why is it so slow, aside from the fact that text files obviously aren't indexed? Is there a better solution that could still be coded without too much hassle in VBA, or is it a lost cause? I'm using Excel 2007 and the ADO 6.0 library. Here is some sample code:
Sub RunSQL()
Dim cn As New ADODB.Connection
Dim rs As New ADODB.Recordset
cn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
"Extended Properties=""text;HDR=YES;FMT=Delimited"";" & _
"Data Source=\\network\share\path\;"
rs.Open "select count(*) from Customers.tab where CHANGE_FLAG = 'Y'", cn
Range("A1").CopyFromRecordset rs
rs.Close
cn.Close
End Sub

Resources