I use Excel VBA with SQLite3 ODBC driver obtained from http://www.ch-werner.de/sqliteodbc/
Both are 32 bits version on a Windows 10 computer.
A simple Select query on a single table with 16 columns and 200 rows takes approximately 200ms per row (I timed it).
This is more than 30 seconds for a query that executes <30ms in SQLite3.
I tried several cursorType , LockType , CursorLocation with no success
I tried Excel 2010 and Excel 365.
Sub TestSqlite()
Dim cnn As ADODB.Connection
Dim rs As ADODB.Recordset
Set cnn = New ADODB.Connection
Set rs = New ADODB.Recordset
cnn.Open "DRIVER=SQLite3 ODBC Driver;Database=C:\Test.sqlite;"
rs.CursorLocation = adUseClient
rs.CursorType = adOpenStatic
rs.LockType = adLockReadOnly
rs.Open "SELECT * from SimpleTable limit 20;", cnn
rs.Close
Set rs = Nothing
cnn.Close
Set cnn = Nothing
End Sub
Is there any way to fix this ?
There is a known problem with this stupid (pardon my French) Microsoft Antimalware Scan Interface (AMSI) feature, which sounds like a complete BS to me, and which grossly degrades performance of VBA7 DLL calls (I tried a few suggested on the Internet workarounds, but nothing worked for me). While when using ADO you don't call a DLL directly, eventually it calls the SQLiteODBC driver, and AMSI is the most likely culprit (together with all Windows built-in and activated by default security features). I have conducted a few tests of my own last year, though in a slightly different environment. Even though I had disabled most security features, the result was pretty astonishing.
I've stumbled accross a problem I am unable to solve :(
Neither Google nor StackOverflow have given me any usable answers, so I'm turning to you.
The problem is this:
I've created a spreadsheet that loads data from an Access database stored on a network drive.
The data-loading part is only done once, i.e. when opening the file.
I open the connection like this:
Dim con As ADODB.Connection
Dim rs As New ADODB.Recordset
Dim sql As String
Set con = GetConString()
rs.Open "SELECT ID, somevalue FROM sometable", con
Where to connection string is something like this
"Provider=Microsoft.Jet.OLEDB.4.0;Data Source='" & (network path to file) & "'"
Then I dump the information in the spreadsheet and terminate the connection like this:
rs.Close
con.Close
However, when I try to hibernate the PC while this Excel spreadsheet is still open, I get an error message.
It translates to something along the lines of "Excel has prevented the computer from going to sleep".
This seems to happen only when using this constellation...
Does anyone have any idea on how to prevent this behavior?
I'd like my PC to go to sleep, when I tell it to - even though the Excel spreadsheet is still open.
Thank you very much :)
After your Close lines you could
Set rs = Nothing
Set con = Nothing
although I cannot guarantee that this is what is causing your computer to not hibernate - more information would be required.
Is it possible to make query like SELECT from VBA in Excel, so I can query a PostgreSQL DB from Excel?
If is possible please explain me how to connect to the database. I was looking in Google but found no results.
Create a table or view in PostgreSQL that describes the data you want.
Use an ODBC or ADO connection from VBA to connect to PostgreSQL. If using ODBC you'll need to create a DSN via odbcad32.exe then use the DSN in VB, it isn't easy to just connect directly.
See:
Using ADO in VBA to connect to PostgreSQL
PostgreSQL Query to Excel Sheet
http://jackdebear.blogspot.com.au/2011/11/connecting-to-postgres-from-excel.html
Enabling import/export flows between a remote postgres database and excel workbooks
Does ADO work with ODBC drivers or only OLE DB providers?
How to put query results into a datatable with Excel VBA and ADO?
Better written eample that uses Oracle, but the principles are the same - ODBC/ADO.
Here's some code can use as reference. Hope it helps.
Sub SelectBasic()
Dim objDb_con
Dim strSomeValue As String
Set objDb_con = CreateObject("ADODB.Connection")
Set Rsdatatype = CreateObject("ADODB.RecordSet")
glbConnString = Trim(ActiveSheet.Range("B1").Value)
//Connection string format:Driver={PostgreSQL Unicode};Database=MyDB;server=192.16*.*.**;UID=USERID;Pwd=pasword //comment it
If glbConnString = "" Then
MsgBox "Enter the Connection String"
Else:
objDb_con.Open glbConnString
strSql = "select strSomeValue from SOMETABLE where Something=1"
Rsdatatype.Open strSql, objDb_con, adOpenKeyset, adLockpessimistic
If Rsdatatype.EOF = False Then strSomeValue = Rsdatatype.Fields(0).Value
Rsdatatype.Close
End If
objDb_con.Close
End Sub
Even for 64-bit Windows, Excel VBA needs the 32-bit ODBC driver.
Create a DSN via %windir%\SysWOW64\odbcad32.exe. Indeed, typing odbcad32.exe points towards the 64-bit version where you can't find the proper 32-bit drivers by default.
Source: https://github.com/windweller/postgresql-excel-addIn
I found this code online to query Access and input the data into excel (2003), but it is much slower than it should be:
Sub DataPull(SQLQuery, CellPaste)
Dim Con As New ADODB.Connection
Dim RST As New ADODB.Recordset
Dim DBlocation As String, DBName As String
Dim ContractingQuery As String
If SQLQuery = "" Then
Else
DBName = Range("DBName")
If Right(DBName, 4) <> ".mdb" Then DBName = DBName + ".mdb"
DBlocation = ActiveWorkbook.Path
If Right(DBlocation, 1) <> "\" Then DBlocation = DBlocation + "\"
Con.ConnectionString = DBlocation + DBName
Con.Provider = "Microsoft.Jet.OLEDB.4.0"
Con.Open
Set RST = Con.Execute(SQLQuery)
Range(CellPaste).CopyFromRecordset RST
Con.Close
End If
End Sub
The problem is that this code takes very long. If I open up Access and just run the query in there it takes about 1/10th the time. Is there anyway to speed this up? Or any reason this might be taking so long? All my queries are simple select queries with simple where statements and no joins. Even a select * from [test] query takes much longer than it should.
EDIT: I should specify that the line
Range(CellPaste).CopyFromRecordset RST
was the one taking a long time.
I'm no expert, but I run almost exactly the same code with good results. One difference is that I use the Command object as well as the Connection object. Where you
Set RST = Con.Execute(SQLQuery)
I
Dim cmd As ADODB.Command
Set cmd.ActiveConnection = con
cmd.CommandText = SQLQuery
Set RST = cmd.Execute
I don't know if or why that might help, but maybe it will? :-)
I don't think you are comparing like-with-like.
In Access, when you view a Query's dataview what happens is:
an existing open connection is used
(and kept open);
a recordset is partially filled
with the first few rows only (and
kept open);
the partial resultset is shown in a
grid dedicated to the task and
optimized for the native data access
method Access employs (direct use of
the Access Database Engine DLLs,
probably).
In your VBA code:
a new connection is opened (then
later closed and released);
the recordset is fully populated
using all rows (then later closed and
released);
the entire resultset is read into a
Excel's generic UI using non-native
data access components.
I think the most significant point there is that the dataview in Access doesn't fetch the entire resultset until you ask it to, usually by navigating to the last row in the resultset. ADO will always fetch all rows in the resultset.
Second most significant would be the time taken to read the fetched rows (assuming a full resultset) into the UI element and the fact Excel's isn't optimized for the job.
Opening, closing and releasing connections and recordsets should be insignificant but are still a factor.
I think you need to do some timings on each step of the process to find the bottleneck. When comparing to Access, ensure you are getting a full resultset e.g. check the number of rows returned.
Since you're using Access 2003, use DAO instead, it will be faster with the Jet engine.
See http://www.erlandsendata.no/english/index.php?d=envbadacexportdao for sample code.
Note that you should never use the "As New" keyword, as it will lead to unexpected results.
I would recommend you to create the Recordset explicitly rather than implicitly using the
Execute method.
When creating explicitly you can set its CursorType and LockType properties which have impact on performance.
From what I see, you're loading data in Excel, then closing the recordset. You don't need to update, count records, etc... So my advice would be to create a Recordset with CursorType = adOpenForwardOnly & LockType = adLockReadOnly:
...
RST.Open SQLQuery, Con, adOpenForwardOnly, adLockReadOnly
Range(CellPaste).CopyFromRecordset RST
...
Recordset Object (ADO)
I used your code and pulled in a table of 38 columns and 63780 rows in less than 7 seconds - about what I'd expect - and smaller recordsets completed almost instantaneously.
Is this the kind of performance you are experiencing? If so, it is consistent with what I'd expect with an ADO connection from Excel to an MDB back-end.
If you are seeing much slower performance than this then there must be some local environment conditions that are affecting things.
Lots of formulas may reference the query. Try temporarially turning on manual calculate in the macro and turning it off when all of your queries are done updating.
This should speed it up a bit, but still doesn't fix the underlying problem.
If you retrieve a lot of records, it would explain why the Range(CellPaste)takes so long. (If you execute the query in Access it wouldn't retrieve all the records, but if you do the CopyFromRecordset it requires all the records.)
There is a MaxRows parameter for CopyFromRecordset:
Public Function CopyFromRecordset ( _
Data As Object, _
<OptionalAttribute> MaxRows As Object, _
<OptionalAttribute> MaxColumns As Object _
) As Integer
Try if settings this to a low value (like 10 or so) changes the performance.
What about the following turnarounds or improvements:
Once opened, save the recordset as xml file (rst.saveToFile xxx) and then have Excel reopen it.
Once opened, put recordset data in an array (rst.getRows xxx), and copy the array on the active sheet
And, at any time, minimise all memory / access requirements: open the recordset as read-only, forward only, close the connection once the data is on your side, etc.
I don't know if it will help, but I am using VBA and ADO to connect to an Excel spreadsheet.
It was retrieving records lightning-fast (<5 seconds), but then all of a sudden it was awfully slow (15 seconds to retrieve one record). This is what lead me to your post.
I realized I accidentally had the Excel file open myself (I had been editing it).
Once I closed it, all was lightening fast again.
The problem 9 times out of 10 is to do with the Cursor Type/Location you are using.
Using dynamic cursors over network connections can slow down the retrieval of data, even if the query executed very fast.
IF you want to get large amounts of data very quickly, you'll need to use CursorLocation = adUseClient on your connection. This mean's you'll only have a static local cursor, so you won't get live updated from other users.
However - if you are only reading data, you'll save ADO going back to the DB for each individual record to check for changes.
I recently changed this as I had a simple loop, populating a list item, and each loop was taking around 0.3s. Not to slow, but even on 1,000 records thats 30 seconds! Changing only the cursor location let the entire process complete in under 1 second.
I am reading data from an Excel 2007 spreadsheet using ADO. Setting up the connection is easy:
Dim ado As ADODB.Connection
Set ado = CreateObject("ADODB.Connection")
ado.ConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=myFilename.xlsx;Extended Properties=""Excel 12.0 Xml;HDR=NO;IMEX=1"";"
ado.Open
I can call ado.OpenSchema without any trouble on this object. However, when I try to query the data:
Dim rs As ADODB.recordSet
Set rs = ado.Execute("SELECT * FROM [Current Work Load$]")
I simply get a table full of Nulls.
This is mentioned as an issue on the Microsoft Support site - but I have explicitly enabled "Import Mode" (as you can see in the code above - IMEX=1).
The Execute method does not return any records as it is for action queries.
Your might want to try the OpenRecordset method.
Dim rs As ADODB.recordSet
Set rs = ado.OpenRecordset("SELECT * FROM [Current Work Load$]")
I've found the ADO connection strings here are unbelievably picky. I've gotten reading the spreadsheets to work but with a slightly different connection string:
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + fileName + #";Extended Properties="Excel 12.0;IMEX=1";
(I don't have the XML after the Excel 12.0 declaration).
SpreadsheetGear for .NET can read Excel workbooks and enables you to access any cells without the kinds of issues / limatations you can run into with ADO.
You can see live C# & VB samples here and download the free trial here.
Disclaimer: I own SpreadsheetGear LLC
As well as using IMEX=1 in the connection string, you need to review a couple of registry keys. For more details, see this answer on SO.