I am using a set of scripts that pull a lot of different data from iSeries via ODBC.
ActiveWorksheets.RefreshAll does not work as it doesn't leave enough time to run the background queries
I've tried the below,but to no avail
Dim qry As Connections
'Set qry =
For Each qry In ActiveWorksheets.Connections
qry.BackgroundQuery = False
qry.RefreshAll
DoEvents
Next qry
This gives me the expected Error 424 Object expected.
I don't expect to use Set qry = here, as I need to run through 30 different connections
Let's just call them connection1, connection2 etc for now, as their names are all over the place
Is the simplest option to stop the background query, refresh, activate background query - before the data import, or is there a better way?
I've looked all over SO - but can't find info on multiple ODBC connections
EDIT:
Dim qry As WorkbookConnection
For Each qry In ActiveWorkbook.Connections
qry.Refresh
DoEvents
Next qry
I believe your
Dim qry As Connections
should read
Dim qry As WorkbookConnection
The ActiveWorksheets.Connections.Item property returns an object of type WorkbookConnection. If you are trying to refresh the connections one at a time as it seems from your For Each statement, that object represents a single connection with methods like Refresh rather than the collection of all connections.
I managed to work this out. So all who may need this in the future can see:
Dim qry As WorkbookConnection
For Each qry In ActiveWorkbook.Connections
qry.ODBCConnection.BackgroundQuery = False
qry.Refresh
qry.ODBCConnection.BackgroundQuery = True
Next qry
Although it doesn't look like BackgroundQuery = True/False doesn't look like it's vital here. Turning it off means when you qry.Refresh, it pulls the data and refreshes it.
Also using For Each qry means that rather than writing out out 20 times, I can just look every connection and turn them off, refresh, and turn them back on
Related
I get Query timeout expired when I run this code.
I have tried putting in timeout limits on the Conn and the Command from Query timeout expired when trying to run a short procedure but not successful.
You can see my comments in CAPS in the code for the timeout clauses. In SSMS it executes in about 45sec. My workbook has multiple queries using the same connection. This is the heaviest query (returning about 7k rows), so I'm pretty sure the problem is a query timeout, not connection.
Sub Units()
Application.ScreenUpdating = False
Dim rs4 As Object
Set rs4 = CreateObject("ADODB.Recordset") '04Unit2
Dim sqlstr04 As String
sqlstr04 = "select * from dbo.[04Units]"
Sheet17.Cells.Clear
Call connectDatabase
rs4.Open sqlstr04, DBCONT
DBCONT.commandtimeout = 120 'CONNECTION TIMEOUT
rs4.commandtimeout = 120 'RECORDSET QUERY TIMEOUT
'Debug.Print sqlstrledger03
For intColIndex = 0 To rs4.Fields.Count - 1
Sheet17.Range("A1").Offset(0, intColIndex).Value =
rs4.Fields(intColIndex).Name
Next
Sheet17.Range("A2").CopyFromRecordset rs4
End Sub
Error
DBCont is defined in a separate module:
The Open-method of your recordset already triggers the execution of the SQL-command. Your TimeOut-commands must be set before the command is executed.
Simply change the order of your code and set the Timeout before you execute the SQL command should do the trick (I think a recordset object has no timeout method)
DBCONT.commandtimeout = 120 'CONNECTION TIMEOUT
rs4.Open sqlstr04, DBCONT
I get Operation is not allowed when the object is closed error.
Why I do get this error?
[Error location][2]
There is nothing wrong with the connection.
I can do SQL queries on the same cmd connection and have no problems with them.
Only this query messes me up.
I usually don't use Set search path or table short names, but just incase got
into very detail this time.
When I run the same query in PostgreSQL I get the result:
[postgres output for the same SQL query][1]
I have been digging around google for hours and I can't find any answers.
Does it have to do something with drivers?
How do I know what driver I use for excel?
What driver I should use?
How I change driver?
[References I use][3]
https://i.stack.imgur.com/R4nLe.png
https://i.stack.imgur.com/k4uJo.png
https://i.stack.imgur.com/Ytrlk.png
CODE
Public Function getDBArray(cmd, strSQL)
Dim recordID As Long, recordAmount As Long, totalFields As Integer, fieldID As Integer
Dim rs As New ADODB.Recordset
Set rs = CreateObject("ADODB.Recordset")
cmd.CommandText = strSQL '' << this is the strSQL i copy pasted directly to pgadmin and it worked
Set rs = cmd.Execute
totalFields = getTableAmount(rs.Fields)
recordAmount = getTableAmount(rs) ' << error in this function
recordID = 0
Apparently I was not suppose to used SET SEARCH PATH
In Excel 2016 VBA, I'm refreshing several queries like this:
MyWorkbook.Connections(MyConnectionName).Refresh
After the code is done, and no errors are encountered, I see that the hourglass icons for most of the queries are still spinning for several seconds.
Is it possible to check for success AFTER all the refreshes are completed? I'm concerned that my code isn't going to know if an error happens after the code finishes but before the queries are done refreshing.
BTW I don't want to do a RefreshAll, because some of the queries are dependent on others (uses them as a source). I refresh them in a certain sequence so that dependent queries are refreshed after the queries they are dependent on.
UPDATE:
I see that the Connection objects have a read-only RefreshDate property, which at first glance looked like it could be used to do this check:
MyWorkbook.Connections(MyConnectionName).OLEDBConnection.RefreshDate
HOWEVER, it doesn't seem to be getting set. I get an error trying to check it. If I set a Variant variable to that RefreshDate property, the variable shows as "Empty". The source is a SQL server database.
The QueryTable object exposes two events: BeforeRefresh and AfterRefresh.
You need to change your paradigm from procedural/imperative to event-driven.
Say you have this code in ThisWorkbook (won't work in a standard procedural code module, because WithEvents can only be in a class):
Option Explicit
Private WithEvents table As Excel.QueryTable
Private currentIndex As Long
Private tables As Variant
Private Sub table_AfterRefresh(ByVal Success As Boolean)
Debug.Print table.WorkbookConnection.Name & " refreshed. (success: " & Success & ")"
currentIndex = currentIndex + 1
If Success And currentIndex <= UBound(tables) Then
Set table = tables(currentIndex)
table.Refresh
End If
End Sub
Public Sub Test()
tables = Array(Sheet1.ListObjects(1).QueryTable, Sheet2.ListObjects(1).QueryTable)
currentIndex = 0
Set table = tables(currentIndex)
table.Refresh
End Sub
The tables variable contains an array of QueryTable objects, ordered in the order you wish to refresh them; the currentIndex variable points to the index in that array, for the QueryTable you want to act upon.
So when Test runs, we initialize the tables array with the QueryTable objects we want to refresh, in the order we want to refresh them.
The implicit, event-driven loop begins when table.Refresh is called and the QueryTable fires its AfterRefresh event: then we report success, and update the event-provider table object reference with the next QueryTable in the array (only if the refresh was successful), and call its Refresh method, which will fire AfterRefresh again, until the entire array has been traversed or one of them failed to update.
Just found this solution at Execute code after a data connection is refreshed
The bottom line is: Excel refreshes data connection in the background and thus the rest of the code is executed without interruption.
Solution: set BackgroundQuery property to False
Example:
For Each cnct In ThisWorkbook.Connections
cnct.ODBCConnection.BackgroundQuery = False
Next cnct
Possible problem: don't know which connection it is...
Remedy: case... when...
Dim cnct as WorkbookConnection ' if option explicit
' ODBC and OLE DB
For Each cnct In ThisWorkbook.Connections
Select case cnct.type
case xlconnectiontypeodbc
cnct.ODBCConnection.BackgroundQuery = False
case xlconnectiontypeoledb
cnct.OledbConnection.BackgroundQuery = False
end select
Next cnct
As you can see, code above only deals with ODBC and OLE DB. Depending on what types of data connection you are using, you can expand the select case clause. Unless changed, once run, connection's BackgroundQuery will remain off.
Is there any way to read all the data from excel and put it in the datatable or any other container so that i can filter the data based on the conditions required. As shown in attached image i want to get the CuValue of a Partnumber whose status is Success and i want the latest record based on the Calculation date(Latest calculation date). In the below example i want the CuValue 11292 as it is the latest record with status Success..lue.
Thanks in advance
Your question seems very broad, but you're right to ask because there are many different possibilities and pitfalls.
As you don't provide any sample code, i assume you are looking for a strategy, so here is it.
In short: create a database, a table and a stored procedure. Copy the
data you need in this table, and then query the table to get the
result.
You may use ADO for this task. If it is not available on your machine you can download and install the MDAC redistributable from the Microsoft web site.
The advantage vs. OLE Automation is that you doesn't need to install Excel on the target machine where the import shall be executed, so you can execute the import also server-side.
With ADO installed, you will need to create two Connection objects, a Recordset object to read the data from the Excel file and a Command object to execute a stored procedure which will do the INSERT or the UPDATE of the subset of the source fields in the destination table.
Following is a guideline which you should expand and adjust, if you find it useful for your task:
Option Explicit
Dim PartNo as String, CuValue as Long, Status as String, CalcDate as Date
' objects you need:
Dim srcConn As New ADODB.Connection
Dim cmd As New ADODB.Command
Dim rs As New ADODB.Recordset
Dim dstConn As New ADODB.Connection
' Example connection with your destination database
dstConn.Open *your connection string*
'Example connection with Excel - HDR is discussed below
srcConn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
"Data Source=C:\Scripts\Test.xls;" & _
"Extended Properties=""Excel 8.0; HDR=NO;"";"
rs.Open "SELECT * FROM [Sheet1$]", _
srcConn, adOpenForwardOnly, adLockReadOnly, adCmdText
' Import
Do Until rs.EOF
PartNo = rs.Fields.Item(0);
CuValue = rs.Fields.Item(1);
CalcDate = rs.Fields.Item(6);
Status = rs.Fields.Item(7);
If Status = "Success" Then
'NumSuccess = NumSuccess + 1
' copy data to your database
' using a stored procedure
cmd.CommandText = "InsertWithDateCheck"
cmd.CommandType = adCmdStoredProc
cmd(1) = PartNo
cmd(2) = CuValue
cmd(3) = CalcDate
cmd.ActiveConnection = dstConn
cmd.Execute
Else
'NumFail = NumFail + 1
End If
rs.MoveNext
Loop
rs.Close
Set rs = Nothing
srcConn.Close
Set srcConn = Nothing
dstConn.Close
Set dstConn = Nothing
'
By using a stored procedure to check the data and execute the insert or update in your new table, you will be able to read from Excel in fast forward-only mode and write a copy of the data with the minimum of time loss, delegating to the database engine half the work.
You see, the stored procedure will receive three values. Inside the stored procedure you should insert or update this values. Primary key of the table shall be PartNo. Check the Calculation Date and, if more recent, update CuValue.
By googling on the net you will find enough samples to write such a stored procedure.
After your table is populated, just use another recordset to get the data and whatever tool you need to display the values.
Pitfalls reading from Excel:
The provider of your Excel file shall agree to remove the first two or three rows, otherwise you will have some more work for the creation of a fictitious recordset, because the intelligent datatype recognition of Excel may fail.
As you know, Excel cells are not constrained to the same data type per-column as in almost all databases.
If you maintain the field names, use HDR=YES, without all the first three rows, use HDR=NO.
Always keep a log of the "Success" and "Fail" number of records read
in your program, then compare these values with the original overall
number of rows in Excel.
Feel free to ask for more details, anyway i think this should be enough for you to start.
There are lots ways you can do this.
1. You can create an access DB table and import by saving your sheet as can file first, into the access table. Then you can write queries.
2. You can create a sql DB and a table, write some code to import the sheet into that table.
3. You can Write some code in VBA and accomplish that task if your data is not very big.
4. You can write c# code to access the sheet using excel.application and office objects, create a data table and query that data table
Depends on what skills you want to employ to accomplish your task.
Preface
I'm making this question specific to conform to SO asking guidelines, but feel free to suggest wholesale redesign if you wish. I may be using some bad practices.
Basic Question
I use ADO to execute a multi-step SQL Server query that takes several minutes to execute. I use Raiserror in my tsql queries to let myself know more verbosely which steps have finished. Is it possible to pass these messages to VBA before the complete query finishes, while still continuing with the query?
Details and Code
I use the vba below to execute the t-SQL query underneath. As you can see, there are two errors raised in the t-SQL that display "Step 1 complete" and "Step 2 complete". Could I pass these messages (or alternately use error numbers and pass those) back to VBA in a way that would allow me to detect them and update a progress bar while continuing to execute the query?
VBA used to execute the query:
Set cmd = New ADODB.Command
cmd.ActiveConnection = cnn
cmd.CommandTimeout = 0
cmd.CommandText = strQuery
Set rst = New ADODB.Recordset
rst.Open cmd
'Go to the second to last recordset of the multi-step query
String1 = Replace(strQuery, ";", "")
For Loop2 = 1 To (Len(strQuery) - (Len(String1) + 1))
Set rst = rst.NextRecordset
Next Loop2
'Copy results
If Not rst.EOF Then
(snip - actions)
Else
MsgBox "Error: No records returned."
End If
Stripped-down piece of multi-step tSQL query:
--#DRS1: The numbers being researched
select distinct numbers
into #DRS1
from Table1 (nolock)
where numbers in ()
--#DRS1: Index
create nonclustered index Idx_DRS1
on #DRS1(numbers);
Raiserror(“Step 1 complete”,1,1) with nowait;
--#DRS2: Table2 for numbers being researched
select distinct
DRS1.numbers
,a.ID
into #DRS2
from #DRS1 DRS1
join Table2 (nolock) a
on DRS1.numbers = a.numbers
Raiserror(“Step 2 complete”,1,1) with nowait;
--MORE STEPS
(more steps)
(more raiserror statements)
Clarification
I am not interested in:
A method that doesn't allow me to update a progress bar until the query is completely done.
A method that uses Progress/MaxProgress, because as I understand it that would return separate numbers for each of the steps in my query, rather than one progress measure for the entire query.
I am less interested in:
Using # records affected messages to determine progress, because some steps may return equal numbers of records to previous steps.
Research
The closest thing I have found to what I'm looking for is here, but as the discussion of that solution here says:
This approach would only work for stored procedures that are not intended to return results, say procs that insert data into tables. Another approach would be needed if your stored proc returns a result set.
Since I return results in the final step of my query to be manipulated in Excel I don't think this would work for me.
External link code for reference
SQL:
CREATE PROCEDURE dbo.updTesting As
Declare #RetVal integer
Exec #RetVal = updTesting2
Return #RetVal
GO
CREATE PROCEDURE dbo.updTesting2 As
raiserror('Error From Testing 2 procedure',16,1)
Return -2
GO
VBA:
Private Sub Command1_Click()
On Error GoTo ErrorHandler
Dim db As ADODB.Connection
Dim cmd As ADODB.Command
Set db = New ADODB.Connection
db.CursorLocation = adUseClient
db.Open "provider=sqloledb;data source=handel;initial catalog=northwind;integrated security=sspi"
Set cmd = New ADODB.Command
With cmd
Set .ActiveConnection = db
.CommandText = "updTesting"
.CommandType = adCmdStoredProc
.Parameters.Append .CreateParameter("#RetVal", adInteger, adParamReturnValue)
.Execute , , adExecuteNoRecords
End With
ExitPoint:
On Error Resume Next
Set cmd.ActiveConnection = Nothing
Set cmd = Nothing
db.Close
Set db = Nothing
Exit Sub
ErrorHandler:
MsgBox "Error # " & Err.Number & vbNewLine & vbNewLine & Err.Description
Resume ExitPoint
End Sub
There are several possibilities to work out a solution for your problem:
(1) Capture the error messages as they occur while the query is running. That's the requested approach.
(2) Break-down the big, long query into several smaller chunks and run them one after the other. Like this you know which part is completed and you can update your progress bar based on that information just before sending the next chunk to the server.
(3) Update the big, long query to log its progress on the server in a temp table and then read out this log while the other query is still running.
While I'd recommend to use errors only when errors occur and not to "abuse" them for logging, tracking, or feedback, both options (1 & 2) are quite feasible with events:
Similar to Worksheet events Worksheet_Change, Worksheet_Activate, or Worksheet_BeforeDoubleClick there are also ADODB events for ADODB.Connection and ADODB.Recordset. Both are well documented and can be easily viewed within the VBE by (1) adding a reference to Microsoft ActiveX Data Objects x.x Library (2) pressing F2 (3) selecting the ADODB library in the drop-down menu at the top (4) and finally looking up Recordset or Connection within the classes. Here are the available events for Connection:
As you can see, all events are marked with a lightning. To capture / use these events you need to create a Class Module in the VBE and add the following line to it:
Dim WithEvents adoConnection As ADODB.Connection
Afterwards, you can make use of the newly created ADODB.Connection event and select the required event from the top of the list:
The applicable event for option (1) is the InfoMessage event which occurs "[...] whenever a warning occurs during a ConnectionEvent operation." The import part here is during a connection. So, this event fires automatically whenever an ADODB connection "gets" an error.
Of course, this means that the original query to the server must be sent without waiting for an answer. Instead you should use the above event to capture any errors while the query is executing and create yet another event to automatically fire when the entire query completed.
For some more help in respect to asynchronous ADODB connection and possible problems with them you might want to have a look at the following two posts here:
ExecuteComplete ADODB Connection event not fired with adAsyncExecute parameter
Running multiple async queries with ADODB - callbacks not always firing
A similar approach can be used with option (3) as described above and asynchronous ADODB connections.
Let me know if this solves your problems or if you have any further questions.
All available ADODB events can be reviewed here https://msdn.microsoft.com/en-us/library/ms675083%28v=vs.85%29.aspx