I’m using a macro that writes data to a SQL Table everyday. Problem is that usually I have more than 300k rows to upload and Macro runs very slowly (more than 60 minutes). Is there anyway to speed up this process? Below part of the code, this is exactly where Macro spent most of time:
tb = "[Table_Test]"
Set Pos = ThisWorkbook.Sheets(“Sheet1”)
For i = 3 To last_row
query = "INSERT INTO " & tb & " ([DATE],[POSITION])"
query2 = " VALUES " & "('" & Format(CDate(Pos.Cells(i, 1)), "yyyy-mm-dd") & "'" & ", '" & Pos.Cells(i, 2) & "'" &");"
query = query & query2
Execute_SQL_218 (query)
query = False
Application.StatusBar = "Uploading Data To SQL..." & i
DoEvents
Next i
The problem is that this loop sends the data to the server row-by-row and runs a query for each. You need to send the whole thing as one recordset and insert it with one query.
As you did not provide what kind of SQL DB you have, I can only give you one suggestion for Microsoft SQL Server: first export your recordset to .csv file then instead of running the "INSERT INTO " query run a BULK INSERT.
Related
OK. I'm sorry for wasting everyone's time. Like a DUMMY i didn't think simple solution first. The amount of data i am dealing with isn't too large and will actually work better just exporting to an excel file (i'm pretty sure). I would like to thank all that helped (June7, Parfait, and HansUp). The support you guys (everyone on this forum) give has made my job easier by far.
I'm trying to export an Excel Table from my active excel file to an Access database file.
I was getting an error at
"con.excecute sql"
"Run-time error '-2147467259 (80004005)': [Microsoft][ODBC Microsoft Access Driver] Query input must contain at least one table or query."
Sub updateAccess()
Dim con As New ADODB.Connection
Dim connectionString As String
Dim sql, newTable As String
Filename = "C:\Desktop\Quote-Size_Contacts.accdb"
connectionString = "Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=" & Filename
con.Open connectionString
' Save current table ("ContactsTbl_Data") to another table ("ContactsTbl_Data_yyyymmdd_hh_mmss")
newTable = "Quote-Size_Contacts_" & Format(Date, "yyyymmdd") & "_" & Format(Now, "hhmmss")
sql = "SELECT CODE, STORE INTO " & newTable & "FROM ContactsTbl_Data"
con.Execute sql
' Delete rows of current table ("ContactsTbl_Data")
sql = "DELETE FROM ContactsTbl_Data"
con.Execute sql
' Insert new rows into current table ("ContactsTbl_Data") from my Excel Sheet
sql = "INSERT INTO ContactsTbl_Data ([CODE], [STORE]) " & _
"SELECT * FROM [Excel 8.0;HDR=YES;DATABASE=" & ThisWorkbook.FullName & "].[" & ThisWorkbook.Sheets("Sheet2").Name & "$]"
con.Execute sql
con.Close
Set con = Nothing
End Sub
EDIT::
I'm not sure standard protocol for these forums on cleaning up the code and asking more questions so i'll just put an "Edit" here.
I applied the suggestions and matched the fields it was trying to save to my access file. I now get the error: "Method 'Execute' of object '_Connection' failed"
Public Sub updateAccess()
Dim con As New ADODB.Connection
Dim connectionString As String
Dim sql, newTable As String
Filename = "C:\Desktop\Quote-Size_Contacts.accdb"
connectionString = "Provider=Microsoft.ACE.OLEDB.12.0; Data Source='" & Filename & "'"
con.Open connectionString
' Save current table ("ContactsTbl_Data") to another table ("ContactsTbl_Data_yyyymmdd_hh_mmss")
newTable = "Quote-Size_Contacts_" & Format(Date, "yyyymmdd") & "_" & Format(Now, "hhmmss")
sql = "SELECT Company, Contact, Initials, Position, Address, AddressContd, CityStatePost, MainNo, CellNo, FaxNo, Email INTO [" & newTable & "] FROM ContactsTbl_Data"
con.Execute sql
' Delete rows of current table ("ContactsTbl_Data")
sql = "DELETE FROM ContactsTbl_Data"
con.Execute sql
' Insert new rows into current table ("ContactsTbl_Data") from my Excel Sheet
sql = "INSERT INTO ContactsTbl_Data ([Company], [Contact], [Initials], [Position], [Address], [AddressContd], [CityStatePost], [MainNo], [CellNo], [FaxNo], [Email]) " & _
"SELECT * FROM [Excel 12.0 Xml;HDR=Yes;Database=" & ThisWorkbook.FullName & "].[" & ThisWorkbook.Sheets("Sheet2").Name & "$]"
con.Execute sql
con.Close
Set con = Nothing
End Sub
See if this helps.
Table name has hyphen (-) character so use [ ] characters to delimit. Add a space in front of FROM so text doesn't run together in compiled SQL string.
sql = "SELECT CODE, STORE INTO [" & newTable & "] FROM ContactsTbl_Data"
As for connection to Access database, don't think I've ever used or seen Driver, I use Provider:
connectionString = "Provider=Microsoft.ACE.OLEDB.12.0; Data Source='" & Filename & "'"
Background: I have a workbook with a lot of tabs that’s brings in data from SQL database (please see example query below). And due to a lot of reasons, I need to create many pivot tables in each tab because each pivot table needs to look at one particular set of data. Every-time I create a pivot table I have to go to “Pivot Table Options” and jot down the pivottable number and then update the VBA code with the appropriate sequential code Set pvt = Me.PivotTables("PivotTable###").
Questions: Is there some VBA code that can reference all the pivot tables within each tab without the need to reference a pivotable number? Cause I have a lot of tabs and I need to create a lot of pivot tables and I’m losing track of the pivotable numbers.
Sub RefreshData()
Dim SQl As String
Dim startDATE As String, endDATE As String
Dim pvt As PivotTable
Set pvt = Me.PivotTables("PivotTable1")
Set pvt = Me.PivotTables("PivotTable2")
Set pvt = Me.PivotTables("PivotTable3")
'pvt.PivotCache.Connection = "ODBC;DSN=" & Application.Range("DSN_Source") & ";DATABASE=" & Application.Range("T123") & ";"
startDATE = Format(Me.Range("SDate"), "yyyy-mm-dd hh:mm:ss")
endDATE = Format(Me.Range("EDate"), "yyyy-mm-dd hh:mm:ss")
SQl = ""
SQl = "WITH ABC As " & vbCrLf
SQl = SQl & "( " & vbCrLf
SQl = SQl & " SELECT Stuff " & vbCrLf
SQl = SQl & ", more stuff, ROW_NUMBER() OVER (PARTITION BY RIGHT(Stuff, 2) " & vbCrLf
SQl = SQl & ", stuff1, stuff2, stuff3 ORDER BY Estuff4 DESC) As RowNum" & vbCrLf
SQl = SQl & " FROM StuffTable " & vbCrLf
SQl = SQl & " WHERE stuff = 'stuffy' AND " & vbCrLf
*snip
With pvt.PivotCache
.CommandType = xlCmdSql
.CommandText = SQl
End With
pvt.RefreshTable
End Sub
You need to loop through all the PivotTables like so:
For Each pvt In Me.PivotTables
With pvt.PivotCache
.CommandType = xlCmdSql
.CommandText = Sql
End With
pvt.RefreshTable
Next
Just to give you background of my work, i have to fetch data from MS Sql on daily basis and for that every time have to go to other server to run the query. Once the query is executed, have to paste into my common drive, which takes a lot time. ~55 mins to paste 5,00,000 row & 30 fields to common or to move file. In total 2 hours for execution & movement from one location to other.
To reduce this i would need your help to use the SQL queries through excel with the below things:
If possible,
Point1: Query will be stored in the text file in the common location
Point2: Query Parameter to be populate to get
Or
Point2:Range to be defined for parameter
If not possible above,
Query will be pasted into the code and parameter to be populated based on the above mentioned suggestion.
Connection type is windows authentication, it will work based on logged in users windows name.
This code will allow you to provide variables that you use within your SQL statement and put those into cells on a spreadsheet (In this case Cred2) and return the results on a separate sheet (Sheet2).
The first portion of the code establishes a connection with the SQL server.
The column Headers will be started in Row 2 and then the data will begin populating starting on row 3. I have used this to pull well over 100,000 records at a time and this works very quickly.
Private Sub CommandButton1_Click()
Dim cn As Object
Dim rs As Object
Dim strCon As String
Dim strSQL As String
strCon = "DRIVER=SQL Server;SERVER=ServerName;DATABASE=DBName;Trusted_Connection=True"
Set cn = CreateObject("ADODB.Connection")
cn.Open strCon
' if not a trusted connection you could replace top line of strCon with
strCon = "DRIVER=SQL Server; Server=myServerAddress;Database=myDataBase;User Id=myUsername; Password=myPassword"
' set up where you are getting your variables to include in the SQL statement
stat = Sheets("Cred2").Range("c7").Value
barg = Sheets("Cred2").Range("c10").Value
worksite = Sheets("Cred2").Range("c11").Value
' Construct SQL statement
strSQL = "select * " _
& " FROM tableName A , table2 B " _
& "WHERE A.[field1] = B.[field1] " _
& " and field1 like '" & stat & "'" _
& "and field2 like '" & barg & "'" _
& "and field3 like '" & worksite & "'" _
& " order by Field? "
' Build Record Set
Set rs = CreateObject("ADODB.RECORDSET")
rs.ActiveConnection = cn
rs.Open strSQL
' Display Data
For intColIndex = 0 To rs.Fields.Count - 1
Sheet2.Range("A2").Offset(0, intColIndex).Value = rs.Fields(intColIndex).name
Next
Sheet2.Range("A3").CopyFromRecordset rs
' Close Database
rs.Close
cn.Close
Set cn = Nothing
end sub
I am trying to extract data from Access using a column of excel.
I have tried the following code but its not taking too much time when # of rows in excel exceed 5k rows. Does anyone know of a better way to reference the excel data to get the results:
Sub ddd()
Const dbloc As String = "C:\Users\mysystem\Downloads\Database11.accdb"
Dim db As DAO.Database
Dim rs As DAO.Recordset
Dim xlbook As Workbook
Dim xlsheet As Worksheet
Dim a As Long
Dim SQL As String
Set xlbook = ActiveWorkbook
Set xlsheet = xlbook.Worksheets(1)
xlsheet.Range("B2:Z100000").ClearContents
Set db = OpenDatabase(dbloc)
SQL = "SELECT Material, MPN "
SQL = SQL & "FROM Sheet2 "
SQL = SQL & "WHERE Material IN ("
Dim r As Range
For Each r In Range("A2:A19098")
SQL = SQL & r.Text & ","
Next r
SQL = Left(SQL, Len(SQL) - 1) 'Drop last comma
SQL = SQL & ")"
' i want to change this for loop because my range might vary from 80-100k
rows and this method is not working. i got a suggestion here that i can
use a table for this. But i am new to macros and access and not sure of
the syntax. Can anyone please help with the syntax. Assuming tablename for
the excel data is column1 and ranges from a2:a100000
Set rs = db.OpenRecordset(SQL)
', dbOpenSnapshot)
If rs.RecordCount = 0 Then
MsgBox "No data retrieved from database", vbInformation + vbOKOnly, "No
Data"
GoTo SubExit
Else
rs.MoveLast
recCount = rs.RecordCount
rs.MoveFirst
End If
xlsheet.Range("C2").CopyFromRecordset rs
End Sub
Any help will me much appreciated. Thanks!!
If you're looking for an alternative way to reference an Excel column, you can use the following:
SQL = "SELECT Material, MPN "
SQL = SQL & "FROM Sheet2 "
SQL = SQL & "WHERE Material IN ("
SQL = SQL & "SELECT * FROM [A1:A19098] IN '"
SQL = SQL & ActiveWorkbook.FullName & "'"
SQL = SQL & " 'Excel 12.0 Macro;')"
This allows the DAO database engine to directly query your Excel file via OleDb, instead of passing each row as text, which should be more efficient.
The Excel file should be saved before you do this (and it's better to trigger it from a different file, but works for me when triggering from the same file).
Note that I am purposefully including 1 row more, because the first row contains column names when using this.
I'm assuming you're using a .xlsm file.
I have a classic ASP web application that updates a table in a Excel spreadsheet.
I am using the following to connect to the Excel file
Set rows = CreateObject("ADODB.Recordset")
objConnection.Open "Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=" & Server.MapPath("data.xlsx") & ";" & _
"Extended Properties=" & Chr(34) & "Excel 12.0 Xml;HDR=Yes" & Chr(34) & ";"
That is a sample query that I am executing
query = "UPDATE [Sheet1$] SET [STATUS] = 'unclaimed', [CLAIMER] = '' ,[SDATE] = '"
& DATE & "' WHERE [JUR] = '" & in_jur & "' AND [WC] = " & in_wc
objConnection.Execute query,numRows,adExecuteNoRecords
The above query gets executed in the process.asp page when the user clicks on a button that calls a javascript function that sends the value of in_jur and in_wc like this
function sendForProcessing() {
somejur = //use the document.getElementByID to get selected jur
somewc = //use the document.getElementByID to get selected wc
window.location = process.asp?in_jur=somejur&in_wc=somewc
}
The application updates the sheet correctly for some time (i counted the number of times it works. 1st test: 10 times 2nd test: 16 times it just varies) and then it stops working.
I do not get error message in the objConnection.Execute.
When i print out the numRows it prints 1.
Is there a better way to update the row in Excel?