Hi I currently have two worksheets in an excel file with one of them acting as a database of all the products we sell, with the columns Product ID, Product Code, and Description (sample below).
I have another worksheet that acts as a product finder tool, where you would paste multiple Product IDs in the first column and it would return the Product code and Description in the adjacent columns (image below).
I currently use an INDEX search to make this happen, but the database sheet has become too big to manage in the same file, leading to severe slow downs. What would be the easiest solution for this? I was thinking of separating the database sheet as an Excel or AccessDB file but I think I will need a lot of VBA manipulation if I do that. Any help would be much appreciated.
You can access your data in Microsoft Access using ADO and doing a SQL query to gather data.
Could you tell me if it's possible to give a cell range to the WHERE clause?
Yes, there is a trick. SQL commands are plain text, you just need to build it with your parameters. Use the operator IN in the WHERE clause.
I made a fake dataset as example. Here's my Excel Product Finder (a table named Table1):
Notice I want the info only of products 6,3 and 2. Now my fake database:
The code to query those specific products:
Sub TEST()
Dim cnn As Object
Dim RST As Object
Dim DatabasePath As String
Dim i As Long
Dim Allid As String
Dim Arrayid As Variant
Dim SQLQuery As String
DatabasePath = "C:\Temp\temp.accdb" 'path to database
'Create a connection object.
Set cnn = CreateObject("ADODB.Connection")
'Create recordset object
Set RST = CreateObject("ADODB.Recordset")
'Open a connection using the OLE DB connection string.
cnn.Open "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & DatabasePath & ";Persist Security Info=False;"
'merge all ID into one single string
Arrayid = Range("Table1[PRODUCT ID]").Value
For i = LBound(Arrayid) To UBound(Arrayid) Step 1
Allid = Allid & Arrayid(i, 1) & ","
Next i
Allid = Left(Allid, Len(Allid) - 1) 'get rid of last comma
Erase Arrayid 'clean array variable
'specify query
SQLQuery = "SELECT PRODUCT_TABLE.[Product Id], PRODUCT_TABLE.[Product Code], PRODUCT_TABLE.Description FROM PRODUCT_TABLE " & _
"WHERE PRODUCT_TABLE.[Product Id] In (" & Allid & ") ORDER BY PRODUCT_TABLE.[Product Id]"
'Open a recordset using the Open method
'and use the connection established by the Connection object.
RST.Open SQLQuery, cnn
'copy all data into cells. This will bring full query without headers
Range("A6").CopyFromRecordset RST
'close and clean variables
RST.Close
cnn.Close
Set RST = Nothing
Set cnn = Nothing
End Sub
After executing code I get this:
NOTICE that the output is not sorted as we had before. We asked the products in order 6,3,2 but the output is 2,3,6!
This is because my SQL query got the operator ORDER BY that sorts by ID field. If there is no ORDER BY clause the output will be sorted as it is in the database stored, not as your Excel.
If you really really really need the output to be exactly in the same order that your Product Finder, you can create an UDF function to query each single id once and return a single row for each product but if you work with a lot of data this can consume a lot of time. So think carefully how to approach this part.
By the way, make sure you use the right connection string. You can find many on Access connection strings
Related
I'm using Excel and Access 365 for the record.
I have information on around 100,000 account numbers, far too many for Excel to handle efficiently. I put them into Access.
In Excel, I have a list of about 10 account numbers. This list changes daily. How do I get the account information from Access into Excel? If I was able to keep everything in Excel I would use INDEX MATCH, what is the equivalent to get information from Access?
I would suggest setting up a linked table to Excel within Access, and running an SQL statement. Much simpler than loops in VBA.
Open Access
Create a linked table to the Excel worksheet, which is nothing more than connection information to the worksheet; it doesn't actually store the records from the worksheet.
This allows the following:
From within Access -- run queries that join data between Access tables and the linked Excel table. You can save such queries, use them as RecordSource for a form or report etc.
From within Excel -- you can open an ADO connection from within Excel and run an SQL statement joining Access tables and the linked Excel worksheet. You can then use the Excel Range.CopyFromRecordset method to paste those results into an Excel worksheet.
It sounds like you need to use the 'In' clause. I have the following data points on Sheet2 in Range A1:A5.
Ryan
Sam
Timmy
Tommy
Teddy
Paste the code below into a Module and set a reference to 'Microsoft Active X Data Objects 2.8 Library' under Tools in the VBE Window.
Sub Import()
Dim connect As ADODB.Connection
Dim rec1 As ADODB.Recordset
Dim wb As Worksheet
Dim Wb2 As Worksheet
Dim Param() As ADODB.Parameter
Dim Command1 As ADODB.Command
Dim lrow As Integer
Dim i As Integer
Dim ConcatSQL As String
Set wb = ActiveWorkbook.Sheets("Sheet1")
Set Wb2 = ActiveWorkbook.Sheets("Sheet2")
lrow = Wb2.Range("A" & Wb2.Rows.Count).End(xlUp).Row
'Concatenate desired range into one cell
For i = 0 To lrow
ConcatSQL = ConcatSQL & "'" & Wb2.Cells(i + 1, 1) & "'" & ","
Next i
ConcatSQL = "(" & Left(ConcatSQL, Len(ConcatSQL) - 1) & ")"
'Open Command Object with One Paramter
Set Command1 = New ADODB.Command
With Command1
.CommandText = " Select ID, Price from TABLE where ID IN " & ConcatSQL
.CommandType = adCmdText
.CommandTimeout = 600
End With
'Connect to Data Source
Set connect = GetNewConnection 'Represents Private Function with Connection String
Command1.ActiveConnection = connect
Set rec1 = New ADODB.Recordset
Set rec1 = Command1.Execute()
'Paste Results
wb.Activate
With wb.QueryTables.Add(Connection:=rec1, Destination:=wb.Range("A1"))
.Name = "data"
.FieldNames = True
.Refresh BackgroundQuery:=False
End With
'Close Connections
rec1.Close
connect.Close
Set rec1 = Nothing
Set connect = Nothing
End Sub
Here is a screen shot to show how the variables are created.
I'm 100% certain that you can run a simple query in Access and export the results of said query to Excel. Or, save that query, and import the records in the object to Excel. When you get into larger data sets like you described, you may want to consider using different tools for the job. Python and R come to mind.
This will probably require VBA to do efficiently.
Loop through the account numbers, and for each account number query the Access database (using ADO) and return only the required data for each account.
I've got some queries I have connected to an excel sheet that I refresh in order to get the latest data.
Is it possible to make excel "ask" for input parameters for one column lets say?
I am looking to get data by company so wondering if it's possible to do it as in access (type [company]:) in the field criteria
If I save the query like that in access it won't let me connect it to excel
Thanks
Alright another edit.
Here it is, I created a database, its simple.
The database is called "Database1.accdb"
There are more records than shown in the screen shot.
I created a workbook with one sheet, its name is "AccessDBtest.xlsm"
I created a button on Sheet1 and entered the field parameter I wanted in the cell beside it, C3 or (3,3) in (row,col) format.
This is the code that works returning the data set (without field names) based on the input criteria. I made a msgBox before the SQL query execution so that I could look at it first. You don't need that if you do not want it, good for testing.
Private Sub CommandButton1_Click()
Dim inputSheet As Worksheet
Dim fieldSTR As String
Dim placementRange As Range
Dim rs As Object 'record set
Dim conn As Object
Dim strQuery As String
Dim myDB As String
Set inputSheet = ThisWorkbook.Worksheets("Sheet1")
Set placementRange = inputSheet.Range("E2")
fieldSTR = CStr(inputSheet.Cells(3, 3).Value) 'C3 cell
myDB = "C:\Users\Documents\0_Excel Projects\Testing\Database1.accdb"
Set conn = CreateObject("ADODB.Connection")
With conn
.Provider = "Microsoft.ACE.OLEDB.12.0" 'For *.ACCDB Databases
.ConnectionString = myDB
.Open
End With
strQuery = "SELECT * FROM " & _
"tbl_test WHERE tbl_test.Color = " & "'" & fieldSTR & "'" & ";"
'The below gives the same result as * but you could limit the fields returned as well
'tbl_test.ID, tbl_test.Color, tbl_test.number
'just using Color also works you do not need to reference the table directly
MsgBox (strQuery)
Set rs = conn.Execute(strQuery)
placementRange.CopyFromRecordset rs
rs.Close
Set rs = Nothing
conn.Close
Set conn = Nothing
End Sub
See if you can map what you want to do using that.
I wanted to capture a string from a cell and then use that string in the query.
Here is the MsgBox before the query executes (what access will see):
I tested it with * for all fields and below you can see that I called up specific fields, they both work.
And here is what happens after I clear the msgBox, the recordset is pulled and pasted starting in the range I specified Range.("E3") on the sheet I specified (Sheet1)
As you can see, we can pull queries from a database using input that is found on a current sheet.
Let me know if this helps your situation.
We can dig deeper if required.
-WWC
I have an Excel worksheet that has a list of about 1000 Item Numbers in column A on Sheet1. Currently, I import Sheet1 into an Access table named ItemNumbers and run the following query:
SELECT MyTable.ItemNumber, MyTable.ItemName, MyTable.ItemPrice
FROM [ItemNumbers] INNER JOIN MyTable ON [ItemNumbers].ItemNumber = MyTable.ItemNumber
ORDER BY MyTable.ItemNumber;
And then I copy/paste the output to Sheet2.
How can I do this in VBA in Excel and put the results in a recordset? I can figure out how to loop through the recordset and put the results in Sheet2. I'm just not sure on the code to run the query.
I have the following so far. It just needs to be modified to use the values in Sheet1 Column A.
Dim cn As Object
Dim rs As Object
Dim strSql As String
Dim strConnection As String
Set cn = CreateObject("ADODB.Connection")
strConnection = "Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=C:\MyDatabase.accdb"
strSql = "SELECT MyTable.ItemNumber, MyTable.ItemName, MyTable.ItemPrice " & _
"FROM MyTable " & _
"WHERE WHERE (((MyTable.ItemNumber)= ??? IS IN Sheet1!A:A ??? )) " & _
"ORDER BY MyTable.ItemNumber;"
cn.Open strConnection
Set rs = cn.Execute(strSql)
rs.Close
Set rs = Nothing
cn.Close
Set cn = Nothing
Thanks!!!
If I understand right; what you ask is to join a table from Access with a table in Excel (ADODB).
Check this link from SO, and see if it's helpful:
Selecting 2 tables from 2 different databases (ACCESS)
I haven't tried to combine Access and Excel before, but my guess is that it will work for Excel as well.
An alternate way (and that will certainly work):
Run the query without the WHERE clause and store the result in a
recordset;
Store the data from the Excel sheet that you require in a dictionary,
where the ItemNumber (PK?) is the key;
Run through the recordset, and check with the typical dictionary Exists function
if the ItemNumber from each record is available in the dictionary;
If the record is availabe, store the
recordset values in a separate array (or dictionary) that you can
use for further manipulation, (or perform direct actions if that's what you want to do).
Does anyone know the fastest way to get data from and Excel table (VBA Array) to a table on SQL 2008 without using an external utility (i.e. bcp)?
Keep in mind my datasets are usually 6500-15000 rows, and about 150-250 columns; and I end up transferring about 20-150 of them during an automated VBA batch script.
I have tried several methods for getting large amounts of data from an Excel table (VBA) to SQL 2008. I have listed those below:
Method 1. Pass table into VBA Array and send to stored procedure (ADO)
-- Sending to SQL is SLOW
Method 2. Create disconnected RecordSet load it, then sync.
-- Sending to SQL VERY SLOW
Method 3. Put table into VBA array, loop though the array and concatenate(using delimiters) then send to stored procedure.
-- Sending to SQL SLOW, but faster than Method 1 or 2.
Method 4. Put table into VBA array, loop though the array and concatenate(using delimiters) then place each row with ADO recordset .addnew command.
--Sending to SQL very FAST (about 20 times faster than methods 1-3), but now I will need to split that data using a separate procedure, which will add significant wait time.
Method 5. Put table in VBA array, serialize into XML, send to stored procedure as VARCHAR and specify XML in stored procedure.
--Sending to SQL INCREDIBLY SLOW (about 100 times slower than methods 1 or 2)
Anything I am missing?
There is no single fastest way, as it's dependent on a number of factors. Make sure the indexes in SQL are configured and optimized. Lots of indexes will kill insert/update performance since each insert will need to update the index. Make sure you only make one connection to the database, and do not open/close it during the operation. Run the update when the server is under minimal load. The only other method you haven't tried is to use a ADO Command object, and issue a direct INSERT statement. When using the 'AddNew' Method of the recordset object, be sure to issue only one 'UpdateBatch' Command at the end of the inserts. Short of that, the VBA can only run as fast as the SQL server accepting the inputs.
EDIT:
Seems like you've tried everything. There is also what is known as 'Bulk-Logged' recovery mode in SQL Server, that reduces the overhead of writting so much to the transaction log. Might be something worth looking into. It can be troublesome since it requires fiddling with the database recovery model a bit, but it could be useful for you.
The following code will transfer the thousands of data in just few seconds(2-3 sec).
Dim sheet As Worksheet
Set sheet = ThisWorkbook.Sheets("DataSheet")
Dim Con As Object
Dim cmd As Object
Dim ServerName As String
Dim level As Long
Dim arr As Variant
Dim row As Long
Dim rowCount As Long
Set Con = CreateObject("ADODB.Connection")
Set cmd = CreateObject("ADODB.Command")
ServerName = "192.164.1.11"
'Creating a connection
Con.ConnectionString = "Provider=SQLOLEDB;" & _
"Data Source=" & ServerName & ";" & _
"Initial Catalog=Adventure;" & _
"UID=sa; PWD=123;"
'Setting provider Name
Con.Provider = "Microsoft.JET.OLEDB.12.0"
'Opening connection
Con.Open
cmd.CommandType = 1 ' adCmdText
Dim Rst As Object
Set Rst = CreateObject("ADODB.Recordset")
Table = "EmployeeDetails" 'This should be same as the database table name.
With Rst
Set .ActiveConnection = Con
.Source = "SELECT * FROM " & Table
.CursorLocation = 3 ' adUseClient
.LockType = 4 ' adLockBatchOptimistic
.CursorType = 0 ' adOpenForwardOnly
.Open
Dim tableFields(200) As Integer
Dim rangeFields(200) As Integer
Dim exportFieldsCount As Integer
exportFieldsCount = 0
Dim col As Integer
Dim index As Integer
index = 1
For col = 1 To .Fields.Count
exportFieldsCount = exportFieldsCount + 1
tableFields(exportFieldsCount) = col
rangeFields(exportFieldsCount) = index
index = index + 1
Next
If exportFieldsCount = 0 Then
ExportRangeToSQL = 1
GoTo ConnectionEnd
End If
endRow = ThisWorkbook.Sheets("DataSheet").Range("A65536").End(xlUp).row 'LastRow with the data.
arr = ThisWorkbook.Sheets("DataSheet").Range("A1:CE" & endRow).Value 'This range selection column count should be same as database table column count.
rowCount = UBound(arr, 1)
Dim val As Variant
For row = 1 To rowCount
.AddNew
For col = 1 To exportFieldsCount
val = arr(row, rangeFields(col))
.Fields(tableFields(col - 1)) = val
Next
Next
.UpdateBatch
End With
flag = True
'Closing RecordSet.
If Rst.State = 1 Then
Rst.Close
End If
'Closing Connection Object.
If Con.State = 1 Then
Con.Close
End If
'Setting empty for the RecordSet & Connection Objects
Set Rst = Nothing
Set Con = Nothing
End Sub
By far the fastest way to do this is via T-SQL's BULK INSERT.
There are a few caveats.
You will likely need to export your data to a csv first (you may be able to import directly from Excel; my experience is in going from Access .mdbs to SQL Server which requires the interim step to csv).
The SQL Server machine needs to have access to that csv (when you run the BULK INSERT command and specify a filename, remember that the filename will be resolved on the machine where SQL Server is running).
You may need to tweak the default FIELDTERMINATOR and ROWTERMINATOR values to match your CSV.
It took some trial and error for me to get this set up initially, but the performance increase was phenomenal compared to every other technique I had tried.
works pretty fine, on the other hand to improve speed we may still modify the query:
Instead: Source = "SELECT * FROM " & Table
We can use: Source = "SELECT TOP 1 * FROM " & Table
Here is we only need column names. So no need to maka a query for entire table, which is extending the process as long as new data imported.
As far as I remember, you can create a linked server to the Excel file (as long as the server can find the path; it's best to put the file on the server's local disk) and then use SQL to retrieve data from it.
Having just tried a few methods, I came back to a relatively simple but speedy one. It's fast because it makes the SQL server do all the work, including an efficient execution plan.
I just build a long string containing a script of INSERT statements.
Public Sub Upload()
Const Tbl As String = "YourTbl"
Dim InsertQuery As String, xlRow As Long, xlCol As Integer
Dim DBconnection As New ADODB.Connection
DBconnection.Open "Provider=SQLOLEDB.1;Password=MyPassword" & _
";Persist Security Info=false;User ID=MyUserID" & _
";Initial Catalog=MyDB;Data Source=MyServer"
InsertQuery = ""
xlRow = 2
While Cells(xlRow, 1) <> ""
InsertQuery = InsertQuery & "INSERT INTO " & Tbl & " VALUES('"
For xlCol = 1 To 6 'Must match the table structure
InsertQuery = InsertQuery & Replace(Cells(xlRow, xlCol), "'", "''") & "', '" 'Includes mitigation for apostrophes in the data
Next xlCol
InsertQuery = InsertQuery & Format(Now(), "M/D/YYYY") & "')" & vbCrLf 'The last column is a date stamp, either way, don't forget to close that parenthesis
xlRow = xlRow + 1
Wend
DBconnection.Execute InsertQuery 'I'll leave any error trapping to you
DBconnection.Close 'But do be tidy :-)
Set DBconnection = Nothing
End Sub
Hey all, have been working on designing a new database for work. They have been using Excel for their daily reports and all the data is stored in there, so I decided to have the back-end of the database in Access and the front-end in Excel, so any analytical work can be easily performed once all the data has been imported into Excel.
Now I'm fairly new to VBA, slowly getting used to using it, have written some code to transfer one of the calculated tables from Access to Excel:
Option Explicit
Public Const DataLocation As String = "C:\Documents and Settings\Alice\Desktop\Database\TestDatabase21.accdb"
Sub Market_Update()
Call ImportFromAccessTable(DataLocation, "Final_Table", Worksheets(2).Range("A5"))
End Sub
Sub ImportFromAccessTable(DBFullName As String, TableName As String, TargetRange As Range)
Dim cn As ADODB.Connection, rs As ADODB.Recordset, intColIndex As Integer
Set TargetRange = TargetRange.Cells(1, 1)
' open the database
Set cn = New ADODB.Connection
cn.Open "Provider=Microsoft.ACE.OLEDB.12.0; Data Source=" & DBFullName & ";"
Set rs = New ADODB.Recordset
With rs
' open the recordset
' .Open TableName, cn, adOpenStatic, adLockOptimistic, adCmdTable
' all records
.Open "SELECT * FROM Final_Table", cn, , , adCmdText
' filter records
For intColIndex = 0 To rs.Fields.count - 1 ' the field names
TargetRange.Offset(0, intColIndex).Value = rs.Fields(intColIndex).Name
Next
TargetRange.Offset(1, 0).CopyFromRecordset rs ' the recordset data
End With
rs.Close
Set rs = Nothing
cn.Close
Set cn = Nothing
End Sub
Sub Company_Information()
Dim companyName As String
On Error GoTo gotoError
companyName = Application.InputBox(Prompt:="Enter Company Name", _
Title:="Company Name", Type:=2)
Exit Sub 'Don't execute errorhandler at end of routine
gotoError:
MsgBox "An error has occurred"
End Sub
The above code works fine and pulls up the desired calculated table and places it in the right cells in Excel.
I've got two problems that I'm having trouble with; firstly I have some cell-formatting already done for the cells where the data is going to be pasted into in Excel; I want it to apply the formatting to the values as soon as they are pasted in Excel.
Secondly; I have an add-on for Excel which updates some daily Stock Market values; these values need to be transferred into Access at the end of each working day, to keep the database maintained, I tried some code but have been having some problems with it running.
The code for this part can be seen following:
Sub UPDATE()
Dim cnt As ADODB.Connection
Dim stSQL As String, stCon As String, DataLocation As String
Dim stSQL2 As String
'database path - currently same as this workbook
DataLocation = ThisWorkbook.Path & DataLocation
stCon = "Provider=Microsoft.Jet.OLEDB.4.0;" & _
"Data Source=" & DataLocation & ";"
'SQL code for GL Insert to Access
stSQL = "INSERT INTO Historical_Stock_Data SELECT * FROM [Portfolio] IN '" _
& ThisWorkbook.FullName & "' 'Excel 8.0;'"
'set connection variable
Set cnt = New ADODB.Connection
'open connection to Access db and run the SQL
With cnt
.Open stCon
.CursorLocation = adUseServer
.Execute (stSQL)
End With
'close connection
cnt.Close
'release object from memory
Set cnt = Nothing
End Sub
I get the following error with this.
Run-time Error '-2147467259 (80004005)'
The Microsoft Jet database engine cannot open the file 'Cocuments and Settings\Alice\Desktop\Database'. It is already opened exclusively by another user or you need permission to view its data.
I'm fairly new to databases, VBA and Access so any help would be greatly appreciated.
Also I have been told that the above method of having an Excel front-end and Access back-end is not recommended but alot of the analysis they conduct is done through Excel, and the charts feature in Excel is much better than Access in my experience atleast; and that is also one of the requirements for this project.
Thank you advance!
Solution to your first problem:
Sorry to be the bearer of bad news, but your entire first module is unnecessary. Instead, try:
Go to Data->Import External Data->Import Data, select your Access file, select your table, and presto! done!
Right-click on your new "External Data Range" to see a number of options, some related to formatting. You can even keep the original cell formatting and just update the values. I do this all the time.
To update the Excel data table later, there is a "External Data Range" toolbar that allows you to refresh it as well as a "refresh all" option to refresh every table in the Excel file. (You can also automate this thru code. It'll take some trial and error, but you're definitely up to the task)
Regarding your second problem
I've never used it, but there is also a "New Web Query" option in there as well. I assume it can be manipulated and updated the same way.
And lastly
Your choice of the Excel front-end and the Access back-end sounds good for your needs. It gets the data to your analysts in a medium they are familiar with (Excel) while keeping the calculations out of the way in Access. Technically, you could try putting all your calculations in Excel, but that might the Excel file much bigger and slower to open.
Do the data entry/updating/reviewing in Access. One of Access' strengths is using forms that allow you to update the tables without any code. Then allow the users to easily export the data to Excel such as by clicking on some command buttons.
Modules: Sample Excel Automation - cell by cell which is slow
Modules: Transferring Records to Excel with Automation
nothing wrong in principle with the excel/access pairing. I'm not familiar with ADO (I use DAO), but your error message seems to be indicating that the path to the datasource is not fully formed; or you already have it opened and hence are locking it.