Creating Dynamic Excel Connection in SSIS - excel

When I use a static ole db > excel connection manager in SSIS I set the file I want to connect to in the excel connection manager, create the data flow excel destination, hook it up to ole db source and map the fields.
In this case I want to do the same but instead of linking to an existing file I want to create a new file and put my data there. Now I know I can do a SQL task and create a new table (sheet effectively) in an existing excel connection but in this case I have made a script task to create the full path where my new file will go e.g. c:\docs\mypjt\fileformarch.xls with this code:
Public Sub Main()
'
' Add your code here
'
Dim strFileName As String
Dim strFolderName As String
Dim strMonthNameShort As String
Dim strFullPath As String
Dim lastMonth As New Date(DateTime.Today.Year, DateTime.Today.Month - 1, 1)
strFileName = Dts.Variables("FileName").Value.ToString
strFolderName = Dts.Variables("FolderName").Value.ToString
strMonthNameShort = MonthName(lastMonth.Month, True)
strFullPath = strFolderName & strFileName.Substring(0, strFileName.IndexOf(".xls")) & strMonthNameShort & ".xls"
Dts.Variables("FullPath").Value = strFullPath
Dts.TaskResult = Dts.Results.Success
End Sub
But I don't understand how to create my dynamic connection and mappings, I tried mucking about with connection strings in my excel connection manager etc to no avial so figure I am not getting the full picture of exactly what I need to do.
Not sure if I need a template with the column headings to set up a default mapping, but then I have read many cases where the excel connection manager only requires an initial connection to "a" file then the dynamic connectionstring proerty takes over when you run it, but then how does it know what the mappings are if you are creating a new file?
I am grateful for any advice you can give.
Thanks
Andrew

Related

Querying from another Excel or Access database

Hi I currently have two worksheets in an excel file with one of them acting as a database of all the products we sell, with the columns Product ID, Product Code, and Description (sample below).
I have another worksheet that acts as a product finder tool, where you would paste multiple Product IDs in the first column and it would return the Product code and Description in the adjacent columns (image below).
I currently use an INDEX search to make this happen, but the database sheet has become too big to manage in the same file, leading to severe slow downs. What would be the easiest solution for this? I was thinking of separating the database sheet as an Excel or AccessDB file but I think I will need a lot of VBA manipulation if I do that. Any help would be much appreciated.
You can access your data in Microsoft Access using ADO and doing a SQL query to gather data.
Could you tell me if it's possible to give a cell range to the WHERE clause?
Yes, there is a trick. SQL commands are plain text, you just need to build it with your parameters. Use the operator IN in the WHERE clause.
I made a fake dataset as example. Here's my Excel Product Finder (a table named Table1):
Notice I want the info only of products 6,3 and 2. Now my fake database:
The code to query those specific products:
Sub TEST()
Dim cnn As Object
Dim RST As Object
Dim DatabasePath As String
Dim i As Long
Dim Allid As String
Dim Arrayid As Variant
Dim SQLQuery As String
DatabasePath = "C:\Temp\temp.accdb" 'path to database
'Create a connection object.
Set cnn = CreateObject("ADODB.Connection")
'Create recordset object
Set RST = CreateObject("ADODB.Recordset")
'Open a connection using the OLE DB connection string.
cnn.Open "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & DatabasePath & ";Persist Security Info=False;"
'merge all ID into one single string
Arrayid = Range("Table1[PRODUCT ID]").Value
For i = LBound(Arrayid) To UBound(Arrayid) Step 1
Allid = Allid & Arrayid(i, 1) & ","
Next i
Allid = Left(Allid, Len(Allid) - 1) 'get rid of last comma
Erase Arrayid 'clean array variable
'specify query
SQLQuery = "SELECT PRODUCT_TABLE.[Product Id], PRODUCT_TABLE.[Product Code], PRODUCT_TABLE.Description FROM PRODUCT_TABLE " & _
"WHERE PRODUCT_TABLE.[Product Id] In (" & Allid & ") ORDER BY PRODUCT_TABLE.[Product Id]"
'Open a recordset using the Open method
'and use the connection established by the Connection object.
RST.Open SQLQuery, cnn
'copy all data into cells. This will bring full query without headers
Range("A6").CopyFromRecordset RST
'close and clean variables
RST.Close
cnn.Close
Set RST = Nothing
Set cnn = Nothing
End Sub
After executing code I get this:
NOTICE that the output is not sorted as we had before. We asked the products in order 6,3,2 but the output is 2,3,6!
This is because my SQL query got the operator ORDER BY that sorts by ID field. If there is no ORDER BY clause the output will be sorted as it is in the database stored, not as your Excel.
If you really really really need the output to be exactly in the same order that your Product Finder, you can create an UDF function to query each single id once and return a single row for each product but if you work with a lot of data this can consume a lot of time. So think carefully how to approach this part.
By the way, make sure you use the right connection string. You can find many on Access connection strings

Uploading data from Excel 365 to Azure Sql Server using VBA only

Although I used to be pretty advanced using VBA many moons ago, apparently that guy has left the building.
I just recovered some very large commodity and equity data sets I had in excel files from years ago. I have compiled them all together into one file and am trying to upload the data to my Azure SQL server. I cannot find ANYTHING anywhere on how to load data TO Azure, but I find many articles on loading FROM Azure.
Inside excel you can only create a connection to Azure SQL to IMPORT data into excel, one way street, no way to make a connection you can load from.
Any help appreciated. I can always use the old CSV's, read them in using c# in my web app, but I can update the data to current each week in this one file and I'd prefer to write the whole routine in VBA to get the data, check it, and load it to Azure.
Note: Yes, I know how to use ADODB connections but I cant find any drivers specific to using Azure SQL
Further update:
Using the following and I've tried 2 dozen variations, produces this error each time. And I installed the Sql Native Client Driver #12
Here is the function with the Driver, which I believ eis the issue, or the permissioning, because the azure sql db exists. This is the connection string from the portal
Public Function ExportExcelDataToAzureDb(wsSource As Worksheet) As Boolean
'I am using Activex Data Object 2.8 reference
Dim connectionString As String
Dim oConn As ADODB.Connection
Dim record As ADODB.Recordset
Dim cmd As ADODB.Command
connectionString = "Driver={SQL Server Native Client 12.0};Server=tcp:myazuresqlserver.database.windows.net,1433;Initial Catalog=myappdb;Persist Security Info=False;User ID=myuser;Password={mypass};MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"
Set oConn = New ADODB.Connection
oConn.connectionString = connectionString
oConn.Open
'load the data
ExportExcelDataToAzureDb = True
End Function
Obviously, I have substituted for the server, db, user and password
After a LOT of searching for various connection strings examples AND about 100 different tests of string variations.
Here is the proper connection string format to use with Excel VBA to import/export data from Azure SQL Database. I am providing my whole function for inserting data, with some parts removed for security concerns.
Option Explicit
Public dbConn As ADODB.Connection
Public adodbRecordSet As ADODB.recordSet
Public rsc As ADODB.recordSet
Public Const AzureDataSource As String = "tcp:yourazuredbservername.database.windows.net,1433"
Public Const AzureDbName As String = "yourazuredbname"
Public Const dbDriverProvider As String = "SQLNCLI11"
'I am using Activex Data Object 2.8 reference, SQL Native Client 11
Public Function ExportExcelDataToAzureDb(wsSource As Worksheet, dataRange As Range) As Boolean
Dim connectionString As String, sql As String
Dim cnt As Integer
Dim cCell As Range
connectionString = "Provider=" & dbDriverProvider & ";" & "Password=yourpassword;User ID=yourusername; " & _
"Initial Catalog=" & AzureDbName & ";" & _
"Data Source=" & AzureDataSource & ";"""
Set dbConn = New ADODB.Connection
Set adodbRecordSet = New ADODB.recordSet
dbConn.connectionString = connectionString
dbConn.Open
' do work here
cnt = 0
With wsSource
For Each cCell In dataRange
sql = "your insert statement here"
adodbRecordSet.Open sql, dbConn, adOpenDynamic, adLockReadOnly, adCmdText
cnt = cnt + 1
Next cCell
End With
'clean up
dbConn.Close
' log count of uploaded data records
Call PrintLog("Records loaded on " & Now & ": " & cnt & " symbol records")
ExportExcelDataToAzureDb = True
End Function
If anyone knows how to get this working with later versions of the ActiveX Data Object or the SQL Native Client Library please post another answer.
After trying many solution, i have come to the conclusion that the most important thing is
using Activex Data Object 2.8 reference, SQL Native Client 11. using the lower version seems to work best
for my connection string i am using what is below format
connection = "Driver={SQL Server Native Client 11.0};" & _
"Server=tcp:YourDataBase.net,1433;" & _
"Database=YourDatabaseName;" & _
"Uid=XXXX;" & _
"Pwd=XXXXXXX;" & _
"Encrypt=yes;Connection Timeout=30;"
Hope this help other people. This was super frustrating for me

How to export data from Excel to Access via Excel Macro when multiple instances of Excel are open

I am using an Excel macro to define a data range in Excel and then call "objAccess.DoCmd.TransferSpreadsheet" to import the data from this range into an Access table. This import does not work all of the time.
I have come to the conclusion that the macro works fine when there is only one instance of Excel open. However, the data import fails when another instance is already open. In the latter case the Access database opens up and the Excel file from which I run the macro is being reopened (in read-only mode) in the other Excel instance. There is no actual error but the desired import is not being carried out. Why does this happen?
Sub Excel_2_Access()
Dim strPath As String
Dim strwbPath As String
Dim strRange As String
Dim objAccess As Access.Application
Dim wbActive As Workbook
'get database path
strPath = Worksheets("error").Range("Access_DB_Path").Value & "\" & Worksheets("error").Range("Access_DB").Value
'open database
Set objAccess = New Access.Application
Call objAccess.OpenCurrentDatabase(strPath)
objAccess.Visible = True
'access import
Worksheets("error").Columns("P:P").Calculate
Set wbActive = ActiveWorkbook
strwbPath = Application.ActiveWorkbook.FullName
strRange = "error!M2:M" & (Worksheets("error").Range("WKN_count").Value + 2)
Call objAccess.DoCmd.TransferSpreadsheet(acImport, 8, "WKN_Mapping", strwbPath, True, strRange)
objAccess.Forms("MX_Import").Refresh
End Sub
As the macro is fairly short I have included the entire code for your reference. However, I don't think the way the range is specified or names are provided is really relevant to the question.
The desired outcome would be to have an Excel macro in place that carries out the transfer from Excel to Access no matter if there are other instances of Excel open or not.
Is there such a thing as the primary instance of Excel (the first one that was opened) which has a special status? Is there a way to provide the specific Excel instance the workbook is in when calling the Access function from Excel? Or is there a more reliable way to transfer the data which generally avoids this problem with multiple instances?
I tested approach that opens Access db and runs TransferSreadsheet. Don't need to set a workbook object (your code sets but then doesn't even utilize). It ran without error every time. I tried setting the Access object Visible but the database appears and immediately closes anyway, although the data import does happen. Set reference libarary: Microsoft Access x.x Object Library.
Sub test()
Dim ac As Access.Application, strRange As String
Set ac = New Access.Application
strRange = "Sheet1!A1:E3"
ac.OpenCurrentDatabase "C:\Users\June\LL\Umpires.accdb"
ac.DoCmd.TransferSpreadsheet acImport, acSpreadsheetTypeExcel12Xml, "Rates", ThisWorkbook.FullName, True, strRange
End Sub
Example Excel VBA code that exports all rows of worksheet to existing table in Access without opening Access file. Setting an ADODB connection makes the Execute method available. This approach runs faster. Set reference library: Microsoft ActiveX Data Objects x.x Library.
Sub test()
Dim cn As ADODB.Connection
Set cn = New ADODB.Connection
cn.Open "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & ThisWorkbook.FullName & ";Extended Properties=Excel 8.0"
cn.Execute "INSERT INTO Rates(RateLevel, Rate, Pos, EffDate) IN 'C:\Users\June\LL\Umpires.accdb' " & _
"SELECT RateLevel,Rate,Pos,EffDate FROM [Sheet1$];"
cn.Close
Set cn = Nothing
End Sub

linking an access database to excel (without importing the whole table)

Is there any way to link tables in an Access database to Excel without importing the entire table? I need to reference/lookup cells in the Access table but don't want to import the whole table into the excel workbook (tables are too big).
My second option is to export the Access tables into a separate excel workbook, then just reference this new workbook instead of the Access database itself. When I try to do this only around 65,000 rows of data from any Access table actually export to Excel, as the rest 'couldn't be copied to the clipboard'. Is there a simple way around this? (I want to actually have a connection between the excel/access files, so the data can be refreshed, not just copy and paste the rows over)
See this ancient article, which should help you get the data you actually need rather than everything:
http://dailydoseofexcel.com/archives/2004/12/13/parameters-in-excel-external-data-queries/
You can work with Access database tables from Excel without importing the table into an Excel worksheet:
Dim cnn As ADODB.Connection ' Needs a reference to the Microsoft ActiveX
Dim rs As ADODB.Recordset ' Data Objects Library
Set cnn = CreateObject("ADODB.Connection")
cnn.Open "DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=database-path\Data.accdb;"
Set rs = cnn.Execute("SELECT * FROM MyTable")
While Not rs.EOF
Debug.Print rs(1)
rs.MoveNext
Wend
You may need the Microsoft Access Database Engine 2010 Redistributable which you should install with the /passive option if you are using a x86 Access version on a x64 OS.
Try the following script. This should give you what you want.
Option Compare Database
Private Sub Command0_Click()
Dim InputFile As String
Dim InputPath As String
InputPath = "C:\your_path_here\Desktop\"
InputFile = Dir(InputPath & "*.xlsx")
Do While InputFile <> ""
DoCmd.TransferSpreadsheet acLink, , InputFile, InputPath & InputFile, True '< The true is for column headers
InputFile = Dir
Loop
End Sub

Export Access database with multiple tables to Excel with multiple sheets

The title says it about all. I'm trying to write a VBA script that would allow me to run inside Access and it would export all database tables as separate sheets into and Excel file with the same name as the database:
Sub exportTablesToXLS()
Dim td As DAO.TableDef, db As DAO.Database
Dim out_file As String
out_file = CurrentProject.Path & "\" & db.DatabaseName & ".xls"
Set db = CurrentDb()
For Each td In db.TableDefs
If Left(td.Name, 4) = "MSys" Then
'We do not need MSys tables in excel file
Else
DoCmd.TransferSpreadsheet acExport, acSpreadsheetTypeExcel9, _
td.Name, out_file, True, Replace(td.Name, "dbo_", "") 'We do not need dbo prefix in sheetnames
End If
Next
End Sub
Problems I'm having I would like your help on:
see line out_file -> db.DatabaseName return an error. How can I correctly get the database name of the current Access database?
I want to output a logfile (simple textfile) as well. How can I read, for each database table, the number of rows that have been exported and report eventual errors that occured?
Any help to improve this script is greatly appreciated :-)
Your db variable doesn't refer to the current database (yet - it is set in the following line), and the property is Name (I haven't encountered DatabaseName):
Sub Test()
Dim db As DAO.Database
Set db = CurrentDb
Dim sLast As String
MsgBox db.Name
'F:\Documents and Settings\student\My Documents\Staff Database.accdb
sLast = InStrRev(db.Name, "\")
MsgBox Right(db.Name, Len(db.Name) - sLast)
'Staff Database.accdb
End Sub
Name gives the full path and filename, the second MsgBox reduces this to just the filename.
(There may be another way to get the filename without having to parse Name..)
To get the number of rows exported you could open a Recordset for the table(s), reading the RecordCount property. A text file could be created using the FileSystemObject object. Sample code:
Set fso = CreateObject("Scripting.FileSystemObject")
Set oFile = FSO.CreateTextFile(strPath)
oFile.WriteLine "test"
oFile.Close
Or even just simple file (VBA) I/O:
Open pathname For mode [Access access] [lock] As [#]filenumber [Len=reclength]
(the FileSystemObject is easier to work with)
To report errors you'll need to create your own error-handling routines, and an error-logging procedure. This has been done before and a little searching will uncover some code. Essentially, each (important) event has error-handling code that calls a logging procedure. This procedure creates a record in a table to store all relevant information: date, time, which form was open, etc., etc.
A quick Google uncovered this page about logging Access errors.

Resources