I created 3 different connection managers in an SSIS package, say connA connB and ConnC. How can I select the connection to use in an ado.net source using a parameter?
Thank you in advance
Carlo
The way that I would go about it is to wrap your logic in a Script Task with a simple IF Else logic
Return your source via SQLDataAdapter and store in an object Variable.
This variable is then available as a table to read from in in any part of your package. Assuming that you assign the scope of the variable correctly.
Related
I have certain expressions used by multiple steps in Azure Data Factory.
For instance, I get the start time of a pipeline often:
#formatDateTime(pipeline().TriggerTime, 'yyyy-MM-dd HH:mm:ss.fff')
If I need to change it for some reason, I'd need to update it everywhere.
I know about global parameters; however, is there a way to create global expressions?
Based on my knowledge, currently there is no way to have a global expression similar to Global parameters.
For you have to have a common source of truth or 1 common expression, I would suggest you to create a common pipeline in the ADF and call that pipeline via execute pipeline activity to avoid re writing the same code every where.
You could add a variable to the pipeline and set it to the appropriate value at the start of your pipeline run: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-set-variable-activity
You can then refer to this variable as you do with parameters or change its value on the go.
I'm trying to use the Power Query source component in a generic way from SSIS (VS2019).
The idea would be to use a for each loop to load and transform Excel files. At run time, I need to set the connection manager properties for each file as well as the PQY script to be executed on the file.
What I did so far is trying to create a JSON connection string inside a script component and assign the connection string to the connection manager. It keeps on saying that the file requires credentials.
Would someone already experienced that kind of dev? All the files do have the same structure so far, do meta-data need to be refreshed too?
[Edit]
1. In the control flow, I'm retrieving the PQY script I want to apply from a DB.
Before transormations, script starts like this:
let Source = Excel.Workbook(File.Contents("path_to_a_file.xlsx"),null,true),RawData_Sheet = Source{[Item="Table1",Kind="Table"]}[Data]..."
In the C# script task, I'm replacing the path to excel file by the current file variable. M Script is stored in a variable used in the PQY component.
C# Script is then updating the PQY connection manager to target the appropriate file:
ConnectionManager _conn = Dts.Connections["Power Query Connection Manager"];
String _ConnectString = "[{kind:File,path:path_to_a_file.xlss,AuthenticationKind:Windows,Username:myusername,Password:mypassword}]";
_conn.ConnectionString = _ConnectString;
The PQY component is left has it is, connected to ["Power Query Connection Manager"] and getting its script from the variable I set.
PQY configuration screen
Thanks for any tip on this,
Olivier
I can't address the specifics of the PQ but generic anything in a Data Flow will not work.
The Data Flow task works because it makes a strict contract between the source(s) and the destination(s). These columns with these data types will be in play during the run. It's a design-time contract because that allows the run-time engine to allocate resources based on how many buffers of data the system can support. Each row is X bytes, we have Y bytes of memory available, so Z buffers worth of data plus parallelism stuff.
Wish I had a better story to tell you.
I have (several) database project in a solution. In one I have a referesnce to a dacpac (this is ACTUALLY a copy of one of the main databases as we take a SQL snapshot at end of day and some code needs to reference this (DBANME_Daily) rather that DBNAME).
now this builds correctly, the code with SELECT * FROM DBNAME_DAILY.schema.table all compiles and builds with no error.
ON deployment however I get the unresolved reference to DBNAME_DAILY.schema.table
You want to add a dacpac for that database as a reference, using variables for the database. Should be different database, different names. You'd then use that variable in your code and you'd pass in the variable name for your build/publish tasks depending on the environment.
This is a little older article, but still pretty accurate:
http://schottsql.com/2012/10/31/ssdt-external-database-references/
You can tweak that a little bit using a variable for the database name. When I wrote this, it was mostly just a different database with the same name across environments. For your case, you'd just use the DB variable. Then replace your "DBName.schema." with "[$DBNameVariable].schema." (or something similar)
Sorted, my mistake was the dacpac reference would only allow the project to BUILD. For deployment the database DBNAME_DAILY MUST EXIST.
Lesson Learnt.
I have a SSIS package that needs to enumerate Excel files in Sharepoint. When I setup the For Each loop container at the package level and then set up the file path in my Excel Source at the task level everything is fine. When I add my expression variable to the Excel Connection Manager properties I lose the connection and can't view my input table anymore. I have "Delay Validation" set to true on the Excel Connection Manager.
Has anybody experienced this?
EDIT:
The following are screenshots of my ForEach Loop Container configuration:
The following is a screenshot of where I am passing the variable. It is in the Expression box of the Excel Connection Manager:
Do you have a default value set for the variable to point to a sample file? Even though you have set the Delay validation to be True it just means that when the package runs it would not do a validation on the connection. It does not mean that while you are working on the package it would not use the variable to translate to the connection. So you should go ahead and set the default value of your connection variable to the path of the file you are working on.
I had set the DelayValidation at the connection level, at the dataflow level, and at the for-each loop in control flow level but none were working. I had spent 3-4 hours until I found this thread and setting a default value of the variable finally worked.
So setting a default value of the connection variable does solve the problem if you have a similar issue.
We have an excel file which contains a connection to a database to retreive data (with a select statement).
We want to update via a (preferrably powershell) script the connection string of that file to make it query another server instead.
So for exemple :
I have report.xlsx file which connects to server A.
I run update-connection.ps1
And when I open report.xlsx it now connects to server B.
Any idea how we could do that?
Thanks.
It should be fairly easy if you decide (are allowed) to store the connection (server name) in a worksheet. Your VBA code can dynamically build the connection string based on the value of a cell. (I would probably create a named range and use it in the code).
I don't know PowerShell but the code can look something like:
$workbook.Range("Server").Value2 = "PROD_01"
You can make the worksheet hidden if you wish, but it is not a serious security.
You could try automating Excel via PowerShell, as in this article: http://kentfinkle.com/PowershellAndExcel.aspx
If you don't want to automate Excel then you could try using something like ClosedXML in your PowerShell script: http://closedxml.codeplex.com/
You can parse the connectionstring with System.Data.Common.DbConnectionStringBuilder. Check this SO thread:
Powershell regex for connectionStrings?