I want to create a dynamic source dataset for copy activity based on parameters or variables.
I tried to edit referenceName on JSON but it's not working.
Creating Linked Service dynamically by passing the parameters.
Parameters were provided while creating the linked service to pass dynamically.
Creating new Dataset with linked service which we created dynamically.
Needed values for the parameters to load the tables while creating the Dataset dynamically.
Result with expected dataset created dynamically.
We can also use dynamic content for tables in the dataset by providing parameters.
Related
While trying to build an ADF pipeline that generates datasets within Data Factory, I ran into an interesting issue. Or maybe I misunderstand some components completely, in which case I'd happily be educated.
I basically read some meta data from a SQL Database table which determines which source system, schema and tables I should pull new data from. The meta data is stored within a bunch of variables, which then feed a Web Request that attempts to generate a new Data Source as per the MS documentation. Yes, I'm trying to use Azure Data Factory to generate Azure Data Factory components.
The URL to create the DataSet and the JSON Body for the request are both generated using #Concat and a number of the variables. The resulting DataSet is a very straightforward file that does not contain references to the columns, but just the table schema and table name. I generated these manually before, and that all seems to work brilliantly. I basically have a dataset connected to the source system, referincing the table from the meta data.
The code runs, but the resulting dataset is directly published, as opposed to being added in my working branch. While this should not be a big issue once I manage to properly test everything, ideally the object would be created in my working branch (using Azure DevOps, thus a local file).
My next thought was to set up a linked service to my local PC, and simply write the same contents as above there. My challenge seems to be that I essentially am creating a file out of nothing. I am trying to use a Copy Data component, and added an empty placeholder file to act as a source.
I configure the sink with Dynamic Content for Copy Behavior, and attempted to add the JSON contents there. This gets the file created, but it's unfortunately empty. I also attempted to add a new column to the source with the data being the same contents.
However, seeing the file to be used as a sink doesn't exist, a mapping error will occur. Apart from this, I'd not want a column header to be written; just the dynamically created contents.
I'm not sure how to continue with this. I feel I'm very close to achieving my goal, but cannot seem to take this final hurdle.
Any hints or suggestions would be very welcome.
I have created a parameter in the linked service of the oracle database connection and would like to get the value of the parameter to all datasets, I have created datasets for each table and would like to get oracleSchemeName parameter from the Linked service, In my scenario schema name will change based on environments, SO I passed schema name as parameter but when I'm trying to read the value of linked service, I am getting error table or view doesn't exist.
Can someone please guide me on how to provide expression here? {#linkedService().oracleSchemaName} --Not working.
Below is the screenshot of dataset
{#linkedService().SchemaName} is invalid as schema name is not part of the linked service connection string. You can parameterize the properties which are part of the linked service connection string properties.
Example:
If you want to pass the schema/table name dynamically, you can create a pipeline level parameter and pass the values in run time.
Steps to pass the table schema value dynamically at runtime:
Create a parameter at database level (SchemaName) and do not pass any value to it currently.
Edit the table and pass dataset parameter in add dynamic content.
Create a parameter at pipeline level and add the parameter dynamically in sink properties and do not pass any value to the parameter currently.
While running the pipeline, it will ask to pass value to the pipeline parameter, then pass the value and run the pipeline.
Reference: Parameterize linked services
I'm working with ADF and trying to leverage parameters to make life easier and reduce the number of objects being created in the ADF itself. What I am trying to do, would appear on the surface to be extremely simple, bu in reality its driving me slowly crazy. Would greatly appreciate any assistance!
I am trying to set up a parameterised dataset to be used as a sink target. Inside that dataset I have added a param named "filenames" of type string. In the connection tab I have added that param to the file part of the path. The folder part point to my Azure Data Lake folder and the file part is set to: #dataset().filename which is the result of choosing 'dynamic content' then selecting the param.
So far so good.. my sink target is, as far as I am aware, ready to receive "filenames" to write out to.
This is where it all goes wrong.
I now create a new pipeline. I want to use a list or array of values inside that pipeline which represent the names of the files I want to process. I have been told that I'll need a Foreach to send each of the values one at a time to the COPY DATA task behind the Foreach. I am no stranger to Foreach type loops and behaviors.. but for the life of me I CANNOT see where to set up the list of filenames. I can create a param as a type "array" but how the heck do you populate it?
I have another use case which this problem is preventing me from completing. This use case is, I think, the same problem but perhaps serves to explain the situation more clearly. It goes like this:
I have a linked service to a remote database. I need to copy data from that database (around 12 tables) into the data lake. At the moment I have about 12 "COPY DATA" actions linked together - which is ridiculous. I want to use a Foreach loop to copy the data from source to data lake one after the other. Again, I can set up the sink dataset to be parameterised, just fine... but how the heck do I create the array/list of table names in the pipeline to pass to the sink dataset?
I add the Foreach and inside the foreach a "COPY DATA" but where do I add all the table names?
Would be very grateful for any assistance. THANK YOU.
If you want to manually populate values of an array as a pipeline parameter, you create the parameter with Array type and set the value with syntax like: ["File1","File2","File3"]
You then iterate that array using a ForEach activity.
Inside the ForEach, you reference #item() to get the current file name value the loop is on.
You can also use a Lookup activity to get data from elsewhere and iterate over that using the ForEach.
I am new to the ADF.
While I am trying to use Copy activity for moving data from API Call output to Blob Json, I am unable to use Lookup output. I am trying to map the fields explicitly in Mapping using #item().SiteID. But JSON output returns only with input fields (not the derived fields). Can someone help me to let me know how to achieve this?
Can I use Copy activity in For Each activity (#activity('LookupAvailableChannelListForExport').output.value) to pass Lookup output value (#item().siteID)in mapping between source and sink?
As i know, the output of Look Up Activity can't be source data in copy activity,even mapping between source and sink. Acutally, Look Up activity prefers the following usage according to official document:
Dynamically determine which objects to operate on in a subsequent
activity, instead of hard coding the object name. Some object examples
are files and tables.
I think the example from above link is a good interpretation.You could see that the output of Look Up activity is configured as dynamic sql db source dataset table name.Not the data in source.
Then back to your requirement,i think you could configure the source dataset as root folder if the files are stored in the same directory with same schema. And keep this option is selected so that all the data in all files will be grabbed.
If you want to implement some variant of source data, copy activity can't cover it but data flow activity could.You could use Derived column.Such as resetting the Json structure.
My question is pretty simple. I am working on Kentico 9 with its SQL Server database which contains several tables which had been added directly from the SQL Management Studio by an external contractor. The fact is that those tables are being used to store custom content which will be displayed for a site, but, in the code they don't have the code for making queries. I mean, they don't have Info and Provider classes.
https://docs.kentico.com/display/K82/Retrieving+database+data+using+ObjectQuery+API
According with this, all tables into the Kentico database can be accessed by invoking methods on these classes, but I don't have it this time.
Something like this, it will not work if I use my table name:
var user = UserInfoProvider.GetUserInfo("administrator");
var items = CustomTableItemProvider.GetItems("MyTable")
.TopN(10)
.WhereEquals("ItemCreatedBy", user.UserID)
.OrderBy("ItemCreatedWhen");
My question is:
can I query any table by its name?
One last thing:
I cannot declared those table as "custom table" because it seems to be a bug in the CMS.
Or you can pull data using your own SQL query:
var ds = ConnectionHelper.ExecuteQuery("select ....", null, QueryTypeEnum.SQLQuery);
Nevertheless I would recommend to create a custom class inside a custom module (much more robust than custom tables) instead and use the generated Info and InfoProvider classes to get and manipulate data.
I think an object has to be registered within the system (created through Kentico UI or API) in order to be pulled from DB with object query.
So I'd choose one of the following options:
Use Entity Framework or something similar to work with that data
Create appropriate custom tables or even custom module and push data there. Not sure why you can't create a custom table... What is an error you're getting?
If you need to present data on the UI only (without processing on the back end) - use just custom queries
Hope this helps.
If you are accessing in code then you could do it the good old fashioned way. If you want to pull data from the database to display on the website you could also do so by creating a custom query and using a transformation to display the fields, then use a repeater on the page to display the transformed data. Alternatively you can use a SQL datasource with a basic repeater, but you still have to create a transformation to display the data. Both methods allow you to access the data in the tables from within the CMS UI, no need to touch any code behind.
If your objective is to read data from these database tables to transform on webpage e.g. using CMS Repeater webpart, you can simply create custom query(s) in Kentico itself and load data using it. You can find the detail here on how to create custom custom queries and load data using it.
On the other hand you can also write your custom classes and define the custom methods where you can pull data using your own SQL query like this:
var ds = ConnectionHelper.ExecuteQuery("select ....", null, QueryTypeEnum.SQLQuery);
Lastly I don't think there should be any issue to create custom table instead of those direct DB tables, only thing we have to ensure code name of custom table should be unique means don't try to use exact same name because it'll cause exception due to same table name already exist in DB. You can please share exception you getting while creating custom table so that I can help you out further.