I'm trying to use ADF to create azure table storage tables from one source SQL table.
Within my pipeline..
I can query a distinct list of customers pass this into a for-each
task.
Inside the for each select the data for each customer
But when I try to to create an Azure table for each customers data with the table name based on the customer ID I hit errors.
The customer ID is a GUID so I'm trying to format this to remove the dashes which are invalid in a table name...
Something along the lines of
#replace('#{item().orgid}','-','')
So 7fb6d90f-2cc0-40f5-b4d0-00c82b9935c4 becomes 7fb6d90f2cc040f5b4d000c82b9935c4
I can't seem to get the syntax right
Any ideas?
try this: #replace(item().orgid,'-','').
Related
I'm working on an ETL pipeline in Azure Synapse.
In the previous version I used an Array set as a parameter of the pipeline and it contained JSON objects. For example:
[{"source":{"table":"Address"},"destination {"filename:"Address.parquet"},"source_system":"SQL","loadtype":"full"}}]
This was later used as the item() and I used a ForEach, switch, ifs and nested pipelines to process all the tables. I just passed down the item parameters to the sub pipelines and it worked fine.
My task is now to create a dedicated SQL pool and a table which stores parameters as columns. The columns are: source_table, destination_file, source_system and loadtype.
Example:
source_table
destination_file
source_system
loadtype
"Address"
"Address.parquet"
"SQL"
"full"
I don't know how to use this table in the ForEach activity and how to process the tables this way since this is not an Array.
What I've done so far:
I created the dedicated SQL pool and the following stored procedures:
create_parameters_table
insert_parameters
get_parameters
The get_parameters is an SQL SELECT statement but I don't know how to convert it in a way that could be used in the ForEach activity.
CREATE PROCEDURE get_parameters AS BEGIN SELECT source_table, destination_filename, source_system, load_type FROM parameters END
All these procedures are called in the pipeline as SQL pool stored procedure. I don't know how to loop through the tables. I need to have every row as one table or one object like in the Array.
Take a lookup activity in Azure data factory/ Synapse pipeline and in source dataset of lookup activity, take the table that has the required parameter details.
Make sure to uncheck the first row only check box.
Then take the for-each activity and connect it with lookup activity.
In settings of for-each activity, click add dynamic content in items and type
#activity('Lookup1').output.value
Then you can add other activities like switch/if inside the for-each activity.
I have been using the Azure cognitive search to create the search index where the data source will be the azure SQL database.
I have created the search index for azure cognitive search and for the key index will be the id column from the table and the datatype will be fixed as edm.string type.
I have seen that we can change the datatype of id column to edm.int32 using the field mapping.
Can anyone help me with how we can able to create the field mapping for id column by changing the datatype to edm.int32
Id column must be string. Here's the statement from the official doc:
Identify a document key. A document key is an index requirement. It's
a single string field and it will be populated from a source data
field that contains unique values. For example, if you're indexing
from Blob Storage, the metadata storage path is often used as the
document key because it uniquely identifies each blob in the
container.
https://learn.microsoft.com/en-us/azure/search/search-how-to-create-search-index?tabs=portal#schema-checklist
You can create a T-SQL view and cast your id from int to varchar over there.
I'm trying to run a pipeline that results a select with where be the system triggername system output.
I've tried to use :
#concat(concat(concat(concat('select * from ',item().DB_CARGA,'.',item().SC_CARGA,'.',item().tb_carga, 'where ' ,item().DS_CARGA ,'=', string(pipeline().TriggerName)))))
But I'm getting the following error:
Could anyone help me with the right syntax?
I reproduced this and got similar error when I used your syntax.
In your dynamic expression, there should be a space between table name and where and also trigger name should be enclosed in single quotes.
Please go through the below procedure to resolve the error.
First, I have a lookup activity which will give database name, table name and SQL table column(trigger column name). Give this to Foreach.
I have created sample tables with trigger_column column and gave some values.
In pipeline I have created trigger with same name mytrigger. Inside Foreach for my demo I have used lookup query. You can use your activity as per the requirement.
Dynamic content:
You can do this with single concat function.
#concat('select * from ',item().database,'.',item().table_name, ' where ' ,item().trigger_name ,'=','''',string(pipeline().TriggerName),'''')
This will give the SQL query like this when Executed.
Output when triggered:
I want list orphaned disk on my Azure but in the result I obtain this message :
'where' operator: Failed to resolve table or column expression named 'Resources'
I have taken the kql from internet and everybody use Resource has column to get result , I want to know why in my azure there is this error;
The column has the name changed ?
Thanks
Your workspace (or database) does not have a table or function called "Resources", you need to check the table list in the connection pane and see the tables you have, here is an example:
I am using Azure Data Factory (ADF) to load data from Dynamics 365 to an Azure Database. However, for some columns I get a GUID value like 8234CCBA-7B01-E211-B551-D48564518CCA instead of the actual value.
Is there a way to retrieve the real data either by using SQL or changing something in Dynamics?
Normally we will use web api formatted values or CRM SDK methods to get the display name field of related table (Foreign key) in C# or javascript. In ADF, I don't think you can fetch the formatted values for picklist (Integer key values) or lookup (GUID) attributes.
Instead you can use ADF data flow to fetch the different datasets and join them for these kind of data transformations. Read more
Otherwise, dump all the datasets you need into Azure DB - for transformations later in DB views.