I am trying to import data from a CSV file into a Dynamics 365 Account table. As I need to do some transformations I am using a dataflow rather than a basic copy activity.
I was having difficulties getting it to work using a dataflow for writing to a multi lookup field so I tried using a copy activity to see if that worked using the exact same source,sink and mappings. I was able to import the
data successfully with the copy activity. I'm confused as to why the data flow does not work using the same source,sink and mappings. Below are screenshots of the various elements I set up and configured. Would appreciate any suggestions to get the dataflow working.
I'm using a cut down version of what will ultimately be my source CSV file. This is just so I can concentrate on getting the writing to the lookup field working.
Source CSV file
Copy Activity Source
Copy Activity Sink
Dynamics 365 Sink
Dataflow Source
Dataflow Sink
Copy Activity Mapping
Dataflow Mapping
Copy Activity Success
Dataflow Failure
Dataflow Error
Details
{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: DF-REST_001 - Rest - Error response received from the server (url:https://##############v9.0/accounts,request body: Some({"accountid":"8b0257ea-de19-4aaa-9945-############","name":"A User","ownerid":"7d64133b-daa8-eb11-9442-############","ownerid#EntityReference":"systemuser"}), request method: POST, status code: 400), response body: Some({"error":{"code":"0x0","message":"An error occurred while validating input parameters: Microsoft.OData.ODataException: A 'PrimitiveValue' node with non-null value was found when trying to read the value of the property 'ownerid'; however, a 'StartArray' node, a 'StartObject' node, or a 'PrimitiveValue' node with null value was expected.\r\n at Microsoft.OData.JsonLight.ODataJsonLightPropertyAndValueDeserializer.ValidateExpandedNestedResourceInfoPropertyValue(IJsonReader jsonReader, Nullable1 isCollection, String propertyName, IEdmTypeReference typeReference)\r\n at Microsoft.OData.JsonLight.ODataJsonLightResourceDeserializ","Details":"com.microsoft.dataflow.Issues: DF-REST_001 - Rest - Error response received from the server (url:https://dev-gc.crm11.dynamics.com/api/data/v9.0/accounts,request body: Some({"accountid":"8b0257ea-de19-4aaa-9945-############","name":"A User","ownerid":"7d64133b-daa8-eb11-9442-############","ownerid#EntityReference":"systemuser"}), request method: POST, status code: 400), response body: Some({"error":{"code":"0x0","message":"An error occurred while validating input parameters: Microsoft.OData.ODataException: A 'PrimitiveValue' node with non-null value was found when trying to read the value of the property 'ownerid'; however, a 'StartArray' node, a 'StartObject' node, or a 'PrimitiveValue' node with null value was expected.\r\n at Microsoft.OData.JsonLight.ODataJsonLightPropertyAndValueDeserializer.ValidateExpandedNestedResourceInfoPropertyValue(IJsonReader jsonReader, Nullable1 isCollection, String propertyName, IEdmTypeReference typeReference)\r\n at Microsoft.OData.JsonLight.ODataJsonLightResourceDeser"}
I am running into a same wall,
but a temporary solution here is to sink the dataflow output to a Csv/or similar file into ADLS and then use a Copy activity to extract those files and Upsert it into the Dynamics.
Other references: https://vishalgrade.com/2020/10/01/how-to-populate-multi-lookup-attribute-in-ce-using-azure-data-factory/
Related
Sorry if this is a bit vague or rambly, I'm still getting to grips with Data Factory and a lot of it seems a bit obtuse...
What I want to do is query my Cosmos Database for a list of Ids of records that need to be updated. For each of these records, I want to call a REST API using the Id (i.e. /Record/{Id}/Details)
I've created a Data Flow that took a string as a parameter and then called the REST API fine.
I then made a pipeline using a Lookup with a query (select c.RecordId from c where...) and pass that into a ForEach with items set to #activity('Lookup1').output.value
I then setup the Activity of the ForEach to my Data flow. From research, I think I'm supposed to set the Parameter value to "#item().RecordId", but that gives an error "parameter [name] does not match parameter type 'string'".
I can change the type of the parameter to any (and use toString([parameter]) to cast it ) and then when I try and debug it passes the parameter in, but it gives an error of "Job failed due to reason: at (Line 2/Col 14): Datatype any not found".
I'm not sure what the solution is. Is there a way to cast the result of the lookup to an integer or string? Is there a way to narrow an any down? Is there a better way than toString() that would work? Is there a better way than ForEach?
I tried to reproduce similar scenario what you are trying.
My sample data in cosmos
To query Cosmos Database for a list of Ids and call a REST API using the Id For each of these records.
First, I took Lookup activity in data factory and selected the id's where the last_name is Bluth
Its output and settings are as below:
Then I passed the output of lookup activity to For-each activity.
Then inside for each activity I created Dataflow activity and for that DataSource I gave the source as Rest API. My Rest API to call specific user is https://reqres.in/api/users/2 I gave base URL as https://reqres.in/api/users.
Then I created parameter called demoId as datatype string and in relative URL I gave that dynamic value as #dataset().demoId
After this I gave value source parameter as #item().id as after https://reqres.in/api/users there is only id should be provided to get data in you case you can try Record/#{item().id}/Details.
For each id it is successfully passing id to rest API and fetching data:
I am trying to set up a data pipeline in ADF. I can't seem to figure out why an API call works as a source in the "copy data" block in the pipeline, but when I try to use it as a source in the data flow it throws an error.
This is the request in the copy data block: https://i.stack.imgur.com/9wzSZ.png'
This is the request in the data flow tab: : https://i.stack.imgur.com/Onpuk.png
And this is the error I get: https://i.stack.imgur.com/f9zdb.png
The request body should be the same. Still I get a good response when I preview the data in the copy block: https://i.stack.imgur.com/cKZsb.png
Is there some setting that is different that I am missing?
EDIT: the url for the request is https://www.daggegevens.knmi.nl/klimatologie/uurgegevens with the request body start=2022010101&end=2022090724&stns=330&vars=DD:FH:FF:FX&fmt=json
I have reproduced same error in Dataflow.
I tried to give Request body in query parameters as in below image
Response of this API had header and body in each line item.
Thus, select transformation is used to select only the body response and copied it to the sink
Outputs that I received from both copy activity and data flow activity are same.
copy activity output
data flow activity output
Using a lookup activity in ADF to get list of tables that I want to output to Databricks notebook which will be used to run the code.
For Loop Object dynamic content #activity('Lookup IngestionControl').output.value
The error I'm getting is
The value type in key 'TABLENAME' is not expected type 'System.String'
Attempted Solution: #String(activity('Lookup IngestionControl').output.value)
Warning: Expression of type: 'String' does not match the field: 'items'
Ran it with the warning and get an error because the object is type array and cannot be converted to a string
You can only pass a string into the databricks API. ADF uses the databricks jobs API when it calls a notebook / jar.
https://docs.databricks.com/dev-tools/api/latest/jobs.html
What i usually do is convert the array into a json string. Can do this in SQL or in ADF doesnt really matter. Depending on which one you are trying to do would change how i would do it.
#activity('Lookup IngestionControl').output.value tells me its a Lookup activity. I would just create the json from sql and pass it through ADF and into your notebook.
I am trying to extract the data from SAP Solution manager using SAP Table connector in ADF.
It is working for some table and throwing a below error few tables.
Could someone please help me with any lead on how to resolve it.
and also my objective is extract the data from SAP solution manager using ADF tool.
Failure happened on 'Source' side.
ErrorCode=UserErrorRfcFunctionInvokeFailed,
'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Failed to invoke function /SAPDS/RFC_READ_TABLE2 with error:
SAP.Middleware.Connector.RfcAbapRuntimeException,
message: Error with ASSIGN ... CASTING in program /SAPDS/SAPLRS_BASIS .
SAP.Middleware.Connector.RfcConnection.ThrowRfcErrorMsg()
at SAP.Middleware.Connector.RfcConnection.ReadBytes(Byte* buffer, Int32 count)
at SAP.Middleware.Connector.RfcConnection.ReadRfcIDBegin(Int32& length)
at SAP.Middleware.Connector.RfcConnection.ReadUpTo(RFCGET readState, RfcFunction function, RFCID toRid)
at SAP.Middleware.Connector.RfcConnection.RfcReceive(RfcFunction function)
...
...
...
I have an Azure data factory v2 pipeline containing an until activity.
Inside the until is a copy activity - if this fails, the error is logged, exactly as in this post, and I want the loop to continue.
Azure Data Factory Pipeline 'On Failure'
Although the inner copy activity’s error is handled, the until activity is deemed to have failed because an inner activity has failed.
Is there any way to configure the until activity to continue when an inner activity fails?
Solution
Put the error-handling steps in their own pipeline and run them from an ExecutePipeline activity. You'll need to pass-in all the parameters required from the outer pipeline.
You can then use the completion (blue) dependency from the ExecutePipeline (rather than success (green)) so the outer pipeline continues to run despite the inner error.
Note that if you want the outer to know what happened in the inner then there is currently no way to pass data out of the ExecutePipeline to its parent (https://feedback.azure.com/forums/270578-data-factory/suggestions/38690032-add-ability-to-customize-output-fields-from-execut).
To solve this, use an sp activity inside the ExecutePipeline to write data to a SQL table, identified with the pipeline run id. This can be referenced inside the pipeline with #pipeline().RunId.
Then outside the pipeline you can do a lookup in the SQL table, using the run ID to get the right row.
HEALTH WARNING:
For some weird reason, the output of ExecutePipeline is returned not as a JSON object but as a string. So if you try to select a property of output like this #activity('ExecutePipelineActivityName').output.something then you get this error:
Property selection is not supported on values of type 'String'
So, to get the ExecutePipeine's run ID from outside you need:
#json(activity('ExecutePipelineActivityName').output).pipelineRunId
I couldn't find this documented in Microsoft's documentation anywhere, hence posting gory details here.