I am trying to copy data from Office 365 Management API using Copy activity in ADF and REST API dataset.
I am stuck on step when I see that data is fetched from copy activity but with error related to Source and of course I could not save data to some sink.
Error is :
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorMoreThanOneObjectsReturned,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=More than one objects returned\r\n{\r\n "CreationTime": "2020-07-15T11:59:57",\r\n .... and some data.
Of course I am expected some json data in response but error 'More than one objects returned' confused me.
//Alexander
Related
I have created a copy data activity in azure data factory and this data pipeline pulls the data from an API (via REST activity source) and writes the response body (json) on a file kept in the azure blob storage.
The API which I am fetching the response from, is paginated and the link to next page is sent in the response headers in response->headers->link.
This URL to next page is in the following general format:
<https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"
I want to fetch the next page token present in the above URL and use it in the pagination rule.
I have tried using some pagination rules:
> AbsoluteURL = Headers.link
But, this did not work as the entire encoded link shown above, is getting appended directly and hence, the pipeline throws an error.
> Query Parameters
I have also tried to use the query parameters but could not get any result.
I have followed questions over stackoverflow and have read the documentations:
Please help me with how can I access this next page token or what can be the pagination rule to support the scenario.
Pasting postman output and ADF data pipeline, for reference.
Postman Response Headers Output
Pagination Rules, I need help on
I want to save the output result of the rowsRead(source destination) and rowsCopied(sink destination) in a file in ADLS Gen2 or send it as an email notification.
This is the output result I want to store in a file or send as email notification(see picture)
You can get the number rows read and number of rows written from the Copy activity output.
To avoid complicate dynamic content in the web activity Body, first store these in two in a string variables.
For rows read use the below dynamic content.
#string(activity('Copy data1').output.rowsRead)
For rows written use the below expression
#string(activity('Copy data1').output.rowsCopied)
For mail, you can use logic apps. Use web activity to invoke logic app.
create a parameter for mail in pipeline.
Web activity:
In the Body of the web activity, give the below dynamic content.
{
"message" : "This is a custom dynamic message from your pipeline with run ID #{pipeline().RunId} and rows read are #{variables('rowsread')} and rows written are #{variables('rowswritten')}.",
"dataFactoryName" : "#{pipeline().DataFactory}",
"pipelineName" : "#{pipeline().Pipeline}",
"receiver" : "#{pipeline().parameters.receiver}"
}
For logic app workflow, please go through this Microsoft Official documentation which has step by step explanation to send mail from ADF using logic app.
I'm trying to ingest the Json array format from US Census API data into Azure either ASA or ADF is fine... Tried using HTTP or REST and none of them is successful.enter image description here
The error after using HTTP connector is shown as
"Error occurred when deserializing source JSON file 'c753cdb5-b33b-4f22-9ca2-778c97a69953'. Check if the data is in valid JSON object format. Error reading JObject from JsonReader. Current JsonReader item is not an object: StartArray. Path '[0]', line 1, position 2. Activity ID: 8608038f-3dd1-474f-86f1-d94bf5a45eba".
I attached the error message as well as sample API data and "test connection successful" screenshots in this post.
Shall I put in some parameters or advanced set up to specify something about the array form for the census data? Please advise.
The sample data link is inserted for your reference.
https://api.census.gov/data/2020/acs/acs5/subject?get=group(S1903)&for=state:51
Greatly appreciate your help in advance!
T.R.
error in azure synapse ingestion
connection test is good
US Census API Sample Test Data
As per official documentation As REST connector only support response in JSON.
Data link provided by you returns json array not Json. That is why ADF can not accept data returned by data link.
I'm trying to implement Azure data factory pipeline. From there trying to execute one SP and need to send output in HTML format from Azure data factory through Logic App. Facing issue while sending output to Web task in ADF.
Any hint ?
Error details
Error code
2108
Troubleshooting guide
Failure type
User configuration issue
Details
{"error":{"code":"InvalidRequestContent","message":"The request content is not valid and could not be deserialized: 'Unexpected character encountered while parsing value: <. Path '', line 0, position 0.'."}}
Source
In Body option of the Setting TAB in Web Activity, Instead of adding the JSON in the "Dynamic Content", add JSON as usual in the box.
Refer below screen shot:
I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest & https://www.alexvolok.com/2019/adfv2-rest-api-part1-oauth2/
The API which I'm trying to connect uses OAuth2 so I need to first get the access token and then do a get request to get actual data.
Below are the steps which I'm following
Creating a Web HTTP request in the pipeline and passing the client_ID, client secret, username, password and grant type in the body of the request. When I debug the pipline I do get the Access_token which I need in step 2.
In Step two I have a copy activity which uses the output(access_token) from web to authenticate the second REST GET request but this is where I'm facing a lot of issues. The code which I'm using is "#concat('Bearer ', activity('GetAccessToken').output.access_token)"
In step 3 I have two datasets and 2 Linked services, Dataset 1 is a REST dataset which has the base url and relative url which is linked to the REST linked service and secondly the sink dataset is connected to AZURE datalake storage.
In Source Dataset I'm passing additional header Authorization = #concat('Bearer ', activity('GetAccessToken').output.access_token) and ideally since the API which I want to call will return empty if no parameters are send so I pass in the parameters inside the "Request Body" is that even correct? the Request body would look something like this "start_date=2020/07/17&end_date=2020/07/18".
The sink is a simple Json dataset stored in DataLake.
When I try to debug I get the error as below
But I'm getting the below error
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 401 indicates failure. { \"Error\": { \"Message\":\"Authentication failed: Invalid headers\", \"Server-Time\":\"2020-07-27T06:59:24\", \"Id\":\"6AAF87BC-5634-4C28-8626-810A19B86BFF\" } },Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError",
"target": "CopyDataFromAPI",
"details": []
}
Please advise if I'm doing anything wrong.
I Knew this was a simple issue,
So for people who are looking for answers.
Please make sure the REST Source URL starts with HTTPS:// instead of HTTP:// I Guess Azure does not pass headers to url which starts with HTTP:// which is strange because POSTMAN and python script has no problem sending the headers.