Pagination error invalid token for Rest API Data Factory - azure

I am trying to use rest api to do pagination as it is just sending the first page in Azure ADF going to blob storage. I am currently using AbsoluteUrl and $['#odata.nextLink'] to get over all the pages, the issue is I am getting this error, I have used first used the token activity to get the token and then used it in copy activity where the source is rest api dataset with headers dynamically coming from token activity and then used pagination. Can you point me in the right direction on if this is the correct approach or am I missing something.
This is how the import schema looks like:
And the error after importing schema
This is how my rest api configuration look like:
And this is how my token all web activity looks like:
Edit 2:
This is how the output is for Web activity:
Including the part of the snip that missed the access token:
This is the output for Copy Activity when Pagination is on:
This is the setup of the pipeline:

HttpStatusCode 401 indicates the authentication was not completed or failed due to invalid credentials. It maybe that the access token is missing in request from copy activity or not referenced properly or is expired. Make sure you already have the right access to this API.
Here is an example with basic configuration requirements:
Get the access token
Ensure you are able to reference it dynamically using Add dynamic content fields. Modify the reference with respect to the output you have received from earlier Login Activity.
Additional headers: Authorization: #concat('Bearer', activity('Login').output.access_token)
AbsoluteUrl: ${result_root}.{nextPageURL}
Here is the official doc on Pagination support refer the supported Key and value pairs.
If you are getting the access token correctly but still seeing the error, try to Import Schema in Mapping Settings of copy activity. And make sure the nextPageUrl or odata.nextLink in your case is mapped correctly.
Recheck $['#odata.nextLink'] , AbsoluteUrl value as:
$.rootElementName.CollectionOfItems.nextLinkURL

Related

HTTP POST to Microsoft DevOps API Returns 404

I have carefully followed the documentation outlined here by Microsoft to create a Bug in Dev Ops via the API. The documentation is well written.
Steps followed:
In Dev Ops I created a Personal Access Token, the string value of which I converted to Base64.
In a separate Power Automate flow, I composed a list of the projects in my target DevOps Organization for testing purposes. The JSON for the projects in this Organization is as follows:
Now if I copy and paste this URL (in the picture above) into my browser it resolves correctly.
I created a Postman HTTP POST Request with the following details:
URL: https://dev.azure.com/{my org}/{Demo guid}/_apis/wit/workitems/{{taskType}}?api-version=7.0
Under Authorization I set the Bearer Token and pasted in my encoded PAT
The JSON Body is as follows:
The POSTMAN variable in the URL taskType is set to Bug
The request Headers are as follows:
Once again please note that if I paste the URL (1) in green into a browser it resolves correctly. However, the request returns a 404 Page not found.
I cannot understand why it is not finding the page, as I would expect a not authorized error if the problem was with the token.
Looks like you have {{taskType}} as a literal in your URL string.
Just try to perform a List to find all the valid taskTypes. Also drop the Bearer token and just use Basic Auth to keep it simple. The username should be empty and password should contain your AzDO PAT.
Needed to include the $ symbol. I am now getting authentication issues but this is outside the scope of the original submission.

Azure Data Factory - Retrieve next pagination link (decoded) from response headers in a copy data activity of Azure Data Factory

I have created a copy data activity in azure data factory and this data pipeline pulls the data from an API (via REST activity source) and writes the response body (json) on a file kept in the azure blob storage.
The API which I am fetching the response from, is paginated and the link to next page is sent in the response headers in response->headers->link.
This URL to next page is in the following general format:
<https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"
I want to fetch the next page token present in the above URL and use it in the pagination rule.
I have tried using some pagination rules:
> AbsoluteURL = Headers.link
But, this did not work as the entire encoded link shown above, is getting appended directly and hence, the pipeline throws an error.
> Query Parameters
I have also tried to use the query parameters but could not get any result.
I have followed questions over stackoverflow and have read the documentations:
Please help me with how can I access this next page token or what can be the pagination rule to support the scenario.
Pasting postman output and ADF data pipeline, for reference.
Postman Response Headers Output
Pagination Rules, I need help on

SharePoint API: Invalid Access Token Resource

I am trying to obtain an access token for use with the SharePoint Rest API. For my organizations base site. I am able to obtain a token and use that token to make subsequent requests successfully.
Next, I followed the same process and created more app permissions for a different site: {{tenant removed}}/sites/testsite. I was initially unable to create the request for the token because the resource parameter was not valid (see image below):
Per the URI encoding standards, I replaced the "/" in the site url with "%2f" and I am able to get a token (see image below):
Next however, the requests using that token to the API fail:
{
"error_description":
"Exception of type 'Microsoft.IdentityModel.Tokens.AudienceUriValidationFailedException' was thrown."
}
In the response header:
3000003;reason="Invalid audience Uri
'00000003-0000-0ff1-ce00-000000000000/{{tenant
removed}}%2fsites%2f{{removed}}#{{realm
removed}}'.";category="invalid_client"
Did I encode the resource incorrectly? What am I missing? How can I use this method to get information from the other site?
I can see many developers making the same assumption when they create requests, since almost all documentation don't point out this scenario. You will be able to obtain a token for the site successfully as long as the resource is in a valid uri format, there is no validation done on the uri itself. Even if you get a token it will not work for any requests.
When fetching the access token for subsites (i.e: {{tenant}}/sites/testsite ). The resource part of the request body does not need to be modified.
So, for example, when you are getting a token for test.sharepoint.com/sites/testsite the resource of the request body should just be:
00000003-0000-0ff1-ce00-000000000000/test.sharepoint.com#{{realm}} (without /sites/testsite)
However, when you make HTTP requests to the API with the token, you should use the full site name. Example:
https://test.sharepoint.com/sites/testsite/_api/web/

How to copydata from RestAPI using datafactory and save it in Datalake?

I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest & https://www.alexvolok.com/2019/adfv2-rest-api-part1-oauth2/
The API which I'm trying to connect uses OAuth2 so I need to first get the access token and then do a get request to get actual data.
Below are the steps which I'm following
Creating a Web HTTP request in the pipeline and passing the client_ID, client secret, username, password and grant type in the body of the request. When I debug the pipline I do get the Access_token which I need in step 2.
In Step two I have a copy activity which uses the output(access_token) from web to authenticate the second REST GET request but this is where I'm facing a lot of issues. The code which I'm using is "#concat('Bearer ', activity('GetAccessToken').output.access_token)"
In step 3 I have two datasets and 2 Linked services, Dataset 1 is a REST dataset which has the base url and relative url which is linked to the REST linked service and secondly the sink dataset is connected to AZURE datalake storage.
In Source Dataset I'm passing additional header Authorization = #concat('Bearer ', activity('GetAccessToken').output.access_token) and ideally since the API which I want to call will return empty if no parameters are send so I pass in the parameters inside the "Request Body" is that even correct? the Request body would look something like this "start_date=2020/07/17&end_date=2020/07/18".
The sink is a simple Json dataset stored in DataLake.
When I try to debug I get the error as below
But I'm getting the below error
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorHttpStatusCodeIndicatingFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 401 indicates failure. { \"Error\": { \"Message\":\"Authentication failed: Invalid headers\", \"Server-Time\":\"2020-07-27T06:59:24\", \"Id\":\"6AAF87BC-5634-4C28-8626-810A19B86BFF\" } },Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError",
"target": "CopyDataFromAPI",
"details": []
}
Please advise if I'm doing anything wrong.
I Knew this was a simple issue,
So for people who are looking for answers.
Please make sure the REST Source URL starts with HTTPS:// instead of HTTP:// I Guess Azure does not pass headers to url which starts with HTTP:// which is strange because POSTMAN and python script has no problem sending the headers.

Forbidden Microsoft Graph

We are trying to get shifts from a team.
Error: MS-APP-ACTS-AS header needs to be set for application context requests
The flow process is as follows:
Get oauth2 v2.0 token (Documentation: https://learn.microsoft.com/en-us/graph/auth-v2-service?context=graph%2Fapi%2Fbeta&view=graph-rest-beta)
Get "installedApps" and get the one with "Teams" on it. (https://learn.microsoft.com/en-us/graph/api/teamsappinstallation-list?view=graph-rest-beta)
Get ID from request above
Query for Shift (https://learn.microsoft.com/en-us/graph/api/resources/schedule?view=graph-rest-beta)
The error happens on the last step.
From the documentation:
Important: Application permissions are currently in private preview only and are not available for public use.

Resources