I am trying to dynamically add/update linked service REST based on certain trigger/events to consume a RESP API to be authenticated using cookie which provides telemetry data. This telemetry data will be stored in Data Lake Gen2 and then will use Data Bricks to move to secondary data storage/SQL Server.
Have someone tried this? I am not able to find the cookie based Auth option while adding the linked service REST.
Also how to create data pipes dynamically or to have the parameters of the rest api to be dynamic ?
Currently, unfortunately this is not possible using Azure data factory native components/activities. For now at least, you cannot get access to the response cookies from a web request in data factory. Someone has put a feature request for this or something that might help, see here
It might be possible to do this via an Azure function to get/save the cookie and then send it as part of a following request. I had a similar problem but resorted to using Azure functions for all of it, but I guess you could just do the authentication part with a function! ;-)
EDIT: update
Actually, after I wrote this I went back to check if this was still the case and looks like things have changed. There now appears (never seen this before) in the web response output, a property called "ADFWebActivityResponseHeaders" and as you can see there is property for the "Set-Cookie"
See example below:-
Related
I am trying to pull azure cognitive search current and quota .
Can anyone help me on:
" how can I get that information to csv using some service principle."
I found one link to pull this kind of data :
https://learn.microsoft.com/en-us/rest/api/searchservice/get-service-statistics
But I am not sure how to use this api to get above screenshot information.
The link you posted is the correct way to do it, it returns all the information you need.
It looks like this endpoint does not support OAuth2 & RBAC. So instead of using service principal, you need to provide the admin api-key in the request header.
You can check here how to access the api-key. If you'll be doing that from a powershell script, you can authenticate with your service principal and fetch the key using Get-AzSearchAdminKeyPair, then use this key to make a http request to get the statistics and finally convert them to CSV format.
I am trying to figure out if Azure LogicApp can be used for files/documents migration from Azure Blob Storage to a custom service, where we have REST API. Here is the shortlist of requirements I have right now:
Files/documents must be uploaded into Azure Storage weekly or daily, which means that we need to migrate only new items. The amount of files/documents per week is about hundreds of thousands
The custom service REST API is secured and any interaction with endpoints should have JWT passed in the headers
I did the following exercise according to tutorials:
Everything seems fine, but the following 2 requirements make me worry:
Getting only new files and not migrate those that already moved
Getting JWT to pass security checks in REST
For the first point, I think that I can introduce a DB instance (for example Azure Table Storage) to track files that have been already moved, and for the second one I have an idea to use Azure Function instead of HTTP Action. But everything looks quite complicated and I believe that there might be better and easier options.
Could you please advise what else I can use for my case?
For the first point, you can use "When a blob is added or modified" trigger as the logic app's trigger. Then it will just do operation on the new blob item.
For the second point, just provide some steps for your reference:
1. Below is a screenshot that I request for the token in logic app in the past.
2. Then use "Parse JSON" action to parse the response body from the "HTTP" action above.
3. After that, your can request your rest api (with the access token from "Parse JSON" above)
I need to get XML Data from API and store the data in Azure Data lake store and finally have a table created for this in Azure SQL DB/DWH. I know ADF can't handle XML data. How do i need to pull XML Data into azure. I have checking some links on using Logicapps.Any suggestions or way to handle it
As I'm not so clear about the details of your requirement, you asked "how to pull XML data into azure", so I post some suggestions for your reference.
If you want to get the xml data of your api, you can use "HTTP" action in your logic app.
Then you can use the output of the api in the next steps of your logic app.
If you want to parse the xml data, you need to transform it to json, please refer to the screenshot below.(input "json(xml(body('HTTP')))" to the content box and provide a schema)
I am trying to setup a data set in Power BI which can be refreshed on a need basis or scheduled to refresh.
I am uploading an excel workbook which has a power query.
the power query is connecting to Replicon Service to get data via service. the query looks as below:
Source = Web.Contents(
"https://na2.replicon.com/services/ClientService1.svc/GetActiveClients",
[
Headers=
[
#"Authorization"="Bearer *ValidToken*",
#"Accept"="application/json",
#"Content-Type"="application/json"
],
Content=Text.ToBinary("{}")
]
)
the request is a POST operation and hence Content field is used in Web.Contents Options argument. Authentication is via Bearer token.
Data source setting is Anonymous Credentials with privacy Level set to None.
This works fine and I am able to retrieve the results and even refresh form within workbook.
Once I upload this to PowerBI and attempt to refresh the newly created Dataset it says:
You cannot refresh yet because you need to provide valid credentials for your data sources in the dataset.
So I go to Manage Data sources. Click Edit Credentials. Select "Anonymous" Authentication method and click on Sign In and it says "Login Failed".
Why is that so? It appears that the Headers are lost when I upload the Excel workbook. How can I accomplish this? Is there any alternate ways of being able to set up a Data Set which can be refreshed - the source being a web service?
this is what i ended up doing finally.
my scenario first:
my requests are POST
Authentication is via bearer token which needed to be passed via Request Header. this was the requirement of the replicon service i am trying to invoke which i couldn't change
the dataset that is created in PowerBI needed to be refreshable.
Since i couldn't get it to work directly from inside PowerBI i introduced an intermediate layer. this would interpret GET requests from PowerBI. process the token from query string. accept the service and operation also as query string parameters. it would then create POST request to the Real Service (replicon services in my case). the service name and operation names were also picked up from request URL. the token was pushed as part of request header
so the request from PowerBI would look as PowerBI needed i.e.
Web.Contents("https://intermediatelayer.com?access_token="*validtoken*"&ServiceName="ClientService"&Operation="GetActiveClients"")
not an ideal solution but works.
Manage Data Sources is validating the Anonymous credentials with a GET request to the URL without the hardcoded headers, like you suggest. It's basically running
Web.Contents("https://na2.replicon.com/services/ClientService1.svc/GetActiveClients")
which fails with "(405): Method Not Allowed", and so Manage Data Sources thinks the credentials are wrong.
Short of making the service reply with a success response for the above M, I don't see any way to set up refresh on this mashup.
I'm trying to get data from a table I created in Azure Mobile Services.
When I setup the service, I created a new database to go along with it.
I've added a row of data to my table called TODOITEM and can query this row of data fine from the db admin tool on Azure.
Now what I'm trying to do is run a GET request on this table, as documented here...
https://msdn.microsoft.com/en-us/library/azure/jj677199.aspx
I am using Fiddler and trying to send the following request with HTTP 1.1...
https://mobilemeshtest.azure-mobile.net/tables/todoitem
X-ZUMO-APPLICATION: my azure mobile services key
X-ZUMO-AUTH: my facebook authentication key
Host: mobilemeshtest.azure-mobile.net
No matter what I try, all I get back is a 404 not found.
I'm even copying what the guy did in a PluralSight video tutorial on Azure Mobile Services, and he gets a correct 200 response.
I can't see what I'm doing wrong.
404 is pretty simple. The url is wrong.
Propably you have not set your Controller right.
If your Controller is named SomethingController then the route is /tables/Something assuming you havent customized the default Routing methods
Application Key and Authorization Key is not needed if you haven't specified the corresponding authorization attribute
for example : [AuthorizeLevel(AuthorizationLevel.User)]
RESOLVED : I was using a .NET backend, it seems one has to publish their data objects and controllers first using Visual Studio.
This blog post has a good example on how to achieve this.
If I was using the Javascript backend option, the controllers are automatically setup for you.