Is there a way to save the output of an Azure Data Factory Web Activity into a dataset?
Here is my current use case:
I have to dynamically build a JSON post request
The API I'm trying to reach requires a SSL certificate so I have to use the Web Activity Authentication Client Certificate option.
The API also requires basic authentication so I input the Content -Type and authorization guid in the header section of the Web Activity.
Once I get the JSON response from my post request I need to save the response into a blob storage some where.
I tried using the Copy Data Set HTTPS or Rest API as a data set source but both only allow one type of authentication certificate or Basic authentication.
Is there a way I can configure the Rest API or HTTPS data set source handle both types of authentications (SSL and Basic Authorization) or capture all the Web Activity output into a blob storage?
Thank you all for your help! I'm desperate at the moment lol..
Here is what my Web Activity looks like (sorry I had hide part of the credentials for security purposes:
Please use Http dataset in your Coput Activity.
When we create the linked service of the Http dataset, select client certificate option and embedded data, then we need to upload the SSL certificate.
The offical document is here.
Related
So I'm trying to use 2 API's from our external data source, an organization that solely focuses on delivering data to customers (me).
They have 2 different API's:
A login API: this API is called with basic authentication (username and password) and will provide a bearer token
An API to extract data and manage filters (this API will require the bearer token acquired through the login API)
Important notes:
The login API requires a self-signed certificate, I have created a certificate on my PC using openSSL and shared the public key with the external data source.
I uploaded the .pfx file in Azure (however i cannot set the domain or port number, this seems to be an issue, I can do this in Postman but can't seem to find it in Azure)
The data source confirmed that the certificate is valid and I am able to use the certificate in Postman but only if i set the correct hostname and also the correct port number (Else i get a TLS error in Postman)
Everything is working as it should in Postman (I can call the Login API and i can use the bearer token to access and download the data using a json file made for postman that i received from the data source organization)
I uploaded the .pfx certificate in Azure using AppServices > TLS/SSL settings and upload the private certificate, however the hostname it says here is my own database name (I am not sure if this is correct, in Postman i set the hostname when i upload the certificate to be the hostname of our data source with the needed port number)
I tried making a web call in Azure Datafactory, here i've used the correct URL i also use in Postman to access the login API and selected basic authentication and filled in the credentials
However when i try to debug pipeline (only the login web call) it gives the following error:
Error code 2108
Troubleshooting guide
Failure type User configuration issue
DetailsError calling the endpoint '[LINK i put with portnumber, which is correct]'. Response status code: 'NA - Unknown'. More details: Exception message: 'NA - Unknown [ClientSideException] An error occurred while sending the request.'.
Request didn't reach the server from the client. This could happen because of an underlying issue such as network connectivity, a DNS failure, a server certificate validation or a timeout.
Source
Pipeline
Punctuality
Am i looking in the wrong direction for this kind of data retrieval? is something going wrong with the certificate? I can't seem to find where i can set the correct hostname and port number (Like i can in Postman when uploading the certificate in .pfx format) Should I use API Management Service rather than Data Factory to make the API call to the external data source?
Select the authentication method as Client Certificate in web activity settings.
Specify base64-encoded contents of a PFX file in Pfx.
Hello I am trying to deploy my Azure Machine Learning pipeline with a REST endpoint. My problem is that I was able to generate an endpoint but has some sensitive information in it (ex: subscription id, resource group, etc). How can I generate a URL that forwards the request body to my Azure ML REST endpoint?
also, here is an approach I've done:
Used Application Gateway Redirect (this approach didn't forward the request body. It instead turned my POST request into a GET request when it redirected to the correct URL.)
The issue is raised because of some of the default security headers dependent on REST API and web based. Need to set the REST API CSP HEADER. Check the request and response headers in config file of the web application.
I have come across a challenge for that I need your opinion. We have a Front-end app in Vue that is publicly open and doesn't authenticate users through Login, the app doesn't have a backend. Currently, the app has a JSON data file that we manually update monthly. Based on my research and knowledge I have come to the conclusion, we can't put our SOAP web service endpoint with credentials since everyone will see the service user name and password in the browser. Even though the data is publicly displayed on the app but we want to secure the SOAP web service endpoint and credentials.
Currently, I only see two options:
We build the backend and authenticate(Login) the user in the app.
Build a backend (Node and Express server with Loopback) that consumes that SOAP service and converts that data in JSON format and consumes that data through Publicly open REST API endpoint with Read-Only option and only allow frontend app IP or Address to request data and reject the other IP request.
Both methods require the backend to secure the SOAP endpoint but the second option reduces the frontend work since it will convert the data from XML to JSON, my concern with this option is that it will be an open endpoint in REST API, can that jeopardize the SOAP credentials?
This is the Loopback diagram converting SOAP to REST.
I would like to know if anyone else has another idea or option to resolve this or do you see a problem with the above methods.
https://learn.microsoft.com/en-us/power-bi/service-real-time-streaming
I am using the REST API to send live data to PowerBi. (From a native application on a Windows)
How can I handle the authentication , encryption and other security features while streaming data to PowerBi?
Can I use the powerBI gateway somehow?
I see following issues currently:
If someone gets the REST Api link to the dataset, they can induce incorrect data.
The json data that I stream is not encrypted
As #silent said in his answer, the communication is fully encrypted. Pushing data to a streaming dataset looks like this (sample code in PowerShell):
$endpoint = "https://api.powerbi.com/beta/08bbc04c-a46d-4c45-b587-9dec9454fc2d/datasets/15e4b6c3-4697-442f-91f9-2ad056eef2a8/rows?key=QINNGFRYZnWHHFA51G6VCDeL%2FYyfh0oDZ0qsV1qwzIh18tNfs2POjWgFIJdnWxxA3bjqJqfMhWPOhzQ6bK3vgw%3D%3D"
$payload = #{
"datetime" ="2019-05-03T17:17:05.830Z"
"somevalue" =98.6
}
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json #($payload))
Note that the endpoint uses HTTPS protocol, i.e. it is encrypted. However, the difference between pushing data to a streaming dataset and pushing data to a "normal" push dataset, is that with streaming you do not use access token generated when you authenticate yourself against Azure AD, but a key in the endpoint URL. You must protect this key the same way as you are protecting your account's password. If someone got an access to them, he can harm you. So in the regard of authentication, there is not much difference. Also, because the communication is encrypted, you should not worry about the streamed data.
You can't use the gateway, because you don't need it. First, it is used to enable communication from Power BI Service to your premise (while in your case it is in the other direction and can be established directly) and second, even if you communicate through the gateway, the data will be encrypted the same way (so there is no difference regarding that).
Not sure if I understand your question correctly, but the REST APIs are all only accessible via TLS encryption and authentication is based on Azure AD.
All requests to REST APIs are secured using Azure AD OAuth.
https://learn.microsoft.com/en-us/power-bi/service-real-time-streaming#using-power-bi-rest-apis-to-push-data
I want to purge an Azure CDN endpoint from Microsoft Flow. The CDN Profile's Pricing Ties is Standard Verzion.
I have created a Microsoft Flow with an HTTP - HTTP action.
I used an URL generated by the tool shown by clicking on Try it on https://learn.microsoft.com/en-us/rest/api/cdn/endpoints/purgecontent and filling the necessary fields.
I want to purge all CDN content so I put the following as a body (I hope this is the right way as the official docs were silent about this use case).
{
"contentPaths": [
"/*"
]
}
I have created a Registered application in the Azure Active Directory and used that to fill in Active Directory OAuth fields of the HTTP - HTTP action. I presume that the OAuth authentication is succeeding as if I put wrong values in any of the fields I get a respective error message.
So the HTTP - HTTP action is filled as on the following image. The image mostly shows where I got the values I used on the real flow.
When I run the flow it fails and shows just Unauthorized. inside the failed HTTP - HTTP action.
I have tried to add the app as a Contributor role of the CDN Profile but the error stayed the same.
How to get it working?
According to the error message, it seems that you missed the access token in the request.
If you want to generate the token for your azure ad app(service principal), you could refer to this article and the thread of generating the access token via postman.
Also, you could try to add the app as a role in the logic app.