I am currently experimenting with Databricks. I'd like to leverage the Python SQL Connector to allow external APIs to access certain tables through Databricks.
Using the Python SQL Connector requires the use of a Personal Access Token to authenticate with Databricks.
The issue here is I do not want the access token for a service to be tied to my personal identity, ideally I'd like the access token to be attached to a service identity. Databricks only allows me to create users with a first name, last name, and email. Is creating a service identity even possible to externally access tables?
Thanks.
Yes, it's really possible. You need to create a service principal (via REST API or Terraform provider), add it to corresponding group that has access to data (via REST API, UI, or Terraform provider), give it permission to use personal access token (via REST API or Terraform provider), and then generate a token for it (via REST API or Terraform provider).
Related
I have created a fhir server using Azure API for FHIR. I have created a smartclient in azure using the MS open source smart on fhir proxy.(https://github.com/microsoft/fhir-proxy/tree/main/scripts) I basically got standalone launch working using the this proxy.
I have used MSI as an authentication system and AAD as an identity provider.
The azure AD consent prompt is all or nothing (the app asks the user to consent to a list of permissions, the user can accept or deny all). The limited access scenario (g)(10)(v)(A)(10,11,12) seems to require that the user is presented with a list of permissions with the ability to accept or deny each one.
How can I achieve this using AAD?
I have seen something like this azure graph explorer(Please see the ss), but I'm not sure how I can achieve this here in AAD.
Here is the more detail information about my case:-
smartclient name :-proxy10391-smart-client-1212
fhir server:- fhirstandalonecli
fhir proxy:-sfp-proxy10391.azurewebsites.net
I have assigned these permissions to my smartclient:-
enter image description here
Whenever I'm using Oauth2.0 flow user is automatically getting permissions to access all of the above resources.
For Limited App case I want user to select individual permissions at run time.
Suppose user has not selected Observation resource then user can't access that resource.
Any idea how can I achieve this using azure AAD?
I will really appreciate your help.
We aim to collect data from the Azure Management APIs. These APIs provide information on the resources we have running in Azure, the consumed budget, etc (example). Following our design choices, we prefer to exclusively use Azure Data Factory to make the HTTP requests and store the data into our data lakes. This is fairly obvious, using the REST linked service. However, we struggle to correctly set up the OAuth2 authentication dance with this method.
Our first idea was to store the token and the refresh token within the Azure Key Vault. A series of HTTP requests within the pipeline would then test whether the token is still valid or otherwise use the refresh token to get a new token. The downside to this approach is that the token within the Azure Key Vault is never updated, when needed, and that the logic becomes more complex.
Alternatively, we were trying to set up the authorization through combination of a registered app and service principal to our Azure AD account. The REST linked service within Data Factory can be created with a service principal, which would then handle most of the information of the scope and consent. The service principal is also accompanied with a Azure app, which would hold the token etc. Unfortunately, we are unable to make this setup function correctly.
Questions we have:
Can we actually use a service principal / app to store our OAuth2 tokens? If so, will these be automatically refreshed within our app?
How do we assign the correct privileges / authorizations to our app that it can use this (external) API?
Is the additional logic with HTTP calls within Azure Data Factory pipeline needed to update the tokens or can these apps / service principals handle this?
Thank you for your time and help!
It is not a good idea to store the tokens in the keyvault, because they will expire.
In your case, two options for you to use.
Use service principal to auth
Use managed identity to auth(best practice)
Steps to use service principal to auth:
1.Register an application with Azure AD and create a service principal.
2.Get values for signing in and create a new application secret.
3.To call the Azure REST API e.g. Resources - List you mentioned, your service principal needs the RBAC role in your subscription.
Navigate to the Azure portal -> Subscription -> add your service principal as a Contributor/Owner role in the subscription like below.
4.In the linked service, configure it like below, fix them with the values got from step 2.
Don't forget to replace the {subscriptionId} in the Base URL.
https://management.azure.com/subscriptions/{subscriptionId}/resources?api-version=2020-06-01
5.Test the linked service with a copy activity, it works fine.
Steps to use managed identity to auth:
1.Make sure your data factory has enabled the MSI(managed identity), if you create it in the portal or powershell, MSI will be enabled automatically, don't worry about that.
2.Navigate to the Subsctiption in the portal, add the role to the MSI like step 3 in Steps to use service principal to auth, just search for your ADF name in the bar, the MSI is essentially a service principal with the same name of your ADF, which is managed by azure.
3.Then in the linked service, just change it like below.
At last, answer your questions.
Can we actually use a service principal / app to store our OAuth2 tokens? If so, will these be automatically refreshed within our app?
As I mentioned, it is not a good idea, just use the service principal/MSI to auth like the steps above.
How do we assign the correct privileges / authorizations to our app that it can use this (external) API?
To use the Azure REST API, just assign the RBAC roles like above, specify the correct AAD resource e.g. https://management.azure.com in this case.
Is the additional logic with HTTP calls within Azure Data Factory pipeline needed to update the tokens or can these apps / service principals handle this?
No need to do other steps, when you use the configuration above, essentially it will use the client credential flow to get the token in the background for you automatically, then use the token to call the API.
I'm trying to use Azure RBAC to secure access to storage blobs, and to use Azure Identity to access those blobs from Apache Spark. I see that recent versions of Hadoop-Azure support abfs, and it supports a few token providers: https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html#Azure_Managed_Identity . For production usage, I can use a service principal with an AD app and the associated client id, secret, and endpoint. Or I can even use Managed Identity.
When developing locally, it would be good to be able to do the same with something like DeviceCodeCredential or InteractiveBrowserCredential, i.e. something that will make the user log in to Azure using a browser, and use the credentials returned to get the access token as pass it to Spark. The reason I'd like this is to have users use their own credentials when accessing data, and not have storage keys / SAS tokens / etc flying about.
Is something like this possible? I could implement a Custom Token Provider that wraps an Azure Identity instance, but was wondering if there were a less nuclear approach.
If you want to use the user credential to auth, the closest way in the supported auth ways is the OAuth 2.0: Username and Password, but essentially it uses the Azure AD ROPC flow to auth, it has some limits, e.g. it will not work with the user account which is MFA-enabled.
Actually, for local development, the way I most want to recommend is to use a service principal to auth i.e. OAuth 2.0 Client Credentials, because the MSI(managed identity) is essentially a service principal managed by azure, if you use MSI for production, the transition of the environment will be smoother, the permissions stuff in azure will be some differences between the user account and service principal in some scenarios(maybe not in this case). Of course, Custom Token Provider is also a feasible way, they all depend on yourself.
Currently, a user has to add my application within the Azure portal. I will then get a ClientID etc. and use that to get access to the users Consumption data. I wondered if I could streamline this process by asking for consent using the Azure authentication.
Pre-thanks!
**
Let me elaborate a bit more, I'm creating an application that will help the user by giving insights on there cloud spend. I get the data using the consumption API. At the moment I get access to that API by them giving my app credentials via the Azure portal. But I want to get the access via Azure authentication
**
In Azure AD, there are several auth flows, not sure which one you want to use. Generally, these flows can be divided into two types, user-interactive or non-interactive.
Per my understanding, you want to call the Azure Consumption REST API via Azure AD auth. If you want the user-interactive way, you could use auth code flow. If you want a non-interactive way, you could use client credential flow.
For more details, see https://learn.microsoft.com/en-us/rest/api/azure/
Is there a way with a single app to access Graph, VSTS, and Azure information? It seems access to each of these requires it's own app with origination and callback urls.
For Azure, I'm using NPM's passport-azure-ad in a node js app.
Ideally, I would like to combine VSTS build info, Azure service usage info, and User profile info.
Each of the services you mentioned has their own API:
Azure REST API
Visual Studio Team Services REST API
Microsoft Graph
This does not however mean that they also each need their own "app". When you register your application in Azure AD via the Azure Portal you're able to request access to a number APIs. Each access_token you receive will be tied to one API (called a "resource") but you can use the refresh_token to switch the targeted resource:
The only exception here is the VSTS REST API. While most APIs use the same identity provider, VSTS has their own. So for the purposes of VSTS, you will need to have the user authenticate separately. Obviously, that isn't a great user experience but there is a useful workaround: Personal Access Tokens.
Using a Personal Access Token for VSTS allows you to authenticate the user via Azure AD OAuth and get an access token you can use with Microsoft Graph and the Azure REST API. Once you've authenticated them, you can ask them to provide a Personal Access Token to access VSTS. This allows you to forgot asking the user to authenticate a second time since you'll store their PAT use it for any calls to VSTS.
First, there is Allow scripts to access OAuth token option in the Phase of Build/Release definition, you can check this option and access the token through System.AcessToken variable.
To grant the permission for that user, you need to grant the permission(s) for Project Collection Build Service (xxxx) account.
Secondly, there are some tasks related to Azure (e.g. Azure PowerShell), that can access azure resources (The AAD application is associated to the Azure endpoint)
You can retrieve the necessary information in multiple task, then store the result in the variables through Logging Commands (##vso[task.setvariable]value), then combine them together.