I am automating creation of HDInsight Clusters. I can create the clusters. However, the template creates ADMIN accounts. We are using Ambari to create the USER accounts manually but would like to automate this. I think I can get a script included as part of the template.
I need a script to create User accounts in a manner Ambari would. I have no idea where to start.
Creating groups would also be helpful.
Almost all Amabri actions can be scripted using the Rest interface of Ambari. See:
How to use RestSharp with Ambari Swagger
I chose to implement a RestSharp interface using Azure Functions triggered by a Cluster
created subscription event. However, these actions can also be implemented with a Curl script defined by a Script Action during or after creation. The Ambari Rest interface is finicky and requires certain headers are OR are not present. Fiddler can be used to listen to the Ambari web client to determine the correct headers. There is a swagger.json file downloadable from a cluster. It is not a very good file, but enough to get started.
I put my client with swagger.json on github:
https://github.com/USStateDept/Azure-HDInsight-Ambari-RestClient
Unfortunately, you cannot create user accounts in HDInsight clusters using script actions and Ambari in Azure HDInsight does not support creating local users and sign in using those users.
You can add users accounts are only available in HDInsight Enterprise Security Package.
OR
You can LDAP users to login into Ambari UI in domain-joined HDInsight clusters.
HDInsight clusters with Enterprise Security Package (ESP) can use strong authentication with Azure Active Directory (Azure AD) users, as well as use role-based access control (RBAC) policies. As you add users and groups to Azure AD, you can synchronize the users who need access to your cluster.
Reference: Synchronize Azure Active Directory users to an HDInsight cluster and also you may checkout similar question addressed in Azure HDInsight MSDN forum.
Related
Azure Databricks issue.
Is there any way to programmatically set this parameter in Admin Console: Cluster, Pool and Jobs Access Control. I think this is a spark_conf property, but I can't find any informations.
Other question: When a parameter is set, is it possible to export a json parameter file showing all these settings?
Thank you
Yes, the permissions API lets you manage permissions in the Azure Databricks.
Note: This feature is in public preview
The Permissions API supports several objects and endpoints:
Token permissions — Manage which users can create or use tokens.
Password permissions — Manage which users can use password login when
SSO is enabled.
Cluster permissions — Manage which users can manage, restart, or
attach to clusters.
Pool permissions — Manage which users can manage or attach to pools.
Some APIs and documentation refer to pools as instance pools.
Job permissions — Manage which users can view, manage, trigger,
cancel, or own a job.
Notebook permissions — Manage which users can read, run, edit or
manage a notebook.
Directory permissions — Manage which users can read, run, edit, or
manage all notebooks in a directory.
MLflow registered model permissions — Manage which users can read,
edit, or manage MLflow registered models.
The permissions API is not completely documented on the Azure Databricks REST API page. I would request you to follow the document on databricks to create Permissions API.
Most of the sample ARM templates I find for deploying an Azure App Service + Azure SQL database end up configuring the connection string for the web app using the administrator login credentials provided when creating the Azure SQL Server (this one from Microsoft, for example). I've also seen the App Service configured for Managed Identity and then that user granted administrative access over the Azure SQL instance via AAD.
I'd never want to provision a production application this way, but I can't find examples in a more secure configuration.
The only other info I find about provisioning non-admin users is via SQL after the Azure SQL instance is already up. This means I have to add another step outside of my ARM deployment to get my system fully functional.
As of the time this question was written, running custom Powershell scripts as part of an ARM deploy is in preview and that could be a path forward, but it's not ideal.
Can non-administrative users be provisioned in Azure SQL via an ARM template, without resorting to PowerShell?
If you want to create non-administrative users in Azure SQL database via arm template, it is impossible. Because Azure ARM template team just defines Administrators type and does not define user type or user property in server. We just can create SQL Admin or Azure AD Admin via ARM template. For more details, please refer to here and here
So if you want to create non-administrative users, you need to write custom scripting.
If you want to create the users as part of a template deployment, you can use the deploymentScripts resource to run TSQL or any arbitrary script:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template?tabs=CLI
If you want to do it as part of the declaration of the Microsoft.SQL/servers resource, no you can't (as Jim mentioned).
Is it possible to temporarily disable Azure Active Directory RBAC in Azure Kubernetes Service? The reason I ask is because we are unable to set up automated tasks (like continuous integration) because authenticating against kubectl now requires human intervention to complete device code auth - I have another post here regarding that. Perhaps even just disabling Kubernetes RBAC will bypass the need to authenticate with AD? I would do this until a solution to the issue is available.
Although there is no document exactly say that you cannot disable the RBAC of an existing AKS cluster. But it shows that enabling role-based access control (RBAC) on existing clusters isn't supported at this time. In my opinion, it also means that you cannot disable the RBAC on existing AKS cluster. And it seems there is no way to achieve it, no matter Azure CLI, PowerShell or REST API.
I think the RBAC is a setting for AKS cluster and it could not be changed after it created now. We can expect that it could be changed in the future. Hope this will help you.
I can able to view Spark Cluster Master & Worker nodes hostname but I couldn't find username to log in on SSH
For Example(In terminal):
ssh username#hostname_or_Ip
As per my knowledge, you cannot SSH on Azure Databricks.
Note: Azure Databricks integrated with Azure Active Directory – So, Azure Databricks users are only regular AAD users.
Azure Active Directory users can be used directly in Azure Databricks
for al user-based access control (Clusters, jobs, Notebooks etc.).
Azure Databricks has delegated user authentication to AAD enabling
single-sign on (SSO) and unified authentication.
Notebooks and their outputs, are stored in the Databricks account.
However, AAD based access control ensures that only authorized users
can access them.
For more details, refer “Azure Databricks Preview Documentation”.
We are migrating an ASP.NET application onto Azure. Target state selected as Cloud Service Web role to host application component and Database onto Azure SQL/SQL Database. Currently the application is configured onto Kerberos in the source environment. Please let me know if Kerberos can be configured onto Target Environment without any issues.
Are they any specific steps to be done for Kerberos to work?
Yes, but you have to use Azure Active Directory. This method of authentication is in preview atm and only available for Azure SQL v12. There's possibly a lot of steps involved for your scenario from populating your Azure AD to setup your application. Check these references:
Integrating your on-premises identities with Azure Active Directory
Azure SQL Database security guidelines and limitations
Connecting to SQL Database By Using Azure Active Directory Authentication