Is there a way to start, stop, list available SQL Endpoints in Databricks SQL?
With "Can Use" permission it's possible to start a SQL Endpoint while connecting to an endpoint using the Databricks JDBC driver, but have no idea how to stop it (not to mention how to list all the available endpoints).
All of this (and more) could be done via REST API for SQL Endpoints. You just need to be authenticated to workspace & have corresponding permissions. To stop you need to have CAN_MANAGE permission (see docs)
Related
Is it possible for me to insert some data from one database to another in Azure sql?
Let's say I have a trigger in db1 that updates some values in db2.
I read about elastic queries but it seems like they are read-only so they don't solve my problem.
You can't use cross-database in Azure Sql Server because databases can't see eachother physically , you could use elastic pools but they are Read Only.
A solution is to use SQL Managed Instance to upload your instance . This supports cross-database queries but it was expensive.
There was some previous discussion here about doing similar:
C# Azure Function trigger when SQL Database has a new row added without polling
There is also the Azure SQL Bindings for Azure Functions but they are input bindings and not triggers and they're still in preview and limited to C#, JavaScript and Python.
Azure SQL bindings for Azure Functions overview (preview)
There was a new announcement last week after MS Build however for Azure SQL Database External REST Endpoints Integration (hopefully they don't refer to it as ASDEREI) but this is currently in preview under Early Adoption Program (EAP).
Announcing the “Azure SQL Database External REST Endpoints Integration” Early Adoption Program
I am trying to access the internal Hive metastore tables like HIVE.SDS, HIVE.TBLS etc.
I have an HDInsight Hadoop Cluster running with the default internal metastore. From Ambari screen, I got the Advanced setting details required for connections like -
javax.jdo.option.ConnectionDriverName,javax.jdo.option.ConnectionURL,javax.jdo.option.ConnectionUserName as well as the password
When I try connecting to the SQL Server instance(internal hive metastore) instance from a local machine, I get the message to add my IP address to the allowed list. However, since this Azure SQL server is not visible in the list of Azure SQL server dbs in the portal, it is not possible for me to whitelist my IP.
So, I tried logging in via the secure shell user- SSHUSER into the Cluster and tried accessing the HIVE database from within the Cluster using the credentials of metastore provided in Ambari. I am still not able to access it. I am using sqlcmd to connect to sql sever.
Does HDInsight prevent direct access to internal Metastores? Is External Metastore the only way to move ahead? Any leads would be helpful.
Update- I created an external SQL Server instance and used it as an external metastore and was able to access it programatically.
No luck with the Internal one yet.
There is not a way to access internal metastores for HDInsight cluster. The internal metastores live in the internal subscription which only PGs are able to access.
If you want to have more control on you metastores it is recommended to bring your own "external" metastore.
We are using Azure SQL as our database across multiple regions, with one primary and multiple secondaries.
The scenario is, we want to find if there are any active replication problems through some sort of API call / so we can integrate this into our overall environment.
Is there an Azure Management API / or SQL query we can run and build an API on top of - so we can some result like replica DB X is having problems and the data is outdated?
Azure doesn't have an API for this yet. But you should be able to build APIs on top of the views.
There are out of box views 'sys.dm_database_replica_states' and 'sys.dm_db_resource_stats' provided on Azure SQL Database that could be used for your requirement. However, I would recommend you to go through the below link and choose them appropriately:
https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out#monitoring-and-troubleshooting-read-only-replicas
https://learn.microsoft.com/en-us/azure/azure-sql/database/read-scale-out#data-consistency
I have a 3-4 clusters in my databricks instance of Azure cloud platform. I want to maintain a common metastore for all the cluster. Let me know if anyone implemented this.
I recommend configuring an external Hive metastore. By default, Detabricks spins its own metastore behind the scenes. But you can create your own database (Azure SQL does work, also MySQL or Postgres) and specify it during the cluster startup.
Here are detailed steps:
https://learn.microsoft.com/en-us/azure/databricks/data/metastores/external-hive-metastore
Things to be aware of:
Data tab in Databricks - you can choose the cluster and see different metastores.
To avoid using SQL user&password, look at Managed Identities https://learn.microsoft.com/en-us/azure/stream-analytics/sql-database-output-managed-identity
Automate external Hive metastore connections by using initialization scripts for your cluster
Permissions management on your sources. In case of ADLS Gen 2, consider using password pass-through
I am working on a solution that uses SQL Azure. Part of the project deals with backups and using the DAC Web Services for backups.
The issue is that there is a different endpoint depending on which region the Azure SQL database is in. As I am working with multiple groups, and cannot ensure which region the database will be in, I am looking for a way to programmatically determine the region.
The region is also important, as I want to copy the backups to a different region just to be on the safe side.
I know that I can look in the Admin console, but I would like to use code to solve this problem.
Additional information:
The application is running on Azure using Worker Roles for functionality.
I do not have access to all of the account-id's to use the full REST API.
I do have access to the master database on the Azure Sql Server.
Working on this in C# (I failed to put the language)
You can use Get Servers request (GET https://management.database.windows.net:8443/<subscription-id>/servers) of Azure REST API to enumerate SQL servers which gives the Location or Region more info at msdn -> http://msdn.microsoft.com/en-us/library/windowsazure/gg715269.aspx