I'm trying to create a cluster in Azure Databricks and getting a such error messgae
Resources were not reachable via SSH. If the problem persists, this usually indicates a network environment misconfiguration. Please check your cloud provider configuration, and make sure that Databricks control plane can reach Spark clusters instances.
I have such the default configuration:
Cluster mode: Standard
Pool: None
Runtime version: 5.5 LTS
Autoscaling enabled
Worker Type: Standard_DS3_v2
Driver Type: Standard_DS3_v2
From Logs Analytics I see Azure tried to create virtual machines and without any reason (I suppose because they were unreachable) had to delete all of them.
Did anyone face such issue?
If the issue is temporary, this may be caused by the driver of the virtual machine going down or a networking issue since Azure Databricks was able to launch the cluster, but lost the connection to the instance hosting the Spark driver referring to this. You could try to remove it and create the cluster again.
If the problem persists, this may happen when you have an Azure Databricks workspace deployed to your own VNet. If the virtual network where the workspace is deployed is already peered or has an ExpressRoute connection to on-premises resources, the virtual network cannot make an ssh connection to the cluster node when Azure Databricks is attempting to create a cluster. You could add a user-defined route (UDR) to give the Azure Databricks control plane ssh access to the cluster instances.
For detailed UDR instructions, see Step 3: Create user-defined routes and associate them with your Azure Databricks virtual network subnets. For more VNet-related troubleshooting information, see Troubleshooting
Hope this could help you.
Issue: Instances Unreachable: Resources were not reachable via SSH.
Possible cause: traffic from control plane to workers is blocked. If you are deploying to an existing virtual network connected to your on-premises network, review your setup using the information supplied in Connect your Azure Databricks Workspace to your On-Premises Network.
Reference: Azure Databricks - Troubleshooting
Hope this helps.
Related
I have an Azure Kubernetes Cluster Running with Azure CNI (virtual network) as the Network. The cluster is running on 1 subnet of the network.
On another subnet, I have a Virtual Machine running as it has a private IP of 10.1.0.4.
Now I have a pod in the K8S cluster, which is trying to connect with the Virtual Machine. But it's not able to do so.
Also, the ping 10.1.0.4 from inside the pod gives a timeout.
Please help me to figure out, what I am doing wrong so that I can connect the Pod with the VM.
• You cannot directly create communication between an AKS cluster pod and a Virtual Machine as the IP assigned to a pod/node in an AKS cluster is a subset range of the address space of the higher CIDR IP address range assigned while deploying the cluster. And communication within the cluster between the nodes is uninterrupted and possible readily. But the same with resources other AKS is restricted as they are governed by Azure CNI framework policy which directs the Kubernetes cluster to direct traffic outbound of the cluster in a regulated and conditional way.
• Thus, the above said can only be achieved by initiating intermediate services such as an internal load balancer between the AKS and the VMs as the CIDR of the VM and the AKS is different. So, leveraging the Azure plugin to deploy an internal load balancer as a service through AKS is only way through which you can achieve communication between AKS pod and a VM deployed in Azure. Below is a diagram for illustration purposes.
To deploy the internal load balancer through YAML files in AKS for external communication with VMs, kindly refer to the link below for details: -
https://fabriciosanchez-en.azurewebsites.net/implementing-virtual-machine-to-pod-communication-in-azure-kubernetes-service-aks/
don´t know if this possible or not.
client wants to create a Windows 2016 cluster with 2 different vms/nodes that are in Azure which are in different subscriptions and virtual networks. No shared storage
the idea is to configure SQL always on between them so that DB and sql config replicates exactly from VM1 to VM2. Then always on config would be removed when this syncs completes. client won´t do a normal backup/restore from one to the other (I already suggest them this aproach), they would go with always on aproach.
Vms are already on the same localdomain and they can ping each other . Command in powershell to test if cluster can be done with both vms was successfull:
PS C:\windows\system32> Test-Cluster -Node VM07.domain.local,VM04.domain.local
WARNING: System Configuration - Validate Software Update Levels: The test reported some warnings..
WARNING: Network - Validate Network Communication: The test reported some warnings..
WARNING:
Test Result:
HadUnselectedTests, ClusterConditionallyApproved
Testing has completed for the tests you selected. You should review the warnings in the Report. A cluster solu
supported by Microsoft only if you run all cluster validation tests, and all tests succeed (with or without war
Test report file path: C:\xxxx\xxxxxx\AppData\Local\Temp\Validation Report 2021.03.26 At 11.13.54.htm
Thing is that this cluster doesn´t have a listener or load balancer IP, as this requires vms on same subnet . Cluster is only going to be used for SQL always on config.
Is it possible to create this cluster without a Loadbalancer Static IP for the cluster name?. Can the IP of one of the 2 nodes be used for this instead. something like:
VM07 IP: 10.1.2.3
VM04 IP: 10.1.2.4
New-Cluster –Name newcluster -Node VM07,VM04 –StaticAddress ClusterIP 10.1.2.3
–NoStorage
I know is a odd idea but want to be sure if it´s possible or not in practice.
thank you!
Use a single NIC per server (cluster node) and a single subnet.
Because the virtual IP access point works differently in Azure, you need to configure Azure Load Balancer to route traffic to the IP address of the FCI nodes or the availability group listener. In Azure virtual machines, a load balancer holds the IP address for the VNN that the clustered SQL Server resources rely on. The load balancer distributes inbound flows that arrive at the front end, and then routes that traffic to the instances defined by the back-end pool. You configure traffic flow by using load-balancing rules and health probes. With SQL Server FCI, the back-end pool instances are the Azure virtual machines running SQL Server.
Refer to this link for best practices and limitations: https://learn.microsoft.com/en-us/azure/azure-sql/virtual-machines/windows/hadr-cluster-best-practices
UPDATE
Azure Load Balancer or App Gateway can be configured with any kind of static or dynamic IP for destination.
https://learn.microsoft.com/en-us/azure/load-balancer/manage
I have hosted my SQL on Azure SQL.
From my AKS, each of the pods, I found out it is not able to connect to Azure SQL.
DB Connection:
Data Source=tcp:dbname.database.windows.net,1433;Initial Catalog=dbname;User Id={account};Password={password}
In Azure Portal > I have enable this below
I double checked the connection string and is able to connect from my local machine, but inside the kubenetes pod, I try to perform telnet to the server it responds
Connection closed by foreign host.
May I know what going wrong on this.
Azure provides two options for pods running on an AKS worker nodes to access a MySQL or PostgreSQL DB instance:
Create a firewall rule on the Azure DB Server with a range of IP addresses that encompasses all IPs of the AKS Cluster nodes (this can be a very large range if using node auto-scaling).
Create a VNet Rule on the Azure DB Server that allows access from the subnet the AKS nodes are in. This is used in conjunction with the Microsoft.Sql VNet Service Endpoint enabled on the cluster subnet.
VNet Rules are recommended and preferable in this situation for several reasons. Nodes are often configured with dynamic IP addresses that can change when a node is restarted resulting in broken firewall rules that reference specific IPs. Nodes can be added to a cluster which would require updating the firewall rule to add additional IPs. VNet Rules avoid these issues by granting access to an entire subnet of AKS nodes.
Manual steps
Configuring a secure networking environment for AKS and Azure DB requires the following:
AKS cluster setup
ResourceGroup: a logical grouping a resources required for all resources.
VNet: creates a virtual network for the AKS cluster nodes.
Subnet has a range of private IPs for AKS cluster nodes
Create an AKS cluster using the above resources.
Configure managed service access
VNet Service Endpoint: update the cluster subnet above with a service endpoint for Microsoft.Sql to enable connectivity for new Azure DB service resource.
Provision managed services with private IPs on the cluster’s network
Provision managed Azure DB service instances: PostgreSQL, MySQL.
VNet Rule for each managed service instance to allow traffic from all nodes in the cluster subnet to a given Azure DB service instance (PostgreSQL, MySQL).
I have found the issue, basically the Issue is on the AKS getting the wrong configuration, For the Identity, It doesn't read the proper appsettings.json, which it should be point to /secrets/*.json
AddEntityFrameworkStores()
I change the code to retrieve the information from the correct secret, the apps is work now.
Sadhus answer is correct and secure. But first you can quickly check by enabling the traffic as follows.
First select your server from your resource group.
Now in your sql server enable "Allow Azure services and resources to access this serve"
We have an application that runs on an Ubuntu VM. This application connects to Azure Redis, Azure Postgres and Azure CosmosDB(mongoDB) services.
I am currently working on moving this application to Azure AKS and intend to access all the above services from the cluster. The services will continue to be external and will not reside inside the cluster.
I am trying to understand how the network/firewall of both the services and aks should be configured so that pods inside the cluster can access the above services or any Azure service in general.
I tried the following:
Created a configMap containing the connection params(public ip/address, username/pwd, port, etc) of all the services and used this configMap in the deployment resource.
Hardcoded the connection params of all the services as env vars inside the container image
In the firewall/inbound rules of the services, I added the AKS API ip, individual node ips
None of the above worked. Did I miss anything? What else should be configured?
I tested the setup locally on minikube with all the services running on my local machine and it worked fine.
I am currently working on moving this application to Azure AKS and
intend to access all the above services from the cluster.
I assume that you would like to make all services to access each other and all the services are in AKS cluster? If so, I advise you configure the internal load balancer in AKS cluster.
Internal load balancing makes a Kubernetes service accessible to
applications running in the same virtual network as the Kubernetes
cluster.
You can take a try and follow the following document: Use an internal load balancer with Azure Kubernetes Service (AKS). In the end, good luck to you!
Outbound traffic in azure is SNAT-translated as stated in this article. If you already have a service in your AKS cluster, the outbound connection from all pods in your cluster will come thru the first LoadBalancer type service IP; I strongly suggest you create one for the sole purpose to have a consistent outbound IP. You can also pre-create a Public IP and use it as stated in this article using the LoadBalancerIP spec.
On a side note, rather than a ConfigMap, due to the sensitiveness of the connection string, I'd suggest you create a Secret and pass that down to your Deployment to be mounted or exported as environment variable.
I have an Azure Cloud Service (Worker Role) that needs to connect to my Azure SQL database and also connect to an external database.
In development the external database was on the public Internet and connectivity was not a problem.
However, the solution now needs to be deployed in a production environment and access to the external database is to be restricted by setting up a Virtual Network.
The Cloud Service, when deployed in the Virtual Network, gets an IP from the subnet, but seems to become inaccessible to the outside world, and is not connecting to the Azure SQL database (I also cannot RDP to it).
This seems to be beyond my level of understanding of networking, but I don't see why it should lose access to its neighbours in the Azure environment.
What am I missing? Do I need to get involved with Endpoints? Is the Virtual Network misconfigured?
Thanks in advance.
Your question is quite vague, in terms that it does describe the whole picture in the best possible way. Let me put my answer based on my understanding about your issue.
First of all - Azure virtual Network is Virtual Network. It is designed to enable secure cross-premisses connectivity with Windows Azure Data Center.
When you deploy a proper PaaS Cloud Service (Worker Role / Web Role) in a Virtual Network, the role instances get IP Addresses allocated from the defined DHCP pool (the VNet Definition).
When you deploy any service in an Azure Virtual Network you have to take care of Name Resolution! Meaning that, if you do not provide a proper DNS Server, your cloud instances will not be able to resolve any address. That includes Azure SQL Database servers. More on Name Resolution can be read here.
Next, but not less important - Azure SQL Database servers are not part of, and, as of March'2013, cannot be added to Azure Virtual Network!
The last statement means that in order for your Worker Role to access Azre SQL Database server, you need to provide a proper DNS server in your Azure Virtual Network.
And lastly, when you deploy a PaaS service into a Virtual Network, in order to access it via Remote Desktop need to:
* Properly enable and configure RDP extension. it will anyway create Input Endpoint. But this is the only way to enable RDP on PaaS right now;
* You could probably enable RDP via PowerShell startup script and access RDP via the VPN tunnel for the Virtual Network - say you configured a Site-to-Site or Point-to-site VPN for your Azure VNet.
Check the building cross-premises Virtual Network guide here.