Is there a way to connect to Azure HDInsight Kafka cluster from the public network without using a VPN ?
Is it only possible to connect to Azure Kafka cluster from with in Azure network ?
Thanks
You don't need a VPN. Just open the firewall ports from Azure
Related
I'm trying to create a cluster in Azure Databricks and getting a such error messgae
Resources were not reachable via SSH. If the problem persists, this usually indicates a network environment misconfiguration. Please check your cloud provider configuration, and make sure that Databricks control plane can reach Spark clusters instances.
I have such the default configuration:
Cluster mode: Standard
Pool: None
Runtime version: 5.5 LTS
Autoscaling enabled
Worker Type: Standard_DS3_v2
Driver Type: Standard_DS3_v2
From Logs Analytics I see Azure tried to create virtual machines and without any reason (I suppose because they were unreachable) had to delete all of them.
Did anyone face such issue?
If the issue is temporary, this may be caused by the driver of the virtual machine going down or a networking issue since Azure Databricks was able to launch the cluster, but lost the connection to the instance hosting the Spark driver referring to this. You could try to remove it and create the cluster again.
If the problem persists, this may happen when you have an Azure Databricks workspace deployed to your own VNet. If the virtual network where the workspace is deployed is already peered or has an ExpressRoute connection to on-premises resources, the virtual network cannot make an ssh connection to the cluster node when Azure Databricks is attempting to create a cluster. You could add a user-defined route (UDR) to give the Azure Databricks control plane ssh access to the cluster instances.
For detailed UDR instructions, see Step 3: Create user-defined routes and associate them with your Azure Databricks virtual network subnets. For more VNet-related troubleshooting information, see Troubleshooting
Hope this could help you.
Issue: Instances Unreachable: Resources were not reachable via SSH.
Possible cause: traffic from control plane to workers is blocked. If you are deploying to an existing virtual network connected to your on-premises network, review your setup using the information supplied in Connect your Azure Databricks Workspace to your On-Premises Network.
Reference: Azure Databricks - Troubleshooting
Hope this helps.
We setup a Point-to-Site azure VPN gateway to allow us to connect to our Virtual Machines and azure SQL instance. The gateway is linked to the Virtual Network where our VM's live, and the Azure SQL has a firewall rule to allow connections from That virtual network. We are able to connect to the VM's but not to the Azure SQL.
I tried linking the Virtual network to the Azure SQL through the firewall settings, it didn't help.
I assume you are using Azure SQL database. It's public and does not deploy in a VNet. Also, VNet Service Endpoints don’t extend to on-premises. So It did not help to connect to Azure SQL database over VPN connection. If you want it to work, you need whitelist your on-premise client public IP address in the firewall of Azure SQL database.
Otherwise, you need to deploy your SQL instance in a VNet, so you need to use Azure SQL Database managed instance. You could follow this quickstart: Configure a point-to-site connection to an Azure SQL Database Managed Instance from on-premises
I have a Azure Container Instance, and I want to connect to a kafka that is also in the azure.
If in kafka configure the advertised.listeners with dns I can connect, however with hostname I can not.
In ACI I also can not ping/wget/telnet to other internal resources, just for other ACIs.
vnet kafka = vnetA
subnet kafka = subnetA
vnet ACI = vnetA
subnet ACI = subnetB
I created the ACI with private ip.
This article explains how to configure your listeners in this situation: https://rmoff.net/2018/08/02/kafka-listeners-explained/
For your issue that the Azure Container Instances connect with other Azure resources. Now the Azure Container Instance just supports a private IP and no DNS labels, and the instances can connect with other resources in the same Vnet or in different Vnet with peering.
Container groups deployed to a virtual network do not currently
support public IP addresses or DNS name labels.
So you can just connect the ACI with the private IP the Vnet. And it just is a preview version. For more details, see Deploy container instances into an Azure virtual network
I'm able to connect to hive externally using the following URL for a HDInsight cluster in Azure.
jdbc:hive2://<host>:443/default;transportMode=http;ssl=true;httpPath=/
However, I'm not able to find such a string for spark. The documentation says the port is 10002, but its not open externally. How do I connect to the cluster to run SparkSQL queries through JDBC?
There is not one available. But you can vote for the feature at https://feedback.azure.com/forums/217335-hdinsight/suggestions/14794632-create-a-jdbc-driver-for-spark-on-hdinsight.
HDInsight is deployed with a gateway. This is the reason why HDInsight clusters out-of-box enable only HTTPS (Port 443) and SSH (Ports 22, 23) communication to the cluster. If you don' t deploy the cluster in a virtual network (vnet) there is no other way you can communicate with HDInsight clusters. So instead of Port 10002 Port 443 is used if you want to access the Spark thrift server. If you deploy the cluster in a vnet, you could also access the thrift server via the ip address it is running on (one of the headnodes) and standard port 10002. See also public and non-public ports in the documentation.
I have a microsoft Azure HDInsight cluster.
On the node I am rdp'ing and starting an application that binds to port 8080. I would like to be able to connect to this application from outside the cluster.
I have my cluster connection string (https://xxxxx.azurehdinsight.net) however when I try to connect to it I am timing out.
I believe this is due to the fact that I have not opened port 8080 to public. How can I do this as under the cluster I only have Hadoop Services and username....
At this point in time, we don't allow you to control / open additional network ports on an HDInsight cluster.
You can deploy an HDInsight cluster into an Azure Virtual network if you'd like to have another machine in Azure to have access to all of the ports/nodes on the cluster. We've documented how to deploy into a vnet in this article.