I have an ingress-nginx ingress for my Kubernetes service.
If I curl the external IP from inside the cluster, like from a pod such as this kubectl run my-shell --rm -i --tty --image ubuntu -- bash, I get the result I expect.
If I curl from the internet, It times out. What could be the problem?
Resources online have pointed me to check my firewall and NSG settings. I see one of my NSGs has an explicit exception for the IP chosen by the ingress-nginx ingress, which seems to update automatically. Is there a way to see if there are other firewalls and network security groups between my AKS cluster and the internet?
This problem began after updating from kubernetes 1.19 to 1.22
I deleted the AKS cluster and recreated it. this resolved the issue but was obviously very destructive.
I'm beginning to build out a kubernetes cluster for our applications. We are using Azure for cloud services, so my K8s cluster is built using AKS. The AKs cluster was created using the portal interface for Azure. It has one node, and I am attempting to create a pod with a single container to deploy to the node. Where I am stuck currently is trying to connect to the AKS cluster from Powershell.
The steps I have taken are:
az login (followed by logging in)
az account set --subscription <subscription id>
az aks get-credentials --name <cluster name> --resource-group <resource group name>
kubectl get nodes
After entering the last line, I am left with the error: Unable to connect to the server: dial tcp: lookup : no such host
I've also gone down a few other rabbit holes found on SO and other forums, but quite honestly, I'm looking for a straight forward way to access my cluster before complicating it further.
Edit: So in the end, I deleted the resource I was working with and spun up a new version of AKS, and am now having no trouble connecting. Thanks for the suggestions though!
As of now, the aks run command adds a fourth option to connect to private clusters extending #Darius's three options posted earlier:
Use the AKS Run Command feature.
Below are some copy/paste excerpts of a simple command, and one that requires a file. It is possible to chain multiple commands with &&.
az aks command invoke \
--resource-group myResourceGroup \
--name myAKSCluster \
--command "kubectl get pods -n kube-system"
az aks command invoke \
--resource-group myResourceGroup \
--name myAKSCluster \
--command "kubectl apply -f deployment.yaml -n default" \
--file deployment.yaml
In case you get a (ResourceGroupNotFound) error, try adding the subscription, too
az aks command invoke \
--resource-group myResourceGroup \
--name myAKSCluster \
--subscription <subscription> \
--command "kubectl get pods -n kube-system"
You can also configure the default subscription:
az account set -s <subscription>
Unable to connect to the server: dial tcp: lookup : no such host
The error is coming because of private cluster. The Private Cluster option is enabled while creating the AKS cluster. You need to disable this option.
Kubectl is a kubernetes control client. It is an external connectivity provider to connect with our kubernetes cluster. We can't connect with the private cluster externally.
Believe me.... just disable the private cluster options And see your success. Thank you.
Note: We can't disable this option after the cluster creation. you need to delete the cluster and again reform it.
Posting this as Community Wiki for better visibility.
Solution provided by OP:
Delete resource and spun up a new version of AKS.
For details, you can check docs Create a resource group, Create AKS cluster and resource create.
Next try worth to try:
kubectl config use-context <cluster-name>
as it was proposed in similar Github issue.
Gaurav's answer pretty much sums it up. In fact you can refer to the documentation which states that
The API server endpoint has no public IP address. To manage the API
server, you'll need to use a VM that has access to the AKS cluster's
Azure Virtual Network (VNet). There are several options for
establishing network connectivity to the private cluster.
To connect to a private cluster, there are only 3 methods:
Create a VM in the same Azure Virtual Network (VNet) as the AKS cluster.
Use a VM in a separate network and set up Virtual network peering. See the section below for more information on this option.
Use an Express Route or VPN connection.
It is more convenient to use Az module from desktop Powershell for any management operation with Azure portal. Microsoft adds a lot of new cmdlets for managing AKS and Service Fabric clusters.
Please take a look Az.Aks
In your case:
Connect-AzAccount
Get-AzAksNodePool
I was also facing the issue, I'm using a private cluster and I have a machine (bastion) in a different vnet with peering enabled but still, I was not able to connect the cluster (I was able to SSH and telnet to the machine).
Then I added a virtual network link in the private DNS zone for the vnet where the bastion host resides. It worked for me, I'm able to access the cluster.
When using a private cluster, the kubernetes api-endpoint is only accessible on the cluster's VNet. Connecting via VPN unfortunately does not work painlessly since the azure private DNS will not be available via for VPN clients (yet).
However, it is possible to connect kubectl directly to the IP-address of the api-endpoint, but that will require you to ignore certificate errors since we are using the IP directly.
If you edit your .kube/config and change the server address to the IP number. Then call kubectl with something like this
kubectl get all --all-namespaces --insecure-skip-tls-verify
Usually, this is all that is required to connect. Check whether firewall is not blocking any traffic. Also, verify subscription id and other identifiers again and make sure you are using the correct values. If the issue still persists, I recommend you ask azure support to help you out.
I had the same issues when running the kubectl command from jenkins. For me it was the permission issues of ~/.kube/config I gave it access to jenkins as well which solved the issue for me.
You can run kubectl commands on a private AKS cluster using az aks command invoke. Refer to this for more info.
As for why you might want to run private AKS clusters, read this
You can simply append "--admin" to the query as seen below.
az aks get-credentials --name <cluster name> --resource-group <resource group name> --admin
I also hit this after restarting my kubernetes cluster, but it turned out I was just not waiting long enough, after about 10 minutes the "kubectrl" commands started working again.
If you are using AWS with kops then this might help you
mkdir autoscaler
cd autoscaler/
git clone https://github.com/kubernetes/autoscaler.git
create a file called ig-policy.json with the contents
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
Then you need to create iam policy
aws iam create-policy --policy-name ig-policy --policy-document file://ig-policy.json
And attach the above create iam policy with the user id to the cluster name
aws iam attach-role-policy --policy-arn arn:aws:iam::005935423478650:policy/ig-policy --role-name nodes.testing.k8s.local
Then update the cluster
kops update cluster testing.k8s.local --yes
Then run
kops rolling-update cluster
Creating private not easy journey, but it has beautiful views so I encourage anyone to get there.
I did it all in terraform, so some names can be little different than they are in portal/azure CLI.
And this is how I did it:
Private DNS zone, with name as privatelink.westeurope.azmk8s.io
VNET where AKS will be placed (let's call it vnet-access)
Virtual network from which you want to access AKS
Private AKS (private_dns_zone_id set to dns zone form first point)
Virtual network link (in private DNS zone, pointing to VNET from point 3)
Peering between networks from points 2 and 3.
This should allow any machine in vnet-access to firstly resolve DNS, and then - to access cluster...
Yet... if you want to get there from your local machine, this is another setup. Fortunately Microsoft have such tutorial here
If you find that something is still not working - put the error in comment and I'll try to adapt my answer to cover this.
For me I had this issue when I was trying to connect a new Linux user to my Elastic Kubernetes Cluster in AWS.
I setup a new user called jenkins-user, then I tried to run the command below to get pods:
kubectl get pods
And then I will run into the error below:
Unable to connect to the server: dial tcp: lookup 23343445ADFEHGROGMFDFMG.sk1.eu-east-2.eks.amazonaws.com on 198.74.83.506:53: no such host
Here's how I solved it:
The issue was because I had not set the context for the Kubernetes cluster in the kube config file of the new linux user (jenkins-user).
All I had to do was either first install the aws-cli for this new user (install it into the home directory of this new user). And then run the command aws configure to configure the necessary credentials. Although, since I already had the aws-cli setup for the other users on the Linux system I simply copied the ~/.aws directory from an already existing user to the jenkins-user home directory using the command:
sudo cp -R /home/existing-user/.aws/ /home/jenkins-user/
Next, I had to create a context for the Kubernetes configuration which will create a new ~/.kube/config file for the jenkins-user using the command below:
aws eks --region my-aws-region update-kubeconfig --name my-cluster-name
Next, I checked the kube config file to confirm that my context has been added using the command:
sudo nano /.kube/config
This time when I ran the command below, it was successful:
kubectl get pods
Resources: Create a kubeconfig for Amazon EKS
I faced the same issue and resolved it by deleting .kube folder which was under the following path C:\Users\<your_username> and then restarting kubernetes cluster.
I am getting issues when trying to getting the information about the nodes created using AKS(Azure Connected Service) for Kubernetes after the execution of creating the clusters and getting the credentials.
I am using the azure-cli on ubuntu linux machine.
Followed the Url for creation of clusters: https://learn.microsoft.com/en-us/azure/aks/kubernetes-walkthrough
I get the following error when using the command kubectl get nodes
after execution of connecting to cluster using
az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>
Error:
kubectl get nodes
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)
I do get the same error when i use :
kubectl get pods -n kube-system -o=wide
When i connect back as another user by the following commands i.e.,
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
I will be able to retrieve the nodes i.e..,
kubectl get nodes
NAME STATUS ROLES AGE VERSION
<host-name> Ready master 20m v1.10.0
~$ kubectl get pods -n kube-system -o=wide
NAME READY STATUS RESTARTS AGE
etcd-actaz-prod-nb1 1/1 Running 0
kube-apiserver-actaz-prod-nb1 1/1 Running 0
kube-controller-manager-actaz-prod-nb1 1/1 Running 0
kube-dns-86f4d74b45-4qshc 3/3 Running 0
kube-flannel-ds-bld76 1/1 Running 0
kube-proxy-5s65r 1/1 Running 0
kube-scheduler-actaz-prod-nb1 1/1 Running 0
But this is actually overwriting newly clustered information from file $HOME/.kube/config
Am i missing something when we connect to AKS-cluster get-credentials command-let that's leading me to the error
*Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)*
After you
az aks get-credentials -n cluster-name -g resource-group
If should have merged to your local configuration:
/home/user-name/.kube/config
Can you check your config
kubectl config view
And check if it is pointing to the right cluster.
Assuming you have chosen default configuartion while deploying AKS. So You need to create SSH key pair to login to AKS Node.
Push above created public key to AKS node using "az vm user update" {plz take help to know what all switch you need to pass. It quite simple)
To create an SSH connection to an AKS node, you run a helper pod in your AKS cluster. This helper pod provides you with SSH access into the cluster and then additional SSH node access.
To create and use this helper pod, complete the following steps:
- Run a debian (or any other container like centos7 etc) container image and attach a terminal session to it. This container can be used to create an SSH session with any node in the AKS cluster:
kubectl run -it --rm aks-ssh --image=debian
The base Debian image doesn't include SSH components.
apt-get update && apt-get install openssh-client -y
Copy private key (the one you created in the begining to pod) using kubelet cmd. kubelet toolkit must be present on your machine from where you created ssh pair.
kubectl cp :/
Now you will see private key file on your container location, change the private key permission to 600 and now able to ssh your AKS node
Hope this helps.
I had pull my azure acs credentials using below command and I can communicate with kubernetes machine on Azure from my local machine
az acs kubernetes get-credentials --resource-group=<cluster-resource-group> --name=<cluster-name>
But Now I wanted to disconnect this connection so that my kubctl can connect with other machine , it can be local or any other machine (I am trying to connect with local).
But everytime I ran kubectl command it communicate with Azure ACS
For your scenario, we can use kubectl config use-context CONTEXT_NAME to switch default cluster to others, in this way, we can switch to another k8s cluster.
We can use this command to list k8s contexts:
root#shui:~# kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
jasontest321mgmt jasontest321mgmt jasontest321mgmt-admin
* jasonk8s321mgmt jasonk8s321mgmt jasonk8s321mgmt-admin
Specify k8s cluster name, we can use this commandkubectl config use-context CONTEXT_NAME:
root#shui:~# kubectl config use-context -h
Sets the current-context in a kubeconfig file
Examples:
# Use the context for the minikube cluster
kubectl config use-context minikube
Usage:
kubectl config use-context CONTEXT_NAME [options]
For example:
root#shui:~# kubectl config use-context jasontest321mgmt
Switched to context "jasontest321mgmt".
I followed the guide to getting Kubernetes running in Azure here:
http://kubernetes.io/docs/getting-started-guides/coreos/azure/
In order to create pods, etc., the guide has you ssh into the master node kube-00 in the cloud service and run kubectl commands there:
ssh -F ./output/kube_randomid_ssh_conf kube-00
Once in you can run the following:
kubectl get nodes
kubectl create -f ~/guestbook-example/
Is it possible to run these kubectl commands without logging to the master node, e.g., how can I set up kubectl to connect to the cluster hosted in Azure from my development machine instead of ssh'ing into the node this way?
I tried creating a context, user and cluster in the config but the values I tried using did not work.
Edit
For some more background the tutorial creates the azure cluster using a script using the Azure CLI. It ends up looking like this:
Resource Group: kube-randomid
- Cloud Service: kube-randomid
- VM: etcd-00
- VM: etcd-01
- VM: etcd-02
- VM: kube-00
- VM: kube-01
- VM: kube-02
It creates a Virtual Network that all of these VM's live in. As far as I can tell all of the machines in the cloud service share a single virtual IP.
The kubectl command line tool is just a wrapper to execute remote HTTPS API REST calls on the kubernetes cluster. If you want to be able to do so from your own machine you need to open the correct port (443) on your master node and pass along some parameters to the kubectl tool as specified in this tutorial:
https://coreos.com/kubernetes/docs/latest/configure-kubectl.html