I am trying to host an application in AWS Elastic Kubernetes Service(EKS). I have configured the EKS cluster using the AWS Console using an IAM user (user1). Configured the Node Group and added a Node to the EKS Cluster and everything is working fine.
In order to connect to the cluster, I had spin up an EC2 instance (Centos7) and configured the following:
1. Installed docker, kubeadm, kubelet and kubectl.
2. Installed and configured AWS Cli V2.
I had used the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID of user1 to configure AWS Cli from within the EC2 Instance in order to connect to the cluster using kubectl.
I ran the below commands in order to connect to the cluster as user1:
1. aws sts get-caller-identity
2. aws eks update-kubeconfig --name trojanwall --region ap-south-1
I am able to do each and every operations in the EKS cluster as user1.
However, I have now create a new user named 'user2' and I have replaced the current AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID with that of user2. Did the same steps and when I try to run 'kubectl get pods', I am getting the following error:
error: You must be logged in to the server (Unauthorized)
Result after running kubectl describe configmap -n kube-system aws-auth as user1:
Name: aws-auth
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
mapRoles:
----
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::XXXXXXXXXXXX:role/AWS-EC2-Role
username: system:node:{{EC2PrivateDNSName}}
BinaryData
====
Events: <none>
Does anyone know how to resolve this?
When you create an EKS cluster, only the user that created a cluster has access to it. In order to allow someone else to access the cluster, you need to add that user to the aws-auth. To do this, in your data section, add
mapUsers: |
- userarn: arn:was:iam::<your-account-id>:user/<your-username>
username: <your-username>
groups:
- systems:masters
You can use different groups, based on the rights you want to give to that user.
If you don't already have a config map on your machine:
Download the config map curl -o aws-auth-cm.yaml https://amazon-eks.s3.us-west-2.amazonaws.com/cloudformation/2020-10-29/aws-auth-cm.yaml
Replace default values with your values (role arn, username, account id...)
add the mapUsers section as described above
from terminal execute kubectl apply -f aws-auth-cm.yaml
You can also follow steps from the documentation (it's more detailed)
Related
We have a Jenkins virtual machine on GCE which deals with deployments, including the ones we do to GKE. We've tried to deploy a project which we have not touched for some time. The deployment failed when calling
kubectl set image deployment my-deployment my-deployment=gcr.io/my-project/my-project:version-tag
getting this error:
Error from server (Forbidden): deployments.extensions "my-deployment" is forbidden: User "client" cannot get resource "deployments" in API group "extensions" in the namespace "default"
The weird thing is, if I log in to the machine, use my Linux user + my gcloud user, I can deploy fine. But when switching to the jenkins user using su - jenkins and then authorizing gcloud with my user I get this same error that our deploy account gets.
Please advise how to fix.
It seems related to cluster RBAC configurations. Did you enable the RBAC fo Google Groups? In this case you should follow the instructions in the documentation above or disable it.
Otherwise, ss Raman Sailopal stated, you can try this:
with your regular user run kubectl config get-contexts to retrieve your current context
copy from /home/Linux user/.kube/config to /home/jenkins/.kube/config
change user to jenkins and be sure you're using the same context by running kubectl config get-contexts and kubectl config set-context ...
try your rights with:
# Check to see if I can create deployments in any namespace
kubectl auth can-i create deployments
# Check to see if I can list deployments in my current namespace
kubectl auth can-i list deployments.extensions
While trying to deploy an application got an error as below:
Error: UPGRADE FAILED: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Output of kubectl api-resources consists some resources along with the same error in the end.
Environment: Azure Cloud, AKS Service
Solution:
The steps I followed are:
kubectl get apiservices : If metric-server service is down with the error CrashLoopBackOff try to follow the step 2 otherwise just try to restart the metric-server service using kubectl delete apiservice/"service_name". For me it was v1beta1.metrics.k8s.io .
kubectl get pods -n kube-system and found out that pods like metrics-server, kubernetes-dashboard are down because of the main coreDNS pod was down.
For me it was:
NAME READY STATUS RESTARTS AGE
pod/coredns-85577b65b-zj2x2 0/1 CrashLoopBackOff 7 13m
Use kubectl describe pod/"pod_name" to check the error in coreDNS pod and if it is down because of /etc/coredns/Corefile:10 - Error during parsing: Unknown directive proxy, then we need to use forward instead of proxy in the yaml file where coreDNS config is there. Because CoreDNS version 1.5x used by the image does not support the proxy keyword anymore.
This error happens commonly when your metrics server pod is not reachable by the master node. Possible reasons are
metric-server pod is not running. This is the first thing you should check. Then look at the logs of the metric-server pod to check if it has some permission issues trying to get metrics
Try to confirm communication between master and slave nodes.
Try running kubectl top nodes and kubectl top pods -A to see if metric-server runs ok.
From these points you can proceed further.
I am getting issues when trying to getting the information about the nodes created using AKS(Azure Connected Service) for Kubernetes after the execution of creating the clusters and getting the credentials.
I am using the azure-cli on ubuntu linux machine.
Followed the Url for creation of clusters: https://learn.microsoft.com/en-us/azure/aks/kubernetes-walkthrough
I get the following error when using the command kubectl get nodes
after execution of connecting to cluster using
az aks get-credentials --resource-group <resource_group_name> --name <cluster_name>
Error:
kubectl get nodes
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)
I do get the same error when i use :
kubectl get pods -n kube-system -o=wide
When i connect back as another user by the following commands i.e.,
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
I will be able to retrieve the nodes i.e..,
kubectl get nodes
NAME STATUS ROLES AGE VERSION
<host-name> Ready master 20m v1.10.0
~$ kubectl get pods -n kube-system -o=wide
NAME READY STATUS RESTARTS AGE
etcd-actaz-prod-nb1 1/1 Running 0
kube-apiserver-actaz-prod-nb1 1/1 Running 0
kube-controller-manager-actaz-prod-nb1 1/1 Running 0
kube-dns-86f4d74b45-4qshc 3/3 Running 0
kube-flannel-ds-bld76 1/1 Running 0
kube-proxy-5s65r 1/1 Running 0
kube-scheduler-actaz-prod-nb1 1/1 Running 0
But this is actually overwriting newly clustered information from file $HOME/.kube/config
Am i missing something when we connect to AKS-cluster get-credentials command-let that's leading me to the error
*Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)*
After you
az aks get-credentials -n cluster-name -g resource-group
If should have merged to your local configuration:
/home/user-name/.kube/config
Can you check your config
kubectl config view
And check if it is pointing to the right cluster.
Assuming you have chosen default configuartion while deploying AKS. So You need to create SSH key pair to login to AKS Node.
Push above created public key to AKS node using "az vm user update" {plz take help to know what all switch you need to pass. It quite simple)
To create an SSH connection to an AKS node, you run a helper pod in your AKS cluster. This helper pod provides you with SSH access into the cluster and then additional SSH node access.
To create and use this helper pod, complete the following steps:
- Run a debian (or any other container like centos7 etc) container image and attach a terminal session to it. This container can be used to create an SSH session with any node in the AKS cluster:
kubectl run -it --rm aks-ssh --image=debian
The base Debian image doesn't include SSH components.
apt-get update && apt-get install openssh-client -y
Copy private key (the one you created in the begining to pod) using kubelet cmd. kubelet toolkit must be present on your machine from where you created ssh pair.
kubectl cp :/
Now you will see private key file on your container location, change the private key permission to 600 and now able to ssh your AKS node
Hope this helps.
I had pull my azure acs credentials using below command and I can communicate with kubernetes machine on Azure from my local machine
az acs kubernetes get-credentials --resource-group=<cluster-resource-group> --name=<cluster-name>
But Now I wanted to disconnect this connection so that my kubctl can connect with other machine , it can be local or any other machine (I am trying to connect with local).
But everytime I ran kubectl command it communicate with Azure ACS
For your scenario, we can use kubectl config use-context CONTEXT_NAME to switch default cluster to others, in this way, we can switch to another k8s cluster.
We can use this command to list k8s contexts:
root#shui:~# kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
jasontest321mgmt jasontest321mgmt jasontest321mgmt-admin
* jasonk8s321mgmt jasonk8s321mgmt jasonk8s321mgmt-admin
Specify k8s cluster name, we can use this commandkubectl config use-context CONTEXT_NAME:
root#shui:~# kubectl config use-context -h
Sets the current-context in a kubeconfig file
Examples:
# Use the context for the minikube cluster
kubectl config use-context minikube
Usage:
kubectl config use-context CONTEXT_NAME [options]
For example:
root#shui:~# kubectl config use-context jasontest321mgmt
Switched to context "jasontest321mgmt".
I followed the guide to getting Kubernetes running in Azure here:
http://kubernetes.io/docs/getting-started-guides/coreos/azure/
In order to create pods, etc., the guide has you ssh into the master node kube-00 in the cloud service and run kubectl commands there:
ssh -F ./output/kube_randomid_ssh_conf kube-00
Once in you can run the following:
kubectl get nodes
kubectl create -f ~/guestbook-example/
Is it possible to run these kubectl commands without logging to the master node, e.g., how can I set up kubectl to connect to the cluster hosted in Azure from my development machine instead of ssh'ing into the node this way?
I tried creating a context, user and cluster in the config but the values I tried using did not work.
Edit
For some more background the tutorial creates the azure cluster using a script using the Azure CLI. It ends up looking like this:
Resource Group: kube-randomid
- Cloud Service: kube-randomid
- VM: etcd-00
- VM: etcd-01
- VM: etcd-02
- VM: kube-00
- VM: kube-01
- VM: kube-02
It creates a Virtual Network that all of these VM's live in. As far as I can tell all of the machines in the cloud service share a single virtual IP.
The kubectl command line tool is just a wrapper to execute remote HTTPS API REST calls on the kubernetes cluster. If you want to be able to do so from your own machine you need to open the correct port (443) on your master node and pass along some parameters to the kubectl tool as specified in this tutorial:
https://coreos.com/kubernetes/docs/latest/configure-kubectl.html