How to use static External IP for Azure kubernetes LoadBalancer? - azure

I created a deployment at AKS:
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: frontend
strategy:
type: Recreate
template:
metadata:
labels:
io.kompose.service: frontend
spec:
containers:
- image: app:latest
name: frontend
volumeMounts:
- mountPath: /app/db
name: db
- mountPath: /root/.aspnet/https
name: https
readOnly: true
env:
- name: ASPNETCORE_URLS
value: "https://+;http://+"
- name: ASPNETCORE_HTTPS_PORT
value: "443"
- name: ASPNETCORE_Kestrel__Certificates__Default__Path
value: "/root/.aspnet/https/cert.pfx"
- name: ASPNETCORE_Kestrel__Certificates__Default__Password
valueFrom:
secretKeyRef:
name: certificate-pass
key: pass
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: db
persistentVolumeClaim:
claimName: db
- name: https
secret:
secretName: certificate
items:
- key: file
path: cert.pfx
and a service:
apiVersion: v1
kind: Service
metadata:
name: frontend-service
spec:
selector:
io.kompose.service: frontend
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
- name: https
protocol: TCP
port: 443
targetPort: 443
type: LoadBalancer
Service is created successfully. I can access it using provided External IP:
Now I want to make this IP static. There is an official docs which tells how to make it: Use a static public IP address and DNS label with the Azure Kubernetes Service (AKS) load balancer
There is also an article which technically duplicates the docs, but in a bit more details: Use a static public IP address outside of the node resource group with the Azure Kubernetes Service (AKS) load balancer
I am able to create an IP address, but when I reach az role assignment create command it fails ($GROUP here is just a placeholder for real Resource group literal):
$ CLIENT_ID=$(az aks show --resource-group Default --name k8s --query "servicePrinci
palProfile.clientId" --output tsv)
$ SUB_ID=$(az account show --query "id" --output tsv)
$ az role assignment create --assignee $CLIENT_ID --role "Network Contributor" --scope /subscriptions/$SUB_ID/resourceGroups/$GROUP
If the assignee is an appId, make sure the corresponding service principal is created with 'az ad sp create --id $CLIENT_ID
When I try proposed command if fails:
$ az ad sp create --id $CLIENT_ID
Another object with the same value for property servicePrincipalNames already exists.
I found similar issue at GitHub Azure/azure-cli repo, but there is no answer.
If I skip this step and set loadBalancerIP: XXX.XXX.XXX.XXX at config level, load balancer has a new property Load balancer IP, but External IP is not changed:
How to mitigate this error?

According to the messages you provide, I'm very confused about why you need to run the command az ad sp create --id $CLIENT_ID, both the docs do not show the necessity to run this command.
As I know, you only need to assign the "Network Contributor" role of the group other than the AKS node group to the service principal of the AKS. That's the right step you need to do. I think you need to read the docs again with more attention.

I had to use literal value instead of a variable $CLIENT_ID. That helped.

Related

Aks SecretProviderClass secret not found

EDIT: Found the issue.I didnt installed the addon for secret driver. Once installed that i was able to make it work
I am facing an issue here and i have no idea what else i can try to figure out the issue.
I have an aks running with a single pod that runs a basic web app todo list. Nothing too fancy or complicated. what i am trying to do here, is to give permission to the aks cluster to access a keyvault and GET a secret to pass to the pod. the secret is just an ASPNETCORE_ENVIRONMENT: Development.
Following the documentations, i used helm to install the repo:
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure
I created a Service Principle in azure:
SERVICE_PRINCIPLE_CLIENT_SECRET = az ad sp create-for-rbac --skip-assignment --name sp-aks-keyvault
i queried the clientId and Secret and passed them to my cluster as follow:
kubectl create secret generic secrets-store-creds --from-literal clientid="ClientID" --from-literal clientsecret="Password"
Once everything was set. I set those deployments.
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp-deployment
namespace: default
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: dockerimage-acr
ports:
- containerPort: 80
env:
- name: ASPNETCORE_ENVIRONMENT
valueFrom:
secretKeyRef:
name: aspenet-environment
key: environment
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: secrets-mount
mountPath: "/mnt/secrets-store"
readOnly: true
restartPolicy: Always
volumes:
- name: secrets-mount
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "kv-name"
nodePublishSecretRef: # Only required when using service principal mode
name: secrets-store-creds
And my secretProvider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: keyvault-secret-class
namespace: default
spec:
provider: azure
secretObjects:
- secretName: aspenet-environment
type: Opaque
data:
- objectName: aspnetcoreenvironment
key: environment
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "false"
userAssignedIdentityID: ""
keyvaultName: "mykeyvault-name"
objects: |
array:
- |
objectName: aspnetcoreenvironment
objectType: secret
objectVersion: ""
tenantId: "<Tenant-Id>"
In my keyvault i gave access policy to the Service principle created and assigned Secret Permissions: GET and created a secret called
Name: aspnetcoreenvironment
value: Development
So far everything went ok, but when i run the deployment. and use the command kubectl describe pod <podname> i see the error, that prevents the container to start
Warning Failed 8s (x3 over 21s) kubelet Error: secret "aspenet-environment" not found
I tried different solutions but nothing works.
if i run the command kubectl get secretproviderclass i get back my provider i created.
As far as i understand, if no service is requiring a specific secret, i wont be able to find the secret i want to create if i run the command: kubectl get secret
And this is correct, i guess, because my pod is not starting.
Any help or enlightenment here about what i am doing wrong or how to fix it?
Thank you so much guys
EDIT:
Some extra debugging i came across the fact that the volume mount is still required. So i did add the volume to the deployment. But this is still giving an error.
The issue is, as i realized. Is when i run the command kubectl apply -f secretProviderClass.yml, no secret get created at all, reason why is failing.
So i think something is wrong here. Applying the SecretProviderClass shouldnt create automatically a secret service?

Secret is not creating in AKS after fetching it with CSI Driver

By using the reference of https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls this document, I'm trying to fetch the TLS secrets from AKV to AKS pods.
Initially I created and configured CSI driver configuration with using User Assigned Managed Identity.
I have performed the following steps:
Create AKS Cluster with 1 nodepool.
Create AKV.
Created user assigned managed identity and assign it to the nodepool i.e. to the VMSS created for AKS.
Installed CSI Driver helm chart in AKS's "kube-system" namespace. and completed all the requirement to perform this operations.
Created the TLS certificate and key.
By using TLS certificate and key, created .pfx file.
Uploaded that .pfx file in the AKV certificates named as "ingresscert".
Created new namespace in AKS named as "ingress-test".
Deployed secretProviderClass in that namespace are as follows.:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-tls
spec:
provider: azure
secretObjects: # secretObjects defines the desired state of synced K8s secret objects
- secretName: ingress-tls-csi
type: kubernetes.io/tls
data:
- objectName: ingresscert
key: tls.key
- objectName: ingresscert
key: tls.crt
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "7*******-****-****-****-***********1"
keyvaultName: "*****-*****-kv" # the name of the AKV instance
objects: |
array:
- |
objectName: ingresscert
objectType: secret
tenantId: "e*******-****-****-****-***********f" # the tenant ID of the AKV instance
Deployed the nginx-ingress-controller helm chart in the same namespace, where certificates are binded with application.
Deployed the Busy Box deployment are as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-one
labels:
app: busybox-one
spec:
replicas: 1
selector:
matchLabels:
app: busybox-one
template:
metadata:
labels:
app: busybox-one
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-tls"
---
apiVersion: v1
kind: Service
metadata:
name: busybox-one
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: busybox-one
Check secret is created or not by using command
kubectl get secret -n <namespaceName>
One thing to notice here is, if I attach shell with the busy box pod and go to the mount path which I provided to mount secrets I have seen that secrets are successfully fetched there. But this secrets are not showing in the AKS's secret list.
I have troubleshooted all the AKS,KV and manifest files but not found anything.
IF there is anything I have missed or anyone has solution for this please let me know.
Thanks in advance..!!!
i added this as a new answer, bcs the formatting was bad in the comments:
As you are using the Helm chart, you have to activate the secret sync in the values.yaml of the Helm Chart:
secrets-store-csi-driver:
syncSecret:
enabled: true
I would still recommend to use the csi-secrets-store-provider-azure as AKS Addon instead of the Helm-Chart
Your config looks good to me. One thing to consider is, that the User Assigned Managed Identity should not be the one you created for the AKS, it should be the managed identity from your nodepool (kubelet) and it also needs permission on the AKV.
I had the same issues while using the wrong Managed identity.
userAssignedIdentityID = Kubelet Client Id ( Nodepool Managed Idendity )
AZ CLI
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

Refer the docker image from another azure container repository (ACR) of different subscription

I am trying to pull the docker image in (QA-ACR) of subscription (QA-Subscription) from another Azure Container Registry (DEV-ACR) in subscription (DEV-Subscription).
Below are the steps in detail.
Created the docker image (example: docker-image-sample) in Subscription DEV-Subscription
Created the secret file by using the following command in Subscription DEV-Subscption
kubectl create secret docker-registry test-secret --docker-server=devsample.azurecr.io --docker-username=**** --docker-password=****
Pod is running in DEV-subscription by referring this secret. below is deployment file
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: test
spec:
replicas: 2
template:
metadata:
labels:
app: test
spec:
containers:
- image: devsample.azurecr.io/test_msdi:latest
imagePullPolicy: Always
name: test
ports:
- containerPort: 443
env:
- name: ASPNETCORE_ENVIRONMENT
value: dev
imagePullSecrets:
- name: test-secret
I am trying to pull the docker image from another ACR in different subscription.
Created the same secret here also like above.
Below is the content of the kubernetes deployment file
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: test
spec:
replicas: 2
template:
metadata:
labels:
app: test
spec:
containers:
- image: devsample.azurecr.io/test_msdi:latest
imagePullPolicy: Always
name: test
ports:
- containerPort: 443
env:
- name: ASPNETCORE_ENVIRONMENT
value: qa
imagePullSecrets:
- name: test-secret
Pod is failing from another ACR of different subscription. Issue is "Back off pulling the image ..."
Since your using an Azure Container Registry you might find it easier to assign the AKS Service Principal permissions on the container registry rather than rely on passing in credentials using a Kubernetes secret.
$Aks = Get-AzAks -ResourceGroupName QaSubscriptionAksResourceGroup -Name QaSubscriptionAks
New-AzRoleAssignment -ApplicationId $Aks.ServicePrincipalProfile.ClientId -RoleDefinitionName AcrPull -ResourceGroupName DevSubscriptionAcrResourceGroup
You might need to run Select-AzSubscription between the two commands to change from the QA subscription to the DEV subscription. Once that's set up remove
imagePullSecrets:
- name: test-secret
from your deployment file and rerun it.
Depending on how your AKS instances were deployed you might find that the AKS Service Principals already have the AcrPull role assigned within their own subscriptions, if that's the case you can remove imagePullSecrets completely.

Unable to assign public ip address to AKS: pending forever

I allocated an IP address for my resource group as the following:
az network public-ip create --resource-group myResourceGroup --name ipName --allocation-method static
Now, I'd like to assign it to my AKS so I just altered the yaml as it follows:
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
loadBalancerIP: xx.xx.xxx.xxx <--the ip generated before
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-sgr
Then I run:
kubectl apply -f mykube.yaml
But it appears to be stuck:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.0.77.234 <pending> 80:32320/TCP 79m
By executing describe I get indeed the following:
Warning CreatingLoadBalancerFailed 21m (x19 over 86m) service-controller Error creating
load balancer (will retry): failed to ensure
load balancer for service default/nginx: user supplied IP Address
xx.xx.xxx.xxx was not found in resource group
MC_**myResourceGroup**_myAKSCluster_westeurope
please note that it seems it's searching in a resource group that is composed by the resource group I specified in the first command (the same as kubernates is) and other information...what am I doing wrong?
As I know, the possible reason is that you need to assign your AKS the permission of the resource group which you create the public IP if you create it in another group. For more details, see Use a static IP address outside of the node resource group. And you need to add the annotations like below:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
name: azure-load-balancer
spec:
loadBalancerIP: 40.121.183.52
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-load-balancer
Or you can just create the public IP in your AKS cluster nodes group. For you, the group name can be found in the error you provide: MC_**myResourceGroup**_myAKSCluster_westeurope.

Cannot apply SSL cert to Kubernetes LoadBalancing Service on Kubernetes

Issue
I am having trouble applying TLS to the DNS name of my LoadBalancer service for my Kubernetes cluster, and I am at a bit of a loss.
This is the first time I have worked with Kubernetes as well as Azure's Manage Container Services. For reasons that are out of my control this api is required to run on Azure's Managed Container Services.
Environment
Cluster is running on Azure using Managed Container Services (preview). I created my environment by following the steps here: https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster
I created a static IP in Azure to use in the yaml for the loadbalancer service. Furthermore, I created a myprefix.cloudapp.azure.com DNS name for the IP using the following commands (https://learn.microsoft.com/en-us/azure/aks/static-ip)
IP="XX.XX.XX.XX"
DNSNAME="myprefix"
RESOURCEGROUP=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[resourceGroup]" --output tsv)
PIPNAME=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[name]" --output tsv)
az network public-ip update --resource-group $RESOURCEGROUP --name $PIPNAME --dns-name $DNSNAME
Deployment
This is the yaml I am using for my deployment:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: my-node-express-api-deployment
spec:
replicas: 2
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
minReadySeconds: 5
template:
metadata:
labels:
app: my-node-express-api
spec:
containers:
- name: my-node-express-api-container
image: myrepo/my-node-express-api-image:latest
ports:
- containerPort: 3000
volumes:
- name: tls
secret:
secretName: my-tls-secret
Service
This is the yaml for my LoadBalancing Service
apiVersion: v1
kind: Service
metadata:
name: my-node-express-api-loadbalancer
spec:
loadBalancerIP: 52.176.148.91
type: LoadBalancer
ports:
- port: 80
targetPort: 3000
port: 443
targetPort: 3000
selector:
app: my-node-express-api
Secret
Yaml for secret
apiVersion: v1
kind: Secret
metadata:
name: my-tls-secret
namespace: default
data:
tls.crt: (base64 for myprefix.cloudapp.azure.com.crt)
tls.key: (base64 for myprefix.cloudapp.azure.com.key)
Note:
Everything works correctly over http when I remove the Secret from my deployment and remove port 443 from the LoadBalancer Service.
On Azure, if you need TLS termination on kubernetes, you can use Nginx Ingress controller(Now, Microsoft working with Azure ingress controller which uses Application gateway).
To archive this, we can follow those steps:
1 Deploy the Nginx Ingress controller
2 Create TLS certificates
3 Deploy test http service
4 configure TLS termination
More information about configure Nginx ingress controller for TLS termination on kubernetes on Azure, please refer to this blog.

Resources