I allocated an IP address for my resource group as the following:
az network public-ip create --resource-group myResourceGroup --name ipName --allocation-method static
Now, I'd like to assign it to my AKS so I just altered the yaml as it follows:
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
loadBalancerIP: xx.xx.xxx.xxx <--the ip generated before
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-sgr
Then I run:
kubectl apply -f mykube.yaml
But it appears to be stuck:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.0.77.234 <pending> 80:32320/TCP 79m
By executing describe I get indeed the following:
Warning CreatingLoadBalancerFailed 21m (x19 over 86m) service-controller Error creating
load balancer (will retry): failed to ensure
load balancer for service default/nginx: user supplied IP Address
xx.xx.xxx.xxx was not found in resource group
MC_**myResourceGroup**_myAKSCluster_westeurope
please note that it seems it's searching in a resource group that is composed by the resource group I specified in the first command (the same as kubernates is) and other information...what am I doing wrong?
As I know, the possible reason is that you need to assign your AKS the permission of the resource group which you create the public IP if you create it in another group. For more details, see Use a static IP address outside of the node resource group. And you need to add the annotations like below:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
name: azure-load-balancer
spec:
loadBalancerIP: 40.121.183.52
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-load-balancer
Or you can just create the public IP in your AKS cluster nodes group. For you, the group name can be found in the error you provide: MC_**myResourceGroup**_myAKSCluster_westeurope.
Related
By default the SKU used when creating a load balancer in AKS is standard, for development if you want to use basic SKU we have to use the command line az aks create -g RGName -n ClusterName --load-balancer-sku basic
But could not find anything on how to specify the --load-balancer-sku in the yaml file.
Current YAML File AS-IS given below, what to add to make the SKU basic?
apiVersion: v1
kind: Service
metadata:
name: hello-world-svc
spec:
selector:
app: hello-world-svc
ports:
- protocol: TCP
port: 80
targetPort: 3000
type: LoadBalancer
you can specify the SKU per ENV with passing a config file to the Cloud Controller Manager as of here
I created a deployment at AKS:
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: frontend
strategy:
type: Recreate
template:
metadata:
labels:
io.kompose.service: frontend
spec:
containers:
- image: app:latest
name: frontend
volumeMounts:
- mountPath: /app/db
name: db
- mountPath: /root/.aspnet/https
name: https
readOnly: true
env:
- name: ASPNETCORE_URLS
value: "https://+;http://+"
- name: ASPNETCORE_HTTPS_PORT
value: "443"
- name: ASPNETCORE_Kestrel__Certificates__Default__Path
value: "/root/.aspnet/https/cert.pfx"
- name: ASPNETCORE_Kestrel__Certificates__Default__Password
valueFrom:
secretKeyRef:
name: certificate-pass
key: pass
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: db
persistentVolumeClaim:
claimName: db
- name: https
secret:
secretName: certificate
items:
- key: file
path: cert.pfx
and a service:
apiVersion: v1
kind: Service
metadata:
name: frontend-service
spec:
selector:
io.kompose.service: frontend
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
- name: https
protocol: TCP
port: 443
targetPort: 443
type: LoadBalancer
Service is created successfully. I can access it using provided External IP:
Now I want to make this IP static. There is an official docs which tells how to make it: Use a static public IP address and DNS label with the Azure Kubernetes Service (AKS) load balancer
There is also an article which technically duplicates the docs, but in a bit more details: Use a static public IP address outside of the node resource group with the Azure Kubernetes Service (AKS) load balancer
I am able to create an IP address, but when I reach az role assignment create command it fails ($GROUP here is just a placeholder for real Resource group literal):
$ CLIENT_ID=$(az aks show --resource-group Default --name k8s --query "servicePrinci
palProfile.clientId" --output tsv)
$ SUB_ID=$(az account show --query "id" --output tsv)
$ az role assignment create --assignee $CLIENT_ID --role "Network Contributor" --scope /subscriptions/$SUB_ID/resourceGroups/$GROUP
If the assignee is an appId, make sure the corresponding service principal is created with 'az ad sp create --id $CLIENT_ID
When I try proposed command if fails:
$ az ad sp create --id $CLIENT_ID
Another object with the same value for property servicePrincipalNames already exists.
I found similar issue at GitHub Azure/azure-cli repo, but there is no answer.
If I skip this step and set loadBalancerIP: XXX.XXX.XXX.XXX at config level, load balancer has a new property Load balancer IP, but External IP is not changed:
How to mitigate this error?
According to the messages you provide, I'm very confused about why you need to run the command az ad sp create --id $CLIENT_ID, both the docs do not show the necessity to run this command.
As I know, you only need to assign the "Network Contributor" role of the group other than the AKS node group to the service principal of the AKS. That's the right step you need to do. I think you need to read the docs again with more attention.
I had to use literal value instead of a variable $CLIENT_ID. That helped.
Got following problem:
- My setup is aks in azure. Inside got few services and ambassador working in front of them.
- I know how to setup public static ip for "regular" load balancer (https://learn.microsoft.com/en-us/azure/aks/static-ip)
- When im trying to do the same in my ambassador yaml:
apiVersion: v1
kind: Service
(..)
spec:
type: LoadBalancer
loadBalancerIP: XX.XXX.XX.XXX // <= line from microsoft documentation
ports:
- name: ambassador
port: 80
targetPort: 8080
- name: ambassador-secure
port: 443
targetPort: 443
selector:
service: my-selector
the static ip ive passed seems to be ignored and each time new one is being created.
Any ideas what im doing wrong?
Thanks in advance.
The resource group location seems to be the root cause here. If you have not put your public IP in the MC_xxxx resource group that AKS creates, the Kubernetes’ controller-manager won’t be able to find it since it is the default resource group that it looks under. Unless one is specified in your service manifest using the right service annotation.
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup
name: azure-load-balancer
spec:
loadBalancerIP: 40.121.183.52
type: LoadBalancer
ports:
- port: 80
selector:
app: azure-load-balancer
I have a simple service
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
And here is how my cluster looks like. Pretty simple.
$kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-shell-95cb5df57-cdj4z 1/1 Running 0 23m 10.60.1.32 aks-nodepool-19248108-0 <none> <none>
nginx-deployment-76bf4969df-58d66 1/1 Running 0 36m 10.60.1.10 aks-nodepool-19248108-0 <none> <none>
nginx-deployment-76bf4969df-jfkq7 1/1 Running 0 36m 10.60.1.21 aks-nodepool-19248108-0 <none> <none>
$kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
internal-ingress LoadBalancer 10.0.0.194 10.60.1.35 80:30157/TCP 5m28s app=nginx-deployment
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 147m <none>
$kubectl get rs -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
my-shell-95cb5df57 1 1 1 23m my-shell ubuntu pod-template-hash=95cb5df57,run=my-shell
nginx-deployment-76bf4969df 2 2 2 37m nginx nginx:1.7.9 app=nginx,pod-template-hash=76bf4969df
I see I have 2 pods wiht my nginx app. I want to be able to send a request from any other new pod to either one of them. If one crashes, I want to still be able to send this request.
In the past I used a load balancer for this. The problem with load balancers is that they open up a public IP and int this specific scenario, I don't want a public IP anymore. I want this service to be invoked by other pods directly, without a public IP.
I tried to use an internal load balancer.
apiVersion: v1
kind: Service
metadata:
name: internal-ingress
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "my-subnet"
spec:
type: LoadBalancer
loadBalancerIP: 10.60.1.45
ports:
- port: 80
selector:
app: nginx-deployment
The problem is that it does not get an IP in my 10.60.0.0/16 network like it is described here: https://learn.microsoft.com/en-us/azure/aks/internal-lb#specify-a-different-subnet
I get this never ending <pending>.
kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
internal-ingress LoadBalancer 10.0.0.230 <pending> 80:30638/TCP 15s app=nginx-deployment
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 136m <none>
What am I missing? How to troubleshoot? Is it even possible to have pod to service communication?
From the message you provide, it seems you want to use a special private IP address which is in the subnet that the same as the AKS cluster use. I think the possible reason is that the special IP address which you want to use is already assigned by the AKS, it means you cannot use it.
Troubleshooting
So you need to guide to the Vnet which your AKS cluster used and check if the IP address is already in use. Here is the screenshot:
Solution
Choose an IP address that is not assigned by the AKS from the subnet the AKS used. Or do not use a special one, let the AKS assign your load balancer dynamic. Then change your YAML file like below:
apiVersion: v1
kind: Service
metadata:
name: internal-ingress
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
ports:
- port: 80
selector:
app: nginx-deployment
Use a ClusterIP Service (the default type) which creates only a cluster-internal IP and no public IP:
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- port: 80
targetPort: 80
Then you can access the Service (and thus the Pods behind it) from any other Pod in the same namespace by using the Service name as the DNS name:
curl nginx-service
If the Pod from which you want to access the Service is in a different namespace, you have to use the fully qualified domain name of the Service:
curl nginx-service.my-namespace.svc.cluster.local
I created a service call portal, then I create ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: portal-ingress
spec:
backend:
serviceName: portal
servicePort: 8080
but the address is empty:
NAME HOSTS ADDRESS PORTS AGE
portal-ingress * 80 33m
The address will remain empty in AKS ingress and that is not a problem. You can still use the external IP address of the ingress controller service as the IP.
kubectl get svc -n <namespaceinwhichnginxcontrollerisdeployed>
For example:kubectl get svc -n nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-nginx-ingress-controller LoadBalancer 10.23.145.21 13.15.230.190 80:31108/TCP,443:31753/TCP
You can access the ingress as http(s)://13.15.230.190/
I think there are probably ways to make the address be populated, but I did not have a need to make it populated. I hope that is not what you want but to use the exposed service.
Ok, 3 years after and k8s apis changed, but, for the records, in my (today's) case this was due to not having installed the ingress controller as described the doc:
https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli
NAMESPACE=ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
--create-namespace \
--namespace $NAMESPACE \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz