I have an aks cluster with 4 nodepools consisting of windows and linux nodepools and a total of 700 namespaces in it. The total node count would be between 50-60 all the time . So i had cleared down more than 200 namespaces which were utilizing the cluster bt still the cluster run between 50-60 average cpu and memory usage of cluster is very low and below 50 all the time. I'm still not sure why the scale down is not happening properly after clearing down namespaces autoscaling vmss is all in place and its working bt only scales in between 50-60 nodes.
I followed the below steps to scale down the AKS node pool
I have created the AKS cluster with name aks-clusterz, The scale up operations are performed by the cluster auto scaler
scale down operations will decide to delete or deallocate the nodes in Aks cluster upon scaling down
I have installed the Aks preview extension
az extension add --name aks-preview
Created the node pool with 20 nodes and specified the scale down and the nodes are to be deallocated via scale down mode
az aks nodepool add --node-count 20 --scale-down-mode Deallocate --node-osdisk-type Managed --max-pods 10 --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
By changing the node count 5 and scaling the node pool I will deallocate to remaining nodes
az aks nodepool scale --node-count 5 --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
Deleted the deallocated nodes using this command
az aks nodepool update --scale-down-mode Delete --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
The default behaviour of cluster without using scale down mode is to delete the nodes when we scale down the cluster, using scale down it can be explicitly achieved by setting scale down delete mode
az aks nodepool add --enable-cluster-autoscaler --min-count 1 --max-count 10 --max-pods 10 --node-osdisk-type Managed --scale-down-mode Delete --name nodepool23z --cluster-name aks-clusterz --resource-group Alldemorg
NOTE:
At a time we cannot delete more than 30 nodes , if we delete more than that it will not scale down properly
The node utilization level is defined as the sum of requested resources divided by its capacity, based on the node utilization it will monitor
Related
I am trying to achieve HPA on azure cluster. But it is not working as expected, as it is not scaling up the pods when it is clearly showing the metric value is double of the target value. As you can see in the below screenshot
Here is the HPA configuration for the same.
Might be your Metrics server is not automatically installed with AKS,The Metrics Server is used to provide resource utilization to Kubernetes, and is automatically deployed in AKS clusters versions 1.10 and higher.
To see the version of your AKS cluster, use the az aks show command, as shown in the following example:
az aks show --resource-group myResourceGroup --name myAKSCluster --query kubernetesVersion --output table
If your AKS cluster is less than 1.10, the Metrics Server is not automatically installed. You can install via url.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
To use the autoscaler, all containers in your pods and your pods must have CPU requests and limits defined.
For more information how to implement you can refer this Microsoft Document
I have spot instance nodes in Azure Kubernetes Cluster. I want to simulate the eviction of a node so as to debug my code but not able to. All I could find in azure docs is we can simulate eviction for a single spot instance, using the following:
az vm simulate-eviction --resource-group test-eastus --name test-vm-26
However, I need to simulate the eviction of a spot node pool or a spot node in an AKS cluster.
For simulating evictions, there is no AKS REST API or Azure CLI command because evictions of the underlying infrastructure is not handled by AKS RP.
Only during creation of the AKS cluster the AKS RP can set eviction Policy on the underlying infrastructure by instructing the Azure Compute RP to do so.
Instead to simulate the eviction of node infrastructure, the customer can use az vmsss simulate-eviction command or the corresponding REST API.
az vmss simulate-eviction
az vmss simulate-eviction --instance-id
--name
--resource-group
[--subscription]
Reference Documents:
https://learn.microsoft.com/en-us/cli/azure/vmss?view=azure-cli-latest#az_vmss_simulate_eviction
https://learn.microsoft.com/en-us/rest/api/compute/virtual-machine-scale-set-vms/simulate-eviction
Use the following commands to get the name of the vmss with nodepool:
1.
az aks nodepool list -g $ClusterRG --cluster-name $ClusterName -o
table
Get the desired node pool name from the output
2.
CLUSTER_RESOURCE_GROUP=$(az aks show –resource-group YOUR_Resource_Group --name YOUR_AKS_Cluster --query
nodeResourceGroup -o tsv)
az vmss list -g $CLUSTER_RESOURCE_GROUP --query "[?tags.poolName == '<NODE_POOL_NAME>'].{VMSS_Name:name}" -o tsv
References:
https://louisshih.gitbooks.io/kubernetes/content/chapter1.html
https://ystatit.medium.com/azure-ssh-into-aks-nodes-471c07ad91ef
https://learn.microsoft.com/en-us/cli/azure/vmss?view=azure-cli-latest#az_vmss_list_instances
(you may create vmss if you dont have it configured. Refer :create a VMSS)
I removed two nodes of my Kubernetes cluster manually first calling "kubectl drain " and then "kubectl delete " for each. While the cluster seems to work without a problem the Azure UI shows me exactly two nodes more than I see when I use "kubectl get nodes". So when I configure Kubernetes to have 9 nodes in the Azure UI only 7 nodes are there if I take a look with kubectl. Scaling up or down does not solve the problem as Azure is always off by two nodes.
How can I solve this problem? Is there a way I can notify Azure that a node has been deleted?
If you want to solve the issue, you need to have a deeper understanding of the k8s cluster.
When you use the command kubectl delete to remove the node from the agent pool, it means the agent pool won't have control over that node. But it does not mean you really delete the machine. So you can find the number of the machine does not change in the Azure portal. This is the truth you find.
How can I solve this problem? Is there a way I can notify Azure that a
node has been deleted?
Here are two questions. For the first, you can express it in this way:
How to restore the node that deleted before to the agent pool?
It's simple to solve. You only need to restart the kubelet service in that node. For example, you use the VMSS as the agent pool of the AKS and that node instance id is 4. Then you can do it like this:
az vmss run-command invoke --resource-group group_name --name vmss_name --instance-id 4 --command-id RunShellScript --scripts "service kubelet restart"
For the second one, you can only use the Azure command to let Azure know the update. Here it means you can scale the agent pool, for example, using the Azure CLI command:
az aks nodepool --resource-group group_name --name agentpool_name --cluster-name cluster_name --node-count 2
I use the command az aks nodepool delete -g $myresource --cluster-name $mycluster --name $mypool to delete node pools in aks, but this command kill any running job. I am trying to optimize azure AKS cots, and some times it would be useful to schedule az aks nodepool delete.
Is it any way to schedule az aks nodepool delete so a node pool will be deleted after a job is finished?
I understand that you are trying to delete your NodePool that are creating VMs for GPU.
Note that while deleting the AgentPool could be a viable solution, you should look at the newly release feature from AKS which allow User Mode NodePool to be scaled down to 0.
https://learn.microsoft.com/en-us/azure/aks/use-system-pools
With this version of AKS: https://github.com/Azure/AKS/releases/tag/2020-04-13
Features
AKS has now introduced a new Mode property for nodepools. This will allow you to set nodepools as System or User nodepools. System nodepools will have additional validations and will be preferred by system pods, while User pool will have more lax validations and can perform additional operations like scale to 0 nodes or be removed from the cluster. Each cluster needs at least one system pool. All details here: https://aka.ms/aks/nodepool/mode
System/User nodepools are available from core CLI version 2.3.1 or greater (or latest preview extension 0.4.43)
Nodepool mode requires API 2020-03-01 or greater
AKS now allows User nodepools to scale to 0.
This will allow you to keep the VMSS for your GPU, enable the AutoScaler for the Node Pool and set the minimum node count to 0.
Your GPU nodes will be created when needed (a pod requiring the GPU nodes, should trigger the autoscaler)
I am working with windows + kubernetes cluster on ACS (Azure Container Service). I have few queries about the auto-scaling of agents/worker nodes in ACS.
Suppose I have a cluster of 1 master and 5 worker nodes. I have 200 running pods and these pods are distributed along the 5 worker nodes and the resources of these 5 nodes are used. Now, if I deploy a new pod or scale the running pods which will requires more resources, so is there any way ACS can auto-scale the worker nodes to like 7 worker nodes based on the resource usage?
Same case, if resource usage is reduced, can ACS descale the worker nodes to 3 worker nodes from 7 nodes?
My question is not related to auto-scaling of pods as provided by kubernetes, I am talking about auto-scaling of worker/agent nodes which are managed by ACS.
My question is not related to auto-scaling of pods as provided by
kubernetes, I am talking about auto-scaling of worker/agent nodes
which are managed by ACS
Currently, autoscaling of agent nodes in a container service cluster is not supported.
For now, we can use Azure CLI2.0 to scale down or up by command az acs scale
For example:
azure acs scale -g myResourceGroup -n containerservice-myACSName --new-agent-count 10
More information about az acs scale command, please refer to this link.
You can use azure-cli. The command you would need is az acs scale