I use the command az aks nodepool delete -g $myresource --cluster-name $mycluster --name $mypool to delete node pools in aks, but this command kill any running job. I am trying to optimize azure AKS cots, and some times it would be useful to schedule az aks nodepool delete.
Is it any way to schedule az aks nodepool delete so a node pool will be deleted after a job is finished?
I understand that you are trying to delete your NodePool that are creating VMs for GPU.
Note that while deleting the AgentPool could be a viable solution, you should look at the newly release feature from AKS which allow User Mode NodePool to be scaled down to 0.
https://learn.microsoft.com/en-us/azure/aks/use-system-pools
With this version of AKS: https://github.com/Azure/AKS/releases/tag/2020-04-13
Features
AKS has now introduced a new Mode property for nodepools. This will allow you to set nodepools as System or User nodepools. System nodepools will have additional validations and will be preferred by system pods, while User pool will have more lax validations and can perform additional operations like scale to 0 nodes or be removed from the cluster. Each cluster needs at least one system pool. All details here: https://aka.ms/aks/nodepool/mode
System/User nodepools are available from core CLI version 2.3.1 or greater (or latest preview extension 0.4.43)
Nodepool mode requires API 2020-03-01 or greater
AKS now allows User nodepools to scale to 0.
This will allow you to keep the VMSS for your GPU, enable the AutoScaler for the Node Pool and set the minimum node count to 0.
Your GPU nodes will be created when needed (a pod requiring the GPU nodes, should trigger the autoscaler)
Related
I have an aks cluster with 4 nodepools consisting of windows and linux nodepools and a total of 700 namespaces in it. The total node count would be between 50-60 all the time . So i had cleared down more than 200 namespaces which were utilizing the cluster bt still the cluster run between 50-60 average cpu and memory usage of cluster is very low and below 50 all the time. I'm still not sure why the scale down is not happening properly after clearing down namespaces autoscaling vmss is all in place and its working bt only scales in between 50-60 nodes.
I followed the below steps to scale down the AKS node pool
I have created the AKS cluster with name aks-clusterz, The scale up operations are performed by the cluster auto scaler
scale down operations will decide to delete or deallocate the nodes in Aks cluster upon scaling down
I have installed the Aks preview extension
az extension add --name aks-preview
Created the node pool with 20 nodes and specified the scale down and the nodes are to be deallocated via scale down mode
az aks nodepool add --node-count 20 --scale-down-mode Deallocate --node-osdisk-type Managed --max-pods 10 --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
By changing the node count 5 and scaling the node pool I will deallocate to remaining nodes
az aks nodepool scale --node-count 5 --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
Deleted the deallocated nodes using this command
az aks nodepool update --scale-down-mode Delete --name nodepool23 --cluster-name aks-clusterz --resource-group Alldemorg
The default behaviour of cluster without using scale down mode is to delete the nodes when we scale down the cluster, using scale down it can be explicitly achieved by setting scale down delete mode
az aks nodepool add --enable-cluster-autoscaler --min-count 1 --max-count 10 --max-pods 10 --node-osdisk-type Managed --scale-down-mode Delete --name nodepool23z --cluster-name aks-clusterz --resource-group Alldemorg
NOTE:
At a time we cannot delete more than 30 nodes , if we delete more than that it will not scale down properly
The node utilization level is defined as the sum of requested resources divided by its capacity, based on the node utilization it will monitor
I am trying to achieve HPA on azure cluster. But it is not working as expected, as it is not scaling up the pods when it is clearly showing the metric value is double of the target value. As you can see in the below screenshot
Here is the HPA configuration for the same.
Might be your Metrics server is not automatically installed with AKS,The Metrics Server is used to provide resource utilization to Kubernetes, and is automatically deployed in AKS clusters versions 1.10 and higher.
To see the version of your AKS cluster, use the az aks show command, as shown in the following example:
az aks show --resource-group myResourceGroup --name myAKSCluster --query kubernetesVersion --output table
If your AKS cluster is less than 1.10, the Metrics Server is not automatically installed. You can install via url.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
To use the autoscaler, all containers in your pods and your pods must have CPU requests and limits defined.
For more information how to implement you can refer this Microsoft Document
When deploying new jobs and services to Azure Kubernetes Service cluster, the pods fail to request valid AAD access tokens with all permissions available. If new permissions were added on the same day, before or after a deployment, the tokens still do not pick them up. This issue has been observed with permissions granted to Active Directory Groups over Key Vaults, Storage Accounts, and SQL databases scopes so far.
Example: I have a .NET 5.0 C# API running on 3 pods with antiaffinity rules located each on a separate node. The application reads information from a SQL database. I made a release and added the database permissions afterwards. Things I have tried so far to make the application reset the access tokens:
kubectl delete pods --all -n <namespace> which essentially created 3 new pods again failing due to insufficient permissions.
kubectl apply -f deployment.yaml to deploy a new version of the image running in the containers, again all 3 pods kept failing.
kubectl delete -f deployment.yaml followed by kubectl apply -f deployment.yaml to erase the old kubernetes object and create a new one. This resolved the issue on 2/3 pods, however, the third one kept failing due to insufficient permissions.
kubectl delete namespace <namespace> to erase the entire namespace with all configuration available and recreated it again. Surprisingly, again 2/3 pods were running with the correct permissions and the last one did not.
The commands were ran more than one hour after the permissions were added to the group. The database tokens are active for 24 hours and when I have seen this issue occur with cronjobs, I had to wait 1 day for the task to execute correctly (none of the above steps worked in a cronjob scenario). The validity of the tokens kept changing which implied that the pods are requesting new access tokens, again excluding the most recently added permissions. The only solution I have found that works 100% of the time is destroy the cluster and recreate it which is not viable in any production scenario.
The failing pod from my example was the one always running on node 00 which made me think there may be an extra caching layer on the first initial node of the cluster. However, I still do not understand why the other 2 pods were running with no issue and also what is the way to restart my pods or refresh the access token to minimise the wait time until resolution.
Kubernetes version: 1.21.7.
The cluster has no AKS-managed AAD or pod-identity enabled. All RBAC is granted to the cluster MSI via AD groups.
Please check if below can be worked around in your case.
To access the Kubernetes resources, you must have access to the AKS cluster, the Kubernetes API, and the Kubernetes objects. Ensure that you're either a cluster administrator or a user with the appropriate permissions to access the AKS cluster
Things you need to do, if you haven't already:
Enable Azure RBAC on your existing AKS cluster, using:
az aks update -g myResourceGroup -n myAKSCluster --enable-azure-rbac
Create Role that allows read access to all other Pods and Services:
Add the necessary roles (Azure Kubernetes Service Cluster User Role , Azure Kubernetes Service RBAC Reader/Writer/Admin/Cluster Admin) to the user. See ( Microsoft Docs).
Also check Troubleshooting
Also check if you need to have "Virtual Machine Contributor" and storage account contributer for your resource group containing pods and see if namespace is mentioned in that pod , if you have missed . Stack Overflow refernce.Also do check if firewall is restricting the access to the network in that pod.
Resetting the kubeconfig context using the az aks get-credentials command may clear the previously cached authentication token for some xyz user:
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing >Reference
Please do check Other References below:
kubernetes - Permissions error - Stack Overflow
create-role-assignments-for-users-to-access-cluster | microsoft docs
user can't access to AKS cluster with RBAC enabled (github.com)
kubernetes - Stack Overflow
I removed two nodes of my Kubernetes cluster manually first calling "kubectl drain " and then "kubectl delete " for each. While the cluster seems to work without a problem the Azure UI shows me exactly two nodes more than I see when I use "kubectl get nodes". So when I configure Kubernetes to have 9 nodes in the Azure UI only 7 nodes are there if I take a look with kubectl. Scaling up or down does not solve the problem as Azure is always off by two nodes.
How can I solve this problem? Is there a way I can notify Azure that a node has been deleted?
If you want to solve the issue, you need to have a deeper understanding of the k8s cluster.
When you use the command kubectl delete to remove the node from the agent pool, it means the agent pool won't have control over that node. But it does not mean you really delete the machine. So you can find the number of the machine does not change in the Azure portal. This is the truth you find.
How can I solve this problem? Is there a way I can notify Azure that a
node has been deleted?
Here are two questions. For the first, you can express it in this way:
How to restore the node that deleted before to the agent pool?
It's simple to solve. You only need to restart the kubelet service in that node. For example, you use the VMSS as the agent pool of the AKS and that node instance id is 4. Then you can do it like this:
az vmss run-command invoke --resource-group group_name --name vmss_name --instance-id 4 --command-id RunShellScript --scripts "service kubelet restart"
For the second one, you can only use the Azure command to let Azure know the update. Here it means you can scale the agent pool, for example, using the Azure CLI command:
az aks nodepool --resource-group group_name --name agentpool_name --cluster-name cluster_name --node-count 2
I am working with windows + kubernetes cluster on ACS (Azure Container Service). I have few queries about the auto-scaling of agents/worker nodes in ACS.
Suppose I have a cluster of 1 master and 5 worker nodes. I have 200 running pods and these pods are distributed along the 5 worker nodes and the resources of these 5 nodes are used. Now, if I deploy a new pod or scale the running pods which will requires more resources, so is there any way ACS can auto-scale the worker nodes to like 7 worker nodes based on the resource usage?
Same case, if resource usage is reduced, can ACS descale the worker nodes to 3 worker nodes from 7 nodes?
My question is not related to auto-scaling of pods as provided by kubernetes, I am talking about auto-scaling of worker/agent nodes which are managed by ACS.
My question is not related to auto-scaling of pods as provided by
kubernetes, I am talking about auto-scaling of worker/agent nodes
which are managed by ACS
Currently, autoscaling of agent nodes in a container service cluster is not supported.
For now, we can use Azure CLI2.0 to scale down or up by command az acs scale
For example:
azure acs scale -g myResourceGroup -n containerservice-myACSName --new-agent-count 10
More information about az acs scale command, please refer to this link.
You can use azure-cli. The command you would need is az acs scale