I have create a VMSS in Azure Portal, to have the autoscale feature for my application. My application resided in Kubernetes cluster - around 10 microservices.
I want to create an Scale out rule, that if there is no enough memory , then increase the VM instance. But I don't see an option to set the rule based on memory. There are rules which we can define based on CPU utilization, disk space etc... But this won't help me to solve the problem. For my 10 microservice to work each service having 5 pods, i need to set a rule based on memory. If I set the rule based on CPU, the VM doesn;t scale up, as the CPU is not utilised much. Issue is with memory.
I get the error "0/3 nodes are available: 3 Insufficient pods.
The node was low on resource: [MemoryPressure]. "
I read that the memory rule is not available in host metrics in Azure, but it can enabled via guest metrics. To enable guest metrics, i see below link .
https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-mvss-guest-based-autoscale-linux
But I don't see an option to edit the template as defined in the above link. There is only "export Template" option visible for VMSS, where you cannot edit the template.
Could anyone please help me on this issue , to define memory rule for VMSS in Azure ?
No option seen to enable guest metrics for VMSS. No option to edit the template, only "export Template" option visible, where you cannot edit the template.
For the AKS autoscale, you just need to enable the autoscale function for your AKS cluster, set the min and max count of the nodes and then it will scale itself. You do not need to set the autoscale rule for it. Take a look at the AKS cluster autoscale.
When does Cluster Autoscaler change the size of a cluster?
Cluster Autoscaler increases the size of the cluster when:
there are pods that failed to schedule on any of the current nodes
due to insufficient resources.
adding a node similar to the nodes currently present in the cluster
would help.
Cluster Autoscaler decreases the size of the cluster when some nodes are consistently unneeded for a significant amount of time. A node is unneeded when it has low utilization and all of its important pods can be moved elsewhere.
And that what you have seen in the VMSS, the metric server is already installed in the high version AKS. If not install, you can install yourself and the steps here.
Related
In Azure K8s service, you can scale up the node pool but only we define the min and max nodes.
When i check the node pool scale set scale settings, i found it set to manual.
So i assume that the Node Pool auto scale does't rely on the belonging scale set, but i wonder, can we just rely on the scale set auto scale with the several metric roles instead of the very limited Node Pool scale settings ?
The AKS autoscaling works slightly different as the VMSS autoscaling.
From the official docs:
The cluster autoscaler watches for pods that can't be scheduled on
nodes because of resource constraints. The cluster then automatically
increases the number of nodes.
The AKS autoscaler is tightly coupled with the control plane and the kube-scheduler, so it takes resource requests and limits into account that is far the better scaling method as the VMSS autoscaler (for k8s workload) that is anyway not supported for AKS:
The cluster autoscaler is a Kubernetes component. Although the AKS
cluster uses a virtual machine scale set for the nodes, don't manually
enable or edit settings for scale set autoscale in the Azure portal or
using the Azure CLI.
I am unable to scale vertical my AKS cluster.
Currently, I have 3 nodes in my cluster with 2 core and 8 ram, I am trying to upgrade it with 16 code and 64 RAM, how do I do it?
I tried scaling the VM scale set, on Azure portal it shows it is scaled but when I do "kubectl get nodes -o wide" it still shows the old version.
Any leads will be helpful.
Thanks,
Abhishek
Vertical scaling or changing the node pool VM size is not supported. You need to create a new node pool and schedule your pods on the new nodes.
https://github.com/Azure/AKS/issues/1556#issuecomment-615390245
this UX issues is due to how the VMSS is managed by AKS. Since AKS is
a managed service, we don't support operations done outside of the AKS
API to the infrastructure resources. In this example you are using the
VMSS portal to resize, which uses VMSS APIs to resize the resource and
as a result has unexpected changes.
AKS nodepools don't support resize in place, so the supported way to
do this is to create a new nodepool with a new target and delete the
previous one. This needs to be done through the AKS portal UX. This
maintains the goal state of the AKS node pool, as at the moment the
portal is showing the VMSize AKS knows you have because that is what
was originally requested.
I have an Azure VM Scaleset on Windows VMs, and I am trying to setup an autoscale rule that would increase the number of instances if the Memory utilization gets over a certain threshold. There's an option for Percentage CPU, but not RAM. Tried creating the autoscale rule through Terraform since I saw it in some documentation somewhere, but got an error saying "MetricUnsupported". Is this really not possible?
Azure VMSS does not support using CPU Utilization in autoscaling rules. It just support CPU Credits Consumed or CPU Credits Remaining. For more details, please refer to the document
I am trying to figure out what is the trigger to scale AKS cluster out horizontally with nodes. I am having a cluster that runs on 103% CPU for 5+ minutes but there is no action taken. Any ideas what the triggers are and how I could customize them? If I start more jobs the cluster will lower the CPU allocation for all pods.
The article that MS has doesn't have anything specific around that https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler
You need to notice that:
The cluster autoscaler is a Kubernetes component. Although the AKS
cluster uses a virtual machine scale set for the nodes, don't manually
enable or edit settings for scale set autoscale in the Azure portal or
using the Azure CLI. Let the Kubernetes cluster autoscaler manage the
required scale settings.
Which brings us to the actual Kubernetes Cluster Autoscaler:
Cluster Autoscaler is a tool that automatically adjusts the size of
the Kubernetes cluster when one of the following conditions is true:
there are pods that failed to run in the cluster due to insufficient resources.
there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing
nodes.
The first condition above is the trigger you are looking for.
To get more details regarding the installation and configuration you can go through the Cluster Autoscaler on Azure. For example, you can customize your CA based on the Resources:
When scaling from an empty VM Scale Set (0 instances), Cluster
Autoscaler will evaluate the provided presources (cpu, memory,
ephemeral-storage) based on that VM Scale Set's backing instance type.
This can be overridden (for instance, to account for system reserved
resources) by specifying capacities with VMSS tags, formated as:
k8s.io_cluster-autoscaler_node-template_resources_<resource name>: <resource value>. For instance:
k8s.io_cluster-autoscaler_node-template_resources_cpu: 3800m
k8s.io_cluster-autoscaler_node-template_resources_memory: 11Gi
I'm trying to run different helm charts and I keep running into this error. It's much more cost effective for me to run 3-4 cheaper nodes than 1 or 2 very expensive nodes that can have more disks attached to them.
Is there a way to configure kubernetes or helm to have a disk attach limit or to set the affinity of one deployment to a particular node?
It's very frustrating that all the deployments try to attach to one node and then run out of disk attach quota.
Here is the error:
Service returned an error. Status=409 Code="OperationNotAllowed"
Message="The maximum number of data disks allowed to be attached to a
VM of this size is 4."
Is there a way to configure kubernetes or helm to have a disk attach
limit or to set the affinity of one deployment to a particular node?
For now, ACS k8s provision PVC based on Azure managed disks or blob disks, so the limit is the number of VM disks.
For now, Azure does not support change the limit about number of VM disks. About VM size and max data disks, we can find the limit here:
More information about limit, please refer to this link.
By the way, the disk maximum capacity is 2TB, maybe we can extend it to 2TB.