Azure Kubernetes - replica vs HPA? - azure

Question

What is the difference between replicas and HPA?
For sample, below deployment is configured with 3 replicas
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
spec:
**replicas: 3**
and the below HPA with 2-20 replicas
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hello
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hello
**minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80**
does it mean that the above HPA will control the overall number of replicas irrespective of what is defined in the "deployment.yaml"? When HPA scales up would it add one more "deployment" replica or three more "deployment" replicas?

Answer 1

Yes the answer is, based on the observations those I had with AKS.
The deployment.yaml, asks for desired number of replicas, and hpa carries the variation around this based on metrics configured.
The desired state or replicas in deployment object ( when you do kubectl get deploy ), will give the current replicas as well as desired replicas always and you can see a variation there with the load.
So it will start with 3 instances and then it will try to keep min replicas always available ( hence the min-replicas in hpa and replicas in deployment file are kept same ), and then based on load computation against the provided metrics, it will scale up or down to min or max defined levels.

Answer 2

It is important to add to previous answer that deployment spec.replicas and HPA spec.minReplicas might be conflicting. When both configured some unexpected behaviour might arise.
If there is an HPA, it manages the amount of replicas according to it's settings. But while deployment is under control of an HPA, if you apply deployment config with set amount of replicas, it would override current desired amount of replicas and might scale your deployment unexpectedly.
For example if you have deployment spec.replicas set to 1, and HPA currently scaled your deployment to 5 replicas, when you apply deployment config it would set amount of desired replicas to 1 and immediately scale down your deployment. Then HPA takes back control, changes desired amount of replicas back to 5 and scales it up again.
Here is how this issue looks on my Grafana dashboard that tracks amount of running replicas
More on the topic:
Blog post about the problem.
Problem explained in Kubernetes GitHub issue.