I have a cluster with an on-demand nodepool and spot instance nodepool. I want to give spot instnace the highest priority of pods to be assigned and once its terminated only all pods should get assigned into my on-demand nodepool.
I know spot instance has a default taint and labels . what are ways to implement this change.
Related
I have kubernetes cluster running on Azure Virtual Machine Scale Set. I use Kubernetes Cluster Autoscaler to scale the number of nodes. It works fine, if i set limit from 1 to 10 but the problem appears when i set limit from 0 in one particular case:
When the number of nodes has been scaled to 0 and after this operation pod with cluster autoscaler restarted. Then i want to run pod on this VMSS (pod with nodeSelector - agentpool: memory), but it looks like autoscaler can't read appropriate labels from VMSS when number of instance is scaled to 0.
According to documentation i add the following tag to the VMSS k8s.io_cluster-autoscaler_node-template_label_agentpool: memory.
I have logs from autoscaler pod:
GeneralPredicates predicate mismatch, reason: node(s) didn't match node selector
Foreword
When you create a Kubernetes cluster on AKS you specify the type of VMs you want to use for your nodes (--node-vm-size). I read that you can't change this after you create the Kubernetes cluster, which would mean that you'd be scaling vertically instead of horizontally whenever you add resources.
However, you can create different node pools in an AKS cluster that use different types of VMs for your nodes. So, I thought, if you want to "change" the type of VM that you chose initially, maybe add a new node pool and remove the old one ("nodepool1")?
I tried that through the following steps:
Create a node pool named "stda1v2" with a VM type of "Standard_A1_v2"
Delete "nodepool1" (az aks nodepool delete --cluster-name ... -g ... -n nodepool1
Unfortunately I was met with Primary agentpool cannot be deleted.
Question
What is the purpose of the "primary agentpool" which cannot be deleted, and does it matter (a lot) what type of VM I choose when I create the AKS cluster (in a real world scenario)?
Can I create other node pools and let the primary one live its life? Will it cause trouble in the future if I have node pools that use larger VMs for its nodes but the primary one still using "Standard_A1_v2" for example?
Primary node pool is the first nodepool in the cluster and you cannot delete it, because its currently not supported. You can create and delete additional node pools and just let primary be as it is. It will not create any trouble.
For the primary node pool I suggest picking a VM size that makes more sense in a long run (since you cannot change it). B-series would be a good fit, since they are cheap and CPU\mem ratio is good for average workloads.
ps. You can always scale primary node pool to 0 nodes, cordon the node and shut it down. You will have to repeat this post upgrade, but otherwise it will work
It looks like this functionality was introduced around the time of your question, allowing you to add new system nodepools and delete old ones, including the initial nodepool. After encountering the same error message myself while trying to tidy up a cluster, I discovered I had to set another nodepool to a system type in order to delete the first.
There's more info about it here, but in short, Azure nodepools are split into two types ('modes' as they call it): System and User. When creating a single pool to begin with, it will be of a system type (favouring system pod scheduling -- so it might be good to have a dedicated pool of a node or two for system use, then a second user nodepool for the actual app pods).
So if you wish to delete your only system pool, you need to first create another nodepool with the --mode switch set to 'system' (with your preferred VM size etc.), then you'll be able to delete the first (and nodepool modes can't be changed after the fact, only on creation).
I have a Terraform module that needs to depend on the resource aws_autoscaling_group ec2 "Instance State" and "Status Checks" (from the ec2 console) to go green before starting. How can this be done? Thanks.
In my understanding that's the default behavior of autoscaling_group. From the docs:
Terraform provides two mechanisms to help consistently manage ASG scale uptime across dependent resources.
The first is the default behavior. Terraform waits after ASG creation for min_size (or desired_capacity, if specified) healthy instances to show up in the ASG before continuing.
[...]
Terraform considers an instance "healthy" when the ASG reports HealthStatus: "Healthy" and LifecycleState: "InService"
Also, from the AWS docs:
When each instance is fully configured and passes the Amazon EC2 health checks, it is attached to the Auto Scaling group and it enters the InService state. The instance is counted against the desired capacity of the Auto Scaling group.
Spark needs lots of resources to does its job. Kubernetes is great environment for resource management. How many Spark PODs do you run per node to have the best resource utilization?
Trying to run Spark Cluster on Kubernetes Cluster.
It depends on many factors. We need to know how much resources do you have and how much is being consumed by the pods. To do so you need to setup a Metrics-server.
Metrics Server is a cluster-wide aggregator of resource usage data.
Next step is to setup HPA.
The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization or other custom metrics. HPA normally fetches metrics from a series of aggregated APIs:
metrics.k8s.io
custom.metrics.k8s.io
external.metrics.k8s.io
How to make it work?
HPA is being supported by kubectl by default:
kubectl create - creates a new autoscaler
kubectl get hpa - lists your autoscalers
kubectl describe hpa - gets a detailed description of autoscalers
kubectl delete - deletes an autoscaler
Example:
kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80 creates an autoscaler for replication set foo, with target CPU utilization set to 80% and the number of replicas between 2 and 5. You can and should adjust all values to your needs.
Here is a detailed documentation of how to use kubectl autoscale command.
Please let me know if you find that useful.
I am creating an EMR cluster using the aws_emr_cluster resource in terraform.
I need to get access to the instance ID of the underlying EC2 hardware, specifically the MASTER node.
It does not appear in the attributes and neither when I perform an terraform show
The data definitely exists and is available in AWS.
Does anyone know how I can get at this value and how to do it it using terraform?
You won't be able to access the nodes (EC2 Instances) in an EMR Cluster through terraform. It is the same case for AutoScaling Groups too.
If terraform includes EMR or ASG nodes, state file will be changed everytime a change happens in EMR/ASG. So, storing the instance information won't be ideal for terraform.
Instead, you can use AWS SDK/CLI/boto3 to see them.
Thanks.