In the official documentation here, the pod.spec.container.resources.limits is defined as follows :
"Limits describes the maximum amount of compute resources allowed."
I understand that k8s prohibits a pod from consuming more resources than specified in limits.
The documentation does not say that a pod is not scheduled in a node that does not have the amount of resources specified in limits.
For instance, if each node in my cluster has 2 cpus, and I try to deploy a pod defining a cpu limit to 3, my pod will never be running and will be in status Pending.
Here is the example template : mypod.yml
apiVersion: v1
kind: Pod
metadata:
name: secondbug
spec:
containers:
- name: container
image: nginx
resources:
limits:
cpu: 3
Is this behaviour intended and why?
You have 3 kinds of pods, pods with no request definition, pods that are burstable which have limits and requests and the third kind which has only limits or requests defined.
For the third case if you only define limits or requests, the other one will be defined by the first one meaning:
resources:
limits:
cpu: 3
Is actually:
resources:
limits:
cpu: 3
requests:
cpu: 3
So you cant really define only one without the other. So in your case, you want to define requests for it so it won't be defined by the limits.
Related
I followed Strimzi blog to resize PV.
I use Openshift v3.11 deployed on Azure VMs with PV as Azure managed disk
My Kafka cluster storage config
..
config:
offsets.topic.replication.factor: 2
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.6"
storage:
type: persistent-claim
size: 256Gi
deleteClaim: false
...
I directly edit pvc and changed the resource request to 257Gi. I wait for few mins , checked the status of PVC like blow
oc get pvc data-0-xx-dev-kafka-0 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
...
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 257Gi
storageClassName: generic
volumeName: pvc-xx-2849-xx-913f-xx
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 256Gi
conditions:
- lastProbeTime: null
lastTransitionTime: 2021-10-08T15:54:06Z
status: "True"
type: Resizing
phase: Bound
In the description of pvc, I see below
Warning VolumeResizeFailed 3s (x2 over 1m) volume_expand Error expanding volume "kafka/data-0-sirius-dev-kafka-0" of plugin kubernetes.io/azure-disk : compute.DisksClient#CreateOrUpdate:
Failure sending request: StatusCode=409 -- Original Error: failed request: autorest/azure:
Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-xx-2849-xx-913f-xx while it is
attached to running VM /subscriptions/xxxx/resourceGroups/xxx-dev-openshift/providers/Microsoft.Compute/virtualMachines/ocp-node-dev-1. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated.
Please stop your VM and retry the operation."
I have also tried, redeployed kafka with jbod with single disk, resize and did rolling update. Same result like above
openshift v3.11.0+cbab8ee-94(K8s v1.11.0+d4cacc0)
Kafka Version: 2.6.0
Operators : 0.20.1
Note that resize PV does support in my cluster(Previously I resized a PV of an app successfully by scaling down replicas to zero)
UPDATE
$ oc describe storageclass generic
Name: generic
IsDefaultClass: Yes
Annotations: storageclass.beta.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/azure-disk
Parameters: kind=managed,location=${location},storageaccounttype=Premium_LRS
AllowVolumeExpansion: True
MountOptions:
discard
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
I tried to scale down reaplica to 0 with oc scale replicas=0 sts/XX-dev, but cluster-operator is not allowing since because of replication factor.
Normally Azure Disk is supporting resizing of Persistent Volumes, but currently, there are some issues in terms of compliancy and implementation check here
So before going any other place; just check one more time following points
check the Storage Class definition with "storageClassName: generic" if allowVolumeExpansion is set to true. (may be you tried resizing with different storage class before) You can only expand a PVC if its storage class's allowVolumeExpansion field is set to true.
it is looking like that you need to de-allocate everything which is pointing to that volume first(check warning at https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi) then resize and lastly scale Pods back.
try to increment with multiplies check the sizes from here(https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types)
maybe hitting volume limits per VM, default is 16 for Azure, for the time being, check how many volumes you have now in that node
The other issue is that if underlying capacity is not enough for resizing you have to deallocate and relocate VM in the new resource group. Yes cloud is not unlimited
I am using Azure Kubernetes, and trying to set TCP_Keepalive on a container basis.
Is there away of achieving that?
You could do this via sysctls on the pod manifest in AKS/Kubernetes:
spec:
securityContext:
sysctls:
- name: "net.ipv4.tcp_keepalive_time"
value: "45"
Here is also further documentation:
https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/
https://docs.syseleven.de/metakube/de/tutorials/confiugre-unsafe-sysctls
I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.
So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.
I am setting it as below
Master image (spark-master:X)
FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
ENTRYPOINT ["/opt/spark/sbin/start-master.sh", "-p", "8081", "<A long running server command that can accept get traffic on 8080 to submit jobs>"]
Worker worker image (spark-worker:X)
FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
ENTRYPOINT ["/opt/spark/sbin/start-worker.sh", "spark//spark-master:8081" ,"-p", "8081", "<A long running server command to keep up the worker>"]
Deplyments
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-master-server
spec:
replicas: 1
selector:
matchLabels:
component: spark-master-server
template:
metadata:
labels:
component: spark-master-server
spec:
containers:
- name: spark-master-server
image: spark-master:X
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
name: spark-master
spec:
type: ClusterIP
ports:
- port: 8081
targetPort: 8081
selector:
component: spark-master-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: spark-worker-instance
spec:
replicas: 3
selector:
matchLabels:
component: spark-worker-instance
template:
metadata:
labels:
component: spark-worker-instance
spec:
containers:
- name: spark-worker-server
image: spark-worker:X
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8081
Questions
Is this setup recommended ?
How can we submit new job from within Kubernetes cluster in absence of yarn (k8s) ?
Reason we are trying not create master and driver dynamically per job (as given in example - http://spark.apache.org/docs/latest/running-on-kubernetes.html) is that it may be an overhead for large no. of small jobs.
Is this setup recommended ?
Don't think so.
Dynamic Resource Allocation is a property of a single Spark application "to dynamically adjust the resources your application occupies based on the workload."
Dynamic Resource Allocation spans its resource requirements regardless of available nodes in a cluster. As long as there are resources available and a cluster manager could assign them to a Spark application these resources are free to go.
What you seem to be trying to set up is how to scale the cluster itself up and down. In your case it's Spark Standalone and although technically it's possible with ReplicaSets (just a guess) I've never heard any earlier attempts at it. You're on your own as Spark Standalone does not support it out of the box.
That I think is an overkill since you're building a multi-layer cluster environment: using a cluster manager (Kubernetes) to host another cluster manager (Spark Standalone) for Spark applications. Given Spark on Kubernetes supports Dynamic Allocation out of the box the only worry of yours should simply be how to "throw in" more CPU and memory on demand while resizing Kubernetes cluster. You should rely on the capabilities of Kubernetes to resize itself up and down rather than Spark Standalone on Kubernetes.
The spark on k8s operator may provide at least one mechanism to dynamically provision the resources you need to do safe scaling of resources based on demand.
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
My thinking is that instead of performing a direct spark submit to a master in a static spark cluster, you could make a call to the k8s API to provision the required instance; or otherwise define these as a cron schedule.
In the dynamic provisioning scenario one thing to consider is how your workloads get distributed across the cluster; we can’t use simple HPA rules for this as it may not be safe to tear down a worker on CPU/Mem levels; spawning a separate cluster for each on demand workload bypasses this but may not be optimal. I would be interested to hear how you get on.
I tried to use K8s to setup spark cluster (I use standalone deployment mode, and I cannot use k8s deployment mode for some reason)
I didn't set any cpu related arguments.
for spark, that means:
Total CPU cores to allow Spark applications to use on the machine (default: all available); only on worker
http://spark.apache.org/docs/latest/spark-standalone.html
for k8s pods, that means:
If you do not specify a CPU limit for a Container, then one of these situations applies:
The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running.
The Container is running in a namespace that has a default CPU limit, and the Container is automatically assigned the default limit. Cluster administrators can use a LimitRange to specify a default value for the CPU limit.
https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/
...
Addresses:
InternalIP: 172.16.197.133
Hostname: ubuntu
Capacity:
cpu: 4
memory: 3922Mi
pods: 110
Allocatable:
cpu: 4
memory: 3822Mi
pods: 110
...
But my spark worker only use 1 core (I have 4 cores on the worker node and the namespace has no resource limits).
That means the spark worker pod only used 1 core of the node (which should be 4).
How can I write yaml file to set the pod to use all available cpu cores?
Here is my yaml file:
---
apiVersion: v1
kind: Namespace
metadata:
name: spark-standalone
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: spark-slave
namespace: spark-standalone
labels:
k8s-app: spark-slave
spec:
selector:
matchLabels:
k8s-app: spark-slave
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
name: spark-slave
namespace: spark-standalone
labels:
k8s-app: spark-slave
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/edge
operator: Exists
hostNetwork: true
containers:
- name: spark-slave
image: spark:2.4.3
command: ["/bin/sh","-c"]
args:
- "
${SPARK_HOME}/sbin/start-slave.sh
spark://$(SPARK_MASTER_IP):$(SPARK_MASTER_PORT)
--webui-port $(SPARK_SLAVE_WEBUI_PORT)
&&
tail -f ${SPARK_HOME}/logs/*
"
env:
- name: SPARK_MASTER_IP
value: "10.4.20.34"
- name: SPARK_MASTER_PORT
value: "7077"
- name: SPARK_SLAVE_WEBUI_PORT
value: "8081"
---
Kubernetes - No upper bound
The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running
Unless you confgure a limit on CPU for your pod, it can use all available CPU resources on the node.
Consider dedicated nodes
If you are running other workload on the same node, they also consume CPU resources, and may be guaranteed CPU resources if they have configured request for CPU. Consider use a dedicated node for your workload using NodeSelector and Taints and Tolerations.
Spark - No upper bound
You configure the slave with parameters to the start-slave.sh e.g. --cores X to limit CPU core usage.
Total CPU cores to allow Spark applications to use on the machine (default: all available); only on worker
Multithreaded workload
In the end, if pod can use multiple CPU cores depends on how your application uses threads. Some things only uses a single thread, so the application must be designed for multithreading and have something parallelized to do.
I hit absolutely the same issue with a Spark worker. Knowing the fact, that Java sometimes failed to calculate CPU correctly, I tried to specify the CPU request or limit in Pod spec and the worker automatically understood what the environment it is. And needed cores will be assigned to the Spark worker executor.
Also, I faced this behavior in k8s only. In Docker Swarm, all available CPU cores had been taken by a worker.
What's more, in default templates for 'cores' parameters for Spark worker 1 core is mentioned. I believe it could be taken in case of the wrong calculation of CPU cores.
I have a nodejs microservice running on GKE that serves html/js assets on our clients' sites. The configuration for this is as follows: Ingress > NodePort Service > Pods (4 replicas). If for some season the pods are failing or the Node goes down, the microservice will then take the full 30s to timeout. That causes delayed page load times for our clients. What I need to happen is in the event of a failure that the NodePort service cuts off the connection or responds with a 502 error after 2 seconds.
I've tried two ways of manipulating the same setting. The first is creating a BackendConfig following the docs: https://cloud.google.com/kubernetes-engine/docs/how-to/configure-backend-service
My config looks like this:
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: timeout-config
spec:
timeoutSec: 2
Then I connected it to my service like this:
apiVersion: v1
kind: Service
metadata:
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"80":"timeout-config"}}'
labels:
run: <MICROSERVICE>
name: <MICROSERVICE>
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 3000
selector:
run: <MICROSERVICE>
type: NodePort
I tested this configuration by having my microservice alternate between returning 200 and 502 for the health check endpoint every 30 seconds. That caused the pod to be restarted about every 30s which would cut off communication with the pod. I expected that once it was being restarted that the request would timeout and default to the 2-second setting I had configured. However, it still took 30 seconds to receive the 502 error.
The second method I tried was to set the timeout to 2 seconds using gcloud. I did so by following the docs here: https://github.com/kubernetes/ingress-gce/blob/e72479ba461fedae5fc5bf64999f28ba3125004d/examples/websocket/README.md#change-backend-timeout
That method did not work either. What other methods can I use to get my service to timeout after 2 seconds on GKE?