I have a cluster running Kubernetes v1.6.7.
The VMs are :
Distributor ID: Debian
Description: Debian GNU/Linux 8.7 (jessie)
Release: 8.7
Codename: jessie
Kernel :
3.16.0-4-amd64
We have set cpu limits in deployments, however pods comsume cpu over this limit if they need to. Is there something am I missing ? Like a parameter in kubelet to enable this limit ? I haven't found anything about this problem.
For example, if I create the following deployment :
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: stress
namespace: default
labels:
k8s-app: stress
spec:
replicas: 1
selector:
matchLabels:
k8s-app: stress
template:
metadata:
labels:
k8s-app: stress
spec:
containers:
- name: stress
image: progrium/stress
imagePullPolicy: Always
args: ["--cpu", "1"]
resources:
limits:
cpu: "500m"
memory: "1Gi"
requests:
cpu: "100m"
memory: "512Mi"
On the node, this makes a CPU 100% used whereas it should be 50%.
Thanks for your help.
From a comments to your question, where you can try to run docker container with CPU limit you getting:
WARNING: Your kernel does not support CPU cfs period or the cgroup is not mounted. Period discarded.
WARNING: Your kernel does not support CPU cfs quota or the cgroup is not mounted. Quota discarded.
That mean your system's kernel build without support of CFS. Here is an issue about it, right about 3.16.0-4-amd64 version.
You need to rebuild or update your kernel.
Here is how to update it from backports:
Add mirror with backports:
$ sudo vi /etc/apt/sources.list.d/sources.list
deb http://http.debian.net/debian jessie-backports main
deb-src http://http.debian.net/debian jessie-backports main
Update apt:
sudo apt-get update
Install a kernel from a backport repo:sudo apt-get install -t jessie-backports linux-image-amd64
Reboot after instalation.
Related
I am toying with the spark operator in kubernetes, and I am trying to create a Spark Application resource with the following manifest.
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
name: pyspark-pi
namespace: spark-jobs
spec:
batchScheduler: volcano
batchSchedulerOptions:
priorityClassName: routine
type: Python
pythonVersion: "3"
mode: cluster
image: "<image_name>"
imagePullPolicy: Always
mainApplicationFile: local:///spark-files/csv_data.py
arguments:
- "10"
sparkVersion: "3.0.0"
restartPolicy:
type: OnFailure
onFailureRetries: 3
onFailureRetryInterval: 10
onSubmissionFailureRetries: 5
onSubmissionFailureRetryInterval: 20
timeToLiveSeconds: 86400
driver:
cores: 1
coreLimit: "1200m"
memory: "512m"
labels:
version: 3.0.0
serviceAccount: driver-sa
volumeMounts:
- name: sparky-data
mountPath: /spark-data
executor:
cores: 1
instances: 2
memory: "512m"
labels:
version: 3.0.0
volumeMounts:
- name: sparky-data
mountPath: /spark-data
volumes:
- name: sparky-data
hostPath:
path: /spark-data
I am running this in kind, where I have defined a volume mount to my local system where the data to be processed is present. I can see the volume being mounted in the kind nodes. But when I create the above resource, the driver pod crashes by giving the error 'no such path'. I printed the contents of the root directory of the driver pod and I could not see the mounted volume. What is the problem here and how do I fix this?
The issue is related to permissions. When mounting a volume to a pod, you need to make sure that the permissions are set correctly. Specifically, you need to make sure that the user or group that is running the application in the pod has the correct permissions to access the data.You should also make sure that the path to the volume is valid, and that the volume is properly mounted.To check if a path exists, you can use the exec command:
kubectl exec <pod_name> -- ls
Try to add security context which gives privilege and access control settings for a Pod
For more information follow this document.
I'm using pm2 to watch the directory holding the source code for my app-server's NodeJS program, running within a Kubernetes cluster.
However, I am getting this error:
ENOSPC: System limit for number of file watchers reached
I searched on that error, and found this answer: https://stackoverflow.com/a/55763478
# insert the new value into the system config
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
However, I tried running that in a pod on the target k8s node, and it says the command sudo was not found. If I remove the sudo, I get this error:
sysctl: setting key "fs.inotify.max_user_watches": Read-only file system
How can I modify the file-system watcher limit from the 8192 found on my Kubernetes node, to a higher value such as 524288?
I found a solution: use a privileged Daemon Set that runs on each node in the cluster, which has the ability to modify the fs.inotify.max_user_watches variable.
Add the following to a node-setup-daemon-set.yaml file, included in your Kubernetes cluster:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-setup
namespace: kube-system
labels:
k8s-app: node-setup
spec:
selector:
matchLabels:
name: node-setup
template:
metadata:
labels:
name: node-setup
spec:
containers:
- name: node-setup
image: ubuntu
command: ["/bin/sh","-c"]
args: ["/script/node-setup.sh; while true; do echo Sleeping && sleep 3600; done"]
env:
- name: PARTITION_NUMBER
valueFrom:
configMapKeyRef:
name: node-setup-config
key: partition_number
volumeMounts:
- name: node-setup-script
mountPath: /script
- name: dev
mountPath: /dev
- name: etc-lvm
mountPath: /etc/lvm
securityContext:
allowPrivilegeEscalation: true
privileged: true
volumes:
- name: node-setup-script
configMap:
name: node-setup-script
defaultMode: 0755
- name: dev
hostPath:
path: /dev
- name: etc-lvm
hostPath:
path: /etc/lvm
---
apiVersion: v1
kind: ConfigMap
metadata:
name: node-setup-config
namespace: kube-system
data:
partition_number: "3"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: node-setup-script
namespace: kube-system
data:
node-setup.sh: |
#!/bin/bash
set -e
# change the file-watcher max-count on each node to 524288
# insert the new value into the system config
sysctl -w fs.inotify.max_user_watches=524288
# check that the new value was applied
cat /proc/sys/fs/inotify/max_user_watches
Note: The file above could probably be simplified quite a bit. (I was basing it on this guide, and left in a lot of stuff that's probably not necessary for simply running the sysctl command.) If others succeed in trimming it further, while confirming that it still works, feel free to make/suggest those edits to my answer.
You do not want to run your container as a privileged container if you can help it.
The solution here is to set the following kernel parameters, then restart your container(s). The container(s) will use the variables from the kernel that your container is running within. This is because containers do not run separate kernels on Linux hosts (containers use the same kernel).
fs.inotify.max_user_watches=10485760
fs.aio-max-nr=10485760
fs.file-max=10485760
kernel.pid_max=10485760
kernel.threads-max=10485760
You should paste the above into: /etc/sysctl.conf.
I created a Azure file share and I am able to connect to it using map network drive in my laptop having windows 10. I created a hello-world spring boot application with volume mount configurations for azure file share and trying to deploy in Kubernetes in docker-desktop. But my pod doesn't starts -
hello-world-9d7479c4d-26mv2 0/1 ContainerCreating 0 15s
Here is the error I can see in events when I describe the POD -
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9h Successfully assigned default/hello-world-9d7479c4d-26mv2 to docker-desktop
Warning FailedMount 9h (x7 over 9h) kubelet, docker-desktop MountVolume.SetUp failed for volume "fileshare-pv" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o file_mode=0777,dir_mode=0777,vers=3.0,<masked> //mystorage.file.core.windows.net/myshare /var/lib/kubelet/pods/425012d1-13ee-4c40-bf40-d2f7ccfe5954/volumes/kubernetes.io~azure-file/fileshare-pv
Output: mount: /var/lib/kubelet/pods/425012d1-13ee-4c40-bf40-d2f7ccfe5954/volumes/kubernetes.io~azure-file/fileshare-pv: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.
Then I updated my Dockerfile to install cifs-utils -
FROM ubuntu:16.04
# Install Java
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get install -y cifs-utils && \
apt-get clean;
ENV PORT 8080
EXPOSE 8080
COPY target/*.jar /opt/app.jar
WORKDIR /opt
CMD ["java", "-jar", "app.jar"]
Still that error doesn't go. I googled a lot for solution but no luck. Is there any limitation in using azure file share with kubernates container in docker-desktop [windows machine]?
Here are my K8 configurations -
secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: storage-secret
namespace: default
type: Opaque
data:
azurestorageaccountname: BASE64-encoded-account-name
azurestorageaccountkey: BASE64-encoded-account-key
pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: fileshare-pv
labels:
usage: fileshare-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
azureFile:
secretName: storage-secret
shareName: myshare
readOnly: false
pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: fileshare-pvc
namespace: default
# Set this annotation to NOT let Kubernetes automatically create
# a persistent volume for this volume claim.
annotations:
volume.beta.kubernetes.io/storage-class: ""
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
selector:
# To make sure we match the claim with the exact volume, match the label
matchLabels:
usage: fileshare-pv
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
namespace: default
labels:
app: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: hello-world-pod
image: 'hello-world-k8:1.0'
volumeMounts:
- name: azure
mountPath: /azureshare
ports:
- containerPort: 8080
volumes:
- name: azure
persistentVolumeClaim:
claimName: fileshare-pvc
---
apiVersion: v1
kind: Service
metadata:
name: hello-world-service
namespace: default
spec:
selector:
app: hello-world
ports:
- name: http
protocol: TCP
port: 8080
targetPort: 8080
type: LoadBalancer
You likely need to install a package that knows how to mount that file system. For NFS this may be nfs-common with Debian/Ubuntu.
sudo apt update && sudo apt install nfs-common -y
It happened on my ubuntu server 22.04 LTS machine. Use sudo apt install nfs-common or sudo apt install nfs-utils to resolve it.
I am trying to install the nfs-kernel-server package on all nodes in my AKS cluster. The kernel module for NFS is not installed by default in AKS Ubuntu 16.04. I am following the guide here: https://medium.com/#patnaikshekhar/initialize-your-aks-nodes-with-daemonsets-679fa81fd20e.
my daemonset:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: installer
namespace: node-installer
spec:
selector:
matchLabels:
job: installer
template:
metadata:
labels:
job: installer
spec:
hostPID: true
restartPolicy: Always
containers:
- image: patnaikshekhar/node-installer:1.3
name: installer
securityContext:
privileged: true
volumeMounts:
- name: install-script
mountPath: /tmp
- name: host-mount
mountPath: /host
volumes:
- name: install-script
configMap:
name: sample-installer-config
- name: host-mount
hostPath:
path: /tmp/install
and this is my configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: sample-installer-config
namespace: node-installer
data:
install.sh: |
#!/bin/bash
# Update and install packages
apt-get update
apt-get install nfs-kernel-server -y
Letting the pods build and complete (x3), when I inspect their logs, only the first pod's logs show the package installed on the node. The rest of the pods have no log at all. Is there a way to reliably accomplish this?
The way to accomplish this is with a daemonset and configmap, with a pod execution policy of privileged and a host path mount. That way the container can install the required packages onto the host node, and the daemonset will apply the configuration to all new nodes on the cluster.
I have nodejs application running on Kubernetes with limits:
apiVersion: apps/v1
kind: Deployment
..
spec:
..
template:
..
spec:
containers:
- name: mynodejsapp
..
resources:
requests:
memory: "1000Mi"
cpu: "500m"
limits:
memory: "2000Mi"
cpu: "1000m"
..
And my Dockerfile for my nodejs application is based on node:10-slim:
FROM node:10-slim
..
COPY . .
RUN npm run build
EXPOSE 3000
ENTRYPOINT ["dumb-init", "--"]
CMD [ "node", "./build/index.js" ]
I find some older posts that --max_old_space_size should be set. Is that correct? Or will nodejs process automatically find existing memory limitations?
With Nodejs 10 it's bad, because on 64bits OS it could exceed your limit value. It should be fine with Nodejs 12, however If you want scale pods using CPU activity, setting max old memory is a good idea. The GC will try to stay under this value, (so under your k8s request memory) and the CPU activity will grow up. I've write a post here on this subject.