I am using ubuntu 22(Linux blah-laptop 5.15.0-43-generic #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux), and docker started to use too much memory (almost 8Gb) even without any container running.
When I kill docker desktop I can free the memory, so I guess I am missing some configuration/optimization?!
Here some details:
Tasks: 397 total, 2 running, 395 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.8 us, 1.4 sy, 0.0 ni, 95.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15623.0 total, 232.4 free, 4956.7 used, 10433.9 buff/cache
MiB Swap: 1404.2 total, 1011.5 free, 392.8 used. 1854.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3211 blah 20 0 6267844 273440 119828 R 25.0 1.7 0:27.97 /usr/bin/gnome-shell
8092 blah 20 0 13540 4516 3404 R 12.5 0.0 0:00.02 top -c
3875 blah 20 0 22.5g 60400 42916 S 6.2 0.4 0:03.08 /usr/share/docker-desktop/frontend/Docker Desktop --name=dashboard --enable-crashpad
8059 blah 20 0 9656992 7.8g 7.8g S 6.2 51.0 0:07.74 qemu-system-x86_64 -accel kvm -cpu host -machine q35 -m 7811 -smp 6 -kernel /usr/share/docker-desktop/linuxkit/kernel -append page_poison=+
Docker vesion:
Client:
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:45:09 2022
OS/Arch: linux/amd64
Context: desktop-linux
Experimental: true
Server: Docker Desktop 4.8.0 (77103)
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:46:14 2022
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Any ideas on how to fix this?
There is an option on Docker Desktop settings, to limit the use of resources.
I didn't see that (silly me). Fixed now.
Related
Docker cgroup file memory.limit_in_bytes use 9223372036854771712 instead of uint64 max as a default value ?
uint64 max = 9223372036854775807.
ahojliu#ubuntu:/sys/fs/cgroup/memory/docker/210401ffb15b18832b6a104d8aebac98cd29ff20fe0bc779f4ddfe04923faf46$ cat memory.limit_in_bytes
9223372036854771712
Ubuntu 20.04
Linux kernel version:
Linux version 5.11.0-41-generic (buildd#lgw01-amd64-005) (gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #45~20.04.1-Ubuntu SMP Wed Nov 10 10:20:10 UTC 2021
Docker version:
Client: Docker Engine - Community
Version: 20.10.11
API version: 1.41
Go version: go1.16.9
Git commit: dea9396
Built: Thu Nov 18 00:37:06 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.11
API version: 1.41 (minimum version 1.12)
Go version: go1.16.9
Git commit: 847da18
Built: Thu Nov 18 00:35:15 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
seems the default value of cgroup memory is set to PAGE_COUNTER_MAX,which is LONG_MAX/PAGE_SIZE on 64-bit platform.when the paltform`s PAGE_SIZE is diferent ,the default value of cgroup memory is diferent, such as 4k and 64k has the different default values:
0x7ffffffffffff000(pagesize=4k)
0x7fffffffffff0000(pagesize= 64k)
reflink:
https://tracker.ceph.com/issues/42059
https://github.com/torvalds/linux/blob/ea4424be16887a37735d6550cfd0611528dbe5d9/mm/memcontrol.c#L5337
Briefly
I'm looking to build docker image from a dockerfile in a directory on an external device.
Context
I have an empty directory /media/nathan/ext/test except for Dockerfile
Dockerfile is : FROM alpine
docker version is : Docker version 20.10.8, build 3967b7d28e
OS is Ubuntu 21.10
I am part of the docker group
mount options :
$> findmnt /media/nathan/ext
TARGET SOURCE FSTYPE OPTIONS
/media/nathan/ext /dev/sda1 ext4 rw,nosuid,nodev,relatime
docker deamon
$> ps aux | grep dockerd
root 919 0.0 0.5 2166356 85600 ? Ssl 09:03 0:08 dockerd --group docker --exec-root=/run/snap.docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/run/snap.docker/docker.pid --config-file=/var/snap/docker/1125/config/daemon.json
nathan 19756 0.0 0.0 11844 2448 pts/0 S+ 11:44 0:00 grep --color=auto dockerd
$DOCKER_HOST is undefined
$> echo $DOCKER_HOST
docker info
$> docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 263
Server Version: 20.10.8
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e25210fe30a0a703442421b0f60afac609f950a3
runc version:
init version: de40ad0
Security Options:
Expected result
I get a docker image
True result
$> docker build .
error checking context: 'can't stat '/media/nathan/ext/test''.
What I have tried
Just sudo everything
$> sudo docker build .
error checking context: 'can't stat '/media/nathan/ext/test''.
Issue is not resolved
Am I the owner of the context folder ?
$> echo $USER
nathan
$> ls -la
total 12
drwxrwxr-x 2 nathan nathan 4096 nov. 12 10:33 .
drwxr-xr-x 8 nathan root 4096 nov. 12 09:39 ..
-rw-rw-r-- 1 nathan nathan 12 nov. 12 10:32 Dockerfile
As per command above, I am the owner of the context directory. Am I missing something ?
add everything to .dockerignore
I've created a .dockerignore that matches everything : '*'.
Running the command [sudo] docker build . gives a very baffling answer:
$> sudo docker build .
open /media/nathan/ext/test/.dockerignore: permission denied
I do not understand how sudo doesn't have the necessary permissions to read (?) the .dockerfile. Permission which I have set to 777 out of astonishement :
ls -la
total 16
drwxrwxr-x 2 nathan nathan 4096 nov. 12 10:41 .
drwxr-xr-x 8 nathan root 4096 nov. 12 09:39 ..
-rw-rw-r-- 1 nathan nathan 12 nov. 12 10:32 Dockerfile
-rwxrwxrwx 1 nathan nathan 2 nov. 12 10:41 .dockerignore
of course, other programms were capable of reading the file without any issue as expected
$> cat .dockerignore
*
Build outside of external drive
$> pwd
/home/nathan/Bureau/test
$> ls -la
total 12
drwxrwxr-x 2 nathan nathan 4096 nov. 12 10:58 .
drwxr-xr-x 3 nathan nathan 4096 nov. 12 10:56 ..
-rw-rw-r-- 1 nathan nathan 12 nov. 12 10:58 Dockerfile
$> docker build .
Sending build context to Docker daemon 2.048kB
Step 1/1 : FROM alpine
---> 14119a10abf4
Successfully built 14119a10abf4
Image is built, but I which to replicate result into external drive.
running docker build . with journalctl
[...]
nov. 12 11:42:52 nathan-pc systemd[1746]: Started snap.docker.docker.ba3da9ef-34ee-4a63-8ff4-6a56327c5cd2.scope.
nov. 12 11:42:52 nathan-pc audit[19690]: AVC apparmor="DENIED" operation="open" profile="snap.docker.docker" name="/media/nathan/ext/workspace/dino/ntrip-client/RTKLIB/" pid=19690 comm="docker" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
nov. 12 11:42:52 nathan-pc kernel: audit: type=1400 audit(1636713772.367:93): apparmor="DENIED" operation="open" profile="snap.docker.docker" name="/media/nathan/ext/workspace/dino/ntrip-client/RTKLIB/" pid=19690 comm="docker" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
nov. 12 11:42:52 nathan-pc systemd[1746]: snap.docker.docker.ba3da9ef-34ee-4a63-8ff4-6a56327c5cd2.scope: Deactivated successfully.
[...]
Thank you for your time
Hi i am new to kubernetes.
1) Could not able to scaled container/pods in worker nodes. and its memory usage always remain zero. any reason ?
2) Whenever i scaled pods/container its always create in master node.
3) Is there any way to limit pod on specific nodes ?
4) How pods divide when i scaled ?
any help appropriated.
kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
kubectl describe nodes
Name: worker-node
Roles: worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=worker-node
node-role.kubernetes.io/worker=worker
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 19 Feb 2019 15:03:33 +0530
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 19 Feb 2019 18:57:22 +0530 Tue, 19 Feb 2019 15:26:13 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure True Tue, 19 Feb 2019 18:57:22 +0530 Tue, 19 Feb 2019 15:26:23 +0530 KubeletHasDiskPressure kubelet has disk pressure
PIDPressure False Tue, 19 Feb 2019 18:57:22 +0530 Tue, 19 Feb 2019 15:26:13 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 19 Feb 2019 18:57:22 +0530 Tue, 19 Feb 2019 15:26:13 +0530 KubeletReady kubelet is posting ready status. AppArmor enabled
OutOfDisk Unknown Tue, 19 Feb 2019 15:03:33 +0530 Tue, 19 Feb 2019 15:25:47 +0530 NodeStatusNeverUpdated Kubelet never posted node status.
Addresses:
InternalIP: 192.168.1.10
Hostname: worker-node
Capacity:
cpu: 4
ephemeral-storage: 229335396Ki
hugepages-2Mi: 0
memory: 16101704Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 211355500604
hugepages-2Mi: 0
memory: 15999304Ki
pods: 110
System Info:
Machine ID: 1082300ebda9485cae458a9761313649
System UUID: E4DAAC81-5262-11CB-96ED-94898013122F
Boot ID: ffd5ce4b-437f-4497-9337-e72c06f88429
Kernel Version: 4.15.0-45-generic
OS Image: Ubuntu 18.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.13.3
Kube-Proxy Version: v1.13.3
PodCIDR: 192.168.1.0/24
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 55m kube-proxy, worker-node Starting kube-proxy.
Normal Starting 55m kube-proxy, worker-node Starting kube-proxy.
Normal Starting 33m kube-proxy, worker-node Starting kube-proxy.
Normal Starting 11m kube-proxy, worker-node Starting kube-proxy.
Warning EvictionThresholdMet 65s (x1139 over 3h31m) kubelet, worker-node Attempting to reclaim ephemeral-storage
This is very strange, by default kubernetes has the label to exclude the master from pod execution.
kubectl get nodes --show-labels
Now check for the label
node-role.kubernetes.io/master=true:NoSchedule
If your master doesn't has this label, you can retain the master with:
kubectl taint nodes $HOSTNAME node-role.kubernetes.io/master=true:NoSchedule
Though I am able to successfully push a newly pulled docker image to Nexus 3 docker hosted repo, an error like "invalid checksum digest format" is thrown at the end. I pulled "jenkins:latest" from dockerhub, then tagged it and then pushed it to a nexus docker hosted repo.
f3e4e0468545: Pushed
656120ad8c56: Pushed
30f9a83f20f3: Pushed
78dbfa5b7cbc: Pushed
invalid checksum digest format
I know Nexus 3 is not LTS yet, but want to be sure that its not my environment settings. I have an insecure docker registry on 18443
docker info
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 53
Server Version: 1.10.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 89
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 3.16.0-53-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.86 GiB
Client:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:27:08 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.1
API version: 1.22
Go version: go1.5.3
Git commit: 9e83765
Built: Thu Feb 11 19:27:08 2016
OS/Arch: linux/amd64
Docker version 1.10 was not out when Nexus 3.0m7 was released. We are working on adding support for it now. This specific issue is being tracked here:
https://issues.sonatype.org/browse/NEXUS-9766
UPDATE: This issue/ticket is resolved now in Nexus Repository Manager 3.0.0-03. For upgrade instructions see https://support.sonatype.com/hc/en-us/articles/217967608-How-to-Upgrade-Nexus-3-Milestone-m7-to-3-0-0-Final.
[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G
It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).