The command docker build fails with error :
Error getting container f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753 from driver devicemapper: Error mounting '/dev/mapper/docker-8:2-41159178-f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753' on '/var/lib/docker/devicemapper/mnt/f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753': no such file or directory
docker info
Containers: 7
Images: 148
Storage Driver: devicemapper
Pool Name: docker-8:2-41159178-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 27.96 GB
Data Space Total: 107.4 GB
Data Space Available: 79.42 GB
Metadata Space Used: 19.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.128 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.14.27-xxxx-grs-ipv6-64
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.66 GiB
Name: libra
ID: KYU6:KECQ:GGF3:QL4W:SB35:C3UX:54EY:NN3A:U4RF:SFIK:5ULD:2THZ
Username: porfus
Registry: https://index.docker.io/v1/
I use root-server OVH Kimsufi and Ubuntu 14.04.2 (3.14.27-xxxx-grs-ipv6-64 #1 SMP Wed Dec 17 14:02:42 CET 2014 x86_64 x86_64 x86_64 GNU/Linux). I read the description of this bug on GitHub (https://github.com/docker/docker/issues/4036), but did not understand how to get rid of it.
Considering you have "Library Version: 1.02.77 (2012-10-15)", consider upgrading docker and that library to its latest version.
That would make sure all the fixes mentioned in issues/4036 are taken into account.
Related
We have deployed jenkins on docker container and recently we started seeing that our jenkins server is not coming up due to disk space issue. Below is the error we see in logs.
2022-09-17 21:41:32.567+0000 [id=32] INFO hudson.slaves.SlaveComputer#tryReconnect: Attempting to reconnect V3LOCITY-SLAVE-02
/usr/local/bin/jenkins.sh: line 38: cannot create temp file for here-document: No space left on device
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Exception in thread "main" java.io.IOException: Jenkins has failed to create a temporary file in /tmp
at Main.extractFromJar(Main.java:498)
at Main._main(Main.java:310)
at Main.main(Main.java:151)
Caused by: java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2063)
at Main.extractFromJar(Main.java:495)
... 2 more
We assume issue with docker container running of out space, See below info for your reference.
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 1 572.5MB 0B (0%)
Containers 1 0 9.467GB 9.467GB (100%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
Assuming container running of space we have increased it to 40 GB by adding below content in /etc/docker/daemon.json file and recreated the contained but still see the same issue after restart of container
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.basesize=40G"
]
}
See below docker info your reference.
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 19.03.11-ol
Storage Driver: devicemapper
Pool Name: docker-249:0-1140851221-pool
Pool Blocksize: 65.54kB
Base Device Size: 42.95GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 10.82GB
Data Space Total: 107.4GB
Data Space Available: 96.56GB
Metadata Space Used: 6.877MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.141GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.170-RHEL7 (2020-03-24)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
runc version: 52de29d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.1.12-124.65.1.2.el7uek.x86_64
Operating System: Oracle Linux Server 7.9
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.56GiB
Name: vm-app-docker-jenkinsqa
ID: TAII:OWLM:Y3BU:65DC:A3SK:SSJQ:H6H2:BLA2:HQA5:ODCP:Y7S5:KCJ2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Registries:
You need to map jenkins home to an external folder (volume) and make sure the host has enough space.
See Jenkins docs for more details.
For example:
docker run --name jenkins -v /var/jenkins_home:/var/jenkins_home ...
I have the same issue as in space issue on docker devmapper and CentOS7
It only specifies to clean up but not how I can increase the space and I dont have any images to clean. I tried several things with dm.min_free_space but nothing worked and want to increase the space.
OS Version/build: Red Hat Enterprise Linux Server release 7.3 (Maipo)
App version:
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Steps to reproduce
I have no containers running currently and have some docker images pertaining to Kubernetes which will be used by the Kubernetes service.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[kubeuser4#kubenode4 Employee]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/busybox latest 00f017a8c2a6 5 days ago 1.11 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 34d3450d733b 6 weeks ago 205 MB
docker.io/java 8 d23bdf5b1b1b 8 weeks ago 643.1 MB
gcr.io/google_containers/heapster_grafana v2.6.0-2 b43443930626 12 months ago 230 MB
When I try to create a docker image of my application that needs to be used, I get the below error.
devmapper: Thin Pool has 8783 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I tried the cleaning up as mentioned in the other forums, but not helped much and getting the same error. When I tried to run with this sudo docker --storage-opt dm.min_free_space=0%, seems like it starts as a daemon, but still it failed with another error "docker-runc not installed on system" and also I dont want to run it as a daemon.
Below are some command outputs
sudo dmsetup status
localvg00-lv_home: 0 20971520 linear
localvg00-lv_home: 20971520 20971520 linear
docker-251:5-134039-pool: 0 209715200 thin-pool 924 848/524288 1629226/1638400 - rw discard_passdown queue_if_no_space
localvg00-lv_tmp: 0 4194304 linear
localvg00-lv_swap: 0 8388608 linear
localvg00-lv_root: 0 2097152 linear
localvg00-lv_root: 2097152 20971520 linear
localvg00-lv_usr: 0 16777216 linear
localvg00-lv_var: 0 8388608 linear
localvg00-lv_var: 8388608 62914560 linear
sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-251:5-134039-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 106.8 GB
Data Space Total: 107.4 GB
Data Space Available: 601.2 MB
Metadata Space Used: 3.473 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: overlay null bridge host
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 4.1.12-61.1.28.el7uek.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.545 GiB
Name: kubenode4
I had also tried increasing all the physical volume size and logical volume size(lv_var) on my linux machine, but still it doesnt work.
sudo lvs
[sudo] password for kubeuser4:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home localvg00 -wi-ao---- 20.00g
lv_root localvg00 -wi-ao---- 11.00g
lv_swap localvg00 -wi-ao---- 4.00g
lv_tmp localvg00 -wi-ao---- 2.00g
lv_usr localvg00 -wi-ao---- 8.00g
lv_var localvg00 -wi-ao---- 34.00g
sudo ls -lsh /var/lib/docker/devicemapper/devicemapper/data
2.3G -rw------- 1 root root 100G Mar 14 22:16 /var/lib/docker/devicemapper/devicemapper/data
Someone please let me know how it can be done.
Thanks,
It is better move away from devicemapper for a few reasons.
devicemapper in loopback unrecoverable storage issue: https://github.com/docker/docker/issues/3182 "devicemapper not recommended for production use".
I found it easy enough to switch to overlay storage driver, YMMV of course but hopefully not too much. 'rm -rf /var/lib/docker' is somewhat optional when switching but easy and I would highly recommend it as long as you can load your images back in. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
systemctl stop docker
rm -rf /var/lib/docker
# if these files do not already exist . . . create them, otherwise you need to edit by hand, you can also just add -s overlay in the systemctl docker script
ls /etc/sysconfig/docker /etc/sysconfig/docker-storage
[[ $? != 0 ]] && {
echo OPTIONS='--selinux-enabled=false' > /etc/sysconfig/docker
echo "DOCKER_STORAGE_OPTIONS= -s overlay" > /etc/sysconfig/docker-storage
}
systemctl start docker
systemctl status docker
docker images
more reading:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/
Was able to get it working and have mentioned it in
https://forums.docker.com/t/devmapper-space-issue/29786/3
I'm running into a problem with docker. I've got here OpenSuse 13.2 with a self-built version of libseccomp library. it's fresh version 2.3.1 from couple of weeks ago. If i'm running any docker container, i get the following error:
hostname:/usr/lib/docker # docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
78445dd45222: Pull complete
Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7
Status: Downloaded newer image for hello-world:latest
container_linux.go:247: starting container process caused "conditional filtering requires libseccomp version >= 2.2.1"
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "conditional filtering requires libseccomp version >= 2.2.1".
ERRO[0002] error getting events from daemon: net/http: request canceled
Of course i can use an option --security-opt seccomp:unconfined when starting a container, but this is not my purpose.
# rpm -qa libseccomp
libseccomp-2.3.1-1.x86_64
docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 1.13.0
Storage Driver: devicemapper
Pool Name: docker-254:2-655361-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 307.2 MB
Data Space Total: 107.4 GB
Data Space Available: 20.64 GB
Metadata Space Used: 806.9 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.03.01 (2011-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: oci runc
Default Runtime: runc
Init Binary: docker-init
containerd version: (expected: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e)
runc version: N/A (expected: 2f7393a47307a16f8cee44a37b262e8b81021e3e)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 3.16.7-53-desktop
Operating System: openSUSE 13.2 (Harlequin) (x86_64)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.868 GiB
Name: hostname
ID: DCOH:JZMG:ZUTM:5MSB:DVAG:SQXS:Z36N:5OXU:GQII:YTMO:RWDA:HYBJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Seems the problem may be with runc. I'm currently running into the same identical error, on Leap 42.1 with docker & runc from OBS Virtualization:containers repo. My setup was running fine until recent set of package updates.
i | runc | package | 0.1.1+gitr2942_2f7393a-33.2 | x86_64 | Virtualization:containers (openSUSE_Leap_42.1)
i | docker | package | 1.13.0-182.1 | x86_64 | Virtualization:containers (openSUSE_Leap_42.1)
strings on /usr/sbin/runc show:
strings /usr/sbin/runc | grep 2.2.1
[..]
conditional filtering requires libseccomp version >= 2.2.1
[..]
Going down further, changelog shows:
* Fri Feb 24 2017
- update to docker-1.13.0 requirement
* Mon Dec 19 2016
- update runc to the version used in docker 1.12.5 (bsc#1016307).
And the source for that package has Godeps/_workspace/src/github.com/seccomp/libseccomp-golang/seccomp_internal.go with this on line 299:
return fmt.Errorf("conditional filtering requires libseccomp version >= 2.2.1")
Looks like there is now an official bug report, and this issue impacts a few different SUSE releases that use that repo:
https://bugzilla.opensuse.org/show_bug.cgi?id=1028639
My host machine OS is OEL7 with kernel
Linux ispaaaems1 3.10.0-123.el7.x86_64 #1 SMP Wed Jul 9 18:59:11 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux
And my docker info is
Containers: 4
Images: 124
Storage Driver: devicemapper
Pool Name: docker-253:0-88356-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 7.43 GB
Data Space Total: 107.4 GB
Data Space Available: 99.94 GB
Metadata Space Used: 9.302 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Udev Sync Supported: true
Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.el7.x86_64
Operating System: Oracle Linux Server 7.2
CPUs: 2
Total Memory: 7.641 GiB
Name: ispaaaems1
ID: 6MUK:HS3D:OQTS:QMWY:WCKE:AZT6:COJP:F7EA:RPNX:7RHY:TKFB:D4LT
I am running a docker container with OS OEL6.6. I am sending a radius request at 1812-1813. All the packets are reaching the host machine, but few packets (3 out of 5) are getting dropped (not reaching inside the container).
Any help will be appreciated. Thanks in adavance.
I am trying to start any of my saved containers in docker but am unable to do it. I have started getting the Error response from daemon:
Error running DeviceCreate (createSnapDevice) dm_task_run failed
This started happening after committing a relatively big docker image and it seemed to have filled up all available docker data space, even though I had lots of space on the host machine. Now I am unable to free up the docker data space anymore, even after deleting the big image. Docker is unable to reclaim the space. I also tried the fix mentioned below so that I can start the docker container but was not successful. Is there anything I can do to fix existing Docker to run images again?
Related question: Can't run Docker container due device mapper error
Here is my host configuration. Data Space used and total has reached max and free is in 0.
# docker info
=========================================================
Containers: 49
Images: 23
Storage Driver: devicemapper
Pool Name: docker-8:3-4998488-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 107.4 GB
Data Space Total: 107.4 GB
Data Space Available: 0 B
Metadata Space Used: 60.36 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.087 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux
CPUs: 4
Total Memory: 7.64 GiB
docker version
=========================================================
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
running
docker system df
will tell you disk usage and reclaimable space.
To free up space and keep your images:
Remove Volumes:
docker volume rm $(docker volume ls --format {{.ID}})
Remove Stopped Containers:
docker rm $(docker ps -aq)
To Just clean everything
optional: Deletes All images/containers/volumes/networks
docker system prune -af
On the RHEL docker host, The only way I could reclaim the space and start a docker image was to actually remove the devicemapper/data. No other docker command to remove containers/images/volumes seems to work. I backed up the image I wanted to a tar ball and did the removal to have it working again.