Docker run, no space left on device - linux

[root#host ~]# docker run 9e7de9390856
Timestamp: 2015-06-15 22:20:58.8367035 +1000 AEST
Code: System error
Message: [/usr/bin/tar -xf /var/lib/docker/tmp/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5090033894/_tmp.tar -C /var/lib/docker/devicemapper/mnt/cde0f3a199597ac2e18e7efc7744c84a6c134adef31fb88b6982a8732f45efa5/rootfs/tmp .] failed: /usr/bin/tar: ./was/fixPack/7.0.0-WS-WASSDK-LinuxX64-FP0000027.pak: Wrote only 4608 of 10240 bytes
/usr/bin/tar: ./was/fixPack/wasFixPackInstallResponseFile: Cannot write: No space left on device
.
.
Cannot write: No spaFATA[0141] Error response from daemon: : exit status 2
df -h:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 6.0G 3.2G 2.9G 52% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/xvdb1 99G 28G 67G 30% /var/lib/docker
docker info:
Containers: 2
Images: 34
Storage Driver: devicemapper
Pool Name: docker-202:17-2621441-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 15.89 GB
Data Space Total: 107.4 GB
Data Space Available: 76.3 GB
Metadata Space Used: 10.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.137 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 2
Total Memory: 3.452 GiB
Name: ip-10-100-128-182.localdomain
ID: 4ZZZ:BSQD:GBKL:4Y3N:J6BL:47QE:3HMQ:GLMY:FPUK:CEPM:3EBP:ZU7G
Debug mode (server): true
Debug mode (client): false
Fds: 13
Goroutines: 18
System Time: Mon Jun 15 22:48:24 AEST 2015
EventsListeners: 0
Init SHA1: 836be3a369bfc6bd4cbd3ade1eedbafcc1ea05d0
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
uname -a:
Linux ip-10-100-128-182.localdomain 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
Anyone can help me?
Not sure this information is enough. But tried couple of solutions, nothing worked.
docker version:
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 8aae715/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 8aae715/1.6.0
OS/Arch (server): linux/amd64
[root#host ~]# service docker status -l
Redirecting to /bin/systemctl status -l docker.service
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled)
Active: active (running) since Tue 2015-06-16 00:31:46 AEST; 2min 2s ago
Docs: http://docs.docker.com
Main PID: 3306 (docker)
CGroup: /system.slice/docker.service
└─3306 /usr/bin/docker -d --storage-opt dm.basesize=30G --storage-opt dm.loopmetadatasize=4G

It sounds like you're trying to start a container from a 14GB image.
A Docker container, when using the devicemapper storage driver, only has 10GB of space available by default. You appear to be using the devicemapper driver, so this is probably the source of your problem.
This article discusses in detail the process you need to use to increase the amount of space available for container filesystems.
Filesystem-based drivers (like the overlay driver) to not have this same limitation (but they may of course suffer from other limitations).

Related

Docker containers consuming all the inodes of linux machine

I am running out of inodes on my Debian 10 machine
I found the culprits
~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 500578 317 500261 1% /dev
tmpfs 504337 547 503790 1% /run
/dev/sda1 3276800 3090639 186161 95% /
tmpfs 504337 1 504336 1% /dev/shm
tmpfs 504337 3 504334 1% /run/lock
tmpfs 504337 17 504320 1% /sys/fs/cgroup
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/c238a2f5bcd5e7e193d572fd4b331510cd2438d113a66e9125f983c929cd8485/merged
overlay 3276800 3090639 186161 95% /var/lib/docker/overlay2/5f3ee0ad57f147f8dba95e1e06ea98fc697c72ff8eaaafa6494faeb8f0742096/merged
tmpfs 504337 11 504326 1% /run/user/10976
tmpfs 504337 11 504326 1% /run/user/10762
shm 504337 1 504336 1% /var/lib/docker/containers/135852b9e95d22ace0cf10b27752a3ad15d1fdfaebcde74f546c26ae4ef01ad8/mounts/shm
shm 504337 1 504336 1% /var/lib/docker/containers/cd9619f7703b1e9fee7c1338379dab5270260d23834b27f278c57500b5e50b5a/mounts/shm
docker overlay /var/lib/docker/overlay2/*/merged
Only 2 containers are running.
no dangling images on the machine as well.
~# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 19
Server Version: 20.10.18
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.7.0-0.bpo.2-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.848GiB
Name: name
ID: SNF2:X4SU:OLKG:ESHV:PX3U:UJGB:MADV:OZ3R:UCZP:KONV:K4ZV:ILL4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
what can I do to reclaim inodes?
PS: Docker version 18.09.0, build 4d60db4
docker system prune and docker image prune does not help either.

Same docker image works in Docker for Windows but not in docker on linux

What I want to achieve
I want to connect to an oracle database within an oracle container via sqlplus
What my problem is
I can connect to the database of the same image within Docker for Windows but not on a linux host
What I did
Windows
c:\>docker pull wnameless/oracle-xe-11g
c:\>docker run -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
c:\>docker exec -it f79f7aa5222c /bin/bash
root#f79f7aa5222c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:28:45 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
Linux
[xx#yy Docker]$ sudo docker pull wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker run --shm-size=2g -d -p 49160:22 -p 49161:1521 wnameless/oracle-xe-11g
[xx#yy Docker]$ sudo docker exec -it cea28583cb1c /bin/bash
root#cea28583cb1c:/# sqlplus system/oracle
Result:
SQL*Plus: Release 11.2.0.2.0 Production on Wed Jul 19 07:34:16 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
Infos
Linux system:
docker info
[xx#yy Docker]$ sudo docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 33
Server Version: 1.13.1
Storage Driver: devicemapper
Pool Name: docker-8:17-4460587-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 4.741 GB
Data Space Total: 107.4 GB
Data Space Available: 23.21 GB
Metadata Space Used: 4.289 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.143 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use --storage-opt dm.thinpooldev to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.137 (2016-11-30)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.11.9-300.fc26.x86_64
Operating System: Fedora 26 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 7.776 GiB
Name: yy.domain.de
ID: 4YVE:4FQ7:27SF:5JZJ:H6RA:P4H5:TK6O:FAGY:CKHS:RQGQ:3BDS:L7W7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://registry.fedoraproject.org/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Registries: registry.fedoraproject.org (secure), registry.access.redhat.com (secure), docker.io (secure)
Docker logs
[xx#yy Docker]$ sudo docker logs cea28583cb1c
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.
/usr/sbin/startup.sh: ignoring /docker-entrypoint-initdb.d/*
df
root#cea28583cb1c:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-8:17-4460587-f479be876efbf45834ea013ac119d63629944a2b315c7cbe7da3a22c1b1e566e 10G 1.9G 8.2G 19% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdb1 110G 89G 17G 85% /etc/hosts
shm 2.0G 0 2.0G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/firmware
stat
root#cea28583cb1c:/# stat /
File: '/'
Size: 253 Blocks: 0 IO Block: 4096 directory
Device: fd01h/64769d Inode: 131 Links: 22
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-07-11 07:12:36.676904944 +0000
Modify: 2017-07-19 07:33:44.026249772 +0000
Change: 2017-07-19 07:33:44.216248030 +0000
Birth: -
lsnrctrl
xx#yy:/etc/selinux# lsnrctl status
LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 19-JUL-2017 11:30:37
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_XE)))
STATUS of the LISTENER
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date 19-JUL-2017 07:33:46
Uptime 0 days 3 hr. 56 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Default Service XE
Listener Parameter File /u01/app/oracle/product/11.2.0/xe/network/admin/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/cea28583cb1c/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_XE)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cea28583cb1c)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
What I already tried
I tried both users, oracle and root.
I changed permissions of /u01/app/oracle to 755
I checked selinux, this seems not to be active/installed. At least thats what the docker info tells and I cant find sestatus command and /etc/selinux only contains semanage.conf.
I tried several different oracle images, same result in every container.
EDIT
As requested, I tried starting oracle service myself:
service oracle-xe start
Result of alert_XE.log:
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =19
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production.
Using parameter settings in server-side spfile /u01/app/oracle/product/11.2.0/xe/dbs/spfileXE.ora
System parameters with non-default values:
sessions = 176
sga_target = 576M
control_files = "/u01/app/oracle/oradata/XE/control.dbf"
compatible = "11.2.0.0.0"
db_recovery_file_dest = "/u01/app/oracle/fast_recovery_area"
db_recovery_file_dest_size= 10G
undo_management = "AUTO"
undo_tablespace = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
dispatchers = "(PROTOCOL=TCP) (SERVICE=XEXDB)"
shared_servers = 4
job_queue_processes = 4
audit_file_dest = "/u01/app/oracle/admin/XE/adump"
db_name = "XE"
open_cursors = 300
pga_aggregate_target = 195840K
diagnostic_dest = "/u01/app/oracle"
Solved the puzzle, finally.
It turned out that SELinux was active. I don't know how I missed that some weeks ago. I definitely checked that, but maybe I was within the container and not the host.
So I ran the container with the --privileged flag and suddenly I can connect to the db.
Thanks for your comments and sorry that I posted it here, you are right #jww. I will make sure to only post programming questions here in the future. Sorry for that.
You can also edit /etc/sysconfig/docker and remove --selinux-enabled from options that's usually added by default. Then restart docker service:
sudo systemctl restart docker
or
sudo service docker restart

Docker Devmapper space issue - increase size

I have the same issue as in space issue on docker devmapper and CentOS7
It only specifies to clean up but not how I can increase the space and I dont have any images to clean. I tried several things with dm.min_free_space but nothing worked and want to increase the space.
OS Version/build: Red Hat Enterprise Linux Server release 7.3 (Maipo)
App version:
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-11.el7.centos.x86_64
Go version: go1.7.4
Git commit: 96d83a5/1.12.6
Built: Tue Mar 7 09:23:34 2017
OS/Arch: linux/amd64
Steps to reproduce
I have no containers running currently and have some docker images pertaining to Kubernetes which will be used by the Kubernetes service.
sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[kubeuser4#kubenode4 Employee]$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/busybox latest 00f017a8c2a6 5 days ago 1.11 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 34d3450d733b 6 weeks ago 205 MB
docker.io/java 8 d23bdf5b1b1b 8 weeks ago 643.1 MB
gcr.io/google_containers/heapster_grafana v2.6.0-2 b43443930626 12 months ago 230 MB
When I try to create a docker image of my application that needs to be used, I get the below error.
devmapper: Thin Pool has 8783 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I tried the cleaning up as mentioned in the other forums, but not helped much and getting the same error. When I tried to run with this sudo docker --storage-opt dm.min_free_space=0%, seems like it starts as a daemon, but still it failed with another error "docker-runc not installed on system" and also I dont want to run it as a daemon.
Below are some command outputs
sudo dmsetup status
localvg00-lv_home: 0 20971520 linear
localvg00-lv_home: 20971520 20971520 linear
docker-251:5-134039-pool: 0 209715200 thin-pool 924 848/524288 1629226/1638400 - rw discard_passdown queue_if_no_space
localvg00-lv_tmp: 0 4194304 linear
localvg00-lv_swap: 0 8388608 linear
localvg00-lv_root: 0 2097152 linear
localvg00-lv_root: 2097152 20971520 linear
localvg00-lv_usr: 0 16777216 linear
localvg00-lv_var: 0 8388608 linear
localvg00-lv_var: 8388608 62914560 linear
sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-251:5-134039-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 106.8 GB
Data Space Total: 107.4 GB
Data Space Available: 601.2 MB
Metadata Space Used: 3.473 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: overlay null bridge host
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 4.1.12-61.1.28.el7uek.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 2
Total Memory: 7.545 GiB
Name: kubenode4
I had also tried increasing all the physical volume size and logical volume size(lv_var) on my linux machine, but still it doesnt work.
sudo lvs
[sudo] password for kubeuser4:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_home localvg00 -wi-ao---- 20.00g
lv_root localvg00 -wi-ao---- 11.00g
lv_swap localvg00 -wi-ao---- 4.00g
lv_tmp localvg00 -wi-ao---- 2.00g
lv_usr localvg00 -wi-ao---- 8.00g
lv_var localvg00 -wi-ao---- 34.00g
sudo ls -lsh /var/lib/docker/devicemapper/devicemapper/data
2.3G -rw------- 1 root root 100G Mar 14 22:16 /var/lib/docker/devicemapper/devicemapper/data
Someone please let me know how it can be done.
Thanks,
It is better move away from devicemapper for a few reasons.
devicemapper in loopback unrecoverable storage issue: https://github.com/docker/docker/issues/3182 "devicemapper not recommended for production use".
I found it easy enough to switch to overlay storage driver, YMMV of course but hopefully not too much. 'rm -rf /var/lib/docker' is somewhat optional when switching but easy and I would highly recommend it as long as you can load your images back in. http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/
systemctl stop docker
rm -rf /var/lib/docker
# if these files do not already exist . . . create them, otherwise you need to edit by hand, you can also just add -s overlay in the systemctl docker script
ls /etc/sysconfig/docker /etc/sysconfig/docker-storage
[[ $? != 0 ]] && {
echo OPTIONS='--selinux-enabled=false' > /etc/sysconfig/docker
echo "DOCKER_STORAGE_OPTIONS= -s overlay" > /etc/sysconfig/docker-storage
}
systemctl start docker
systemctl status docker
docker images
more reading:
https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/
Was able to get it working and have mentioned it in
https://forums.docker.com/t/devmapper-space-issue/29786/3

Docker instance cannot run images anymore and unable to reclaim free space

I am trying to start any of my saved containers in docker but am unable to do it. I have started getting the Error response from daemon:
Error running DeviceCreate (createSnapDevice) dm_task_run failed
This started happening after committing a relatively big docker image and it seemed to have filled up all available docker data space, even though I had lots of space on the host machine. Now I am unable to free up the docker data space anymore, even after deleting the big image. Docker is unable to reclaim the space. I also tried the fix mentioned below so that I can start the docker container but was not successful. Is there anything I can do to fix existing Docker to run images again?
Related question: Can't run Docker container due device mapper error
Here is my host configuration. Data Space used and total has reached max and free is in 0.
# docker info
=========================================================
Containers: 49
Images: 23
Storage Driver: devicemapper
Pool Name: docker-8:3-4998488-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 107.4 GB
Data Space Total: 107.4 GB
Data Space Available: 0 B
Metadata Space Used: 60.36 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.087 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: Red Hat Enterprise Linux
CPUs: 4
Total Memory: 7.64 GiB
docker version
=========================================================
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
running
docker system df
will tell you disk usage and reclaimable space.
To free up space and keep your images:
Remove Volumes:
docker volume rm $(docker volume ls --format {{.ID}})
Remove Stopped Containers:
docker rm $(docker ps -aq)
To Just clean everything
optional: Deletes All images/containers/volumes/networks
docker system prune -af
On the RHEL docker host, The only way I could reclaim the space and start a docker image was to actually remove the devicemapper/data. No other docker command to remove containers/images/volumes seems to work. I backed up the image I wanted to a tar ball and did the removal to have it working again.

Error when building a Docker container

The command docker build fails with error :
Error getting container f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753 from driver devicemapper: Error mounting '/dev/mapper/docker-8:2-41159178-f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753' on '/var/lib/docker/devicemapper/mnt/f43128eda488c88a3b2e111aafb30b80a44faaead33bcf02f8bffd7ae1832753': no such file or directory
docker info
Containers: 7
Images: 148
Storage Driver: devicemapper
Pool Name: docker-8:2-41159178-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 27.96 GB
Data Space Total: 107.4 GB
Data Space Available: 79.42 GB
Metadata Space Used: 19.27 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.128 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.14.27-xxxx-grs-ipv6-64
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.66 GiB
Name: libra
ID: KYU6:KECQ:GGF3:QL4W:SB35:C3UX:54EY:NN3A:U4RF:SFIK:5ULD:2THZ
Username: porfus
Registry: https://index.docker.io/v1/
I use root-server OVH Kimsufi and Ubuntu 14.04.2 (3.14.27-xxxx-grs-ipv6-64 #1 SMP Wed Dec 17 14:02:42 CET 2014 x86_64 x86_64 x86_64 GNU/Linux). I read the description of this bug on GitHub (https://github.com/docker/docker/issues/4036), but did not understand how to get rid of it.
Considering you have "Library Version: 1.02.77 (2012-10-15)", consider upgrading docker and that library to its latest version.
That would make sure all the fixes mentioned in issues/4036 are taken into account.

Resources