Arch Linux, Docker "No space left on device." - linux

Question

All of the similar questions I see are resolved by cleaning up the images or containers or orphaned volumes but I am not having any of those problems. I even completely deleted /var/lib/docker and still nothing.
Relevant output:
[N] ⋊> ~/W/W/cocagne on master ⨯ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro -v /var/lib/docker:/var/lib/docker martin/docker-cleanup-vol
umes
docker: Error response from daemon: Container command '/usr/local/bin/docker-cleanup-volumes.sh' not found or does not exist..
[N] ⋊> ~/W/W/cocagne on master ⨯ docker-compose build 11:56:23
mysql uses an image, skipping
Building vitess
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 9, in <module>
load_entry_point('docker-compose==1.7.1', 'console_scripts', 'docker-compose')()
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 58, in main
command()
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 109, in perform_command
handler(command, command_options)
File "/usr/lib/python3.5/site-packages/compose/cli/main.py", line 213, in build
force_rm=bool(options.get('--force-rm', False)))
File "/usr/lib/python3.5/site-packages/compose/project.py", line 300, in build
service.build(no_cache, pull, force_rm)
File "/usr/lib/python3.5/site-packages/compose/service.py", line 718, in build
buildargs=build_opts.get('args', None),
File "/usr/lib/python3.5/site-packages/docker/api/build.py", line 54, in build
path, exclude=exclude, dockerfile=dockerfile, gzip=gzip
File "/usr/lib/python3.5/site-packages/docker/utils/utils.py", line 103, in tar
t.add(os.path.join(root, path), arcname=path, recursive=False)
File "/usr/lib/python3.5/tarfile.py", line 1938, in add
self.addfile(tarinfo, f)
File "/usr/lib/python3.5/tarfile.py", line 1966, in addfile
copyfileobj(fileobj, self.fileobj, tarinfo.size)
File "/usr/lib/python3.5/tarfile.py", line 244, in copyfileobj
dst.write(buf)
File "/usr/lib/python3.5/tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -a 11:56:30
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[I] ⋊> ~/W/W/cocagne on master ⨯ docker ps -q 11:57:25
[I] ⋊> ~/W/W/cocagne on master ⨯ docker image -q 11:57:28
docker: 'image' is not a docker command.
See 'docker --help'.
[I] ⋊> ~/W/W/cocagne on master ⨯ docker images -a 11:57:39
REPOSITORY TAG IMAGE ID CREATED SIZE
martin/docker-cleanup-volumes latest 8c41df286c03 12 weeks ago 22.12 MB
[I] ⋊> ~/W/W/cocagne on master ⨯ df -h 11:57:41
Filesystem Size Used Avail Use% Mounted on
dev 3.9G 0 3.9G 0% /dev
run 3.9G 832K 3.9G 1% /run
/dev/sda4 27G 9.1G 17G 36% /
tmpfs 3.9G 64M 3.8G 2% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 32K 3.9G 1% /tmp
/dev/sda1 42G 16G 25G 39% /home
/dev/sda2 42G 9.4G 30G 24% /var
/dev/sda5 1.3G 32M 1.3G 3% /boot
tmpfs 790M 12K 790M 1% /run/user/1000
[I] ⋊> ~/W/W/cocagne on master ⨯ 11:57:54
docker info
[I] ⋊> ~/W/W/cocagne on master ⨯ docker info 12:01:55
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.11.2
Storage Driver: devicemapper
Pool Name: docker-8:2-2359321-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 11.8 MB
Data Space Total: 107.4 GB
Data Space Available: 34.57 GB
Metadata Space Used: 581.6 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.131 (2016-07-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 4.6.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.706 GiB
Name: crockford
ID: HO2U:ELWR:LDB3:PMEY:5YOJ:D7YJ:2HJA:PVYG:45K2:J6KI:D6WO:4RUE
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
One thing that makes my issue a little different (Where I think the root of the issue comes from)
Before I created a separate partition for /var, it was on my root partition, which eventually maxed out. Once it maxed out, I shrunk my home partition, create a /var partition, copied my root's /var to my new /var, and removed my old /var. But for some reason, docker still think's it's maxed out? I have no idea.
I also tried to resinstall docker with sudo pacman -S docker but nothing.
Edit: I just tried it with a normal docker build . and that works fine. Somehow docker-compose thinks it's out of memory though?

Answer 1

The python stack trace from docker-compose indicates that it can't seem to create a temporary file. This would indicate there's no space left in /tmp.
OP mentioned that his RAM is completely consumed when he runs docker-compose in the comments. Given that and the fact that /tmp is mounted on tmpfs it makes sense that there is no space left for Python/docker-compose to create any temporary files in /tmp.
The possible solutions are:
Temporarily switch the default tempfile generation location by setting one of the following environment variables: TMPDIR, TEMP, TMP (ref: Python doc)
Change /tmp to not use tmpfs and use disk instead.
Increase the amount of RAM/Swap space on your machine. (You can increase swap without messing with your partitions like so). tmpfs is backed by volatile storage, which means both RAM and Swap should theoretically work.
Note, most of these cases will result in a slowdown of your application, especially if the docker build process is I/O heavy.

Answer 2

Try this:
mount -o remount,size=4G,noatime /tmp