AWS ECS volumes do not share any files - linux

Question

I have an EBS volume I have mounted to an AWS ECS cluster instance. This EBS volume is mounted under /data:
$ cat /etc/fstab
...
UUID=xxx /data ext4 defaults,nofail 0 2
$ ls -la /data
total 28
drwxr-xr-x 4 1000 1000 4096 May 14 06:11 .
dr-xr-xr-x 26 root root 4096 May 15 21:18 ..
drwxr-xr-x 4 root root 4096 May 14 06:11 .ethereum
drwx------ 2 1000 1000 16384 May 14 05:29 lost+found
Edit: Output of /proc/mounts
[ec2-user#xxx ~]$ cat /proc/mounts
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
/dev/xvda1 / ext4 rw,noatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4078988k,nr_inodes=1019747,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/perf_event cgroup rw,relatime,perf_event 0 0
/dev/xvdf /data ext4 rw,relatime,data=ordered 0 0
Now, I would like to mount /data/.ethereum as a Docker volume to /geth/.ethereum in my ECS task definition:
{
...
"containerDefinitions": [
{
...
"volumesFrom": [],
"mountPoints": [
{
"containerPath": "/geth/.ethereum",
"sourceVolume": "ethereum_datadir",
"readOnly": null
}
],
...
}
],
...
"volumes": [
{
"host": {
"sourcePath": "/data/.ethereum"
},
"name": "ethereum_datadir"
}
],
...
}
It appears that the volume is correctly mounted after running the task:
$ docker inspect -f '{{ json .Mounts }}' f5c36d9ea0d6 | python -m json.tool
[
{
"Destination": "/geth/.ethereum",
"Mode": "",
"Propagation": "rprivate",
"RW": true,
"Source": "/data/.ethereum"
}
]
How ever, if I create file inside the container inside the mount point, it will not be there on the host machine.
[ec2-user#xxx .ethereum]$ docker exec -it f5c36d9ea0d6 bash
root#f5c36d9ea0d6:/geth# cat "Hello World!" > /geth/.ethereum/hello_world.txt
cat: Hello World!: No such file or directory
root#f5c36d9ea0d6:/geth# echo "Hello World!" > /geth/.ethereum/hello_world.txt
root#f5c36d9ea0d6:/geth# cat /geth/.ethereum/hello_world.txt
Hello World!
root#f5c36d9ea0d6:/geth# exit
exit
[ec2-user#xxx ~]$ cat /data/.ethereum/hello_world.txt
cat: /data/.ethereum/hello_world.txt: No such file or directory
Somehow the file systems are not being shared. Any ideas?

Answer 1

Found the issue.
It seems like with Docker, any mount point (e.g. for EBS volumes) on the host instance has to be created before the Docker daemon has started, otherwise Docker will write the files into the instance's root file system without you even noticing.
I stopped Docker, unmounted the EBS volume, cleaned everything up, mounted the EBS volume again and started Docker afterwards. Now Docker seems to recognize the mount point and writes everything into my EBS volume as it should.