Shared storage with CoreOS - coreos

Question

I have a test cluster of 4 CoreOS machines. I want to have shared storage between them for example to put my docker images there and not have to pull them to each machine.
It seems however that CoreOS does not support NFS. What are my options for creating shared storage on CoreOS?

Answer 1

CoreOS does in fact support NFS-- We configured an NFS mount for EFS shared storage in AWS.
This is an example cloud-config to mount AWS EFS on /mnt:
#cloud-config
write_files:
- path: /etc/conf.d/nfs
permissions: '0644'
content: |
OPTS_RPC_MOUNTD=""
coreos:
units:
- name: rpc-statd.service
command: start
enable: true
- name: mnt.mount
content: |
[Mount]
What=AZ_ZONE.fs-xxxxxxxx.efs.us-west-2.amazonaws.com:/
Where=/mnt
Type=nfs
- name: runcmd.service
command: start
content: |
[Unit]
Description=command
[Service]
Type=oneshot
ExecStart=/bin/sh -c "AZ_ZONE=$(curl -L http://169.254.169.254/latest/meta-data/placement/availability-zone); sed -i \"s/AZ_ZONE/$AZ_ZONE/\" /etc/systemd/system/mnt.mount; systemctl daemon-reload; systemctl restart mnt.mount"
update:
group: stable
reboot-strategy: off
Replacing xxxx with the unique alphanumeric ID of your EFS share.
Before EFS was available we used BitTorrent Sync, alternatively.
Quite curious though why you would want to share your image layers. The layers themselves are a deployment strength of Docker. If you have enough in common, very little gets re-pulled between applications. Say two different applications share the Ubuntu:latest base which is the largest layer. You won't have to re-pull Ubuntu when you spin up the second application on that host. You can't use any old storage backend either-- CoreOS uses OverlayFS which might be interesting to read up on.

Answer 2

I have recently solved this exact problem using Deis. Amongst other useful functionality, it sets up a Ceph storage cluster and private docker registry right out of the box. The storage volume is fault-tolerant and spans all machines in the cluster.

Answer 3

I Use EFS(Amazon Elastic File System) for shared Storage in CoreOS cluster.
Another option will be Flocker from ClusterHQ. But its not yet available in CoreOS

Answer 4

You could have your own docker registry server that would start only on the master machine and pull images from s3. I haven't tried this one, but it looks good. The service files below are to work with the official registry repo. This is not a full solution, but should get you into a direction you are looking for, I think.
https://registry.hub.docker.com/u/blalor/docker-s3-registry/
You would have a service file for docker registry and a service discovery file to notify your cluster where the registry lives. An alternate would be to use CoreOS private registry called http://quay.io
registry#.service file:
[Unit]
Description=Docker Registry
After=docker.service
Requires=docker.service
[Service]
Restart=always
RestartSec=10s
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill registry
ExecStartPre=-/usr/bin/docker rm registry
ExecStartPre=/usr/bin/docker pull registry
ExecStart=/usr/bin/docker run -p 5000:5000 --name registry -e SETTINGS_FLAVOR=s3 -e AWS_BUCKET=my-registry-bucket -e STORAGE_PATH=/registry -e AWS_KEY=... -e AWS_SECRET="..." registry
ExecStartPost=/usr/bin/etcdctl set /domains/domain.com/registry `ifconfig|grep 'broadcast 10'|awk '{print $2}'`
ExecStop=/usr/bin/docker stop registry
[Install]
WantedBy=multi-user.target
[X-Fleet]
Global=true
Conflicts=registry#*.service
registry-discovery#.service file:
[Unit]
Description=Announce registry
BindsTo=registry#*.service
[Service]
ExecStart=/bin/sh -c "while true; do etcdctl set /services/registry/%H:%i '{ \"host\": \"%H\", \"port\": \"%i\", \"version\": \"52c7248a14\" }' --ttl 60;sleep 45;done"
ExecStop=/usr/bin/etcdctl rm /services/registry/%H:%i
[X-Fleet]
X-ConditionMachineOf=registry#*.service