I've played around with Panamax as a solution for managing groups of containers on a single server CoreOS installation, but it lacks several features, notably fleet management and user based access restriction.
Finally, the project does not seem to be maintained anymore.
Are there any active and production ready alternatives that make the management of multiple CoreOS servers possible via a UI (Web or desktop)?
Have not yet tried this yet, but Mist.io seems like a very promising option. Comes in both an open source and SaaS version. Open Source version is being actively maintained and their service is production ready for an impressive list of cloud providers. The UI gives options for managing and monitoring CoreOS clusters at both the OS and container levels: you can spin up new hosts and new containers from the same UI. Might be what you are looking for.
The container manager for fleet is fleet-browser:
https://github.com/cloudwalkio/fleet-browser
Here is the unit file I use to deploy it, just change the lines where I have put "YOURBLAH", etc.
[Unit]
Description=Expose Fleet API in a nice GUI
Requires=docker.service
After=docker.service
[Service]
EnvironmentFile=/etc/environment
KillMode=none
TimeoutStartSec=0
Restart=always
RestartSec=10s
ExecStartPre=-/usr/bin/docker kill fleet-browser
ExecStartPre=-/usr/bin/docker rm fleet-browser
ExecStartPre=/usr/bin/docker pull cloudwalk/fleet-browser:latest
ExecStart=/usr/bin/docker run --net=host --rm --name fleet-browser \
-e FLEET_ENDPOINT=YOURENDPOINT:8081 \
-e ETCD_ENDPOINT=YOURENDPOINT:2379 \
-p 5000:5000 cloudwalk/fleet-browser:latest
ExecStop=/usr/bin/docker stop fleet-browser
[X-Fleet]
MachineMetadata="server-type=YOURSERVERTYPE" "node=YOURNODENAME"
Related
All my unix host use the ldap backend.
docker group is existing on the ldap, this is also why docker.service must start after nslcd.service.
I have tried to edit systemctl startup configuration for docker.service:
$ sudo systemctl edit --full docker.service
And I add nslcd.service to After, Wants, Requires:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service nslcd.service
Wants=network-online.target nslcd.service
Requires=docker.socket nslcd.service
I still can't get docker to run after that service:
sudo service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: https://docs.docker.com
Oct 10 19:35:02 dev-08 systemd[1]: Dependency failed for Docker Application Container Engine.
There is no problem to start container manually after starts, since I login through ldap.
docker group is existing on the ldap, this is also why docker.service must start after nslcd.service.
It is generally a bad idea to have system services depend on users and groups in a remote directory service (because problems with the directory service can impact service availability on your host).
And I add nslcd.service to After, Wants, Requires
Specifying both a Wants= and a Requires= relationship is redundant. The Requires= relationship is simply a stronger version of Wants=: using Requires= means that if you start the docker service, and nslcd is not already running, it will be started as well. Using Wants= in the same situation, docker would start without starting nslcd.
I still can't get docker to run after that service
Is is entirely likely that nslcd takes a moment to connect to the directory service. In this case, it's possible that the process has started, which satisfies the After= dependency, so docker starts even though your groups are not yet available.
There are a few ways of addressing this situation:
In light of my initial comment, just create a local docker group. This is by far the simplest and most robust solution.
Create a new oneshot unit that spins until the docker group exists. Make this unit depend on nslcd, and make docker depend on the new unit.
Possibly replacing nslcd with something that implements local caching (e.g., sssd) would also resolve this problem.
On a different note, it is a bad idea to directly edit the unit files as you have done in this example, because if you've installed Docker via packaging tools (apt/yum/etc), your modifications will be overwritten next time you upgrade the package. A better solution is to create drop-in files to augment the unit configuration.
Update
Option 2 might look like this:
[Unit]
Requires=nslcd.service docker.service
After=nslcd.service
Before=docker.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c "while ! getent group docker; do sleep 1; done"
I just created a secure Docker Registry and ran it on a remote VM (using docker run ...). I then ran docker ps and saw that it is in fact running. I exited the machine and then SSHed back in. Again, I ran docker ps and verified it "survived" me exiting the SSH session.
This has me wondering: do Docker containers actually run as Linux services? If not, is there any way of getting them to run as traditional (upstart- or systemd-based) services? Is there even any reason/merit to do so?
The docker engine runs as a daemon.
That is mentioned in "Host integration":
As of Docker 1.2, restart policies are the built-in Docker mechanism for restarting containers when they exit. If set, restart policies will be used when the Docker daemon starts up, as typically happens after a system boot. Restart policies will ensure that linked containers are started in the correct order.
If restart policies don’t suit your needs (i.e., you have non-Docker processes that depend on Docker containers), you can use a process manager like upstart, systemd or supervisor instead.
That involves (when a container runs with some options) some security fraught, by the way: see issue 14767 and issue 6401:
The container (with --net host option) is the host when it comes to the network stack so any services running on the host are accessible to the container. It just so happens that you communicate to upstart ( and others ) this way.
This feature is a runtime only option, just like the --privileged flag, therefore an image cannot request this, it must be explicitly set at runtime.
this is a rather conceptual question. I am running three node.js webservers as Docker containers behind a HaProxy instance, also in a docker container. The containers get started by docker-compose, so everything pretty standard.
My problem: HaProxy does health checks to see if one of my node.js containers dies to redirect traffic, so far so good. But I cannot find a good solution on how to restart dead containers automatically.
Are there any good practices for this?
You could use --retry=always policy when you run the container so that upon exit it will be automatically restarted by docker daemon.
Take a look at documentation for more details on restart policies.
You can start special container willfarrell/autoheal that monitors and restarts unhealthy containers labeled with autoheal label on the host.
docker run -d \
--name autoheal \
--restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
willfarrell/autoheal
See https://github.com/willfarrell/docker-autoheal for details.
May be you can try to setup sensu for configure same helth checks and to restart unhelth containers.
While running the docker images just mention
restart: always
Option in your docker-compose.yaml file
I have a test cluster of 4 CoreOS machines. I want to have shared storage between them for example to put my docker images there and not have to pull them to each machine.
It seems however that CoreOS does not support NFS. What are my options for creating shared storage on CoreOS?
CoreOS does in fact support NFS-- We configured an NFS mount for EFS shared storage in AWS.
This is an example cloud-config to mount AWS EFS on /mnt:
#cloud-config
write_files:
- path: /etc/conf.d/nfs
permissions: '0644'
content: |
OPTS_RPC_MOUNTD=""
coreos:
units:
- name: rpc-statd.service
command: start
enable: true
- name: mnt.mount
content: |
[Mount]
What=AZ_ZONE.fs-xxxxxxxx.efs.us-west-2.amazonaws.com:/
Where=/mnt
Type=nfs
- name: runcmd.service
command: start
content: |
[Unit]
Description=command
[Service]
Type=oneshot
ExecStart=/bin/sh -c "AZ_ZONE=$(curl -L http://169.254.169.254/latest/meta-data/placement/availability-zone); sed -i \"s/AZ_ZONE/$AZ_ZONE/\" /etc/systemd/system/mnt.mount; systemctl daemon-reload; systemctl restart mnt.mount"
update:
group: stable
reboot-strategy: off
Replacing xxxx with the unique alphanumeric ID of your EFS share.
Before EFS was available we used BitTorrent Sync, alternatively.
Quite curious though why you would want to share your image layers. The layers themselves are a deployment strength of Docker. If you have enough in common, very little gets re-pulled between applications. Say two different applications share the Ubuntu:latest base which is the largest layer. You won't have to re-pull Ubuntu when you spin up the second application on that host. You can't use any old storage backend either-- CoreOS uses OverlayFS which might be interesting to read up on.
I have recently solved this exact problem using Deis. Amongst other useful functionality, it sets up a Ceph storage cluster and private docker registry right out of the box. The storage volume is fault-tolerant and spans all machines in the cluster.
I Use EFS(Amazon Elastic File System) for shared Storage in CoreOS cluster.
Another option will be Flocker from ClusterHQ. But its not yet available in CoreOS
You could have your own docker registry server that would start only on the master machine and pull images from s3. I haven't tried this one, but it looks good. The service files below are to work with the official registry repo. This is not a full solution, but should get you into a direction you are looking for, I think.
https://registry.hub.docker.com/u/blalor/docker-s3-registry/
You would have a service file for docker registry and a service discovery file to notify your cluster where the registry lives. An alternate would be to use CoreOS private registry called http://quay.io
registry#.service file:
[Unit]
Description=Docker Registry
After=docker.service
Requires=docker.service
[Service]
Restart=always
RestartSec=10s
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill registry
ExecStartPre=-/usr/bin/docker rm registry
ExecStartPre=/usr/bin/docker pull registry
ExecStart=/usr/bin/docker run -p 5000:5000 --name registry -e SETTINGS_FLAVOR=s3 -e AWS_BUCKET=my-registry-bucket -e STORAGE_PATH=/registry -e AWS_KEY=... -e AWS_SECRET="..." registry
ExecStartPost=/usr/bin/etcdctl set /domains/domain.com/registry `ifconfig|grep 'broadcast 10'|awk '{print $2}'`
ExecStop=/usr/bin/docker stop registry
[Install]
WantedBy=multi-user.target
[X-Fleet]
Global=true
Conflicts=registry#*.service
registry-discovery#.service file:
[Unit]
Description=Announce registry
BindsTo=registry#*.service
[Service]
ExecStart=/bin/sh -c "while true; do etcdctl set /services/registry/%H:%i '{ \"host\": \"%H\", \"port\": \"%i\", \"version\": \"52c7248a14\" }' --ttl 60;sleep 45;done"
ExecStop=/usr/bin/etcdctl rm /services/registry/%H:%i
[X-Fleet]
X-ConditionMachineOf=registry#*.service
Is there a brief guide to explain how to start up a application when the instance starts up and running? If it were one of the services installed through yum then I guess I can use /sbin/chkconfig to add it to the service. (To make it sure, is it correct?)
However, I just want to run the program which was not installed through yum. To run node.js program, I will have to run script sudo node app.js at home directory whenever the system boots up.
I am not used to Amazon Linux AMI so I am having little trouble finding a 'right' way to run some script automatically on every boot.
Is there an elegant way to do this?
One way is to create an upstart job. That way your app will start once Linux loads, will restart automatically if it crashes, and you can start / stop / restart it by sudo start yourapp / sudo stop yourapp / sudo restart yourapp.
Here are beginning steps:
1) Install upstart utility (may be pre-installed if you use a standard Amazon Linux AMI):
sudo yum install upstart
For Ubuntu:
sudo apt-get install upstart
2) Create upstart script for your node app:
in /etc/init add file yourappname.conf with the following lines of code:
#!upstart
description "your app name"
start on started mountall
stop on shutdown
# Automatically Respawn:
respawn
respawn limit 99 5
env NODE_ENV=development
# Warning: this runs node as root user, which is a security risk
# in many scenarios, but upstart-ing a process as a non-root user
# is outside the scope of this question
exec node /path_to_your_app/app.js >> /var/log/yourappname.log 2>&1
3) start your app by sudo start yourappname
You can use forever-service for provisioning node script as a service and automatically starting during boots. Following commands will do the needful,
npm install -g forever-service
forever-service install test
This will provision app.js in the current directory as a service via forever. The service will automatically restart every time system is restarted. Also when stopped it will attempt a graceful stop. This script provisions the logrotate script as well.
Github url: https://github.com/zapty/forever-service
As of now forever-service supports Amazon Linux, CentOS, Redhat support for other Linux distro, Mac and Windows are in works..
NOTE: I am the author of forever-service.
Quick solution for you would be to start your app from /etc/rc.local ; just add your command there.
But if you want to go the elegant way, you'll have to package your application in a rpm file,
have a startup script that goes in /etc/rc.d so that you can use chkconfig on your app, then install the rpm on your instance.
Maybe this or this help. (or just google for "creating rpm packages")
My Amazon Linux instance runs on Ubuntu, and I used systemd to set it up.
First you need to create a <servicename>.service file. (in my case cloudyleela.service)
sudo nano /lib/systemd/system/cloudyleela.service
Type the following in this file:
[Unit]
Description=cloudy leela
Documentation=http://documentation.domain.com
After=network.target
[Service]
Type=simple
TimeoutSec=0
User=ubuntu
ExecStart=/usr/bin/node /home/ubuntu/server.js
Restart=on-failure
[Install]
WantedBy=multi-user.target
In this application the node application is started. You will need a full path here. I configured that the application should simply restart if something goes wrong. The instances that Amazon uses have no passwords for their users by default.
Reload the file from disk, and then you can start your service. You need to enable it to make it active as a service, which automatically launches at startup.
ubuntu#ip-172-31-21-195:~$ sudo systemctl daemon-reload
ubuntu#ip-172-31-21-195:~$ sudo systemctl start cloudyleela
ubuntu#ip-172-31-21-195:~$ sudo systemctl enable cloudyleela
Created symlink /etc/systemd/system/multi-user.target.wants/cloudyleela.service → /lib/systemd/system/cloudyleela.service.
ubuntu#ip-172-31-21-195:~$
A great systemd for node.js tutorial is available here.
If you run a webserver:
You probably will have some issues running your webserver on port 80. And the easiest solution, is actually to run your webserver on a different port (e.g. 4200) and then to redirect that port to port 80. You can accomplish this with the following command:
sudo iptables -t nat -A PREROUTING -i -p tcp --dport 80 -j REDIRECT --to-port 4200
Unfortunately, this is not persistent, so you have to repeat it whenever your server restarts. A better approach is to also include this command in our service script:
ExecStartPre to add the port forwarding
ExecStopPost to remove the port forwarding
PermissionStartOnly to do this with sudo power
So, something like this:
[Service]
...
PermissionsStartOnly=true
ExecStartPre=/sbin/iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4200
ExecStopPost=/sbin/iptables -t nat -D PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4200
Don't forget to reload and restart your service:
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl daemon-reload
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl stop cloudyleela
[ec2-user#ip-172-31-39-212 system]$ sudo systemctl start cloudyleela
[ec2-user#ip-172-31-39-212 system]$
For microservices (update on Dec 2020)
The previously mentioned solution gives a lot of flexibility, but it does take some time to set it up. And for each additional application, you need to go through this entire process again. By the time you'll be installing your 5th node application, you'll certainly start wondering: "there has to be a shortcut".
The advantage of PM2 is that it's just 1 service to install. Next it's PM2 which manages the actual applications.
Even the initial setup of PM2 is easy, because it automatically installs the pm2 service for you.
npm install pm2 -g
And adding new services is even easier:
pm2 start index.js --name "foo"`.
When everything's up and running, you can save your setup, to have it automatically start on reboot.
pm2 save
If you want an overview of all your running node applications,
you can run pm2 list
And PM2 also offers an online (webbased) dashboard to monitor your application remotely. You may need a license to access some of the dashboard functionality though (which is a bit over-priced imho).
You can create a script that can start and stop your app and place it in /etc/init.d; make the script adhere to chkconfig's conventions (below), and then use chkconfig to set it to start when other services are started.
You can pick an existing script from /etc/init.d to use as an example; this article describes the requirements, which are basically:
An executable script that identifies the shell needed (i.e., #!/bin/bash)
A comment of the form # chkconfig: where is often 345, startprio indicates where in the order of services to start, and stopprio is where in the order of services to stop. I generally pick a similar service that already exists and use that as a guide for these values (i.e., if you have a web-related service, start at the same levels as httpd, with similar start and stop priorities).
Once your script is set up, you can use
chkconfig --add yourscript
chkconfig yourscript on
and you should be good to go. (Some distros may require you to manually symlink to the script to /etc/init.d/rc.d, but I believe your AWS distro will do that for you when you enable the script.
Use Elastic Beanstalk :) Provides support for auto-scaling, SSL termination, blue/green deployments, etc
If you want the salty sysadmin way for a RedHat based linux distro (Amazon Linux is a flavor of RedHat), learn systemd, as mentioned by #bvdb in the answer above:
https://en.wikipedia.org/wiki/Systemd
Set everything up as described on an EC2 instance, snapshot a custom AMI, and use this custom AMI as your base for EC2 instances hosting your apps. This way you don't have to go through all that setup multiple times. You'll probably want to get acquainted with load balancers too, if you are running in a production environment with uptime requirements.
Or, yes, as mentioned by #bvdb, you could also use pm2 to interface with systemd. Though I don't think pm2 helps with running your app across multiple EC2 instances, which is definitely recommended for production environments with uptime requirements.
All of which is a very steep learning curve. Since the OP seemed to be new to all this, Elastic Beanstalk, Google App Engine, and others are a great way to get code running in the cloud without all that.
These days I dev in TypeScript, deploying to serverless function execution in the cloud for most things, and don't have to think about package installs or app startup at all.
You can use screen. Run crontab -e and add this line:
#reboot screen -d -m bash -c "cd /home/user/yourapp/; node app"
Have been using forever on AWS and it does a good job. Install using
[sudo] npm install forever -g
To add an application use
forever start path_to_application
and to stop the application use
forever stop path_to_application
This is a useful article that helped me with setting it up.