I decided to increase security by enabling userns-remap option in Docker running in swarm mode.
Installation is not new, there's plenty of running services.
Followed configuration with official manual: https://docs.docker.com/engine/security/userns-remap/
Docker service is starting but docker service ls throws error:
Handler for GET /v1.40/services returned error: This node is not a swarm manager. Use docker swarm init or docker swarm join to connect this node to swarm and try again
Error getting services: This node is not a swarm manager. Use docker swarm init or docker swarm join to connect this node to swarm and try again
cat /etc/docker/daemon.json is simple as
{
"userns-remap": "default"
}
cat /etc/subuid /etc/subgid
dockremap:100000:65536
dockremap:100000:65536
id dockremap
uid=1000(dockremap) gid=1000(dockremap) groups=1000(dockremap)
ls -ld /var/lib/docker/100000.100000/
drwx------ 11 231072 231072 26 Mar 21 20:19 /var/lib/docker/100000.100000/
Removing userns-remap from config brings services back to normal.
Running CentOS 7.7 and docker 19.03.8
How can I make it work?
From https://docs.docker.com/engine/security/userns-remap/:
Enabling userns-remap effectively masks existing image and container
layers, as well as other Docker objects within /var/lib/docker/. This
is because Docker needs to adjust the ownership of these resources and
actually stores them in a subdirectory within /var/lib/docker/. It is
best to enable this feature on a new Docker installation rather than
an existing one.
Ergo - all existing images and containers will not be available after enabling user namespace.
Related
Given that using Docker alongside with LXC containers in the same host can create problems in iptables (if I understood correctly) (source: https://github.com/docker/for-linux/issues/103), the same applies when using Docker INSIDE lxc container ?
In other words, could we create a LXC container on the host A, and inside of that container, install Docker, use it and be not affected by this issue?
Context: Why I am asking this? because I want to create and run a gitlab-runner inside a Docker container (with other things such as Docker, maven, etc) in a Host that has lxc containers already running, I do not want to touch these containers.
Two examples include snap and certbot. I used to type sudo certbot and would be able to add ssl certs to my nginx servers. Now I get this every time I enter certbot. The same thing goes for snap. I'm new to docker and don't understand what is going on. Can somebody explain what is ging on?
Usage: docker compose [OPTIONS] COMMAND
Docker Compose
Options:
--ansi string Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
--compatibility Run compose in backward compatibility mode
--env-file string Specify an alternate environment file.
-f, --file stringArray Compose configuration files
--profile stringArray Specify a profile to enable
--project-directory string Specify an alternate working directory
(default: the path of the, first specified, Compose file)
-p, --project-name string Project name
Commands:
build Build or rebuild services
convert Converts the compose file to platform's canonical format
cp Copy files/folders between a service container and the local filesystem
create Creates containers for a service.
down Stop and remove containers, networks
events Receive real time events from containers.
exec Execute a command in a running container.
images List images used by the created containers
kill Force stop service containers.
logs View output from containers
ls List running compose projects
pause Pause services
port Print the public port for a port binding.
ps List containers
pull Pull service images
push Push service images
restart Restart service containers
rm Removes stopped service containers
run Run a one-off command on a service.
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker Compose version information
Run 'docker compose COMMAND --help' for more information on a command.
NEVER INSTALL DOCKER WITH SNAP
I solved the problems. Not sure where everything went wrong, but I completely destroyed snapd from my system following this https://askubuntu.com/questions/1280707/how-to-uninstall-snap. Then I installed snap again and everything works.
INSTALL DOCKER WITH THE OFFICIAL GUIDE (APT)
Go here to install docker the correct way. https://docs.docker.com/engine/install/ubuntu/
If you are new to docker follow this advice and NEVER TYPE snap install docker into you terminal. Follow these words of wisdom or use the first half if you already messed up.
I have the following 3 nodes..
C:\Users\yadamu>docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
4tpray751pk50bl6o6gtbjfq2 YADAMU-DB5 Ready Active 20.10.8
xzx0gxu1m0qo59z6gtr4j2i1p * yadamu-db3 Ready Active Leader 20.10.8
x1yh3l6m6k73gytkxx3ipimq4 yadamu-db4 Ready Active 20.10.10
On yadamu-db3 (Manager), a Windows 11 box, I created an overlay network with
docker network create --driver overlay --attachable Y_OVERLAY
I then started a set of containers using docker-compose on YADAMU-DB3. They came up as expected and could talk to each other. I then started a second set of containers on YADAMU-DB5, which is also a Windows 11 box using a different docker compose file and they could also talk to each other and to the to containers running on YADAMU-DB5.
I then started a third set of containers using docker compose on YADAMU-DB4, which is running Oracle Enterpise Linux 8. These container can talk to each other but are isolated from the containers running on YADAMU-DB3 and YADAMU-DB5.
All three docker compose files contain the following 'networks' sections
networks:
YADAMU-NET:
name: Y_OVERLAY0
attachable : true
However when I run docker-compose on the linux box I see
C:\Development\YADAMU\docker\dockerfiles\swarm\YADAMU-DB4>docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating network "Y_OVERLAY" with the default driver
Creating volume "YADAMU_01-SHARED" with default driver
and when I list the networks I see
C:\Development\YADAMU\docker\dockerfiles\swarm\YADAMU-DB4>docker network ls
NETWORK ID NAME DRIVER SCOPE
b87206cfac4c Y_OVERLAY bridge local
393beef01a7f bridge bridge local
e19e5f965e8d docker_gwbridge bridge local
1b4cbfa566f0 host host local
y58mwnbratkj ingress overlay swarm
32a41d9b3d7c none null local
Where as when I list the networks on YADAMU-DB5 I see
C:\Users\Mark D Drake>docker network ls
NETWORK ID NAME DRIVER SCOPE
luh9dw47k5a1 Y_OVERLAY overlay swarm
y58mwnbratkj ingress overlay swarm
8fd8ef298f47 nat nat local
dce21ec8e1ae none null local
So it appears the on the LINUX box it has not resolved Y_OVERLAY as the overlay network defined by the swarm manager.
Any ideas what I'm missing here..
Note the intent here is not to build a resiliant swarm it's to build a qa environment that can us used for testing interaction between windows and linux hosts on limited hardware.
I found a workaround.. If I down the containers on the linux node, and then promote it to a 'Manager', and then run docker-compose up on the linux node, the contains join the Overlay network...
It's doesn't explain why this did not work with the linux node as a worker, but it means I can move ahead.
I have an Ubuntu 16.04 (Xenial) running inside an Azure VM. I have followed the instructions to install Docker and all seems fine and dandy.
One of the things that I need to do when I trigger docker run is to pass --net=host, which allows me to run apt-get update and other internet-dependent commands within the container.
The problem comes in when I try to trigger docker build based on an existing Ubuntu image. It fails:
The problem here is that there is no way to pass --net=host to the build command. I see that there are issues open on the Docker GitHub (#20987, #10324) but no clear resolution.
There is an existing answer on Stack Overflow that covers the scenario I want, but that doesn't work within a cloud VM.
Any thoughts on what might be happening?
UPDATE 1:
Here is the docker version output:
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:11:10 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:11:10 2016
OS/Arch: linux/amd64
UPDATE 2:
Here is the output from docker network ls:
NETWORK ID NAME DRIVER SCOPE
aa69fa066700 bridge bridge local
1bd082a62ab3 host host local
629eacc3b77e none null local
Another approach would be to try letting docker-machine provision the VM for you and see if that works. There is a provider for Azure, so you should be able to set your subscription id on a local Docker client (Windows or Linux) and follow the instructions to get a new VM provisioned with Docker and it will also setup your local environment variables to communicate with the Docker VM instance remotely. After it is setup running docker ps or docker run locally would run the commands as if you were running them on the VM. Example:
#Name at end should be all lower case or it will fail.
docker-machine create --driver azure --azure-subscription-id <omitted> --azure-image canonical:ubuntuserver:16.04.0-LTS:16.04.201608150 --azure-size Standard_A0 azureubuntu
#Partial output, see docker-machine resource group in Azure portal
Running pre-create checks...
(azureubuntu) Completed machine pre-create checks.
Creating machine...
(azureubuntu) Querying existing resource group. name="docker-machine"
(azureubuntu) Resource group "docker-machine" already exists.
(azureubuntu) Configuring availability set. name="docker-machine"
(azureubuntu) Configuring network security group. location="westus" name="azureubuntu-firewall"
(azureubuntu) Querying if virtual network already exists. name="docker-machine-vnet" location="westus"
(azureubuntu) Configuring subnet. vnet="docker-machine-vnet" cidr="192.168.0.0/16" name="docker-machine"
(azureubuntu) Creating public IP address. name="azureubuntu-ip" static=false
(azureubuntu) Creating network interface. name="azureubuntu-nic"
(azureubuntu) Creating virtual machine. osImage="canonical:ubuntuserver:16.04.0-LTS:16.04.201608150" name="azureubuntu" location="westus" size="Standard_A0" username="docker-user"
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env azureubuntu
#Set environment using PowerShell (or login to the new VM) and see containers on remote host
docker-machine env azureubuntu | Invoke-Expression
docker info
docker network inspect bridge
#Build a local docker project using the remote VM
docker build MyProject
docker images
#To clean up the Azure resources for a machine (you can create multiple, also check docker-machine resource group in Azure portal)
docker-machine rm azureubuntu
Best I can tell that is working fine. I was able to build a debian:wheezy DockerFile that uses apt-get on the Azure VM without any issues. This should allow the containers to run using the default bridged network as well instead of the host network.
According to I can't get Docker containers to access the internet? using sudo systemctl restart docker might help, or enable net.ipv4.ip_forward = 1 or disable the firewall.
Also you may need to update the dns servers in /etc/resolv.conf on the VM
I just created a secure Docker Registry and ran it on a remote VM (using docker run ...). I then ran docker ps and saw that it is in fact running. I exited the machine and then SSHed back in. Again, I ran docker ps and verified it "survived" me exiting the SSH session.
This has me wondering: do Docker containers actually run as Linux services? If not, is there any way of getting them to run as traditional (upstart- or systemd-based) services? Is there even any reason/merit to do so?
The docker engine runs as a daemon.
That is mentioned in "Host integration":
As of Docker 1.2, restart policies are the built-in Docker mechanism for restarting containers when they exit. If set, restart policies will be used when the Docker daemon starts up, as typically happens after a system boot. Restart policies will ensure that linked containers are started in the correct order.
If restart policies don’t suit your needs (i.e., you have non-Docker processes that depend on Docker containers), you can use a process manager like upstart, systemd or supervisor instead.
That involves (when a container runs with some options) some security fraught, by the way: see issue 14767 and issue 6401:
The container (with --net host option) is the host when it comes to the network stack so any services running on the host are accessible to the container. It just so happens that you communicate to upstart ( and others ) this way.
This feature is a runtime only option, just like the --privileged flag, therefore an image cannot request this, it must be explicitly set at runtime.