What are the options to secure Jenkins slave nodes that have docker daemon? Someone could launch docker container and essentially access files owned by root.
Environment and problem details:
Jenkins
Docker daemon available on Jenkins slaves
Jenkins agent run as user “build”
User build is added to group “docker” so that
the jobs can build and start containers
We are considering to use
docker pipeline https://jenkins.io/doc/book/pipeline/docker/
Problem:
A user can mount the server root as a docker volume
They could execute a command like “rm -rf /hostroot/*” or print the contents of a secret file owned by root.
Couple options:
Option 1: Use the –user argument to run the commands as a specific user, but since the Jenkinsfile is managed by development team this can’t be enforced strictly
Option 2: Enable user namespace on the docker daemon (there may be some permissions issues – especially on the workspace folder but some workarounds can be put in place)
Do you use user namespace?
Option 3: build using AWS Fargate (need to look in to this a bit) – so the docker image is not built on Jenkins slave nodes and the build won’t affect the files on the nodes
Example:
pipeline {
agent {
docker {
image 'maven:3-alpine'
args '-v /:/hostroot'
(args '—user 500 ' --> since Jenkinsfile is managed by development team, ensuring everyone puts this parameter is tricky… someone could forget to put it)
}
}
stages {
stage('Build') {
steps {
sh 'rm -rf /hostroot/*' --> delete the entire filesystem!
sh 'cat /path/to/secret/file' --> see some files owned by root!
}
}
}
}
A similar question was posted here but that is 2 years old and doesn't have a good answer:
Building Docker images with Jenkins that runs inside a Docker container
Related
I've got a python package running in a container.
Is it best practice to install it in /opt/myapp within the container?
Should the logs go in /var/opt/myapp?
Should the config files go in /etc/opt/myapp?
Is anyone recommending writing logs and config files to /opt/myapp/var/log and /opt/myapp/config?
I notice google chrome was installed in /opt/google/chrome on my (host) system, but it didn't place any configs in /etc/opt/...
Is it best practice to install it in /opt/myapp within the container?
I place my apps in my container images in /app. So in the dockerfile I do
WORKDIR /app at the beginning
Should the logs go in /var/opt/myapp?
In container world the best practice is that your application logs go into stdout, stderr and not into files inside the container because containers are ephemeral by design and should be treated that way so when a container is stopped and deleted all of its data on its filesystem is gone.
On local docker development environment you can see the logs with docker logs and you can:
start a container named gettingstarted from the image docker/getting-started:
docker run --name gettingstarted -d -p 80:80 docker/getting-started
redirect docker logs output to a local file on the docker client (your machine from where you run the docker commands):
docker logs -f gettingstarted &> gettingstarted.log &
open http://localhost to generate some logs
read the log file with tail realtime or with any text viewer program:
tail -f gettingstarted.log
Should the config files go in /etc/opt/myapp?
Again, you can put the config files anywhere you want, I like to keep them together with my app so in the /app directory, but you should not modify the config files once the container is running. What you should do is instead pass the config variables to the container as environment variables at startup with the -e flag, for example to create MYVAR variable with MYVALUE value inside the container start it this way:
docker run --name gettingstarted -d -p 80:80 -e MYVAR='MYVALUE' docker/getting-started
exec into the container to see the variable:
docker exec -it gettingstarted sh
/ # echo $MYVAR
MYVALUE
From here it is the responsibility of your containerized app to understand these variables and translate them to actual application configurations. Some/most programming languages support reaching env vars from inside the code at runtime but if this is not an option then you can do an entrypoint.sh script that updates the config files with the values supplied through the env vars. A good example for this is the postgresql entrypoint: https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh
Is anyone recommending writing logs and config files to
/opt/myapp/var/log and /opt/myapp/config?
As you can see, it is not recommended to write logs into the filesystem of the container you would rather have a solution to save them outside of the container if you need them persisted.
If you understand and follow this mindset especially that containers are ephemeral then it will be much easier for you to transition from the local docker development to production ready kubernetes infrastructures.
Docker is Linux, so almost all of your concerns are related to the best operative system in the world: Linux
Installation folder
This will help you:
Where to install programs on Linux?
Where should I put software I compile myself?
and this: Linux File Hierarchy Structure
As a summary, in Linux you could use any folder for your apps, bearing in mind:
Don't use system folders : /bin /usr/bin /boot /proc /lib
Don't use file system folder: /media / mnt
Don't use /tmp folder because it's content is deleted on each restart
As you researched, you could imitate chrome and use /opt
You could create your own folder like /acme if there are several developers entering to the machine, so you could tell them: "No matter the machine or the application, all the custom content of our company will be in /acme". Also this help you if you are a security paranoid because will be able to guess where your application is. Any way, if the devil has access to your machine, is just a matter of time to find all.
You could use fine grained permissions to keep safe the chosen folder
Log Folder
Similar to the previous paragraph:
You could store your logs the standard /var/log/acme.log
Or create your own company standard
/acme/log/api.log
/acme/webs/web1/app.log
Config Folder
This is the key for devops.
In a traditional, ancient and manually deployments, some folders were used to store the apps configurations like:
/etc
$HOME/.acme/settings.json
But in the modern epoch and if you are using Docker, you should not store manually your settings inside of container or in the host. The best way to have just one build and deploy n times (dev, test, staging, uat, prod, etc) is using environment variables.
One build , n deploys and env variables usage are fundamental for devops and cloud applications, Check the famous https://12factor.net/
III. Config: Store config in the environment
V. Build, release, run: Strictly separate build and run stages
And also is a good practice on any language. Check this Heroku: Configuration and Config Vars
So your python app should not read or expect a file in the filesystem to load its configurations. Maybe for dev, but no for test and prod.
Your python should read its configurations from env variables
import os
print(os.environ['DATABASE_PASSWORD'])
And then inject these values at runtime:
docker run -it -p 8080:80 -e DATABASE_PASSWORD=changeme my_python_app
And in your developer localhost,
export DATABASE_PASSWORD=changeme
python myapp.py
Before the run of your application and in the same shell
Config of a lot pf apps
The previous approach is an option for a couple of apps. But if you are driven to microservices and microfrontends, you will have dozens of apps on several languages. So in this case, to centralize the configurations you could use:
spring cloud
zookeeper
https://www.vaultproject.io/
https://www.doppler.com/
Or the Configurator (I'm the author)
I need help in creating the container through Jenkins job.
Let me know the steps to be followed: I have already created 3 jobs in Jenkins, and I want to create httpd container through the jobs created.
Should I install any plugins or write any script ?
Assuling we are not talking about Jenkins in docker, or Jenkins agents in Docker, you need to create your http container manually first, without Jenkins.
That means:
validate your SSH access to the remote server
Check it has Docker installed
execute docker commands to run an http container, s described in Docker httpd
Once that is working, you can replicate the process in a Jenkins Job, provided your remote server (the one with Docker, where you want to run your httpd container) is declared as agent to the main Jenkins controller.
I am following the official tutorial from Jenkins website. I have a blueocean Docker container that is running the pipeline in the Jenkinsfile as per tutorial:
pipeline {
agent {
docker {
image 'node:6-alpine'
args '-p 3000:3000'
}
}
stages {
stage('Build') {
steps {
sh 'npm install'
}
}
}
}
The problem is that the pipeline fails when it tries to pull the Docker image:
[ode-js-react-npm-app_master-6PEWX3VWDA4SAdDMJA4YKJCZSABJSAQCSGVYMKHINXGDDJLA] Running shell script
+ docker pull node:6-alpine
Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
script returned exit code 1
After some troubleshooting, I realized that this is failing because Jenkins is trying to pull the Docker container itself, rather than the host. This is not what I want, and the documentation in fact states:
This image parameter (of the agent section’s docker parameter) downloads the node:6-alpine Docker image (if it’s not already available on your machine) and runs this image as a separate container. This means that:
You’ll have separate Jenkins and Node containers running locally in Docker.
The Node container becomes the agent that Jenkins uses to run your Pipeline project. However, this container is short-lived - its lifespan is only that of the duration of your Pipeline’s execution.
Can someone explain what I am doing wrong and why the Node.js Docker image is tried being pulled inside Jenkins instead of the local machine? I want to have a separate Jenkins container from the Nodejs container that orchestrates the app.
I solved this problem by running the container as root, as otherwise it would not have access to the Docker daemon socket /var/run/docker.sock ...
In my application user uploads his jmeter test plan (*.jmx file) and I need to execute it on my server. I want to verify that the jmx file does not contain any code that can harm my server. Are there any plugins, tools that can help me?
JMeter is very flexible and there is no way to stop the user from doing the harm as for example:
It is possible do delete any file or folder using Beanshell or JavaScript
It is possible to read any file and send it over to anyone via email
It is possible to fork too many processes or kick off too much threads and put your server on its knees by overloading it
So there is no any guaranteed way to verify a JMeter test, the best thing you can do is running it in isolated mode like:
Create a user with a very limited permissions set before executing the test and execute the test as this user
Use container mechanism like:
Windows Containers
Linux Containers
FreeBSD Jails
After looking through solutions like chroot, FreeBSD Jails and dockers, we choosed Dockers. The advantages we found were:
very easy setup and cool documentation
the docker starts in less than a second and there are lots of actions you can do with container - copy file into container, mount directory, run process inside container, etc.
I've created one container with jmeter in it. Every time I want to run some jmeter file I start the container, copy the jmx file into the container and run jmeter inside the container. Note that I call jmeter.sh outside of container and get the jmeter output into console again outside of container. When jmeter process is over, I stop the container.
Some commands I have used:
docker create --name container_name -it my_image_with_jmeter //create container from an image. my_image_with_jmeter is the name of the image I've created
docker start container_name
docker cp /path/to/main/server/file container_name:/path/to/container/ //copy file from main server to container
docker exec -it container_name /usr/local/jmeter/jmeter.sh // run jmeter inside container
docker stop container_name
I'm trying to use a stack built with Docker container to run a Symfony2 application (SfDocker). The stack consists of interlinked containers where ubuntu:14.04 is a base:
mysql db
nginx
php-fpm
The recurring problem that I'm facing is managing directory permission inside the container. When I mount a vloume from the host, e.g.
volumes:
- symfony-code:/var/www/app
The mounted directories will always be owned by root or an unidentified user (only user ID visible when running ls -al) inside the container.
This, essentially, makes it impossible to access the application through the browser. Of course running chown -R root:www-data on public directories solves the problem, but as soon as I want to write to e.g. 'cache' directory as from the host (where the user is ltarasiewicz) I'd get permission denied error. On top of that, whenever an application running inside a container creates new directories (e.h. 'logs'), they again are owned byroot and later inaccessible by the browser or my desktop user.
So my question are:
How I should manage permission accross the host and container
environments (when I want to run commands on the container from both
environments) ?
Is it possible to configure Docker so that directories mounted as volumes receive specific ownership/permissions (e.g. 'root:www-data') automatically?
Am I free to create new users and user groups inside my 'nginx' container built from the Ubuntu:14.04 image ?
A few general points, apologies if I don't answer your questions directly.
Don't run as root in the container. Create a user in the Dockerfile and switch to it, either with the USER statement or in an entrypoint or command script. See the Redis official image for a good example of this. (So the answer to Q3 is yes, and do, but via a Dockerfile - don't make changes to containers by hand).
Note that the official images often do a chown on volumes in the entrypoint script to avoid this issue you describe in 2.
Consider using a data container rather than linking directly to host directories. See the official docs for more information.
Don't run commands from the host on the volumes. Just create a temporary container to do it or use docker exec (e.g. docker run -v /myvol:/myvol myimage touch /myvol/x).