I am trying to run a Jenkins pipeline to run my testcase. My testcase uses testcontainer frameworks. Jenkins has been installed over Azure Kubernetes Service (AKS).
When I try to execute the pipeline, Azure doesnt allow modifiy Docker socket, and this command doesnt work:
-v /var/run/docker.sock:/var/run/docker.sock
Reading AKS guidelines, I found next comment in Container limitations:
You can no longer access the docker engine, /var/run/docker.sock, or use Docker-in-Docker (DinD).
https://learn.microsoft.com/en-us/azure/aks/cluster-configuration#containerd-limitationsdifferences
How can I map Docker socket using AKS in order to execute docker images // testcontainer? Its possible to use testcontainers over AKS or I would need to change my cloud?
According to the Testcontainers documentation:
Testcontainers requires a Docker-API compatible container runtime.
This isn't available in Kubernetes, as you note, and it means you can't use Testcontainers in a Kubernetes-hosted CI system.
You might consider breaking up your test suite into a set of pure unit tests and separate integration tests. The unit tests don't depend on containers or any other external resources, possibly using stub implementations of interfaces or mocks of external API calls; they necessarily only simulate reality, but can run anywhere your build system runs. The integration tests would depend on a fully-deployed application and make API calls to your container's endpoints. This split would let you avoid needing to run external dependencies in the unit-test phase, and thereby avoid Testcontainers.
Related
We have created our own self hosted agents on an AKS cluster.
We have 3 replicas, using a StatefulSet which mounts a Persistent Volume (PV) in order to use docker's cache. This means that /var/lib/docker is a mount on each replica.
Up until everything works fine.
However, it seems like that agent-0 does most of the work while agent-1 and agent-2 are working way less. This causes a problem, as PV-0 is almost using it's entire capacity and PV-1 and PV-2 aren't:
agent-0
/dev/sdc 98.3G 83.8G 14.5G 85% /var/lib/docker
agent-1
/dev/sdd 98.3G 35.3G 63.0G 36% /var/lib/docker
agent-2
/dev/sdc 98.3G 2.1G 96.2G 2% /var/lib/docker
Is there a way to control Azure DevOps' pipelines so that the load will be spread between the agents instead of always using agent-0?
We have ended up with a different solution, moving from Docker in docker (dind) to Docker outside of docker (DooD).
The main reason that we initially wanted to use our own self hosted agents was the need for docker layer caching, as it takes too much time to build docker images using the hosted agents.
At some point, we figured that we would have to support ARM based images, and start to use docker buildx which made us use the docker dind solution.
We are aware of the risk that is included by mounting the docker.sock:
In order to use Docker from within a Docker container, you need to bind-mount the Docker socket. This has very serious security implications - namely, code inside the container can now run as root on your Docker host. If you're sure you want to do this, see the bind mount documentation on Docker.com.
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#using-docker-within-a-docker-container
But for the time being, this solves our issue as we won't have to manage PVs per pods, but rather use the same docker to build all of our non-ARM images.
A side note: as for ARM, we would still be using a different agent, which will be running docker buildx in order to build docker images.
What you could try is to configure multi-agent for your agent job and type 3 for your 3 agents, if you want to spread the load:
I also think that it will only consume the amount of agents corresponding to your parallel job count for your organization. As stated here:
Each agent can run only one job at a time. To run multiple jobs in
parallel you must configure multiple agents. You also need sufficient
parallel jobs.
EDIT
The above suggested method works only with tasks who support slicing. Otherwise it will just parallel the same execution. In case your tasks dont support slicing, but you have independent Tasks in your pipeline, you can split the Agent Job in 2 or more, without setting the Dependency on previous job, and the jobs will run in parallel if you have sufficient parallel jobs.
Is there a way to control Azure DevOps' pipelines so that the load will be spread between the agents instead of always using agent-0?
I am afraid there is no such out of box way to control the build/release workloads to be evenly distributed to all agents, if we could not use the feature multi-agent.
As workaround, we could clone the our pipeline/job into three, each with different demands. Then set custom capabilities defined for agents:
Check this thread for some more details.
Hope this helps.
I currently use Ansible to manage and deploy a fleet of servers.
I wish to start using Docker for some applications and would like to build a Docker image using the same scripts we use to configure on non Dockerized hosts.
For example we have an Ansible role that builds Nginx with 3rd party modules, would like to use the same role to build a Docker image with the custom Nginx.
Any ideas how I would get this done?
There is the "Ansible Container" project, https://www.ansible.com/integrations/containers/ansible-container. That page points also to the github repo.
It is not clear how well maintained it is, but their reasoning and approach makes sense.
Consider that you might have some adjustments to do regarding two aspects:
a container should do only one thing (microservice)
how to pass configuration to the container at runtime (Docker has some guidelines, such as environmental variables if possible or mounting a volume with the configuration files)
That's a perfect example where the docker-systemctl-replacement script should be used.
It has been developed to allow ansible scripts to target both virtual machines and docker containers. It had been developed when distros did switch to systemd which was hard to enable for containers. When overwriting /usr/bin/systemctl then the docker container will then look good enough for ansible that all the old scripts will continue to run, installing rpm/deb, and having 'service:'s started and enabled.
I have a simple question hopefully; I'm new to Docker and Linux.
Most of the articles/stackoverflow posts suggest installing cron INSIDE the docker container to get it to work as can be seen at this link
However, based on the picture attached below, we can see that the Docker Engine is an abstraction layer between the HOST OS's system and utility libraries and the application container.
Why are we not REUSING the system cron that comes with the HOST instead of installing cron INSIDE the container? It almost feels redundant.
My understanding of docker is you'd install application level libraries and packages like npm node modules INSIDE your nodejs app container for example but if you need a system utility like cron, then you would somehow call back out to the HOST OS's native cron utility; so why not use the HOST's cron within our container somehow, why reinstall cron inside the container?
Lastly, would you use docker-compose instead and separate out the cron service into its own container then somehow have the cron service talk to the application container and referencing its environment vars?
I mean the environment variables defined in the app container; making those accessible to the cron container?
So that we may follow best practice of one service per container?
Use cron on the host machine if you want, eg
0 0 * * * /usr/local/bin/docker run image
As far as I'm aware, modern container applications use some form of scheduling from the host (eco)system. You could use cron on the host machine to trigger docker run commands. You could use a general purpose scheduler like Airflow. You could use a full-fletched container platform like DC/OS, which come with built in scheduling services.
There is nothing wrong with running cron together with your application inside a container per se. However, if you trigger the application container from a scheduling service outside your application container, your container would terminate after the job is finished, thus releasing any resources to other applications.
Secondly, it's considered good practice to have one container per service. Cron is a service in itself.
I have an application with nodejs and R code. The last one runs in a Docker container.
I am planning some end-2-end tests, where I need to have the docker containers running. The service inside the container is stateful, so I would need to restart it for each test (for instance in beforeEach).
I would like to know what is the common way of doing this. I was thinking of executing an external command from the code in nodejs. Somethink like exec(docker run ...), but I don't know whether it is correct and elegant.
Any help is welcome
Docker deamon exposes RESTFul apis that you might want to take a look at. The Docker Engine API api is documented and versioned.
It might be much cleaner to interact with this api rather than forking docker commands.
The company where I work (strictly regulated/audited environment) is yet to embrace containers but would like to adopt them for some applications. There is the view that as the image build process issues commands as root (or could be overridden by the user by use of the USER command), that building (not running) a container is effectively giving a user unfettered access as root during the build process. This is anathema to them and goes against all manner of company policies. Access to certain commands for computers is restricted via PowerBroker, i.e. access to certain commands requires explicit permissioning and is logged/subject to audit.
We need to allow container images to be built by a CI/CD system as well as ideally to allow developers to be able to build containers locally. Containers will generally be run in Kubernetes, but may be run directly on a VM. I'd like to be able to have CI build agents spin up on demand, as there are a lot of developers, so I want to run the build process within Kubernetes.
What is the best practice for building docker containers in this sort of environment please? Should we look to restrict access to commands within the Dockerfile?
My current thinking for this approach:
CI/CD:
Define "company-approved" image to act as build agent within
Kubernetes.
Build image defines a user that the build process runs as (not
root).
Build agent image contains PowerBroker, enabling locking down access
to sensitive commands.
Scan docker file for use of user command and forbid this.
Build agent runs docker-in-docker, as per here
(https://applatix.com/case-docker-docker-kubernetes-part-2/). This
achieves isolation between multiple build instances whilst ensuring
all containers are controlled via Kubernetes.
Images are scanned for security compliance via OpenSCAP or similar.
Passing the scan is part of the build process. Passing the scan
allows the image to be tagged as compliant and pushed to a registry.
I'm uncomfortable with the thinking around (4), as this seems a bit rule bound (i.e. it's a sort of blacklist approach) and I'm sure there must be a better way.
Developer's localhost:
Define "company-approved" base images (tagged as such inside a
trusted registry).
Image defines a user that the build process runs
as (not root).
Base image contains PowerBroker, enabling locking
down access to sensitive commands.
Create wrapper script on localhost that wraps docker build. No direct access to docker build: user must use script instead. Access to script is secured via PowerBroker. Script can also scan docker file for use of user command and forbid this.
Pushing of images to registry requires tagging which requires scanning for security compliance via OpenSCAP or similar as above.
I'd like to use the OpenSCAP results plus the CI system to create an audit trail of the images that exist; similarly for the deploy process. The security team that monitor for CVEs etc should be able to understand what containers exist and have been deployed and be able to trigger rebuilds of images to make use of updated libraries, or to flag up to developers when containers need to be rebuilt/redeployed. I want to be able to demonstrate that all containers meet a security configuration policy that is itself defined as code.
Is this a sensible way to go? Is there even a risk for allowing a user to build (but not run) a container image without restriction? If there is not, what's the best way to ensure that a foolish/malicious developer has not undone the best practices inside the "approved base image", other than a manual code review (which is going to be done anyway, but might miss something)?
By the way, you must assume that all code/images are hosted in-house/on-premises, i.e. nothing is allowed to use a cloud-based product/service.
When docker build runs each layer executes in the context of a container. So the risks presented by that command executing are constrained by what access is available to the container.
Locking down the build environment could be achieved by restricting what the Docker engine instance which will complete the build can do.
Things like ensuring that user namespaces are used can reduce the risk of a command run inside a container having a wider effect on the environment.
Of course that doesn't mitigate the risks of a developer curl|bashing from an untrusted location, but then what's to stop that being done outside of Docker? (i.e. what additional risk is being introduced by the use of Docker in this scenario)
If you have a policy of restricting externally hosted code, for example, then one option could be to just restrict access from the Docker build host to the Internet.
If you're making use of Kubernetes for the build process and are concerned about malicious software being executed in containers, it could be worth reviewing the CIS Kubernetes standard and making sure you've locked down your clusters appropriately.
There is the view that as the image build process issues commands as
root (or could be overridden by the user by use of the USER command),
that building (not running) a container is effectively giving a user
unfettered access as root during the build process
This view is not correct. When you build an image, all what you are doing is creating new docker layers (files) which are stored under /var/lib/docker/aufs/layers. There are simply no security concerns when building docker images.
There are tools to analyze the security of images you already built. One is the image analyzer built into Dockerhub.