I am deploying an Azure Self hosted agent on a Kubernetes Cluster 1.22+ following steps in:
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linuxInstructions
I am adding podman to self hosted agent as container manager, following code is added to self hosted agent Dockerfile:
# install podman
ENV VERSION_ID=20.04
RUN apt-get update -y && apt-get install curl wget gnupg2 -y && . ./etc/os-release && sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list" && wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O- | apt-key add - && apt-get update -y && apt-get -y install podman && podman --version
Everything runs smoothly when running the container in privileged mode.
...
securityContext:
privileged: true
...
When swith to privileged: false and try to connect to podman, I get following error
level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied
the Command I use for connecting is:
podman login private.container.registry \
--username $USER \
--password $PASS \
--storage-opt mount_program=/usr/bin/fuse-overlayfs
How can I use podman with unprivileged mode ?
Issue was related to Containerd's apparmor profile denying the mount syscall,
I fixed it for now by disabling apparmor for the container while running unprivileged mode
...
template:
metadata:
labels:
app: vsts-agent-2
annotations:
container.apparmor.security.beta.kubernetes.io/kubepodcreation: unconfined
...
securityContext:
privileged: false #true
A better way would be creating an apparmor profile that allows the mount and apply it to the container
Related
We have been tasked with setting up a container-based Jenkins deployment, and there is strong pressure to do this in AKS. Our Jenkins needs to be able to build other containers. Normally I'd handle this with a docker-in-docker approach by mounting /var/run/docker.sock & /usr/bin/docker into my running container.
I do not know if this is possible in AKS or not. Some forum posts on GitHub suggest that host-mounting is possible but broken in the latest AKS relase. My limited experimentation with a Helm chart was met with this error:
Error: release jenkins4 failed: Deployment.apps "jenkins" is invalid:
[spec.template.spec.initContainers[0].volumeMounts[0].name: Required
value, spec.template.spec.initContainers[0].volumeMounts[0].name: Not
found: ""]
The change I made was to update the volumeMounts: section of jenkins-master-deployment.yaml and include the following:
-
type: HostPath
hostPath: /var/run/docker.sock
mountPath: /var/run/docker.sock
Is what I'm trying to do even possible based on AKS security settings, or did I just mess up my chart?
If it's not possible to mount the docker socket into a container in AKS, that's fine, I just need a definitive answer.
Thanks,
Well, we did this a while back for VSTS (cloud TFS, now called Azure DevOps) build agents, so it should be possible. The way we did it is also with mounting the docker.sock
The relevant part for us was:
... container spec ...
volumeMounts:
- mountPath: /var/run/docker.sock
name: docker-volume
volumes:
- name: docker-volume
hostPath:
path: /var/run/docker.sock
I have achieved the requirement using following manifests.
Our k8s manifest file carries this securityContext under pod definition.
securityContext:
privileged: true
In our Dockerfile we were installing Docker-inside-Docker like this way
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install curl wget -y
RUN apt-get install \
ca-certificates \
curl \
gnupg \
lsb-release -y
RUN mkdir -p /etc/apt/keyrings
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update
RUN apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
# last two lines of Dockerfile
COPY ./agent_startup.sh .
RUN chmod +x /agent_startup.sh
CMD ["/usr/sbin/init"]
CMD ["./agent_startup.sh"]
Content of agent_startup.sh file
#!/bin/bash
echo "DOCKER STARTS HERE"
service --status-all
service docker start
service docker start
docker version
docker ps
echo "DOCKER ENDS HERE"
sleep 100000
Sample k8s file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: build-agent
labels:
app: build-agent
spec:
replicas: 1
selector:
matchLabels:
app: build-agent
template:
metadata:
labels:
app: build-agent
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: build-agent
image: myecr-repo.azurecr.io/buildagent
securityContext:
privileged: true
When Dockerized agent pool was up, docker daemon was running inside docker container.
My Kubectl version
PS D:\Temp\temp> kubectl.exe version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.22.6
WARNING: version difference between client (1.25) and server (1.22) exceeds the supported minor version skew of +/-1
pod shell output:
root#**********-bcd967987-52wrv:/# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
**Disclaimer: Our kubernetes cluster version is 1.22 and base image is Ubuntu-18.04 and tested only to check if docker-inside-docker is running and not registered with Azure DevOps. You can modify startup script according to your need **
I have an application running in Elastic Beanstalk (AWS) smoothly and fast, but when I run it in Docker in my local, it takes a long time to load a single page (before it was not like this, it just start happened a few weeks ago). I am working on Ubuntu 22.04 LTS.
I also installed Docker desktop, but it does not matter what I do, the result is always the same (very very slow responses), is there something I can do?
The application is running an php:7.4.8-apache image.
This is how I configurated the “Resources”
CPUs: 10
Memory: 26Gb (the host machine has 32Gb)
Swap: 2.5GB ( I tried many different configurations but it does not make any difference)
Disk Image size: 64Gb
And the host machine:
SO: Ubuntu 22.04 LTS
Memory: 32Gb
Processor Inter Core i7 CPU 2.60ghz
Disk: 1Tb
DockerFile
FROM php:7.4.8-apache
ENV NVM_DIR=/root/.nvm
ENV NODE_VERSION=16.17
ENV USER=www-data
#set our application folder as an environment variable
ENV APP_HOME /var/www/html
ENV PATH="/root/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"
# # Add cake and composer and cake command to system path
ENV PATH="${PATH}:/var/www/html/lib/Cake/Console"
ENV PATH="${PATH}:/var/www/html/vendor/bin"
# COPY apache site.conf file
COPY ./config-dev-server/apache/* /etc/apache2/
COPY ./config-dev-server/php/conf.d/* /usr/local/etc/php/conf.d/
COPY ./config-dev-server/php/php.ini-development.ini /usr/local/etc/php/php.ini
# Project structure.
COPY . $APP_HOME
#install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin/ --filename=composer \
#change uid and gid of apache to docker user uid/gid
&& usermod -u 1000 www-data && groupmod -g 1000 www-data \
&& mkdir -p /var/www/html/logs \
&& mkdir -p /var/www/html/tmp \
## Clear cache
&& apt-get clean && apt-get autoclean && apt-get autoremove && rm -rf /var/lib/apt/lists/* \
# Install system dependencies
&& apt-get -o Acquire::Check-Valid-Until="false" update \
&& apt-get update && apt-get install --no-install-recommends -y \
# git \
curl \
npm \
libpng-dev \
libonig-dev \
libxml2-dev \
zip \
unzip \
# Install PHP extensions
&& docker-php-ext-install pdo_mysql mbstring exif pcntl bcmath gd intl \
&& pecl install xdebug \
&& docker-php-ext-enable xdebug \
# fix npm - not the latest version installed by apt-get && install amplify
&& npm install -g \
npm#8.15 \
#aws-amplify/cli \
&& a2enmod rewrite \
&& curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash \
&& . "$NVM_DIR/nvm.sh" && nvm install ${NODE_VERSION} \
&& . "$NVM_DIR/nvm.sh" && nvm use v${NODE_VERSION} \
&& . "$NVM_DIR/nvm.sh" && nvm alias default v${NODE_VERSION} \
&& node --version \
## rebuild node-sass node
&& npm rebuild node-sass \
&& groupadd docker \
&& usermod -aG docker $USER \
# && composer install --no-interaction --no-plugins --no-scripts \
#change ownership of our applications
&& chown -R www-data:www-data $APP_HOME/ \
# Changing log owner:group and permissions.
&& chown -R www-data:www-data /var/log/* \
# change permissions
&& chmod 755 -R $APP_HOME/ \
&& chmod 777 -R /var/log/* \
&& echo "Development environment ready, please install composer depdencies"
WORKDIR $APP_HOME/webroot
EXPOSE 8 8080
docker info
docker info
Client:
Context: desktop-linux
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.10.2)
extension: Manages Docker extensions (Docker Inc., v0.2.9)
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
scan: Docker Scan (Docker Inc., v0.19.0)
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.17
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.10.124-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 25.46GiB
Name: docker-desktop
ID: ZXOQ:5FED:TV2Y:KX5O:L7TF:Q626:4COZ:NWJO:WAJH:72ST:KBGC:X7NI
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false
Thanks in advance.
I have a docker image and I can run the docker image as a container. But I have to bind two folders local to the docker container. And I do it like this:
docker run -d -p 80:80 --name cntr-apache2
and then I go to: localhost:80. And the app is running.
But now I want to deploy it to the azure environment.
So Logged in in the azure portal: azure login.
and in docker I logged in: docker login webadres.azurecr.io.
and then I did a push: docker push webaddres.azurecr.io/webaddress:latest.
And I got this response:
latest: digest: sha256:7eefd5631cbe8907afb4cb941f579a7e7786ab36f52ba7d3eeadb4524fc5dddd size: 4909
And I made a web app for docker.
But now if I go to the url: https://docker-webaddress.azurewebsites.net
the index is empty. And that is logical. Because I didnt bind the two directories to the docker container in azure.
So my question is:
How now to bind the two direcories with azure docker?
Thank you
And this is my dockerfile:
FROM php:7.3-apache
# Copy virtual host into container
COPY 000-default.conf /etc/apache2/sites-available/000-default.conf
# Enable rewrite mode
RUN a2enmod rewrite
# Install necessary packages
RUN apt-get update && \
apt-get install \
libzip-dev \
wget \
git \
unzip \
-y --no-install-recommends
# Install PHP Extensions
RUN docker-php-ext-install zip pdo_mysql
COPY ./install-composer.sh ./
COPY ./php.ini /usr/local/etc/php/
EXPOSE 80
# Cleanup packages and install composer
RUN apt-get purge -y g++ \
&& apt-get autoremove -y \
&& rm -r /var/lib/apt/lists/* \
&& rm -rf /tmp/* \
&& sh ./install-composer.sh \
&& rm ./install-composer.sh
WORKDIR /var/www
RUN chown -R www-data:www-data /var/www
CMD ["apache2-foreground"]
One option is to use App Service persistent storage.
You need to enable the functionality first:
az webapp config appsettings set --resource-group <group-name> --name <app-name> --settings WEBSITES_ENABLE_APP_SERVICE_STORAGE=TRUE
Then create a mapping in a Docker Compose file even if you have a single container to deploy. The {WEBAPP_STORAGE_HOME} environment variable will point to /home in the App Service VM. You can use the Advanced Tools (Kudu) to create the folders and copy the needed files.
Wordpress:
image: <image name:tag>
volumes:
- ${WEBAPP_STORAGE_HOME}/folder1:/etc/apache2/sites-enabled/
- ${WEBAPP_STORAGE_HOME}/folder2:/var/www/html/ docker_webcrawler2
Documentation
Another option is to mount Azure Storage.
Note that mounts using Blobs are read-only and File Share are read-write.
Start by creating a storage account and a File Share.
Create the mount. I find that it's easier using the Portal. See how in the doc link below.
az storage share-rm create \
--resource-group $resourceGroupName \
--storage-account $storageAccountName \
--name $shareName \
--quota 1024 \
--enabled-protocols SMB \
--output none
Documentation
I keep getting this error when I try to use twistcli to scan a container using podman:
failed to augment data: Error: error mounting storage for container c494c177d35d905aa267f0eebccf67dce2bcd0b61bc1511ef7039fde07baf152: error creating aufs mount to /var/lib/containers/storage/aufs/mnt/b5da5a775dc2fcd81fb1ca5b658415cb5e94450ed1cf0861dc4214a3d3fe9285: invalid argument
In the current configuration I'm trying to run twistcli in the gitlab ci pipeline, using Ubuntu 21.04 as an image on which podman is then installed on top.
Pipeline .gitlab-ci.yml
stages:
- scan
scan:
stage: scan
image: ubuntu:21.04
script:
- apt-get update
- apt-get -y install curl
#- apt install software-properties-common uidmap
#- add-apt-repository ppa:projectatomic/ppa
- apt-get -y upgrade
- apt-get -y install podman
- podman info
- podman login docker.io -u sim55649 -p $DOCKER_PASS
- podman pull docker.io/alpine:latest
#- cat ~/etc/containers/storage.conf
- curl -k -u $PRISMA_USER:$PRISMA_PASS --output ./twistcli $PRISMA_ADD/api/v1/util/twistcli
- more ./twistcli
- chmod a+x ./twistcli
- df /var/lib/containers/.
- ./twistcli images scan --address $PRISMA_ADD --user $PRISMA_USER --password $PRISMA_PASS --details alpine:latest
How can this error be solved?
Thanks in advance
I have installed Docker on Ubuntu 16.04 server, using the manual on this page: https://docs.docker.com/cs-engine/1.13/, so, using these steps:
curl -fsSL 'https://sks-keyservers.net/pks/lookup?op=get&search=0xee6d536cf7dc86e2d7d56f59a178ac6c6238f52e' | sudo apt-key add -
sudo add-apt-repository "deb https://packages.docker.com/1.13/apt/repo/ \
ubuntu-$(lsb_release -cs) \
main"
sudo apt-get update
sudo apt-get -y install docker-engine
I have installed it on two servers and I need them to see each other, I needed to let Docker daemon listen on port 2375 (probably doesn't matter, but using this manual: https://github.com/yeasy/cello/blob/master/docs/deployment.md)
So I created the conf file:
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo vim /etc/systemd/system/docker.service.d/override.conf
Added this to the override.conf:
[Service]
DOCKER_OPTS="$DOCKER_OPTS -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --api-cors-header='*' --default-ulimit=nofile=8192:16384 --default-ulimit=nproc=8192:16384"
EnvironmentFile=-/etc/default/docker
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS
Then:
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker.service
Tested the connection between servers like this:
$ docker -H 10.101.35.61:2375 version
The response:
Client:
Version: 1.13.1-cs4
API version: 1.27
Go version: go1.7.5
Git commit: e46aec0
Built: Mon May 22 18:46:40 2017
OS/Arch: linux/amd64
Cannot connect to the Docker daemon at tcp://10.101.35.61:2375. Is the docker daemon running?
Tried restarting the server, same problem. Tried to run with sudo. Tried adding the user to group docker:
sudo usermod -aG docker $USER
Didn't help. I have disabled firewall on both servers. When I check the ports open on the server with sudo lsof -i, I can't see anything listening to port 2375 - I am guessing Docker should be listening to it?
Try the config file in this location, create it if it does not exist:
/etc/docker/daemon.json
Put this and restart the docker service:
{"hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]}
You can add more configs, documented here.