I have a Service Fabric cluster with 5 Windows VMs on it. I deploy an application that is a collection of about 10 difference containers. Each time I deploy, I increment the tag of the containers with a build number. For example:
foo.azurecr.io/api:50
foo.azurecr.io/web:50
Our build system continuously builds each service, tags it, pushes it to Azure, increments all the images in the ApplicationManifest.xml file, and then deploys the app to Azure. We probably build a new version a few times a day.
This works great, but over the course of a few weeks, the disk space on each VM fills up. This is because each VM still has all those old Docker images taking up disk space. Looking at it right now, there's about 50 gigs of old images sitting around. Eventually, this caused the deployment to fail.
My Question: Is there a standard way to clean up Docker images? Right now, the only idea I have is create some sort of Windows Scheduler task that runs a docker image prune --all every day or something. However, at some point we want to be able to create new VMs on the fly as needed so I'd rather each VM be a "stock" image. The other idea would be to use the same tag each time, such as api:latest and web:latest. However, then I'd have to figure out a way to get each VM to issue a docker pull command to get the latest version of the image.
Has anyone solved this problem before?
You can configure PruneContainerImages to True. This will enable the Service Fabric runtime to remove the unused container images. See this guide
Related
I have a docker image, I want to run it in azure VMSS . It should be scaled automatically (custom scaling based on metrics). Need some help on how to achieve this..
I can create a VM, install docker and run docker image there. But not sure how to do the same in VMSS. Do we need to get into the instances of VMSS and install docker and run docker image there in each VM? If so, how it will work when it scaled out to new instances..
It seems that there is an autoscale mechanism in place: https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-autoscale-overview
I wish I had a reliable answer.
I created the base VM, modified /etc/docker/daemon.json so it sets data-root to run in /mnt/docker, which would have tons of space.
tested out docker there using docker version 20.10.12. docker pull worked, docker run, ps -a, docker rm, docker image ls, docker system prune, all worked.
generalized vm, created image, hosted it in compute gallery, built a VMSS from the image version in the compute gallery.
Logged into a running instance.
successfully did a docker pull. but on docker run (same command I used in base VM), and poof, kernel panic, not syncing error.
but then, later on, after googling around (some said out-of-diskspace (not the problem)), found this site, tried to reproduce, and poof, it worked, docker run worked. I changed nothing.
Which kinda sucks. I built the VMSS for Azure Devops build/release pipelines. having it sometimes work (or fail the first time, dunno yet) is not a good solution.
I'm running an Azure Container Instance of a rather large image (~13GB). When I create a new instance it takes around 20 minutes to pull the image from the Azure Registry. When I update the image and then restart the container it also says it's pulling, but it only takes a few seconds. I tested it by changing the console output and it actually seems to update the image, but why is it taking so much less time?
ACI creates containers without you having to care about the underlying infrastructure, however under the hood these containers are still running on hosts. The first time you start your container, unless you are very lucky, it is unlikely that the underlying host has your container image cached and so it has to download the image, which for a large image will take a while.
When you restart a running container, most of the time it will restart on the same host, and so already have the old image cached. To update to the new image it will only need to download the difference, which is quick.
I've got a web app running on a Docker swarm in Azure that was set up through the Docker EE For Azure (17.06) bootstrapper. As the website isn't currently processing a large amount of traffic, I scaled the VM size down to the cheapest option (A0). After scaling these down website was unresponsive.
I SSHed on to a manager node and typing commands was slow. Figuring that I'd scaled down too much I scaled back up to the previous tier I had been on (D1_v2).
My website remained unresponsive so I SSHed onto a manager again. The terminal was responsive, however docker commands such as docker service ls and docker node ls do nothing. The VM in general seems to be working fine, however. I can cat files, run docker version etc.
Does anyone have any ideas why this may have happened? Is there any way I can fix it, or is my best option to just provision a new environment?
Thanks
I contacted Azure Support to see if they could provide any further assistance with this. They advised scaling the number of VMs in my Scale Set which I couldn't do because we had reached our allocated limit in that zone.
Azure were unable to provide any information as to how this may have happened and recommended provisioning a new environment as the best solution.
I'm trying to figure out if best practices would dictate that when deploying a new version of my web app (nodejs running in its own container) I should:
Do a git pull from inside the container and update "in place"; or
Create a new container with the new code and perform a hot swap of the two docker containers
I may be missing some technical details as I'm very new to the idea of containers.
The second approach is the best practice: you would make a second version of your image (with the new code), stop your container, and run a second container based on that second version.
The idea is that you can easily roll-back as the first version of your image can be used to run the container that was initially in production at any time.
Trying to modify a running container is not a good idea as, once it is stopped and removed, running it again would be from the original image, with its original state. Unless you commit that container to a new image, those changes would be lost. And even if you did commit, you would not be able to easily rebuild that image. (plus you would commit the all container: its new code, but also a bunch of additional files created during the execution of the server: logs and other files: not very clean)
A container is supposed to be run from an image that you can precisely build from the specifications of a Dockerfile. It is not supposed to be modified at runtime.
Couple of caveat though:
if your container is used (--link) by other containers, you would beed to stop those first, stop your container and run a new one from a new version of the image, then restart your other containers.
don't forget to remount any data containers that you were using in order to get your persistent data.
I'm planning to set up a jenkins-based CD workflow with Docker at the end.
My idea is to automatically build (by Jenkins) a docker image for every green build, then deploy that image either by jenkins or by 'hand' (I'm not yet sure whether I want to automatically run each green build).
Getting to the point of having a new image built is easy. My question is about the deployment itself. What's the best practice to 'reload' or 'restart' a running docker container? Suppose the image changed for the container, how do I gracefully reload it while having a service running inside? Do I need to do the traditional dance with multiple running containers and load balancing or is there a 'dockery' way?
Suppose the image changed for the container, how do I gracefully reload it while having a service running inside?
You don't want this.
Docker is a simple system for managing apps and their dependencies. It's simple and robust because ALL dependencies of an application are bundled with it. If your app runs today on your laptop, it will run tomorrow on your server. This is because we have captured 100% of the "inputs" for your application.
As soon as you introduce concepts like "upgrade" and "restart", your application can (accidentally) store state internally. That means it might behave differently tomorrow than it does today (after being restarted and upgraded 100 times).
It's better use a load balancer (or similar) to transition between your versions than to try and muck with the philosophy of Docker.
The Docker machine itself should always be immutable as you have to replace it for a new deployment. Storing state inside the Docker container will not work when you want to ship new releases often that you've built on your CI.
Docker supports Volumes which will let you write files that are permanent into some folder on the host. When you then upgrade the Docker container you use the same volume so you've got access to the same files written by the old container:
https://docs.docker.com/userguide/dockervolumes/