New images are not being deployed to AKS - azure

Question

I've split out the initial azure-pipelines.yml to use templates, iteration, etc... For whatever reason, the new images are not being deployed despite using latest tag and/or imagePullPolicy: Always.
Also, I basically have two pipelines PR and Release:
PR is triggered when a PR request is submitted to merge to production. It automatically triggers this pipeline to run unit tests, build the Docker image, do integration tests, etc. and then pushes the image to ACR if everything passed.
When the PR pipeline is passing, and the PR is approved, it is merged into production which then triggers the Release pipeline.
Here is an example of one of my k8s deployment manifests (the pipeline says unchanged when these are applied):
apiVersion: apps/v1
kind: Deployment
metadata:
name: admin-v2-deployment-prod
namespace: prod
spec:
replicas: 3
selector:
matchLabels:
component: admin-v2
template:
metadata:
labels:
component: admin-v2
spec:
containers:
- name: admin-v2
imagePullPolicy: Always
image: appacr.azurecr.io/app-admin-v2:latest
ports:
- containerPort: 4001
---
apiVersion: v1
kind: Service
metadata:
name: admin-v2-cluster-ip-service-prod
namespace: prod
spec:
type: ClusterIP
selector:
component: admin-v2
ports:
- port: 4001
targetPort: 4001
And here are the various pipeline related .yamls I've been splitting out:
Both PR and Release:
# templates/variables.yaml
variables:
dockerRegistryServiceConnection: '<GUID>'
imageRepository: 'app'
containerRegistry: 'appacr.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)'
tag: '$(Build.BuildId)'
imagePullSecret: 'appacr1c5a-auth'
vmImageName: 'ubuntu-latest'
PR:
# pr.yaml
trigger: none
resources:
- repo: self
pool:
vmIMage: $(vmImageName)
variables:
- template: templates/variables.yaml
stages:
- template: templates/changed.yaml
- template: templates/unitTests.yaml
- template: templates/build.yaml
parameters:
services:
- api
- admin
- admin-v2
- client
- template: templates/integrationTests.yaml
# templates/build.yaml
parameters:
- name: services
type: object
default: []
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
steps:
- ${{ each service in parameters.services }}:
- task: Docker#2
displayName: Build and push an ${{ service }} image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)-${{ service }}
dockerfile: $(dockerfilePath)/${{ service }}/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
Release:
# release.yaml
trigger:
branches:
include:
- production
resources:
- repo: self
variables:
- template: templates/variables.yaml
stages:
- template: templates/publish.yaml
- template: templates/deploy.yaml
parameters:
services:
- api
- admin
- admin-v2
- client
# templates/deploy.yaml
parameters:
- name: services
type: object
default: []
stages:
- stage: Deploy
displayName: Deploy stage
dependsOn: Publish
jobs:
- deployment: Deploy
displayName: Deploy
pool:
vmImage: $(vmImageName)
environment: 'App Production AKS'
strategy:
runOnce:
deploy:
steps:
- task: KubernetesManifest#0
displayName: Create imagePullSecret
inputs:
action: createSecret
secretName: $(imagePullSecret)
kubernetesServiceConnection: 'App Production AKS'
dockerRegistryEndpoint: $(dockerRegistryServiceConnection)
- ${{ each service in parameters.services }}:
- task: KubernetesManifest#0
displayName: Deploy to ${{ service }} Kubernetes cluster
inputs:
action: deploy
kubernetesServiceConnection: 'App Production AKS'
manifests: |
$(Pipeline.Workspace)/k8s/aks/${{ service }}.yaml
imagePullSecrets: |
$(imagePullSecret)
containers: |
$(containerRegistry)/$(imageRepository)-${{ service }}:$(tag)
Both PR and Release pass...
The new images are in ACR...
I've pulled the images to verify they have the latest changes...
They just aren't getting deployed to AKS.
Any suggestions for what I am doing wrong here?

Answer 1

For whatever reason, the new images are not being deployed despite using latest tag
How should Kubernetes know that there is a new image? Kubernetes config is declarative. Kubernetes is already running what once was "latest" image.
Here is an example of one of my k8s deployment manifests (the pipeline says unchanged when these are applied)
Yeah, it is unchanged because the declarative disered state has not changed. The Deployment-manifest states what should be deployed, it is not a command.
Proposed solution
Whenever you build an image, always give it a unique name. And whenever you want to deploy something, always set a unique name of what should be running - then Kubernetes will manage this in an elegant zero-downtime way using rolling deployments - unless you configure it to behave different.

Answer 2

In your deployment you pull
image: appacr.azurecr.io/app-admin-v2:latest
Since there is no hash but simply the tag latest referenced the deployment says:
"You want latest? I have latest running!".
Important part is the "running". The pull policy always doesn't help if there is no need to pull in the first place.
Potential solutions:
Change something in your deployment that will cause a redeployment of the pods. Then it will actually pull the image again.
Cleaner solution: Don't use latest! Use some semantic versioning or date or whatever strategy matches your approach. In this case the tag will change always and it will always pull that image.