Well, everyone keeps saying to try to learn something new during this pandemic, so that's what I'm trying to do. I'm brand new to YAML and Docker, and fairly new to ARM templates.
I've been able to use YAML to build and publish .NET Core apps using the dotnet YAML commands. I've been able to use YAML to build a Docker image of a .NET Core 3.1 Web API app.
I'm publishing the docker image to an Azure Resource Container.
Where I'm stuck:
Ideally, I want to use ARM templates in the .NET Core project to be able to deploy my Azure Portal resources via code vs creating manually in the portal. This should include the Azure Resource Container.
It seems like what I need to do to accomplish this is build the docker image during the Build phase and then publish that to the $(System.ArtifactsDirectory) so that I can hold it there until the Azure Resources have been deployed via ARM templates.
I can't find anything about publishing the docker image as a pipeline artifact though. Only things about pushing to DockerHub, GitHub, and Azure Resource Containers, which isn't what I want to do right now.
Anyone have any ideas how I can pull this off?
I actually found a way to pull this off. The article I read about this is over on dev.to
There are a few things that I needed to do to pull this off.
First, don't set the containerRegistry on build.
- task: Docker#2
displayName: Build Docker Image
inputs:
repository: $(imageName)
command: build
dockerfile: '**/Dockerfile'
buildContext: 'HopefulMommaDesignsAPI'
tags: $(Build.BuildId)
The reason for this is because if you set the container registry then, with ACR at least, it prefixes the ACR URL to the docker image name. However, that URL is encrypted when doing this in Azure DevOps. There's nothing wrong with that, but it causes confusion when you list images because it will show up as ***/ImageName.
Second, save the image as a TAR file in the artifact staging directory.
- task: Docker#2
displayName: 'Save image to TAR'
inputs:
repository: '$(imageName)'
command: save
arguments: '--output $(build.artifactstagingdirectory)/$(imageName).image.tar $(imageName):$(Build.BuildId)'
addPipelineData: false
That will include the .tar file when you publish your build.
The next steps are during the deploy jobs.
After you have downloaded the build artifact and deployed the ARM templates, you'll need to load the docker image from the .tar file.
- task: Docker#2
displayName: 'Load Image from Tar'
inputs:
command: load
arguments: '--input $(build.artifactstagingdirectory)/$(buildArtifactName)/$(imageName).image.tar'
Now that the docker image has been loaded you'll want to ReTag the image and publish it.
- task: Docker#2
displayName: 'ReTag Image with ACR URL - BuildId'
inputs:
containerRegistry: 'Hopeful Momma Designs ACR'
repository: $(imageName)
command: tag
arguments: '$(imageName):$(Build.BuildId) $(containerRegistryUrl)/$(imageName):$(Build.BuildId)'
- task: Docker#2
displayName: 'ReTag Image with ACR URL - latest'
inputs:
containerRegistry: 'Hopeful Momma Designs ACR'
repository: $(imageName)
command: tag
arguments: '$(imageName):$(Build.BuildId) $(containerRegistryUrl)/$(imageName):latest'
- task: Docker#2
displayName: push
inputs:
containerRegistry: 'Hopeful Momma Designs ACR'
repository: $(imageName)
command: push
dockerfile: '**/Dockerfile'
buildContext: 'HopefulMommaDesignsAPI'
tags: |
$(Build.BuildId)
latest
NOTE: If your Azure Container Registry is part of your ARM template, the Docker commands where you include the containerRegistry will fail because you have to manually set up the service connection from Azure DevOps to the Azure Container Registry before you can reference it in your YAML files as the containerRegistry.
This isn't a huge deal for me because I will normally work on pipelines in sections anyway. I'll make sure the build works, then I'll make sure the ARM templates work, then I'll make sure the docker images pushes to the ACR. If you're rebuilding an environment from scratch, just comment out the deployment steps after the ARM template step to deploy the infrastructure and then uncomment the remaining steps. It isn't the cleanest solution, but I like that better than having a separate pipeline just to deploy infrastructure.
You publish your container to an Azure Container Registry. Containers aren't build artifacts in the traditional sense.
I suspect that you have a chicken/egg problem. You want to create the registry during your deployment process, but you don't have a registry during the build process. So how can you deploy your container to a registry that won't exist until the next phase, right? In fact, you'll discover that you need to create a service connection for the ACR before you can publish resources to it. The ARM template isn't going to be able to do that for you.
This is a case where it's common to separate the application's requirements from larger infrastructural requirements. The ACR is a supporting piece of infrastructure. It's common in cases like this to have a separate pipeline that creates required supporting infrastructure to solve the chicken-and-egg problem.
Also, please be aware of correct terminology. There's nothing called an "Azure Resource Container". There is something called an "Azure Container Registry", which is what I assume you're talking about.
Related
I've tried so many different things here, so i'm super excited to see what the issue is :) (but, im assuming it's something silly on my end)
Here's the pertinent bit of my GitHub actions YML:
deploy:
name: Update Azure App Service
needs: [tests]
runs-on: ubuntu-latest
steps:
- name: Deploy to Azure
uses: azure/webapps-deploy#v2
with:
app-name: $APP_SERVICE_NAME
images: $IMAGE_NAME
publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
GitHub actions logs (debug mode):
##[debug]Evaluating condition for step: 'Deploy to Azure'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Deploy to Azure
##[debug]Loading inputs
##[debug]Evaluating: secrets.AZURE_WEBAPP_PUBLISH_PROFILE
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'AZURE_WEBAPP_PUBLISH_PROFILE'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run azure/webapps-deploy#v2
::add-mask::***
::add-mask::***
##[debug][GET] https://SNIP.scm.azurewebsites.net:443/diagnostics/runtime
##[debug]loaded affinity cookie ["ARRAffinity=SNIP;Path=/;HttpOnly;Secure;Domain=SNIP-qa-ause.scm.azurewebsites.net","ARRAffinitySameSite=SNIP;Path=/;HttpOnly;SameSite=None;Secure;Domain=SNIP.scm.azurewebsites.net"]
##[debug]getAppRuntime. Data: {"statusCode":200,"statusMessage":"OK","headers":{"connection":"close","content-type":"application/json; charset=utf-8","date":"Fri, 22 Jul 2022 06:27:05 GMT","server":"Kestrel","set-cookie":["ARRAffinity=SNIP;Path=/;HttpOnly;Secure;Domain=app-be-SNIP.scm.azurewebsites.net","ARRAffinitySameSite=SNIP;Path=/;HttpOnly;SameSite=None;Secure;Domain=app-be-SNIP.azurewebsites.net"],"transfer-encoding":"chunked"},"body":{"nodejs":[],"system":{"os_name":"Unix 4.15.0.169","os_build_lab_ex":"","cores":2}}}
##[debug]App Runtime OS: Unix 4.15.0.169
Error: Deployment Failed with Error: Error: Publish profile is invalid for app-name and slot-name provided. Provide correct publish profile credentials for app.
##[debug]Deployment failed
##[debug]Node Action run completed with exit code 1
##[debug]AZURE_HTTP_USER_AGENT='GITHUBACTIONS_DeployWebAppToAzure_0d7e9cbfcd2b52a8a34111a0798ab22d77203fc1f56b2732d2431bfaeb336577'
##[debug]AZURE_HTTP_USER_AGENT=''
##[debug]Finishing: Deploy to Azure
The obvious solution here is what the error says: secret is wrong. It's not. I've added secret to GitHub repository, with the value of the Azure App Service. Done this twice, to make sure i've downloaded it correct, and copied as is.
Other things i've tried:
Tried setting WEBSITE_WEBDEPLOY_USE_SCM to true in the App Service configuration.
Tried setting slot-name to Production in the GitHu action step (even though, there is no slots here)
Other things, that might help solve my problem:
I'm building a .NET image as a Docker image, and pushing this to Azure Containr Registry. I'm then trying to 'update' this, in Azure App Service
The Azure App Service is running Linux, not Windows
"Deployment Center" in App Service still has "Source: Container Registry", pinned to a static image. This is because this was required when i setup App Service. But i assume that my GitHub actions builds will override this?
Any help would be great. :) Thanks in advance!
EDIT
I know i can:
Push to a constant tag, e.g latest
Set the image name to latest in Azure App Service
Set continuous deployment to "on" in Azure App Service
And this will automatically update App Service, because of the webhook.
But this will not help production scenarios, as i need to create different image tags, for different versions i can rollback. Since the version is different, i can't "pin" to a constant tag. Therefore, i need some kind of task to update Azure App Service with a specific tag. I'd prefer to have the same CI for dev/prod.
I have a pipeline in Azure DevOps that build a container and push it in my Azure Container Registry.
# Docker
# Build and push an image to Azure Container Registry
# https://learn.microsoft.com/azure/devops/pipelines/languages/docker
trigger:
- main
resources:
- repo: self
variables:
# Container registry service connection established during pipeline creation
dockerRegistryServiceConnection: 'myguyd'
imageRepository: 'p2005'
containerRegistry: 'myacr.azurecr.io'
dockerfilePath: '$(Build.SourcesDirectory)/api/DOCKERFILE'
tag: '$(Build.BuildId)'
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
tags: latest
In the container I also have some raw data file that the application needs. In the security point of view, it is not great to have the raw data in the container.
So, I was thinking if there is a way to encrypt the Docker container. And if so, how to do it in the Azure pipeline.
I don't know there is a way to encrypt via the Azure pipeline. If you're using AKS by chance, you can take a look at Confidential Computing.
A hardware-based Trusted Execution Environment (TEE) provides strong
assurances. A TEE provides hardware and software measurements from
trusted computing base (TCB) components. Confidential containers
offerings on Azure allow verification of these measurements and
validate if the container applications run in a verifiable execution
environment.
Confidential containers support custom applications developed with any
programming languages. You can also run Docker containers off the
shelf.
You can achieve this with Intel SGX and AKS.
To run an existing Docker container, applications on confidential
computing nodes require an abstraction layer or Intel Software Guard
Extensions (SGX) software to use the special CPU instruction set.
Configure SGX to protect your sensitive application code. SGX creates
a direct execution to the CPU to remove the guest operating system
(OS), host OS, or hypervisor from the trust boundary. This step
reduces the overall surface attack areas and vulnerabilities.
Azure Kubernetes Service (AKS) fully supports confidential containers.
You can run existing containers confidentially on AKS.
I have an Azure pipeline in which I create an artifact (a folder around 400 MB)
...
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifact: 'PublishArtifact'
publishLocation: 'pipeline'
Is it possible to download it to my local folder (let's say: C:\test) with a command within this same pipeline?
(I can download it manually if I go to Pipelines -> 'The latest build' -> Under Related, I have published artifact, but I want to do this automatically within pipeline with some command)
If you want to do it during pipeline execution you should install self hosted agent on your machine. Here you have links:
Self-hosted Linux agents
Self-hosted Windows agents
Then you need to condogure pipeline to use this agent.
Another option would be FTP upload. In this way you need to configure you machine as FTP server and use for instance this task. In this approach you can still use MS Hosted agent.
I think this should be possible if you add your computer as a deployment group (See Provision deployment groups). Then your build could trigger a classic Release pipeline where you can copy the artifact to a folder within you local machine
You can use Copy file task as explained in this doc
Copy file task
I'm pretty familiar with Azure DevOps, pipelines all that stuff, and now I'm trying to dig into GitHub Actions. The question is basically pretty simple, I want to deploy my .NET Core 5 App to Azure. The only problem is, that all examples include this publish profile.
Since I provision the infrastructure with an ARM template, the publish profile is simply not there yet. I could find some examples that deploy the ARM template and a couple of examples that deploy the Web App, but no example combining both. Maybe I'm a little bit polluted by the way Azure DevOps works and the (wonderful) idea of service connections.
So my question is, how do I publish a web app to Azure when I don't have the ability to download a publish profile and store that in my GitHub secrets, using GitHub Actions?
OK, I found this one out myself. Apparently there's an action you can use that will download the publish profile for you. This means that you don't have to have the publish profile up front. The step looks like this:
- name: Get WebApp/FunctionApp publish profile
id: webapp-dev
uses: aliencube/publish-profile-actions#v1
env:
AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }}
with:
resourceGroupName: 'your-resource-group-name'
appName: 'your-app-name'
This leaves you with an output variable called `profile' which can be used in following steps like so:
- name: 'Run Azure webapp deploy action using publish profile credentials'
uses: azure/webapps-deploy#v1
with:
app-name: 'your-app-name'
publish-profile: ${{ steps.webapp-dev.outputs.profile }}
package: './'
This means you can now provision resources using ARM templates, get the publish profile (just-in-time) and use that to deploy your system... Everybody happy...
We have a collection of Azure Function Apps in c# net core. Each App contains a small number of Azure Functions. All Function Apps reside in a single git repository.
We would like some of our environments to deploy automatically from source (e.g. bitBucket or gitHub).
How do we configure the project so that Azure knows which project in source relates to which created Function App?
I have searched around this problem for a number of days and have not seen any results that sit outside of "it just works" so can only assume that we are missing something fundamental.
I'd recommend using Azure DevOps (formerly VSTS) to deploy to Azure, you use YAML to define a build pipeline which can publish an artifact from each of your function apps. The artifacts then get picked up by a release pipeline and can be deployed to Azure.
The basic building blocks of this are, firstly some YAML like this in your build pipeline for each project:
...
steps:
# a script task that let's you use any CLI available on the DevOps build agent, also uses a variable for the build config
- script: dotnet build MyFirstProjectWithinSolution\MyFirstProject.csproj --configuration $(buildConfiguration)
displayName: 'dotnet build MyFirstProject'
# other steps removed, e.g. run and publish tests
- script: dotnet publish MyFirstProjectWithinSolution\MyFirstProject.csproj --configuration $(buildConfiguration) --output MyFirstArtifact
displayName: 'dotnet publish MyFirstProject'
# a DevOps named task called CopyFiles (which is version 2 = #2), DevOps supplies lots of standard tasks you can make use of
- task: CopyFiles#2
inputs:
contents: 'MyFirstProjectWithinSolution\MyFirstArtifact\**'
targetFolder: '$(Build.ArtifactStagingDirectory)'
# now publish the artifact which makes it available to the release pipeline, doing so into a sub folder allows multiple artifacts to be dealt with
- task: PublishBuildArtifacts#1
displayName: 'publish MyFirstArtifact artifact'
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)\MyFirstProjectWithinSolution\MyFirstArtifact'
artifactName: MyFirstArtifact
# now repeat the above for every project you need to deploy, each in their own artifact sub-folder
Next you create a release, which in its simplest form picks up the artifacts and does one or more deployment, here's a simple one which deploys two function app projects:
Within a deployment stage (right hand side above), you can define your release process, again in its simplest form you can just deploy straight to production or to a slot, although until function slots are out of preview you could also spin up another function app and deploy and test there.
This screenshot shows a simple deployment which uses a standard Azure Function App deployment from Azure DevOps:
Within your deployment stage you can define which artifact is deployed and after running your build pipeline for the first time you'll get to see all the available artifacts that it created.
All or parts of the above can be automated from pushing a branch (or other triggers such as on a schedule). Notifications and "gates" can be added as well if you want manual intervention before release or between release stages.
There are also other ways to cut this up, eg with multiple build pipelines, it’s basically completely flexible but the above are the elements you can use to deploy one or more function apps at a time.