Deploy/update Azure ML Model Using Az Ml CLI in Azure pipelines - azure

I'm trying to deploy the azure ml model if not exists in the workspace and when the model is already available in the registered workspace then update the model with the latest version only when an update is available, but I don't know how this works in practice.
The Azure pipelines will run on a weekly schedule.
steps:
- task: AzureCLI#2
displayName: 'Install AML CLI'
inputs:
azureSubscription: $(ml_ws_connection)
scriptLocation: inlineScript
scriptType: 'bash'
inlineScript: 'az extension add -n azure-cli-ml'
- task: AzureCLI#2
displayName: 'Attach folder to workspace'
inputs:
azureSubscription: $(ml_ws_connection)
workingDirectory: $(ml_path)
scriptLocation: inlineScript
scriptType: 'bash'
inlineScript: 'az ml folder attach -w $(ml_ws) -g $(ml_rg)'
# Add potential automated tests
- task: AzureCLI#2
displayName: 'Create AKS cluster'
inputs:
azureSubscription: $(ml_ws_connection)
workingDirectory: $(ml_path)
scriptLocation: inlineScript
scriptType: 'bash'
inlineScript: 'az ml computetarget create aks --name $(ml_aks_name) --cluster-purpose DevTest'
- task: AzureCLI#2
displayName: 'Deploy model to AKS '
inputs:
azureSubscription: $(ml_ws_connection)
workingDirectory: $(ml_path)
scriptLocation: inlineScript
scriptType: 'bash'
inlineScript: 'az ml model deploy --name model1_aks --ct $(ml_aks_name) --ic config/inferenceConfig.json -e $(ml_env_name) --ev $(ml_env_version) --dc config/aksDeploymentConfig-aks.json --overwrite'
- task: AzureCLI#2
displayName: 'Update model in AKS '
inputs:
azureSubscription: $(ml_ws_connection)
workingDirectory: $(ml_path)
scriptLocation: inlineScript
scriptType: 'bash'
inlineScript: 'az ml service update --name $(deploy_service_name)

We can do CICD for automated model release as documented here Continuously deploy Azure Machine Learning models - Azure Machine Learning | Microsoft Docs.
Please follow the references for CLI spec.
Here is link to update a deployment using configuration.
The ml extension to the Azure CLI is the improved interface for Azure Machine Learning users. It enables you to train and deploy models from the command line, with features that accelerate scaling the data science process up and out, all while tracking the model lifecycle.
Using the CLI enables you to run distributed training jobs on GPU compute, automatically sweep hyperparameters to improve your results, and then monitor jobs in the AML studio user interface to see all details including important metrics, metadata and artifacts like the trained model, checkpoints and logs.
Additionally, the CLI is optimized to support YAML-based job, endpoint, and asset specifications to enable users to create, manage, and deploy models with proper CI/CD (or GitOps) best practices for an end-to-end MLOps solution.
To get started with the 2.0 machine learning CLI extension for Azure, please check the link here.

Related

Change ARR affinity property from General Settings with Azure Devops pipeline

I'm trying to modify the "ARR affinity" (clientAffinityEnabled) property in App Service General Settings with a pipeline task but it doesn't work, the value doesn't change.
This pipeline works OK with other General Settings properties.
Another approach to solve this?
Azure DevOps pipeline task:
- task: AzureAppServiceSettings#1
inputs:
azureSubscription: XXXXXXX
ResourceGroupName: XXXXXXX
appName: XXXXXXX
generalSettings: |
[
{
"clientAffinityEnabled": false
}
]
Test the same settings in the AzureAppServiceSettings task, I can reproduce the same situation. It seems that the AzureAppServiceSettings task is not able to update the ARR affinity value.
For a workaround, you can change to use Azure CLI Task to run the Azure CLI: az webapp update to update the ARR affinity value.
For example:
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: xx
scriptType: ps
scriptLocation: inlineScript
inlineScript: 'az webapp update --name xx --resource-group xx --client-affinity-enabled false'

Deploy packages to multiple webapp service by azure pipeline single stage

I have more than 100 webapp service in azure. I want to deploy packages in 100 webapps by azure pipeline with one pipeline yml file. But I couldn't find any documentation like this. I got one microsoft documentation and they prefer to increase pipeline steps. If I have 100 webapps service, then have to add 100 steps for each deployment. This is not an efficient way and its time consuming. I want just like this step.
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: 'JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........'
package: '$(build.artifactstagingdirectory)/**/*.zip'
This yaml file is showing error. I couldn't find any essential extensions to fix it. I also couldn't find any azure powershell deployment command regarding to this issue. How can I get the solution?
You will not be able to do this like this. However you can use Azure Cli task:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$apps= #('JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........')
foreach ($app in $apps) {
az webapp deployment source config-zip -g $(group) -n $app --src '$(build.artifactstagingdirectory)/SOME_FOLDER/Artifact.zip'
}
And here you have more details about deployment itself
Annother approach with multiple task bu continuation if one fail is:
parameters:
- name: apps
type: object
default:
- JustGoTestAgain
- justgotesttwo
- and so on
steps:
- ${{ each app in parameters.apps}}:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ app }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ app }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
Thete was issue with space. Now is fine. Apart from that there is only one issue with connectedServiceName
Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
Which I skipped here as you already have it on your solution.

Execute simple SQL query though Azure Devops CICD pipeline

I am new to Azure Devops and CICD pipeline. I have been tasked to create a CICD pipeline to execute a SQL query from Azure Devops CI/CD pipeline. How do I do that? Any suggestion would be helpful.
SQL queries can be ran via the ADO tasks
Something like:
variables:
AzureSubscription: '<Azure service connection>'
ServerName: '<Database server name>'
DatabaseName: '<Database name>'
AdminUser: '<SQL user name>'
AdminPassword: '<SQL user password>'
SQLFile: '<Location of SQL file in $(Build.SourcesDirectory)>'
steps:
- task: AzurePowerShell#2
displayName: Azure PowerShell script: FilePath
inputs:
azureSubscription: '$(AzureSubscription)'
ScriptPath: '$(Build.SourcesDirectory)\scripts\SetAzureFirewallRule.ps1'
ScriptArguments: '$(ServerName)'
azurePowerShellVersion: LatestVersion
- task: CmdLine#1
displayName: Run Sqlcmd
inputs:
filename: Sqlcmd
arguments: '-S $(ServerName) -U $(AdminUser) -P $(AdminPassword) -d $(DatabaseName) -i $(SQLFile)'
- task: AzurePowerShell#2
displayName: Azure PowerShell script: FilePath
inputs:
azureSubscription: '$(AzureSubscription)'
ScriptPath: '$(Build.SourcesDirectory)\scripts\RemoveAzureFirewallRule.ps1'
ScriptArguments: '$(ServerName)'
azurePowerShellVersion: LatestVersion
Note the extra steps is to remove the firewall if you are utilizing firewall rules

how to execute azure cli command in azure devops pipeline

To execute ACR commands in Azure CLI, we need upgrade AZ CLI. Need to have complete steps in YML to execute the AZ commands in pipeline.
Please refer link: https://learn.microsoft.com/en-us/azure/devops/cli/azure-devops-cli-in-yaml?view=azure-devops
For Linux: azure-pipelines-steps-linux.yml:
steps:
Updating to latest Azure CLI version.
script: pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
displayName: 'upgrade azure cli'
script: az --version
displayName: 'Show Azure CLI version'
script: az extension add -n azure-devops
displayName: 'Install Azure DevOps Extension'
task: AzureCLI#2
inputs:
azureSubscription: 'AzureServiceConnection'
scriptType: bash
scriptLocation: 'inlineScript'
inlineScript: |
az account show

How to securely login in Az CLI from a DevOps Pipeline

I want to execute AZ cli commands from my Azure DevOps Pipeline. In my YAML file I have this:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
buildConfiguration: 'Release'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.x'
architecture: 'x64'
# Updating pip to latest
- script: python -m pip install --upgrade pip
displayName: 'Upgrade pip'
# Updating to latest Azure CLI version.
- script: pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
displayName: 'upgrade azure cli'
- script: az --version
displayName: 'Show Azure CLI version'
- script: az extension add -n azure-devops
displayName: 'Install Azure DevOps Extension'
- script: echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
env:
AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
displayName: 'Login Azure DevOps Extension'
- script: az aks show --name census-k8s --resource-group Census
displayName: 'Show AKS'
The echo ${AZURE_DEVOPS_CLI_PAT} | az devops login step is completed (with success apparently) with a warning message
Failed to store PAT using keyring; falling back to file storage.
You can clear the stored credential by running az devops logout.
Refer https://aka.ms/azure-devops-cli-auth to know more on sign in with PAT.
The az aks show step fails:
Please run 'az login' to setup account.
I am a little bit lost. The az devops login command should enable me to use the az cli, right? If not, Am I supposed to use az login instead of az devops login? And if I am supposed to use az login, how can I pass my credentials in a secure way?
No, you don't need az devops login. What you need is Azure CLI Task:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: <Name of the Azure Resource Manager service connection>
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
az --version
az account show
but then you don't have to do any login. Please call there your az aks show --name census-k8s --resource-group Census
Just to Add to Krzysztof's answer (and jeromerg question in the comment): in Azure CLI step you can also use other tools then az, which require being logged in with AzureCLI:
- task: AzureCLI#2
displayName: Publish Function
inputs:
azureSubscription: <Name of the Azure Resource Manager service connection>
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
func azure publish <function-name>
If your scriptLocation is a scriptPath use the following example
- task: AzureCLI#2
displayName: 'update function appsettings'
inputs:
azureSubscription: 'MY-AzureSubscriptionName'
scriptType: ps
scriptLocation: 'scriptPath'
scriptPath: '$(System.DefaultWorkingDirectory)/Scripts/updateSettings.ps1'
arguments:
-ResourceGroupName 'MY-ResourceGroupName' `
-FunctionAppName 'MY-FunctionAppName'
updateSettings.ps1
param (
[string]$ResourceGroupName,
[string]$FunctionAppName)
)
.
. script body here
.
To use Azure CLI from a script (powershell or batch) you must assign $(System.AccessToken) to an environment variable named AZURE_DEVOPS_EXT_PAT.
- pwsh: |
az pipelines build list
displayName: 'Show build list'
env:
AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
Source: https://learn.microsoft.com/en-us/azure/devops/cli/azure-devops-cli-in-yaml?view=azure-devops

Resources