Deploy terraform to azure through a pipeline in azure devops - azure

I have deployed terraform to azure locally from the azure cli to success so I know the TF is working
I am now trying to build a pipeline to automate this process, however i get this error...
Error: Error building ARM Config: obtain subscription(xxxxxx) from Azure CLI: Error parsing json result from the Azure CLI: Error waiting for the Azure CLI: exit status 1
This error appears when the init is running in the pipeline.
This is what the pipeline looks like...
pool:
vmImage: ubuntu-latest
stages :
- stage: validate
jobs:
- job: validate
continueOnError: false
steps:
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller#0
displayName: 'install'
inputs:
terraformVersion: '0.14.10'
- bash: echo $(ls -latR)
- task: TerraformTaskV1#0
displayName: 'init'
inputs:
provider: 'azurerm'
command: 'init'
workingDirectory: "$(System.DefaultWorkingDirectory)"
backendServiceArm: 'Service-Fabric'
backendAzureRmResourceGroupName: 'fvs-uks-sf-rg-02'
backendAzureRmStorageAccountName: 'fvsukssfstorage'
backendAzureRmContainerName: 'fvs-uks-sf-backend01'
backendAzureRmKey: 'terraform.tfstate'
- task: TerraformTaskV1#0
displayName: 'validate'
inputs:
provider: 'azurerm'
command: 'validate'
- task: TerraformTaskV1#0
displayName: 'plan'
inputs:
provider: 'azurerm'
command: 'plan'
workingDirectory: "$(System.DefaultWorkingDirectory)"
environmentServiceNameAzureRM: 'test'
environmentServiceNameAWS: 'AWS-Azure-VPN'
This is what the backend.tf looks like...
provider "azurerm" {
alias = "hub"
subscription_id = "xxxxxxx"
features {}
}
provider "azurerm" {
features {}
}
terraform {
backend "azurerm" {
subscription_id = "xxxxxx"
resource_group_name = "xxxxxx"
storage_account_name = "xxxxxx"
container_name = "xxxxxx"
key = "terraform.tfstate"
}
}
I have also set up a service connection to azure via a service principal, that when authenticated connects successfully.
Any help is greatly appreciated

Deploy terraform to azure through a pipeline in azure devops
This issue should be related to the authorization.
When we use the service connection to authorization, it use the Service Principal instead of username/password to authenticate.
Then according to the terraform docs:
If you're authenticating using a Service Principal then it must have
permissions to both Read and write all applications and Sign in and
read user profile within the Windows Azure Active Directory API.
Please try to add a API permission for your principal, which you used to create the service connection:
Navigate to the Azure Active Directory overview within the Azure
Portal and select the App Registrations blade. Locate your registered
Application and click on its display name to manage it.
Go to the API Permissions blade for the Application and click the "Add
a permission" button. In the pane that opens, select "Azure Active
Directory Graph" (under the Supported Legacy APIs subheading). Do not
select "Microsoft Graph", as the provider does not currently make use
of this API.
Choose "Application Permissions" for the permission type, and check
the permissions you would like to assign. The permissions you need
will depend on which directory objects you are seeking to manage with
Terraform. We suggest the following permissions:
Application.ReadWrite.All
Directory.ReadWrite.All

Related

Cannot build angular project in Azure DevOps

I have Angular project, that I want to build to Azure Web App
I created this yaml to build and deploy
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '18.x'
displayName: 'Install Node.js'
# - task: Npm#1
# inputs:
# command: 'install'
# workingDir: 'schooly-angular'
- script: npm install -g #angular/cli
displayName: 'npm install -g #angular/cli'
- script: yarn
displayName: 'yarn install'
- script: ng build --prod
displayName: 'ng build'
- task: PublishBuildArtifacts#1
- task: AzureRmWebAppDeployment#4
inputs:
ConnectionType: 'AzureRM'
azureSubscription: 'Azure subscription 1 (*********)'
appType: 'webAppLinux'
WebAppName: 'marifit-admin'
packageForLinux: 'dist/'
But I get this error when try to build
Clicking Authorize, doesn't helps
ow I can solve this?
It seems you do not have permission regard on the subscription Azure subscription 1 (*********) or this subscription does not exist. When edit YAML pipeline, click the Settings to show assistance >> choose your subscription >> click Authorize to create a service connection for this subscription.
If the authorization failed, means you do not have enough permission regard on that subscription. You need to get contributor or owner role regard on the subscription. To get related permission, you need get help from subscription owner. Here is Assign a user as an administrator of an Azure subscription for reference.
I have tried reproducing in my environment and got below results so sharing it here as further reference:
task: AzureRmWebAppDeployment#4
    inputs:
    ConnectionType: 'AzureRM'
    azureSubscription: 'Azure subscription 1 (*********)'
    appType: 'webAppLinux'
    WebAppName: 'marifit-admin'
    packageForLinux: 'dist/'
 The error seems the service connection used here was not existed. To create a proper service connection, follow the below steps. 
Step 1: Create a service principle for the subscription where the application resources are hosted using the below command in the cloud shell.    
$az ad sp create-for-rbac –name <service-principle-name> --scope /subscriptions/<subscription-id>
  
Make note of make note of password, appID and tenantID. 
Step 2: Create a service connection using the service principle just like mentioned in Step 1.
 Go to project settings > service connections > click on create new service connection.  
After filing all the details click on verify to confirm it is valid connection.
.
  
Make sure the azure subscription name above task match with the service connection name

Give Azure DevOps Pipelines access to publish gradle package to Azure DevOps Artifact feed

I am setting up a shared code library and have managed to build artifacts and publish them to a feed using a personal access token, but I want to move the publishing to an Azure Pipeline. I am trying to use the build account with the access token available at build time (System.AccessToken) but I get a 403 error from Azure Artifacts. The [Project] Build Service ([Org]) account is added as a contributor to the feed.
I have a build.gradle.kts file that has the following config (sensitive info replaced):
publishing {
publications {
create<MavenPublication>("private") {
groupId = "[groupId]"
artifactId = "shared"
version = "0.0.1"
artifact("./build/libs/lib.jar")
}
}
repositories {
maven(url = "https://pkgs.dev.azure.com/[Org]/[Project]/_packaging/maven-private/maven/v1") {
name = "maven-private"
credentials {
username = System.getenv("AZURE_DEVOPS_USER")
password = System.getenv("AZURE_DEVOPS_ACCESS_TOKEN")
}
}
}
}
And the Azure pipelines config:
trigger:
- main
pr: none
resources:
- repo: self
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Gradle#2
inputs:
gradleWrapperFile: 'gradlew'
workingDirectory: '$(Build.SourcesDirectory)'
tasks: 'build'
javaHomeOption: 'JDKVersion'
- task: Gradle#2
env:
AZURE_DEVOPS_USER: "[Project] Build Service ([Org])"
AZURE_DEVOPS_ACCESS_TOKEN: $(System.AccessToken)
inputs:
gradleWrapperFile: 'gradlew'
workingDirectory: '$(Build.SourcesDirectory)'
tasks: 'publish'
javaHomeOption: 'JDKVersion'
The publish step fails with the following error:
Execution failed for task ':lib:publishPrivatePublicationToMaven-privateRepository'.
> Failed to publish publication 'private' to repository 'maven-private'
> Could not PUT 'https://pkgs.dev.azure.com/[Org]/[Project]/_packaging/maven-private/maven/v1/[groupId]/shared/0.0.1/shared-0.0.1.jar'. Received status code 403 from server: Forbidden - User '[userId]' lacks permission to complete this action. You need to have 'ReadPackages'.
I have no idea how to proceed with debugging, does anyone have any suggestions?
For the error ' You need to have 'ReadPackages'', you should enter your Azure Artifacts–>your feed–>feed settings–>Permissions。
Depending on your choice scope(organization scope or project scope),
You should click Add Users/groups and search for Project Collection Build Service ({OrgName}) and add as Contributor for organization-level scoped feed.
Or click Add Users/groups and search for {Project Name} Build Service ({Org Name}) and add as Contributor for project-level scoped feed.
Please see the details in the Configure permissions doc.
I hope this could do some help.

Deploy packages to multiple webapp service by azure pipeline single stage

I have more than 100 webapp service in azure. I want to deploy packages in 100 webapps by azure pipeline with one pipeline yml file. But I couldn't find any documentation like this. I got one microsoft documentation and they prefer to increase pipeline steps. If I have 100 webapps service, then have to add 100 steps for each deployment. This is not an efficient way and its time consuming. I want just like this step.
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: 'JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........'
package: '$(build.artifactstagingdirectory)/**/*.zip'
This yaml file is showing error. I couldn't find any essential extensions to fix it. I also couldn't find any azure powershell deployment command regarding to this issue. How can I get the solution?
You will not be able to do this like this. However you can use Azure Cli task:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$apps= #('JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........')
foreach ($app in $apps) {
az webapp deployment source config-zip -g $(group) -n $app --src '$(build.artifactstagingdirectory)/SOME_FOLDER/Artifact.zip'
}
And here you have more details about deployment itself
Annother approach with multiple task bu continuation if one fail is:
parameters:
- name: apps
type: object
default:
- JustGoTestAgain
- justgotesttwo
- and so on
steps:
- ${{ each app in parameters.apps}}:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ app }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ app }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
Thete was issue with space. Now is fine. Apart from that there is only one issue with connectedServiceName
Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
Which I skipped here as you already have it on your solution.

Error when deploying DACPAC via Azure DevOps Pipelines

I am flushing out a CI/CD process with Azure SQL DB deployed via Azure DevOps Pipelines. I am using the Adventure works database and set up a visual studio project importing the schema.
I have a pipeline configured to publish the dacpac and run a subsequent deployment using the SqlAzureDacpacDeployment#1 and am getting the below error:
2020-10-10T02:36:34.1421137Z ##[error]Unable to connect to target server 'server.database.windows.net'. Please verify the connection information such as the server name, login credentials, and firewall rules for the target server.
2020-10-10T02:36:34.1605855Z ##[error]Windows logins are not supported in this version of SQL Server.
2020-10-10T02:36:34.2143924Z ##[error]The Azure SQL DACPAC task failed. SqlPackage.exe exited with code 1.Check out how to troubleshoot failures at https://aka.ms/sqlazuredeployreadme#troubleshooting-
2020-10-10T02:36:34.2522414Z ##[section]Finishing: SqlAzureDacpacDeployment
I am using windows latest and here is my YAML pipeline:
trigger:
- master
pool:
vmImage: 'windows-latest'
jobs:
- job: BuildDeploySQL
variables:
- group: SQLServerLogin
steps:
- task: VSBuild#1
inputs:
solution: '**\*.sln'
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
publishLocation: 'pipeline'
- task: SqlAzureDacpacDeployment#1
inputs:
azureSubscription: 'Subscription Name here'
AuthenticationType: 'server'
ServerName: 'server.database.windows.net'
DatabaseName: 'AdventureWorks'
SqlUsername: 'sqladmin'
SqlPassword: ${{ variables.Password }}
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: '$(Pipeline.Workspace)\s\AdventureWorks\bin\Debug\*.dacpac'
IpDetectionMethod: 'AutoDetect'
I have tried to deploy from my local machine and it is successful using the same sql credentials. Additionally I have confirmed that the SQL Database has allow Azure Services enabled. I have also tried to deploy the dacpac to a new empty database and get this same error.
I do believe this could be just a generic error message as my deployment logs do show a successful connection to the server:
2020-10-10T02:36:18.7912964Z Invoke-Sqlcmd -ServerInstance "server.database.windows.net" -Database "AdventureWorks" -Username "sqladmin" -Password ****** -Inputfile
....
2020-10-10T02:36:33.0554895Z Initializing deployment (Start)
** Update
Just to rule out I did create a new SQL Login with DBO_owner permissions and ran the deployment using that and got the same error message.
Above error is probably because the build agent ip is not allow-listed in the firewall rules of your Azure SQL Database. See the this link about IP ranges for Microsoft-hosted agents.
You can check the firewall rules setting of your azure database, and try allowing all IP ranges.
You can aslo add Azure CLi task to get the agent ip and set a firewall rule for your azure database to allow the agent ip dynamically in your pipeline. See this thread.
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: 'azureSubscription'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$agentIp = (New-Object net.webclient).downloadstring("http://checkip.dyndns.com") -replace "[^\d\.]"
az sql server firewall-rule create -g $(rg) -s $(server) -n test --start-ip-address $agentIp --end-ip-address $agentIp
You can also create a self-hosted agent on your local machine/Azure VM. and run your pipeline on this self-hosted agent. Note to allow-list your local machine ip for the azure database.
The root issue was the password secret contained characters which escaped Powershell. Wrapping the secret in "" resolved it.

Is it possible to use Azure CLI on DevOps (hosted/self-hosted) while connecting with managed identity

I have a small test project with a crude start of a pipeline at https://github.com/veikkoeeva/dockerservice/blob/main/azure-pipelines.yml. It's currently just to check it's possible to connect to the Azure, so
trigger:
- master
pool:
vmImage: windows-latest
steps:
- task: AzureCLI#2
displayName: Az --version
inputs:
azureSubscription: 'TestManagedIdentityConnection'
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
az --version
- task: AzureCLI#2
inputs:
azureSubscription: 'TestManagedIdentityConnection'
scriptType: 'pscore'
scriptLocation: 'scriptPath'
scriptPath: '$(System.DefaultWorkingDirectory)\devops.ps1'
But this fails on login step like so
The service connection scope is on subscription level. It appears the hosted image tries to connect to an Internal Azure token endpoint. Is there a way to use managed identity that can sign in the CLI using hosted images? What could it look like using self-hosted and managed identity?
This seem to work with "the usual" service principal. But it appears developers are often forbidden to create SPNs to company AD so creating a service connections fails. It appears often it's possible to create a service connection using managed identity, but here we are with this problem. :)
<edit: Reading from https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli?view=azure-cli-latest the options could be either az login --identity or a certificate. With az login --identity it appears there is still the same problem of calling the same endpoint as earlier and it errors with the same reason.
<edit 2: Duh! In the image it's called with --identity switch already!

Resources