I am new to Azure Devops and CICD pipeline. I have been tasked to create a CICD pipeline to execute a SQL query from Azure Devops CI/CD pipeline. How do I do that? Any suggestion would be helpful.
SQL queries can be ran via the ADO tasks
Something like:
variables:
AzureSubscription: '<Azure service connection>'
ServerName: '<Database server name>'
DatabaseName: '<Database name>'
AdminUser: '<SQL user name>'
AdminPassword: '<SQL user password>'
SQLFile: '<Location of SQL file in $(Build.SourcesDirectory)>'
steps:
- task: AzurePowerShell#2
displayName: Azure PowerShell script: FilePath
inputs:
azureSubscription: '$(AzureSubscription)'
ScriptPath: '$(Build.SourcesDirectory)\scripts\SetAzureFirewallRule.ps1'
ScriptArguments: '$(ServerName)'
azurePowerShellVersion: LatestVersion
- task: CmdLine#1
displayName: Run Sqlcmd
inputs:
filename: Sqlcmd
arguments: '-S $(ServerName) -U $(AdminUser) -P $(AdminPassword) -d $(DatabaseName) -i $(SQLFile)'
- task: AzurePowerShell#2
displayName: Azure PowerShell script: FilePath
inputs:
azureSubscription: '$(AzureSubscription)'
ScriptPath: '$(Build.SourcesDirectory)\scripts\RemoveAzureFirewallRule.ps1'
ScriptArguments: '$(ServerName)'
azurePowerShellVersion: LatestVersion
Note the extra steps is to remove the firewall if you are utilizing firewall rules
Related
I am deploying a database to an Azure SQL Server with Azure Pipelines. The deployment is working really well. So I added the creation and deployment of a temporary database (one that is removed at the end of the pipeline) for unit testing of my USP's.
I created a unit test .dll with SSDT. I configured the app.config and tested it on my laptop in Visual Studio. On my laptop it works, the unit tests are run an all pass.
I tried to connect to the server with the credentials from the app.config in SSMS. The connection is established.
But, I cant get the pipeline (or hosted agent) task to connect to the database from within the pipeline.
I tried adding the IP of the agent to the server whitelist. I checked the "allow azure services to access the database". These did not solve the problem.
The error I get is this:
Initialization method
BuilEnDeployTest_Unittest.SqlServerUnitTest001.TestInitialize threw
exception. System.Data.SqlClient.SqlException:
System.Data.SqlClient.SqlException: A transport-level error has
occurred when receiving results from the server. (provider: TCP
Provider, error: 0 - An existing connection was forcibly closed by the
remote host.) ---> System.ComponentModel.Win32Exception: An existing
connection was forcibly closed by the remote host.
pipeline yml:
trigger:
- test
name: $(TeamProject)_$(Date:yyyyMMdd)$(Rev:_r)
variables:
- group: "DatabaseDeploy"
- name: databaseName
value: sqldb-AzureDevOps_$(Build.BuildNumber)
#- name: ExecutionContext.__Database__
# value: $(variables.databaseName)
stages:
### Building the code in the Pull-Request
- stage: Build
displayName: Build the project
jobs:
- job: Build
displayName: Build DB project
pool:
vmImage: windows-latest
steps:
- script: echo '$(Build.BuildNumber)'
displayName: BuildName
#- powershell:
# Get-ChildItem -Path $(Agent.WorkFolder)\1\s\BuilEnDeployTest_Unittest -recurse
# displayName: Folder structure
#- script: echo '$(Agent.WorkFolder)\1\s\BuilEnDeployTest_Unittest\app.config'
# displayName: app.config
- powershell:
Get-Content -path $(Agent.WorkFolder)/1/s/BuilEnDeployTest_Unittest/app.config
displayName: app.config
- powershell: >
(Get-Content -path $(Agent.WorkFolder)/1/s/BuilEnDeployTest_Unittest/app.config -Raw)
-replace 'MyDatabase', '$(databaseName)'
-replace 'MyServer', '$(ServerName_unittst).database.windows.net'
-replace 'MyUser', '$(DatabaseUser_unittst)'
-replace 'MyPassword', '$(DatabasePassword_unittst)'| Set-Content -Force -Path $(Agent.WorkFolder)/1/s/BuilEnDeployTest_Unittest/app.config
- powershell:
Get-Content -path $(Agent.WorkFolder)/1/s/BuilEnDeployTest_Unittest/app.config
displayName: app.config
- task: VSBuild#1
displayName: Building the database project
inputs:
solution: '**\*.sln'
- task: CopyFiles#2
displayName: Copy Build Artifacts
inputs:
SourceFolder: '$(agent.builddirectory)\s'
Contents: '**'
TargetFolder: '$(build.artifactstagingdirectory)'
### In the commands below, 'pipeline' is used instead of 'Container', allthough
### 'Container' is often seen in the online documentation.
### I have noticed that changing 'pipeline' to 'Container' breaks my code
### I haven't spend time in figuring out why. It could be because of the path I use
### down the line.
- task: PublishPipelineArtifact#1
displayName: Publish Build Artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'database'
publishLocation: 'pipeline'
#### Unit testing the Pull-Request
- stage: Unit_testing
displayName: Database UnitTest
condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
jobs:
- job: Deploy_to_temporary_database
displayName: Deploy to temporary database
pool:
vmImage: windows-latest
steps:
############ Create the Azure Resources as defined in the AzureSQLARMTemplate.json
## When the Azure SQL server does not exist it is also created by the ARM deployment.
## NOTE: I experienced errors when deleting and recreating the Azure SQL server. This could be
## because of the little time between deletion and creation.
## Deployment through the ARM template can deal with an existing server. In that case
## the server creation is skipped. So best to not delete the Azure SQL server untill the project
## is abandoned.
- task: AzurePowerShell#5
displayName: 'Create unittest DB'
inputs:
azureSubscription: $(connectedServiceNameARM)
ScriptType: 'FilePath'
ScriptPath: './ARMtemplate/CreateAzureSQLAndDB.ps1'
ScriptArguments: -azuresqlserverName $(ServerName_unittst) -sqlserverAdminLogin $(DatabaseUser_unittst) -sqlserverAdminPassword $(DatabasePassword_unittst) -databaseName $(databaseName) -Location 'northeurope' -ResourceGroupName 'rg-temp' -TemplateFile ARMtemplate\AzureSQLARMTemplate.json
azurePowerShellVersion: 'LatestVersion'
############ Download the build artifacts from the build stage
- task: DownloadPipelineArtifact#2
displayName: 'Download artifacts'
inputs:
buildType: 'current'
artifactName: 'database'
targetPath: '$(Pipeline.Workspace)/database'
- powershell:
Get-ChildItem -Path $(Pipeline.Workspace)/database -recurse
displayName: Folder structure
- powershell:
Get-Content -path $(Pipeline.Workspace)/database/a/BuilEnDeployTest_Unittest/bin/Debug/BuilEnDeployTest_Unittest.dll.config
displayName: BuilEnDeployTest_Unittest.dll.config
############ Deploy the database dacpac file to the newly created database
- task: SqlAzureDacpacDeployment#1
displayName: 'Deploy unittest DB'
inputs:
connectedServiceNameARM: $(connectedServiceNameARM)
ServerName: $(ServerName_unittst).database.windows.net
DatabaseName: $(databaseName)
SqlUsername: $(DatabaseUser_unittst)
SqlPassword: $(DatabasePassword_unittst)
DacpacFile: $(Pipeline.Workspace)/database/a/$(dacpacfile_location)
PublishProfile: $(Pipeline.Workspace)/database/a/$(publishprofile_location)
## Create a firewall rule for the IP adress of the build agent
- task: AzurePowerShell#5
displayName: 'Add buildserver public ip'
inputs:
azureSubscription: $(connectedServiceNameARM)
ScriptType: InlineScript
Inline: |
$ip = (Invoke-WebRequest -uri "http://ifconfig.me/ip").Content
$AzureSQLFirewallRule = Get-AzSqlServerFirewallRule -ResourceGroupName 'rg-temp' -ServerName $(ServerName_unittst) -FirewallRuleName "azuredevops"
if ($AzureSQLFirewallRule -eq $null) {New-AzSqlServerFirewallRule -ResourceGroupName 'rg-temp' -ServerName $(ServerName_unittst) -FirewallRuleName "azuredevops" -StartIpAddress $ip -EndIpAddress $ip}
azurePowerShellVersion: 'LatestVersion'
############ Run the unittest
- task: VSTest#2
inputs:
testAssemblyVer2: $(Pipeline.Workspace)/database/a/$(Unittestdll_location)
testConfiguration: 'Debug'
runOnlyImpactedTests: true
runInParallel: false
############ Delete the newly created database
- task: AzurePowerShell#5
displayName: 'Delete unittest DB'
inputs:
azureSubscription: $(connectedServiceNameARM)
ScriptType: 'FilePath'
ScriptPath: './ARMtemplate/CleanupAzureSQLAndDB.ps1'
ScriptArguments: -azuresqlserverName $(ServerName_unittst) -databaseName $(databaseName) -ResourceGroupName 'rg-temp'
azurePowerShellVersion: 'LatestVersion'
## Delete a firewall rule for the IP adress of the build agent
- task: AzurePowerShell#5
displayName: 'remove buildserver public ip'
inputs:
azureSubscription: $(connectedServiceNameARM)
ScriptType: InlineScript
Inline: |
$ip = (Invoke-WebRequest -uri "http://ifconfig.me/ip").Content
Remove-AzSqlServerFirewallRule -ResourceGroupName 'rg-temp' -ServerName $(ServerName_unittst) -FirewallRuleName "azuredevops"
azurePowerShellVersion: 'LatestVersion'
## Deploying the merged result to the database
- stage: Deploy_to_tst
displayName: Deploy
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- job: Deploy
displayName: Deploy
pool:
vmImage: windows-latest
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'current'
artifactName: 'database'
targetPath: '$(Pipeline.Workspace)/database'
- powershell:
#Get-ChildItem -Path $(Pipeline.Workspace)/database -recurse
#Get-ChildItem -Path $(Agent.Workspace) -recurse
- task: SqlAzureDacpacDeployment#1
displayName: 'Deploying the database to Azure'
inputs:
connectedServiceNameARM: $(connectedServiceNameARM)
ServerName: $(Server_tst)
DatabaseName: $(DatabaseName_tst)
SqlUsername: $(DatabaseUser_tst)
SqlPassword: $(DatabasePassword_tst)
DacpacFile: $(Pipeline.Workspace)/database/a/$(dacpacfile_location)
PublishProfile: $(Pipeline.Workspace)/database/a/$(publishprofile_location)
For hosted agent check if you can create a local db with admin username and password. You can use these credentials later to connect to the database.
Normal approach is usually just test if your sql project is deployable. You create local db and nuke it.
For your unit tests you can always keep a developer databae in your release pipeline where you run your test cases.
I've been working on integrating PSRule into an Azure Devops pipeline. The pipeline is succeeding, but the following error: Target object <FileName> has not been processed because no matching rules were found is being displayed for every file in my repository, at the PSRule stage in the pipeline.
Below, I've included the code for my pipeline. Can anyone help me understand where I've gone wrong with implementing PSRule? Custom rules can be defined, but from my understanding, even without defining any, PSRule should run the 270 or so default rules associated with the module.
trigger:
batch: true
branches:
include:
- main
pool:
vmImage: ubuntu-latest
stages:
- stage: Lint
jobs:
- job: LintCode
displayName: Lint code
steps:
- script: |
az bicep build --file $(MainBicepFile)
name: LintBicepCode
displayName: Run Bicep linter
- stage: PSRule
jobs:
- job: PSRuleRun
displayName: PSRule Run
steps:
- task: ps-rule-assert#1
displayName: Analyze Azure template files
inputs:
modules: 'PSRule.Rules.Azure'
- stage: Validate
jobs:
- job: ValidateBicepCode
displayName: Validate Bicep code
steps:
- task: AzureCLI#2
name: RunPreflightValidation
displayName: Run preflight validation
inputs:
azureSubscription: $(ServiceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az deployment group validate \
--resource-group $(ResourceGroupName) \
--template-file $(MainBicepFile) \
- stage: Preview
jobs:
- job: PreviewAzureChanges
displayName: Preview Azure changes
steps:
- task: AzureCLI#2
name: RunWhatIf
displayName: Run what-if
inputs:
azureSubscription: $(ServiceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az deployment group what-if \
--resource-group $(ResourceGroupName) \
--template-file $(MainBicepFile) \
- stage: Deploy
jobs:
- deployment: DeployBicep
displayName: Deploy Bicep
environment: 'Test'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureCLI#2
name: DeployBicepFile
displayName: Deploy Bicep file
inputs:
azureSubscription: $(ServiceConnectionName)
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
set -e
deploymentOutput=$(az deployment group create \
--name $(Build.BuildNumber) \
--resource-group $(ResourceGroupName) \
--template-file $(MainBicepFile))
The warning on by default but not critical to the process after Bicep code has been expanded.
You can disable the warning which shows by default by setting the execution.notProcessedWarning option to false. You can do this in a number of ways however the easiest is to configure it in the ps-rule.yaml option file. See doc for additional options to set this.
Your pipeline configuration is fine however you need to enable expansion for Bicep or parameter files to be processed by the PSRule.Rules.Azure module.
Configuring expansion for Bicep code is done by setting the configuration.AZURE_BICEP_FILE_EXPANSION option to true. This is covered in Using Bicep source.
Hope that helps.
I would want to create a pull request environment in Azure which gets deployed when a pull request is opened. Users can play around in that environment and find bugs. Once the bugs are fixed, and the PR is closed, I would want to delete that environment.
I am following through Sam Learns Azure and I was able to go through most steps, but its all .NET related.
Does anyone have an idea to do the same for a react-app?
I am also in favor of creating a new slot inside the app-service.
This is my modified code:
jobs:
- deployment: DeployWebServiceApp
displayName: "Deploy webservice app"
environment: ${{parameters.environment}}
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifacts'
inputs:
artifactName: "$(Build.BuildId)"
buildType: 'current'
- task: AzureCLI#2
displayName: 'Deploy infrastructure with ARM templates'
inputs:
azureSubscription: "${{parameters.serviceConnection}}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az webapp deployment slot create --name ui-dev-$(prLC)
--resource-group rg-dev
--slot $(prLC)
--configuration-source app-dev
- task: AzureRmWebAppDeployment#3
displayName: 'Azure App Service Deploy: web service'
inputs:
azureSubscription: "${{parameters.serviceConnection}}"
appName: "${{parameters.appServiceName}}"
DeployToSlotFlag: true
ResourceGroupName: '${{parameters.resourceGroupName}}'
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
RemoveAdditionalFilesFlag: true
TakeAppOfflineFlag: true
RenameFilesFlag: true
Why Not. You can do that in below steps-
Create the Build Pipe which build your react app
In the build pipe set a VSTS var only when it is a PR. eg varPR = x
Create a Release pipe which deploys the app
Add one more task at the end of build pipe to call the release using webhook based on condition varPR = x.
Ref - Trigger azure pipeline via webhook?
The link you have shared is also cool that does the job same job in a single build pipe.
I have more than 100 webapp service in azure. I want to deploy packages in 100 webapps by azure pipeline with one pipeline yml file. But I couldn't find any documentation like this. I got one microsoft documentation and they prefer to increase pipeline steps. If I have 100 webapps service, then have to add 100 steps for each deployment. This is not an efficient way and its time consuming. I want just like this step.
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: 'JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........'
package: '$(build.artifactstagingdirectory)/**/*.zip'
This yaml file is showing error. I couldn't find any essential extensions to fix it. I also couldn't find any azure powershell deployment command regarding to this issue. How can I get the solution?
You will not be able to do this like this. However you can use Azure Cli task:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$apps= #('JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........')
foreach ($app in $apps) {
az webapp deployment source config-zip -g $(group) -n $app --src '$(build.artifactstagingdirectory)/SOME_FOLDER/Artifact.zip'
}
And here you have more details about deployment itself
Annother approach with multiple task bu continuation if one fail is:
parameters:
- name: apps
type: object
default:
- JustGoTestAgain
- justgotesttwo
- and so on
steps:
- ${{ each app in parameters.apps}}:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ app }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ app }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
Thete was issue with space. Now is fine. Apart from that there is only one issue with connectedServiceName
Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
Which I skipped here as you already have it on your solution.
parameters:
- name: AzureSubscription
default: 'abc'
- name: BlobName
type: string
default: ""
stages:
- stage: MyStage
displayName: 'My Stage'
variables:
- name: sas
jobs:
- job: ABC
displayName: ABC
steps:
- task: AzureCLI#2
displayName: 'XYZ'
inputs:
azureSubscription: ${{ parameters.AzureSubscription }}
scriptType: pscore
arguments:
scriptLocation: inlineScript
inlineScript: |
$sas=az storage account generate-sas --account-key "mykey" --account-name "abc" --expiry (Get-Date).AddHours(100).ToString("yyyy-MM-dTH:mZ") --https-only --permissions rw --resource-types sco --services b
Write-Host "My Token: " $sas
- task: PowerShell#2
inputs:
targetType: 'filepath'
filePath: $(System.DefaultWorkingDirectory)/psscript.ps1
arguments: >
-Token "????"
-BlobName "${{parameters.BlobName}}"
displayName: 'some work'
In this Azure Devops yaml, i have created 2 tasks. AzureCLI#2 and PowerShell#2
In AzureCLI#2 i get value in $sas varaible. Write-Host confirms that, but $sas does not get passes as parameter to PowerShell#2 powershell file as parameter.
"${{parameters.BlobName}}" is working fine. In powershell i am able to read that value.
How to pass sas variable value?
Tried
-Token $sas # not worked
-Token "${{sas}}" # not worked
Different tasks in Azure Pipeline don't share a common runspace that would allow them to preserve or pass on variables.
For this reason Azure Pipelines offers special logging commands that allow to take string output from a task to update an Azure Pipeline environment variable that can be used in subsequent tasks: Set variables in scripts (Microsoft Docs).
In your case you would use a logging command like this to make your sas token available to the next task:
Write-Host "##vso[task.setvariable variable=sas]$sas"
In the argument of your subsequent task (within the same job) use the variable syntax of Azure Pipelines:
-Token '$(sas)'