My pipeline is stuck on deployment job with message The agent request is not running because all potential agents are running other requests. Current position in queue: 1
This message is wrong. If I remove the SqlAzureDacpacDeployment task below and if I only keep the simple echo command, it will execute just fine. So the problem has something to do with the SQL task. Only there is no log and I can't figure out what's wrong.
I have triple checked all parameters and they are correct. I run the build on a dedicated VM and I have added the VM IP Address to the Firewall rules of SQL Server.
What else can I do to troubleshoot this problem ?
- script:
echo "hello"
- task: SqlAzureDacpacDeployment#1
displayName: 'Azure SQL InlineSqlTask'
inputs:
azureSubscription: 'mysubscription id'
ServerName: my-server.database.windows.net
DatabaseName: mydb
SqlUsername: admin
SqlPassword: my password
deployType: sqlTask
SqlFile: $(workFolder)/Scripts.sql
Related
Is there such a thing in Azure Release Pipelines (YAML) as a while loop? My use case is that I am using runOnce strategy to deploy artifacts to a clean environment, deploy a clients data and before I move onto the next client I need to run a query to ensure all the processing is finished and health checks are done. All checks can be done via SQL scripts into an Azure SQL Database and eventually I need to compare the results and task timings against an expected set.
i.e Does processing the client data across branches yield the expected results and timings.
Might be a square peg round hole so happy to use a different approach if there is an easier way.
- deployment : Install
pool:
vmImage: ubuntu-latest
environment:
name: 'Test_Env'
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
# Remove and re-create blank database on the Elastic Pool.
- task: SqlAzureDacpacDeployment#1
displayName: Drop DB
inputs:
azureSubscription: 'Azure'
AuthenticationType: 'server'
ServerName: '$(DB_SERVER)'
DatabaseName: 'master'
SqlUsername: '$(DB_USERNAME)'
SqlPassword: '$(DB_PASSWORD)'
deployType: 'InlineSqlTask'
SqlInline: |
IF EXISTS (SELECT name FROM master.sys.databases WHERE name = N'$(DB_DATABASE)') DROP DATABASE [$(DB_DATABASE)]
IpDetectionMethod: 'AutoDetect'
- task: SqlAzureDacpacDeployment#1
displayName: Create DB
inputs:
azureSubscription: 'Azure'
AuthenticationType: 'server'
ServerName: '$(DB_SERVER)'
DatabaseName: 'master'
SqlUsername: '$(DB_USERNAME)'
SqlPassword: '$(DB_PASSWORD)'
deployType: 'InlineSqlTask'
SqlInline: |
CREATE DATABASE $(DB_DATABASE) ( SERVICE_OBJECTIVE = ELASTIC_POOL (name = [SQL_ElasticPool] ));
IpDetectionMethod: 'AutoDetect'
- task: CmdLine#2
displayName: Install Product
inputs:
script: |
start /wait msiexec.exe /i "$(System.ArtifactsDirectory)\installer.msi" client_data= $(client_data) DB_USERNAME=$(DB_USERNAME) DB_PASSWORD=$(DB_PASSWORD)
workingDirectory: $(System.ArtifactsDirectory)
- task: CmdLine#2
displayName: Start Service
inputs:
script: |
sc start $(WIN_SERVICE)
# This is where I would want a while-loop
- task: SqlAzureDacpacDeployment#1
displayName: Check if processing finished
inputs:
azureSubscription: 'Azure'
AuthenticationType: 'server'
ServerName: '$(DB_SERVER)'
DatabaseName: '$(DB_DATABASE)'
SqlUsername: '$(DB_USERNAME)'
SqlPassword: '$(DB_PASSWORD)'
deployType: 'InlineSqlTask'
SqlInline: |
select 1 from eventlog if complete = 0
IpDetectionMethod: 'AutoDetect'
Azure Devops YAML Pipelines - While loop?
You could create a template which will have a set of Check if processing finished task, and pass loop time as parameters across during your build, like:
- template: CheckProcessingFinished.yaml
parameters:
param: ["1","2","3"]
CheckProcessingFinished.yaml:
parameters:
param : []
steps:
- ${{each Looptimes in parameters.param}}:
- task: SqlAzureDacpacDeployment#1
displayName: Check if processing finished
inputs:
azureSubscription: 'Azure'
AuthenticationType: 'server'
ServerName: '$(DB_SERVER)'
DatabaseName: '$(DB_DATABASE)'
SqlUsername: '$(DB_USERNAME)'
SqlPassword: '$(DB_PASSWORD)'
deployType: 'InlineSqlTask'
SqlInline: |
select 1 from eventlog if complete = 0
IpDetectionMethod: 'AutoDetect'
- task: PowerShell#2
inputs:
targetType: 'Sleep 30 seconds'
script: |
Start-Sleep 30
You could check the document Solving the looping problem in Azure DevOps Pipelines for some more details.
In YAML we use each keyword, and it is an equivalent of the for loop .
You cannot run a task until a condition is met , you can run multiple
task either by writing them one by one or by using the each
statement which is equivalent to a for loop.
But you can use a GATE which is a way to control a deployment. They are predominantly used for heath checkups of infrastructure, external approvals for deployment, etc.
The gates can either be at the start of the pipeline or at the end of
the pipeline.
Gates are of many types like the ones which use invokes functions and azure monitors and some gates use rest api too. Also you can create your custom gates too.
Gates will continue to check for a user specified condition until it is met.
So, you can use a azure function gate run the required scripts and return the results in the gate for validation.
Refence:
Azure Pipeline Gate
Pushing and pulling of image to Azure Container Registry task in Azure DevOps pipeline fails. When tried to pull or push from local system, there's no problem but when tried to do it using the Azure Devops pipeline it fails. Docker login was successful but it fails when I want to pull the image from the ACR with the following result:
**Error response from daemon: Head "***/a2/abcd/manifest/latest": unauthorized: Invalid clientid or client secret.
##[error]Bash exited with code '1'.
##[debug]Processed: ##vso[task.issue type=error;]Bash exited with code '1'.
I checked all the service connections in Az Devops, and they all look correctly configured. Checked the associated service principals as well if they have AcrPull and AcrPush permissions, all of them are in place. Just couldn't understand what's going wrong.
My Yaml looks like this:
trigger: none
schedules:
- cron: "0 0 0 * *"
displayName: **** *
branches:
include:
- abcd
always: true
pool:
vmImage: 'ubuntu-latest'
variables:
- name: acrname
value: *****.azurecr.io
stages:
- stage: abcd
displayName: "pull images from acr"
jobs:
- job: abcdef
displayName: "abcdef"
pool:
vmImage: ubuntu-latest
steps:
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: '*****.azurecr.io'
- bash: |
docker pull $(acrname)/abc-def:latest
docker pull $(acrname)/igh-jkl:latest
name: pull
displayName: 'pull acr images'
Can anyone help?
I had the same issue with the ACR service connection was expired. So I had to create a new service connection by using these steps.
Docker ID and Docker Password can be obtained from ACR --> Settings --> Access keys
Update your pipeline with this new service connection and you are good to go. Please rate if this solution helps you.
In my case, when I ran into this issue, the simple and clean resolution was to use the docker login. In your situation, it looks like this would be a good solution :
docker login $(acrname)
prior to your calls to get your images
docker pull $(acrname)/abc-def:latest
In my case, the docker login password expired. So I have to do the following:
Go azure and generate the new password for the docker app authentication.
Copy the newly generated password.
Go to your virtual machine where docker is running.
Try the below command
docker login blah.azurecr.io --username your-user-name-here --password yourhaspasswordhere~5Crf9b
Now you are good to go.
I have my normal and working release pipeline that, by given a certain deployment group, performs some tasks:
Copies a script
Executes that powershell script (on the target machines defined in the Deployment Group)
Deletes the script
I know that YAML doesn't support deployment groups, but (lucky me!) so far my deployment group has only one machine, let's call it MyTestVM .
So what I am trying to achieve mainly is simply executing a powershell script on that vm . Normally, what happenes with the release pipeline, is that you have a tentacle/release agent installed on the VM, your deployment target (which is inside the Deployment Group) is hooked up to that, and your release pipeline (thanks to the Deployment Group specification) is able to use that release agent on the machine and do whatever it wants on the VM itself.
I need the same... but through YAML ! I know there is PowerShellOnTargetMachines command available in YAML but I don't want to use that. It uses PSSession, it requires SSL certificates and many other things. I just want to use the already existing agent on the VM !
What I have in place so far:
pool: 'Private Pool'
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: 'blahblah'
definition: 'blah'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
- task: CopyFiles#2
displayName: 'Copy Files to: C:\TestScript'
inputs:
SourceFolder: '$(Pipeline.Workspace)/Scripts/'
Contents: '**/*.ps1'
TargetFolder: 'C:\TestScript'
CleanTargetFolder: true
OverWrite: true
The first part just downloads the Artifact containing my script. And then to be honest I am not even sure I need to copy the script in the second part.. first because I don't think it is copying the script to the VM target workspace, but it is copying it to the VM where the azure pipeline agent is installed. And second: I think I can just reference it from my artifact.. but this is not the important part.
How can I make my YAML pipeline make use of the release agent installed on the VM in the same way that a normal release pipeline does?
Reached somehow a solution. First of all worth mentioning that since deployment groups don't work with YAML pipelines the way to proceed is to create an Environment and add as resource your target VM.
So I didn't need to create my own hosted agent or anything special since the problem was the target itself and not the agent running the pipeline.
By creating an Environment and adding a resource (in my case a VM) to that environment, we create also a new release agent on the target itself. So my target VM will now have 2 release agents: the old one that can be used by normal release pipelines, and the new one, attached to the Environment resource on Azure Devops that can be used by YAML pipelines.
Now I am finally able to hit my VM:
- stage: PerformScriptInVM
jobs:
- deployment: VMDeploy
pool:
vmImage: 'windows-latest'
# watch out: this creates an environment if it doesn’t exist
environment:
name: My Environment Name
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: 'blahblahblah'
definition: 'blah'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
- task: PowerShell#2
displayName: 'PowerShell Script'
inputs:
targetType: filePath
filePath: '$(Pipeline.Workspace)/Scripts/TestScript.ps1'
arguments: 'whatever your script needs..'
To get the job to run on the specific release agent you want, you can do two things:
Create a pool and only put your release agent into it.
pool: 'My Pool with only one release agent'
Use an existing pool, and publish/demand a capability for your agent.
On the agent machine itself, add a system environment variable (for example, MyCustomCapability. Give it a value like 1
then your pipeline becomes:
pool:
name: 'My pool with potentially more than one agent'
demands: 'MyCustomCapability'
If only this agent has this environment variable set, then only this agent can execute the job
Hello all
I've added all the IP range listed in https://devblogs.microsoft.com/devops/new-ip-firewall-rules-for-azure-devops/ into my firewall. But still im not able to run powershell command from azure devops pipeline YAML to my on-premises windows server. Every time im getting new IP from microsoft, and that are not listed in above mentioned site.
Im able to connect to server with my local machine using below powershell command
below is my yaml pipeline script
pool:
vmImage: 'windows-latest'
stages:
- stage: 'Deploy'
jobs:
- deployment: 'Deploy'
environment: 'stage'
strategy:
runOnce:
deploy:
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$so = New-PsSessionOption –SkipCACheck -SkipCNCheck
$username='myUserName'
$password='myPassword'
$secpasswd = ConvertTo-SecureString $password -AsPlainText -Force
$credentials = New-Object System.Management.Automation.PSCredential($username, $secpasswd)
Enter-PSSession -ComputerName xyz.com -Credential $credentials -UseSSL -SessionOption $so
write-output 'host name is '+$(hostname)
Im getting error :
Enter-PSSession : Connecting to remote server xyz.com failed with the following error message : WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet. For more information, see the about_Remote_Troubleshooting.
Is it issue releated to firewall or my approach is wrong to execute Powershell script using azure devops?
Your on-premises remote server is not accessible to Microsoft cloud agents. You cannot connect to your on-premises remote server if you run your pipeline on Microsoft cloud agents. Unless your on-premises remote server can be accessed from public network.
Since you are able to connect to on-premise server with your local machine. You can create a self-hosted agent on your local machine. And run your deployment job on this self-hosted agent by specify the pool to your local agent pool for the deployment job. Then you will be able to connect to the remote local server. See below
- deployment: Deploy
displayName: deploy Web App
#specify the local agent pool to run deployment job on self-hosted agent
pool: default
environment: stage
strategy:
I am flushing out a CI/CD process with Azure SQL DB deployed via Azure DevOps Pipelines. I am using the Adventure works database and set up a visual studio project importing the schema.
I have a pipeline configured to publish the dacpac and run a subsequent deployment using the SqlAzureDacpacDeployment#1 and am getting the below error:
2020-10-10T02:36:34.1421137Z ##[error]Unable to connect to target server 'server.database.windows.net'. Please verify the connection information such as the server name, login credentials, and firewall rules for the target server.
2020-10-10T02:36:34.1605855Z ##[error]Windows logins are not supported in this version of SQL Server.
2020-10-10T02:36:34.2143924Z ##[error]The Azure SQL DACPAC task failed. SqlPackage.exe exited with code 1.Check out how to troubleshoot failures at https://aka.ms/sqlazuredeployreadme#troubleshooting-
2020-10-10T02:36:34.2522414Z ##[section]Finishing: SqlAzureDacpacDeployment
I am using windows latest and here is my YAML pipeline:
trigger:
- master
pool:
vmImage: 'windows-latest'
jobs:
- job: BuildDeploySQL
variables:
- group: SQLServerLogin
steps:
- task: VSBuild#1
inputs:
solution: '**\*.sln'
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
publishLocation: 'pipeline'
- task: SqlAzureDacpacDeployment#1
inputs:
azureSubscription: 'Subscription Name here'
AuthenticationType: 'server'
ServerName: 'server.database.windows.net'
DatabaseName: 'AdventureWorks'
SqlUsername: 'sqladmin'
SqlPassword: ${{ variables.Password }}
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: '$(Pipeline.Workspace)\s\AdventureWorks\bin\Debug\*.dacpac'
IpDetectionMethod: 'AutoDetect'
I have tried to deploy from my local machine and it is successful using the same sql credentials. Additionally I have confirmed that the SQL Database has allow Azure Services enabled. I have also tried to deploy the dacpac to a new empty database and get this same error.
I do believe this could be just a generic error message as my deployment logs do show a successful connection to the server:
2020-10-10T02:36:18.7912964Z Invoke-Sqlcmd -ServerInstance "server.database.windows.net" -Database "AdventureWorks" -Username "sqladmin" -Password ****** -Inputfile
....
2020-10-10T02:36:33.0554895Z Initializing deployment (Start)
** Update
Just to rule out I did create a new SQL Login with DBO_owner permissions and ran the deployment using that and got the same error message.
Above error is probably because the build agent ip is not allow-listed in the firewall rules of your Azure SQL Database. See the this link about IP ranges for Microsoft-hosted agents.
You can check the firewall rules setting of your azure database, and try allowing all IP ranges.
You can aslo add Azure CLi task to get the agent ip and set a firewall rule for your azure database to allow the agent ip dynamically in your pipeline. See this thread.
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: 'azureSubscription'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$agentIp = (New-Object net.webclient).downloadstring("http://checkip.dyndns.com") -replace "[^\d\.]"
az sql server firewall-rule create -g $(rg) -s $(server) -n test --start-ip-address $agentIp --end-ip-address $agentIp
You can also create a self-hosted agent on your local machine/Azure VM. and run your pipeline on this self-hosted agent. Note to allow-list your local machine ip for the azure database.
The root issue was the password secret contained characters which escaped Powershell. Wrapping the secret in "" resolved it.