Good morning,
I have problems checking out a secure file during the build process in azure devops 2019. My task is defined as:
- task: DownloadSecureFile#1
inputs:
secureFile: 'oimPictureEditor_test'
displayName: 'download configuration'
but it fails with:
2022-12-30T10:10:27.9053899Z ##[section]Starten: download configuration
2022-12-30T10:10:28.0009766Z ==============================================================================
2022-12-30T10:10:28.0010142Z Task : Sichere Datei herunterladen
2022-12-30T10:10:28.0010245Z Description : Hiermit wird eine sichere Datei an einen temporären Speicherort auf dem Agent-Computer heruntergeladen.
2022-12-30T10:10:28.0010357Z Version : 1.151.2
2022-12-30T10:10:28.0010489Z Author : Microsoft Corporation
2022-12-30T10:10:28.0010653Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-secure-file
2022-12-30T10:10:28.0010783Z ==============================================================================
2022-12-30T10:10:28.5506559Z ##[error]Error: unable to get local issuer certificate
2022-12-30T10:10:28.5593478Z ##[section]Abschließen: download configuration
does anyone has any idea how to fix this?
thx in advance
iisiggi
Place your secure files on Azure Pipeline and download it.
*Here are the steps:
Upload the secure file in Library on Pipeline
Download the files in the agent machine with using DownloadSecureFile#1 task
Download the secure files
use the download task DownloadSecureFIle#1 task like below.
- task: DownloadSEcureFile#1
name: <nameof the Task>
inputs:
secureFile: <secure file Name>
The secure file is downloaded to $(Agent.TempDirectory). you can check the path with using the prepared variable such as $(<task name>.secureFIlePath)
Reference taken from MSDoc.
This is a known issue for Azure DevOps Server, and you can try the way below to resolve the issue.
steps:
. . .
- task: PowerShell#2
displayName: 'Set CA Cert'
inputs:
targetType: inline
script: |
if ($env:AGENT_HOMEDIRECTORY -ne $null) { $TargetFolder = $env:AGENT_HOMEDIRECTORY }
else { $TargetFolder = [System.Environment]::GetEnvironmentVariable('TEMP','Machine') }
Get-ChildItem -Path Cert:\LocalMachine\CA | ForEach-Object {
$Cert = "-----BEGIN CERTIFICATE-----`n"
$Cert+= $([System.Convert]::ToBase64String($_.export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert),'InsertLineBreaks'))
$Cert+= "`n-----END CERTIFICATE-----`n"
$Chain+= $Cert
}
$CertFile = "$TargetFolder\TrustedRootCAs.pem"
$Chain | Out-File $CertFile -Force -Encoding ASCII
$Chain = $null
Write-Host "##vso[task.setvariable variable=NODE.EXTRA.CA.CERTS]$CertFile"
- task: DownloadSecureFile#1
displayName: 'download configuration'
inputs:
secureFile: 'oimPictureEditor_test'
. . .
The step 'Set CA Cert' will try to get the CA certificate and set it as the variable "NODE.EXTRA.CA.CERTS" for use.
For more details about this issue and the solution, you can reference the following tickets:
Azure DevOps Server pipeline build fails when using self-signed SSL certificate with "unable to get local issuer certificate" during NuGet restore
https://github.com/microsoft/azure-pipelines-tasks/issues/11508
I put the content of my secret file into a secret variable. That worked for me, but is for sure no general solution.
Related
I am using Azure DevOps to build a python wheel. I want to make it as generic as possible so that everyone in the team can use the same pipeline to build their own python wheels and deploying them in some databricks workspace. For that I need to know what the name of the file(s) in build pipeline output is to use it in my release pipeline.
Currently in this case the file is a python wheel saved in build pipeline output. I am using the following code in my pipeline yaml to publish it both at build pipeline output and an Azure artifact feed.
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: dfordbx
- task: UniversalPackages#0
displayName: Publish
inputs:
command: publish
publishDirectory: $(Build.ArtifactStagingDirectory)/dist/
vstsFeedPublish: 'MyProject/py_artifacts_1732'
vstsFeedPackagePublish: $(packageBuildName)
After build pipeline run ends, there is one wheel file in dist folder. I need to get the name of this wheel. For my own code, I of course know the name. But when others run the pipeline for their code, this is not clear to me. I require to get this name in my release pipeline.
In other words, I am looking for a way in my yaml file to get "py_sample_package-0.6.5-py3-none-any.whl" name in the following structure:
By choosing published artifact:
Getting to the file:
The highlighted part is what I need to get in pipeline. Thank you.
Classic Release Pipelines
In a Classic Release pipeline, you reference a build artifact and use it as a trigger. Artifacts are downloaded to $(System.DefaultWorkingDirectory)\$(Build.DefinitionName).
To identify the .whl file in the artifact, add a powershell script into your release pipeline that identifies the file and creates a new variable using the ##vso logging syntax...
For example:
# find the first .whl file in the artifact folder
$whlFile = Get-ChildItem `
-Filter *.whl `
-Path "$(System.DefaultWorkingDirectory)\$(Build.DefinitionName)\dfordbx" |
ForEach-Object { $_.fullname } |
Select-Object -First 1
# create a variable with the full path to the file
Write-Host "##vso[task.setvariable variable=whlFile]$whlFile"
Now you can use the variable like any other defined pipeline variable $(whlFile)
Multi-Stage YAML Pipelines
If you're using a multi-stage YAML pipeline and need to use the artifact between stages, you can't assume that the file will be present on the machine because each job is potentially running on a different machine. You will need to download the artifact at the start of the job.
variables:
- artifactName: dfordbx
stages:
- stage: build
jobs:
- job: buildJob
steps:
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifactName: $(artifactName)
artifactType: 'pipeline'
# or use the 'publish' alias
- publish: '$(Build.ArtifactStagingDirectory)'
artifact: '$(artifactName)'
- stage: deploy
dependsOn: build
condition: success('build')
jobs:
- job: deployJob
steps:
- task: DownloadPipelineArtifact#1
inputs:
source: 'current'
artifact: $(artifactName)
path: '$(Pipeline.Workspace)/$(artifactName)'
# or use the 'download' alias
- download: current
artifact: $(artifactName)
- pwsh: |
$whlFile = Get-ChildItem -Path "$(Pipeline.Workspace)/$(artifactName)" -Filter *.whl |
ForEach-Object { $_.fullName } |
Select-Object -First 1
Write-Host "##vso[task.setvariable variable=whlFile]$whlFile"
I currently have an azure build pipeline that needs to access two different Key Vaults. Unfortunately both of the secrets I am trying to access have a name of SQLUserName. I am trying to pass these as arguments to a python script. I am looking for a way that I could qualify or differentiate between the secrets when passing the arguments.
Ideally I would like to access the variable qualified something like $(ServiceConnection1.SQLUserName) But I can't find any information on this.
I have been researching a way to rename a variable so I could possibly run the first Key Vault task then rename $(SQLUserName) to $(SQLUserNamefoo) then run the second Key Vault task and rename to $(SQLUserName) to $(SQLUserNamebar). I can't seem to find anyway to rename a variable in YML.
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: 'ServiceConnection1'
KeyVaultName: 'Vault1'
SecretsFilter: '*'
RunAsPreJob: true
- task: AzureKeyVault#1
inputs:
azureSubscription: 'ServiceConnection2'
KeyVaultName: 'Vault2'
SecretsFilter: '*'
RunAsPreJob: true
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- task: PythonScript#0
inputs:
scriptSource: 'filePath'
scriptPath: 'keyVaultTest.py'
arguments: '$(SQLUserName))'
#ideal way to work
arguments: '$(SQLUserName1) $(SQLUserName2))'
Azure DevOps accessing two Key Vaults with duplicate secret names
We could add a Inline powershell task with Logging Command to set the variable SQLUserNamefoo with value $(SQLUserName) after the first AzureKeyVault task.
Write-Host ("##vso[task.setvariable variable=SQLUserNamefoo]$(SQLUserName)")
Then we could use the $(SQLUserNamefoo) in the next tasks.
And we could set the another Inline powershell task to set the variable SQLUserNamebar with value $(SQLUserName) after the second AzureKeyVault task
Write-Host ("##vso[task.setvariable variable=SQLUserNamebar]$(SQLUserName)")
As test, I created a Key Vault SQLUserName with value Leotest. In order to verify the SQLUserNamefoo is set to $(SQLUserName), I defined SQLUserNamefoo in the Variables with value 123:
And add another powershell task to output the value of SQLUserNamefoo to a txt file to verify it:
cd $(System.DefaultWorkingDirectory)
New-Item $(System.DefaultWorkingDirectory)\temp -type directory
cd temp
New-Item a.txt -type file
write-output $(SQLUserNamefoo)| out-file -filepath $(System.DefaultWorkingDirectory)\temp\a.txt
The result of txt file:
I'm trying to use azure pipeline to upload certificate and binding the app service.
First I use a DEV-stage,all works well.Currently I have to create a new stage for QUAL env.Just clone a new stage from DEV-stage and update the variables,but we run the pipeline can not find the certificate(file) I uploaded.
My download task is:
steps:
- task: DownloadSecureFile#1
displayName: 'Download ***.**.com Certificate for API App'
inputs:
secureFile: dev.pfx
and then use a azure powershell task，but in my script such error happens:
Certificate does not exist at path D:\a\_temp/
It seems can not find the download file in the agent.
Uploaded task:
steps:
- task: AzurePowerShell#3
displayName: 'Upload Certificate to API app and Bind Domain'
inputs:
azureSubscription: 'Azure: CDA NextGen DEV'
ScriptPath: '$(System.DefaultWorkingDirectory)/CdaApi-ArmTemplates/ArmTemplates/InstallSSLAndCustomDomain.ps1'
ScriptArguments: '-ResourceGroupName $(ResourceGroupName) -AppServiceName $(ApiSiteName) -CustomDomains $(ApiHostName) -CertificatePassword $(Password) -CertificateFileName $(CertificateFileName)'
azurePowerShellVersion: LatestVersion
power shell script:
$CertificateFilePath = $env:AGENT_TEMPDIRECTORY + "/" + $CertificateFileName
$ResourceGroupName -ResourceType Microsoft.Web/sites -ApiVersion 2014-11-01
if ([System.IO.File]::Exists($CertificateFilePath))
{
Write-Host ("Certificate found at {0}" -f $CertificateFilePath)
}
else
{
Write-Error ("Certificate does not exist at path {0}" -f $CertificateFilePath)
throw
}
How to check it?
Updated:
Based on your comment the files has been exists there. Also, combine your powershell script and your error message.
Since you just share the part of your YAML, I could not know how do you define variables. Please ensure your CertificateFileName variable has been stored and passed to powershell successfully.
Because, the complete file name should be displayed in your powershell error message even it does not exists in path.
In fact, it is very easy to cause some issue after you change the agent environment used.
After the Download secure file executed, it will generated one environment variable which name is secureFilePath. You just need set is as output variable and use it directly in your powershell script.
Little changes on your YAML and powershell script:
YAML:
steps:
- task: DownloadSecureFile#1
displayName: 'Download ***.**.com Certificate for API App'
inputs:
secureFile: dev.pfx
name: Path
- task: AzurePowerShell#3
displayName: 'Upload Certificate to API app and Bind Domain'
inputs:
azureSubscription: 'Azure: CDA NextGen DEV'
ScriptPath: '$(System.DefaultWorkingDirectory)/CdaApi-ArmTemplates/ArmTemplates/InstallSSLAndCustomDomain.ps1'
ScriptArguments: '-ResourceGroupName $(ResourceGroupName) -AppServiceName $(ApiSiteName) -CustomDomains $(ApiHostName) -CertificatePassword $(Password) -CertificateFileName $(CertificateFileName) -SecureFilePath $(Path.secureFilePath)'
azurePowerShellVersion: LatestVersion
Powershell:
$CertificateFilePath = $SecureFilePath
$ResourceGroupName -ResourceType Microsoft.Web/sites -ApiVersion 2014-11-01
if ([System.IO.File]::Exists($CertificateFilePath))
{
Write-Host ("Certificate found at {0}" -f $CertificateFilePath)
}
else
{
Write-Error ("Certificate does not exist at path {0}" -f $CertificateFilePath)
throw
}
I have a published a build artifact published to $(Build.ArtifactStagingDirectory)/drop with artifactName "some_sidebar" and the artifact publish location is Azure Pipeline.
How can I retrieve that artifact now inside my release Pipeline if I have only a PowerShell Skript in release task?
here is the code specific part:
$path = ".\_some_sidebar\drop"
#$path = $(Build.Repository.LocalPath)
$SPFolderName = "Style Library/_some_sidebar";
# Upload template list
$status = "Uploading template list to Location: " + $SPFolderName
Write-Host $status
$te = Add-PnPFile -Path $path"\some_sidebar.js" -Folder $SPFolderName -Checkout
Set-PnPFileCheckedIn -Url $te.ServerRelativeUrl
I get the following error:
Uploading template list to Location: Style Library/_some_sidebar
2020-01-16T09:51:20.5062033Z Add-PnPFile : Local file was not found.
2020-01-16T09:51:20.5062546Z At D:\_work\_temp\6d682160-e8a7-4c56-ab30-7ff8c40f2958.ps1:51 char:7
2020-01-16T09:51:20.5062832Z + $te = Add-PnPFile -Path $path"\some_sidebar.js" -Folder $SPFolderName ...
I assume the build artifact path in azure pipeline is some path in the Virtual machine... but I don't know how to specify that path inside the shell script, or what that path is anyway...?
Azure DevOps: How to retrieve a build artifact from build Azure Pipeline from a PowerShell Script in Release Pipeline?
There are three questions in your post that cause this issue.
First, since you select the artifact publish location is Azure Pipeline, you could not set the targetPath. You could check the document Publish Build Artifacts task:
I assume what you said should be set the pathtoPublish to $(Build.ArtifactStagingDirectory)/drop with artifactName "some_sidebar"like:
But this pathtoPublish is used to set The folder or file path to publish, in other words, it is the artifact source location, not the target.
So, we do not need to use the \drop in the powershell scripts to get the artifact.
Second, MS provides a series of Release variables so that we can use them directly.
You could use the System.DefaultWorkingDirectory, System.ArtifactsDirectory or Agent.ReleaseDirectory:
So, we could use one of above three variables in the powershell scripts to get the artifact, but the variable not the full path to the file, it is the path for the artifact in the release pipeline, we need to do one more step.
Third, when you use release pipeline to get the artifact, which will set the artifact to the folder contain the Source alias:
As test, I create a sample with following powershell scripts:
$path = "$(System.DefaultWorkingDirectory)\<SourceAliasVlaue>\<AartifactName>"
#$path = $(Build.Repository.LocalPath)
$SPFolderName = "Style Library/_some_sidebar";
# Upload template list
$status = "Uploading template list to Location: " + $SPFolderName
Write-Host $status
Get-ChildItem -Path $path
I use the powershell scripts Get-ChildItem -Path $path to list the file in the artifact:
Now, I could get artifact file some_sidebar.js in the powershell task.
Note: You could try to use the wildcard to get the artifact, like:
$te = Add-PnPFile -Path "$(System.DefaultWorkingDirectory)\**\some_sidebar.js"
Hope this helps.
You should be able to use System.ArtifactsDirectory.
Here are my pipeline with example how I use the artifact from previous step. Same variable should be possible to use in powershell script.
(This example is from a yaml pipeline for build and release.)
stages:
- stage: build
displayName: 'Build and package solution'
jobs:
- job: buildsteps
displayName: 'Steps to build and package'
pool: 'PrivateVS2017'
steps:
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.SourcesDirectory)/Web'
includeRootFolder: true
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
replaceExistingArchive: true
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'it-service-wiki-build'
publishLocation: 'Container'
- stage: deploy_to_development
displayName: 'Deploy to development Environment'
dependsOn: build
jobs:
- deployment: deploy
displayName: 'Deploy the solution to Dev'
pool: 'PrivateVS2017'
environment: 'ITServiceWiki-Dev'
strategy:
runOnce:
deploy:
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
buildVersionToDownload: 'latest'
downloadType: 'single'
ArtifactName: 'it-service-wiki-build'
downloadPath: '$(System.ArtifactsDirectory)'
- task: ExtractFiles#1
inputs:
archiveFilePatterns: '../a/**/$(Build.BuildId).zip'
destinationFolder: '$(Build.DefaultWorkingDirectory)/$(Build.BuildId)'
cleanDestinationFolder: true
Note the ../a/**/ when searching for the zip after downloading the artifact. Not sure if /a/ is the same om all build agents can prob use system.artifactsDirectory here too.
I have a vue spa application that I host in azure. However, I am not able to get it running as a build pipeline after setting it up in Azure DevOps
The npm install and npm run build work perfectly, however, the script to copy my dist directory to my blob store fails.
Here is what I tried and the results. Does anyone have experience with this?
AzureFileCopy
- task: AzureFileCopy#3
inputs:
SourcePath: '$(System.DefaultWorkingDirectory)/dist'
azureSubscription: '[my subscription details]'
Destination: 'AzureBlob'
storage: 'mystorageaccountname'
ContainerName: '$web'
Result
##[section]Starting: AzureFileCopy
==============================================================================
Task : Azure file copy
Description : Copy files to Azure Blob Storage or virtual machines
Version : 3.1.11
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
==============================================================================
##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\AzureRM\2.1.0\AzureRM.psd1 -Global
##[warning]The names of some imported commands from the module 'AzureRM.Websites' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
##[warning]The names of some imported commands from the module 'AzureRM' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\AzureRM.Profile\2.1.0\AzureRM.Profile.psm1 -Global
##[command]Add-AzureRMAccount -ServicePrincipal -Tenant *** -Credential System.Management.Automation.PSCredential -EnvironmentName AzureCloud
##[command] Set-AzureRmContext -SubscriptionId dc6a0ce7-adcd-49fd-ad85-e1c082994145 -TenantId ***
Uploading files from source path: 'D:\a\1\s\dist' to storage account: 'mystorageaccountname' in container: '$web' with blob prefix: ''
##[command] & "AzCopy\AzCopy.exe" /Source:"D:\a\1\s\dist" /Dest:"https://mystorageaccountname.blob.core.windows.net/`$web" /#:"D:\a\_temp\ead7e7cf-0b6e-4b16-928f-c84cf3e3a7ab" /XO /Y /SetContentType /Z:"AzCopy" /V:"AzCopy\AzCopyVerbose_ae491d97-a7a8-44e6-b7b0-4b932a5e6c08.log" /S
[2019/06/13 00:31:08][ERROR] Error parsing source location "D:\a\1\s\dist": Failed to enumerate directory D:\a\1\s\dist\ with file pattern *. The system cannot find the path specified. (Exception from HRESULT: 0x80070003) For more details, please type "AzCopy /?:Source" or use verbose option /V.
##[error]Upload to container: '$web' in storage account: 'mystorageaccountname' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
##[section]Finishing: AzureFileCopy
Thanks to the comment suggesting I run a "dir". It actually didn't work. Upon inspection, the following command was not even building the app.
- script: |
npm install
npm run build
dir
displayName: 'npm install and build'
I presume this is because I changed the agent to run on windows (I discovered earlier that AzureFileCopy only runs on windows) and windows agent does not allow stacked scripts the way the ubuntu agent does. So I split the install and build into separate tasks and now it runs with only verb warnings. Here is the working script:
pool:
vmImage: 'vs2017-win2016'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- script: |
npm run build
displayName: 'npm run build'
- script: |
dir
displayName: 'list cwd contents (verify build)'
- task: AzureFileCopy#3
inputs:
SourcePath: '$(System.DefaultWorkingDirectory)/dist'
azureSubscription: '[my subscription details]'
Destination: 'AzureBlob'
storage: 'mystorageaccountname'
ContainerName: '$web'