Anyone with experience with PowerShell scripts in an Azure pipeline? Running into an issue where the command
steps:
- checkout: self
persistCredentials: true
- task: PowerShell#2
displayName: "Do some PowerShelling"
inputs:
targetType: 'inline'
script: |
$pattern = "\d+.\d+.\d+"
$s = (git branch -r | Select-String "origin\/release\/name\-[0-9.]+")
$versions = [regex]::Matches($s, $pattern).Value
Works as intended locally, the $s variable contains the results of the git command, but in the Azure DevOps task it does not evaluate the git command. The task fails because the $s is null.
Anyone know what I am doing wrong and how to fix it?
Turns out that PowerShell in Azure YAML can run the git function perfectly fine, the problem was that git branch -r didn't return any results when executed in Azure. Switch it out with git ls-remote did the job.
Related
I'm working with azure pipeline to checkout source code from azure repo and execute setup of inbuilt script to which is provided by webmethod SAG, Using build.yaml i can able to build my application but not able to publish the artifacts.
cat build.yaml
trigger:
- devops-build
pool:
name: CICD
steps:
# Create Target Directory to keep git repo for later use
- bash: |
mkdir -p /home/user/cicd_source/dev/packages/packages
displayName: 'create directory'
- bash: |
echo "webname=${{parameters.projectName}}" > $(Build.ArtifactStagingDirectory)/devpackagename.properties
echo "BuildNumber=$(Build.BuildNumber)" > $(Build.ArtifactStagingDirectory)/devBuildNumber.txt
Above script will create devpackagename.properties and devBuildNumber.txt following path inside my self hosted agent directory work location.
pwd
/home/user/agent/CICD/_work/1/a
ls -lrt
devpackagename.properties
devBuildNumber.txt
cat devpackagename.properties
webname=package
cat devBuildNumber.txt
BuildNumber=20221004.83
After ran the successful pipeline i don't see any artefacts published inside my pipeline
after your build steps add below task
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifact: 'drop'
publishLocation: 'pipeline'
you would see artifact get published on the pipeline
This is the reference doc I have followed to set up the Azure pipeline
https://medium.com/adessoturkey/owasp-zap-security-tests-in-azure-devops-fe891f5402a4
below i am sharing screenshort of the pipeline failed:
Could you please help here to resolve the issue I have exactly followed the medium article to implement the task....
Those who aware on this could you please share your taughts.
This is the pipeline script i am using.
trigger: none
stages:
stage: 'buildstage'
jobs:
job: buildjob
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- checkout: owasap-zap
bash: "docker run -d -p 80:80 nginx:1.14.2"
displayName: "App Container"
bash: |
chmod -R 777 ./
docker run --rm -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-full-scan.py -t http://$(ip -f inet -o addr show docker0 | awk '{print $4}' | cut -d '/' -f 1):80 -x xml_report.xml
true
displayName: "Owasp Container Scan"
- displayName: "PowerShell Script"
powershell: |
$XslPath = "owasp-zap/xml_to_nunit.xslt"
$XmlInputPath = "xml_report.xml"
$XmlOutputPath = "converted_report.xml"
$XslTransform = New-Object System.Xml.Xsl.XslCompiledTransform
$XslTransform.Load($XslPath)
$XslTransform.Transform($XmlInputPath, $XmlOutputPath)
displayName: "PowerShell Script"
task: PublishTestResults#2
displayName: "Publish Test Results"
inputs:
testResultsFiles: converted_report.xml
testResultsFormat: NUnit
# task: PublishTestResults#2
stage: buildstage
According to the YAML file, you want to checkout multiple repositories in your pipeline, but it seems you haven't define a repository resource like mentioned in the document you shared.
resources:
repositories:
- repository: <repo_name>
type: git
name: <project_name>/<repo_name>
ref: refs/heads/master
And according to the screenshot you shared, you only checkout out one repo. Which cause the location of file xml_to_nunit.xslt is different from owasp-zap/xml_to_nunit.xslt. If you only checkout one repo, the location of xml_to_nunit.xslt should be current directory, thus, just define $XslPath in the PowerShell script as "xml_to_nunit.xslt".
Edit
If the repository that contain "xml_to_nunit.xslt" file is in the same organization as the repository run for your pipeline, you need to checkout the repository by using Inline syntax checkout like below or define repository resource.
- checkout: git://MyProject/MyRepo # Azure Repos Git repository in the same organization
You could also add one more command ls before the PowerShell script to list the files in current directory. Aim to figure out where is "xml_to_nunit.xslt".
Task is running on Node.
Part of the yaml file for the pipeline is:
steps:
- script: |
#!/bin/bash
./plan.sh
displayName: "example deploy"
continueOnError: false
Now, when sometimes the ./plan.sh script fails: but it still shows as a success (green tick) in the pipeline. See below:
How do I make it show a "failed" red cross whenever it fails?
What you are doing now is actually calling the bash script file from PowerShell. Your way of writing it cannot capture the error message. The correct way is as follows:
plan2.sh
xxx
pipeline YAML definition:
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- script: |
bash $(System.DefaultWorkingDirectory)/plan2.sh
displayName: "example deploy"
continueOnError: false
Successfully get the issue:
But usually, we use the bash task directly to run the bash script.
plan.sh
{
xxx
# Your code here.
} || {
# save log for exception
echo some error output here.
exit 1
}
pipeline YAML definition part:
steps:
- task: Bash#3
displayName: 'Bash Script'
inputs:
targetType: filePath
filePath: ./plan.sh
Successfully get the issue:
For your script step to signal failure, you need to make the script as a whole return a non-0 exit code. You may need some conditional logic in your script, checking the exit code from plan.sh after it returns.
I was able to solve this by adding
set -o pipefail
In the start of the yaml file.
How could I rewrite this python script so that it can run within azure devops pipeline and export the dataframe as a csv to the devops repository. I'm able to achieve this locally but would like to achieve this remotely.
Put different, how can I export a pandas dataframe to devops repos folder as a csv file using an azure devops pipeline task. Below is the python script that needs to run as a pipeline task.
local_path in this case should be azure devops path.
from azureml.core import Workspace, Dataset
local_path = 'data/prepared.csv'
dataframe.to_csv(local_path)
⚠️You really should not do this. Azure pipelines are for building code, not for processing data. Assuming that you meant Azure DevOps Pipelines, opposed to Azure ML Pipelines.
Also you should not commit data to your repository.
If you still want to proceed, here is an example for what you try to achieve. Note that for the last line, i.e. git push, you need to give the agent permission to write the repository. See Run Git commands in a script for an approximate☹️ documentation on how to do that on your account.
trigger: none
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
persistCredentials: true
- task: UsePythonVersion#0
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- script: |
python your_data_generating_script.py
git config --global user.email "you#example.com"
git config --global user.name "Your Name"
git add data/prepared.csv
git commit -m'test commit'
git push origin HEAD:master
displayName: 'push data to master'
Currently i am in the process of converting my pipelines from classic over to azurepipelines.yml and im having an issue trying to find the correct syntax to reference release variables in a bash step.
The existing code in a bash task
namebuilder=$(RELEASE.ENVIRONMENTNAME)-$(RELEASE.RELEASEID)
will output the following
dev-2049
however when converted over to my new pipeline file the above code produces the the following error
/home/vsts/work/_temp/ac39e1d7-11bd-4c32-9b1b-1520dae11c5a.sh: line 1: RELEASE.ENVIRONMENTNAME: command not found
/home/vsts/work/_temp/ac39e1d7-11bd-4c32-9b1b-1520dae11c5a.sh: line 1: RELEASE.RELEASEID: command not found
[extracted from pipeline.yml]
- bash: |
namebuilder=$(RELEASE.ENVIRONMENTNAME)-$(RELEASE.RELEASEID)
i have even created a step trying a few different approaches without much luck
steps:
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
echo '$(release.environmentname)'
echo $(release.environmentname)
echo '$(RELEASE.ENVIRONMENTNAME)'
echo $(RELEASE.ENVIRONMENTNAME)
produces
This multiline script always runs in Bash.
Even on Windows machines!
$(release.environmentname)
$(RELEASE.ENVIRONMENTNAME)
/home/vsts/work/_temp/260dd504-a42d-45d6-bb1b-bf1f4b015cf8.sh: line 4: release.environmentname: command not found
/home/vsts/work/_temp/260dd504-a42d-45d6-bb1b-bf1f4b015cf8.sh: line 6: RELEASE.ENVIRONMENTNAME: command not found
is it also possible (in a much cleaner approach) to define this as a pipeline variable and reference at a global scope like below ?
variables:
namebuilder: '$(release.environmentname)-$(release.releaseid)'
stages:
- stage: Deploy
displayName: deploy infra
jobs:
- job: deploy_infra
displayName: deploy infra
continueOnError: true
workspace:
clean: outputs
steps:
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
echo '$(namebuilder)'
tia
It doesn't look like release.environment or any release variables are available for multi-stage pipelines. You could use the new environment concept and at that point environment.name would be available. I think you would likely go with $(environment.name)-$(build.buildid) for what you are after.
So I am not sure if the release pipelines you are converting are deploying to say an app service, or to a VM, or just using a hosted agent to publish something else? Disclaimer I have not used the Environment concept extensively yet just some reading and limited testing. Its all new!
So for deploying to VMs You can configure a Virtual Machine resource in an environment. This concept has a bunch of parallels with classic deployment group agents. You register an agent on a target machine. From there your pipeline steps can execute in that machine's context and you get a further set of environment variables.
The example pipeline below outputs any environment variables from the context the steps are running in and also outputs $(environment.name)-$(build.buildid)
A normal job in a hosted pipeline
A Deployment to an Environment
A Deployment to an Environment with a VM resource
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
namebuilder: '$(environment.name)-$(build.buildid)'
jobs:
- job: NormalJobInHostedPipeline
steps:
- task: PowerShell#2
name: EnvironmentVariables
inputs:
targetType: 'inline'
script: 'gci env:* | sort-object name'
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
echo '$(namebuilder)'
# track deployments on the environment
- deployment: DeploymentHostedContext
displayName: Runs in Hosted Pool
pool:
vmImage: 'Ubuntu-16.04'
# auto creates an environment if it doesn't exist
environment: 'Dev'
strategy:
runOnce:
deploy:
steps:
- task: PowerShell#2
name: EnvironmentVariables
inputs:
targetType: 'inline'
script: 'gci env:* | sort-object name'
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
echo '$(namebuilder)'
# Similar to Deployment Group Agent need to register them -stage will fail if resource does not exist
# https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-virtual-machines?view=azure-devops
- deployment: DeploymentVirtualMachineContext
displayName: Run On Virtual Machine Agent
environment:
name: DevVM
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- task: PowerShell#2
name: EnvironmentVariables
inputs:
targetType: 'inline'
script: 'gci env:* | sort-object name'
- task: PowerShell#2
name: VariableName
inputs:
targetType: 'inline'
script: 'echo $(namebuilder)'
Use $(System.StageName) in place of $(Release.EnvironmentName), as for release Id, you'd need to use $(Build.BuildId)
I found that $(Environment.Name) doesn't work unless you're using environments. I'm not since it's still quite limited.