This is the reference doc I have followed to set up the Azure pipeline
https://medium.com/adessoturkey/owasp-zap-security-tests-in-azure-devops-fe891f5402a4
below i am sharing screenshort of the pipeline failed:
Could you please help here to resolve the issue I have exactly followed the medium article to implement the task....
Those who aware on this could you please share your taughts.
This is the pipeline script i am using.
trigger: none
stages:
stage: 'buildstage'
jobs:
job: buildjob
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- checkout: owasap-zap
bash: "docker run -d -p 80:80 nginx:1.14.2"
displayName: "App Container"
bash: |
chmod -R 777 ./
docker run --rm -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-full-scan.py -t http://$(ip -f inet -o addr show docker0 | awk '{print $4}' | cut -d '/' -f 1):80 -x xml_report.xml
true
displayName: "Owasp Container Scan"
- displayName: "PowerShell Script"
powershell: |
$XslPath = "owasp-zap/xml_to_nunit.xslt"
$XmlInputPath = "xml_report.xml"
$XmlOutputPath = "converted_report.xml"
$XslTransform = New-Object System.Xml.Xsl.XslCompiledTransform
$XslTransform.Load($XslPath)
$XslTransform.Transform($XmlInputPath, $XmlOutputPath)
displayName: "PowerShell Script"
task: PublishTestResults#2
displayName: "Publish Test Results"
inputs:
testResultsFiles: converted_report.xml
testResultsFormat: NUnit
# task: PublishTestResults#2
stage: buildstage
According to the YAML file, you want to checkout multiple repositories in your pipeline, but it seems you haven't define a repository resource like mentioned in the document you shared.
resources:
repositories:
- repository: <repo_name>
type: git
name: <project_name>/<repo_name>
ref: refs/heads/master
And according to the screenshot you shared, you only checkout out one repo. Which cause the location of file xml_to_nunit.xslt is different from owasp-zap/xml_to_nunit.xslt. If you only checkout one repo, the location of xml_to_nunit.xslt should be current directory, thus, just define $XslPath in the PowerShell script as "xml_to_nunit.xslt".
Edit
If the repository that contain "xml_to_nunit.xslt" file is in the same organization as the repository run for your pipeline, you need to checkout the repository by using Inline syntax checkout like below or define repository resource.
- checkout: git://MyProject/MyRepo # Azure Repos Git repository in the same organization
You could also add one more command ls before the PowerShell script to list the files in current directory. Aim to figure out where is "xml_to_nunit.xslt".
Related
This is my simple pipeline,
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
name: LinuxJavaCIBuildAgents #CheckmarxAgents #LinuxJavaCIBuildAgents
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
- release
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: echo ....??? what to echo to list all repositories and their source branch from resources.repositories
How do I list all Build SourceBranches that are involved in the above build pipeline? we have 2 repos: repo_a and repo_b, I want to list them using bash and list their source branches.
Thanks
How do I list all Build SourceBranches that are involved in the above
build pipeline? we have 2 repos: repo_a and repo_b, I want to list
them using bash and list their source branches.
Azure DevOps supports multiple repositories check out as a built-in function, Refer to the YAML code below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
ref: main
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
ref: main
trigger:
- main
- release
steps:
- checkout: repo_a
- checkout: repo_b
- script: dir $(Build.SourcesDirectory)
While running the pipeline it will ask for authorization to allow both the repositories to run like below:-
After granting the permission:-
To list the build source branches of the above repos you can use the below echo command as a Bash Inline script in your YAML code:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
- release
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
#!/bin/bash
echo
for repo in $(echo $(echo "${resources}" | jq -r '.repositories[].name')); do
echo "Repository: $repo"
echo "Source branches: $(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]')"
echo "Source branches"
done
Pipeline Run:-
If you want to make sure all repos run on the same branch and require a warning if another repo runs on a different branch, Use below YAML code:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
#!/bin/bash
# Get the source branch of the first repository
source_branch=$(echo "${resources}" | jq -r '.repositories[0].trigger[]')
# Loop through the other repositories and compare the source branch with the first repository
for repo in $(echo $(echo "${resources}" | jq -r '.repositories[].name' | tail -n +2)); do
if [[ $(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]') != "$source_branch" ]]; then
echo "Error: Repository $repo is running on a different branch ($(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]')) than the first repository ($source_branch)"
exit 1
fi
done
echo "All repositories are set to run on branch $source_branch"
Output:-
Reference :-
Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Learn
I would like to run a docker container, which i pull from Azure Container Registry. All of this i would like to do in Azure DevOps pipeline.
Firstly i created sample Node.js app and Dockerized it with this tutorial: https://nodejs.org/en/docs/guides/nodejs-docker-webapp/
Then i did my Azure Pipeline which firstly do build&push and then pull and run.
My pipeline:
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(imageRepository)
dockerfile: $(dockerfilePath)
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(tag)
- task: Docker#2
displayName: Docker pull
inputs:
command: pull
containerRegistry: $(dockerRegistryServiceConnection)
arguments: container01.azurecr.io/devopsnodejs:latest
- task: Docker#2
displayName: Login to ACR
inputs:
command: login
containerRegistry: $(dockerRegistryServiceConnection)
- script: |
docker run -p 49160:8080 -d container01.azurecr.io/devopsnodejs:latest
The pipelines runs every step sucessfully, the last script with docker run prints this to Azure DevOps console
Generating script.
Script contents:
docker run -p 49160:8080 -d ***/devopsnodejs:latest
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/b117892d-e34c-484c-ad8c-f99cd0a97e18.sh
7c6c9d548c4be3e4568e56ffc87cca27e698fc53b5ec15a1595cd45fe72dd143
And now the problem is that, I cannot acces the app which should return simply get request saying 'Hello World'
Im trying to go to localhost:49160, to curl -i localhost:49160 but there is only curl: (7) Failed to connect to localhost port 49160 after 2258 ms: Connection refused
Also, if i do it locally, not in azure pipelines, so I simply run docker pull container01.azurecr.io/devopsnodejs:latest and docker run -p 49160:8080 -d container01.azurecr.io/devopsnodejs:latest in powershell, the docker ps will show me this container, as well as the curl -i localhost:49160 will work. Am I able to access this locally, or if i run it in Azure Pipelines it will work only there?
Have you seen this question already, which is I think your scenario?
Cannot conect to Docker container running in VSTS
You could use a bash script with ssh to logon to the target machine and perform docker commands on the machine. The following is just an example. I have something like this in one of my pipelines.
- task: Bash#3
displayName: "Deploy image $(imageName)"
inputs:
targetType: 'inline'
failOnStderr: false
script: |
sshpass -p '$(localroot-userpassword)' ssh -o StrictHostKeyChecking=no $(localroot-username)#$(remoteHost) "{ sudo docker stop $(imageName) ; sudo docker rm $(imageName) ; sudo docker pull $(imageName) ; sudo docker run ...#add other commands separated by semicolon}"
I'm working with azure pipeline to checkout source code from azure repo and execute setup of inbuilt script to which is provided by webmethod SAG, Using build.yaml i can able to build my application but not able to publish the artifacts.
cat build.yaml
trigger:
- devops-build
pool:
name: CICD
steps:
# Create Target Directory to keep git repo for later use
- bash: |
mkdir -p /home/user/cicd_source/dev/packages/packages
displayName: 'create directory'
- bash: |
echo "webname=${{parameters.projectName}}" > $(Build.ArtifactStagingDirectory)/devpackagename.properties
echo "BuildNumber=$(Build.BuildNumber)" > $(Build.ArtifactStagingDirectory)/devBuildNumber.txt
Above script will create devpackagename.properties and devBuildNumber.txt following path inside my self hosted agent directory work location.
pwd
/home/user/agent/CICD/_work/1/a
ls -lrt
devpackagename.properties
devBuildNumber.txt
cat devpackagename.properties
webname=package
cat devBuildNumber.txt
BuildNumber=20221004.83
After ran the successful pipeline i don't see any artefacts published inside my pipeline
after your build steps add below task
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifact: 'drop'
publishLocation: 'pipeline'
you would see artifact get published on the pipeline
Im trying to run an Helm deployment via Azure Devops. The problem is that the variable i set in my Bash step is not being read in the actual Upgrade step. When i run this command stand alone from my CLI it works fine.
So it is actually about this line:
arguments: "--reuse-values --version $(helmChartVersion)"
The full thing below:
- task: Bash#3
name: repoAdd
displayName: Add repo and deploy
inputs:
targetType: 'inline'
script: |
# Add the repo
helm repo add \
stapp \
https://$(containerRegistry)/helm/v1/repo \
--username $(registryUsername) \
--password '$(registryPassword)'
# Extra version file
export helmChartVersion=$(jq .helmChartVersion $(pipeline.workspace)/ci-pipeline/build-artifact/variables.json -r)
cat $(pipeline.workspace)/ci-pipeline/build-artifact/variables.json
# Lets update the repo
helm repo update
- task: HelmDeploy#0
inputs:
connectionType: 'Azure Resource Manager'
azureSubscription: 'Microsoft Azure(1fafaf-8012-4035-b8f3-fafaffa)'
azureResourceGroup: 'production-rg'
kubernetesCluster: 'production'
namespace: 'stapp-test'
command: 'upgrade'
chartType: 'Name'
chartName: 'stapp/stapp'
releaseName: 'stapp'
install: false
arguments: "--reuse-values --version $(helmChartVersion)"
Best,
Pim
In Azure DevOps you must explicitly set the variable with a legacy label from Visual Studio Online.
# Extra version file
helmChartVersion=$(jq .helmChartVersion $(pipeline.workspace)/ci-pipeline/build-artifact/variables.json -r)
echo "##vso[task.setvariable variable=helmChartVersion]$helmChartVersion"
Alessandro Segala has written a great article about this (https://medium.com/microsoftazure/how-to-pass-variables-in-azure-pipelines-yaml-tasks-5c81c5d31763)
I have Gitlab CI pipeline which is triggered by bitbucket webhook with current and last commit ids. I also want to re-run pipeline manually whenever the build created Gitlab CI file, triggered by webhook is not working as expected.
I tried RUN-PIPELINE option but shows the error:
The form contains the following error:
No stages/jobs for this pipeline.
Here is the GitLab CI file. Include refers to other project where standard yaml file for the pipeline is kept:
include:
- project: Path/to/project
ref: bb-deployment
file: /bitbucket-deployment.yaml
variables:
TILLER_NAMESPACE: <namespace>
NAMESPACE: testenv
REPO_REF: testenvbranch
LastCommitSHA: <commit sha from webhook>
CurrentCommitSHA: <Current commit she from webhook>
Here is the detailed gitlab-ci file that is provided in other project which has stages:
stages:
- pipeline
- build
variables:
ORG: test
APP_NAME: $CI_PROJECT_NAME
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIIVATE_KEY2" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
Building CI Script:
stage: pipeline
image: python:3.6
only:
refs:
- master
script:
- |
curl https://github.com/org/scripts/branch/install.sh | bash -s latest
source /usr/local/bin/pipeline-variables.sh
git clone git#bitbucket.org:$ORG/$APP_NAME.git
cd $APP_NAME
git checkout $lastCommit
cp -r env old
git checkout $bitbucketCommit
$CMD_DIFF old env
$CMD_BUILD
$CMD_INSTALL updatedReposList.yaml deletedReposList.yaml /tmp/test $NAMESPACE $REPO_REF $ORG $APP_NAME $lastCommit $bitbucketCommit
cat cicd.yaml
mv cicd.yaml ..
artifacts:
paths:
- cicd.yaml
Deplopying Apps:
stage: build
only:
refs:
- master
trigger:
include:
artifact: cicd.yaml
job: Building CI Script
strategy: depend
In the manual trigger, instead of considering the last and current commit she, it should rebuild the application.
Any help will be appreciated.
Thank you for your comment (below), I see you are using the include directive (https://docs.gitlab.com/ce/ci/yaml/#include) in one .gitlab-ci.yml to include a GitLab CI YAML file from another project.
I can duplicate this error (No stages / jobs for this pipeline) by invoking "run pipeline" on project 1 which is configured to include GitLab CI YAML from project 2 when the project 2 GitLab CI YAML is restricted to the master branch but I'm running the project on another branch.
For example, let's say project 1 is called "stackoverflow-test" and its .gitlab-ci.yml is:
include:
- project: atsaloli/test
file: /.gitlab-ci.yml
ref: mybranch
And project 2 is called "test" (in my own namespace, atsaloli) and its .gitlab-ci.yml is:
my_job:
script: echo hello world
image: alpine
only:
refs:
- master
If I select "Run Pipeline" in the GitLab UI in project 1 on a branch other than "master", I then get the error message "No stages / jobs for this pipeline".
That's because there is no job defined for my non-master branch, and then without any job defined, I don't have any stage defined.
I hope that sheds some light on what's going on with your webhook.