how to read copied file from github from yaml and jq - azure

I am using devops pipeline to read contents of a json file hosted in a private repo in git hub. I can see the file in the pipeline output but jq is not reading the file its giving this error: "jq: error: Could not open file /home/vsts/work/1/s/config.json: No such file or directory"
this is my yaml code:
---
resources:
repositories:
- repository: testrepo
type: github
endpoint: testendpoint
name: test/test01
trigger:
- none
pool:
name: Hosted Ubuntu 1604
steps:
- script: |
displayName: 'Update the build number in readme.txt'
name: JQ
sudo apt-get install jq
echo 'installing jq'
- checkout: testrepo
path: mytest # will checkout at $(Pipeline.Workspace)/PutMyCodeHere
- script: dir ../mytest/
data=$(jq 'to_entries | map(select(.value.datavalue=="true")) | from_entries' $(Agent.BuildDirectory)/s/data.json )
echo "$data"
how can i jq to my json file?

I can get the same error message and I added task Copy files and Publish build artifacts to check the S folder, then we can see the data.json file.
It seems that jq issue instead of azure devops pipeline, you can raise the issue to jq support.
As a workaround, we can read the json file via cmd jq . {file name}.json
Result:
Update1
My test code:
resources:
repositories:
- repository: vitol
type: github
endpoint: GitHubTest
name: vitoliutest/vitol
trigger:
- none
pool:
name: Hosted Ubuntu 1604
steps:
- script: |
displayName: 'Update the build number in readme.txt'
name: JQ
sudo apt-get install jq
echo 'installing jq'
- checkout: vitol
path: mytest # will checkout at $(Pipeline.Workspace)/PutMyCodeHere
- task: CopyFiles#2
inputs:
SourceFolder: '$(Agent.BuildDirectory)'
Contents: '**'
TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
- script: dir ../mytest/
jq . $(Agent.BuildDirectory)/*/data.json

Related

Azure DevOps Pipeline - list all Build SourceBranches

This is my simple pipeline,
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
name: LinuxJavaCIBuildAgents #CheckmarxAgents #LinuxJavaCIBuildAgents
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
- release
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: echo ....??? what to echo to list all repositories and their source branch from resources.repositories
How do I list all Build SourceBranches that are involved in the above build pipeline? we have 2 repos: repo_a and repo_b, I want to list them using bash and list their source branches.
Thanks
How do I list all Build SourceBranches that are involved in the above
build pipeline? we have 2 repos: repo_a and repo_b, I want to list
them using bash and list their source branches.
Azure DevOps supports multiple repositories check out as a built-in function, Refer to the YAML code below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
ref: main
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
ref: main
trigger:
- main
- release
steps:
- checkout: repo_a
- checkout: repo_b
- script: dir $(Build.SourcesDirectory)
While running the pipeline it will ask for authorization to allow both the repositories to run like below:-
After granting the permission:-
To list the build source branches of the above repos you can use the below echo command as a Bash Inline script in your YAML code:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
- release
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
#!/bin/bash
echo
for repo in $(echo $(echo "${resources}" | jq -r '.repositories[].name')); do
echo "Repository: $repo"
echo "Source branches: $(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]')"
echo "Source branches"
done
Pipeline Run:-
If you want to make sure all repos run on the same branch and require a warning if another repo runs on a different branch, Use below YAML code:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
#!/bin/bash
# Get the source branch of the first repository
source_branch=$(echo "${resources}" | jq -r '.repositories[0].trigger[]')
# Loop through the other repositories and compare the source branch with the first repository
for repo in $(echo $(echo "${resources}" | jq -r '.repositories[].name' | tail -n +2)); do
if [[ $(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]') != "$source_branch" ]]; then
echo "Error: Repository $repo is running on a different branch ($(echo "${resources}" | jq -r '.repositories[] | select(.name == "$repo") | .trigger[]')) than the first repository ($source_branch)"
exit 1
fi
done
echo "All repositories are set to run on branch $source_branch"
Output:-
Reference :-
Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Learn

Error "No module named 'azure.storage.cloudstorageaccount'" when using azure_rm_resourcegroup in Ansible playbook run by an Azure Devops pipeline

As far as I am aware I installed all dependencies to use the azure modules in my Ansible playbook, but I am still getting this error.
The full traceback is:
Traceback (most recent call last):
File "/tmp/ansible_azure_rm_resourcegroup_payload_7l31ymh4/ansible_azure_rm_resourcegroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 250, in <module>
from azure.storage.cloudstorageaccount import CloudStorageAccount
ModuleNotFoundError: No module named 'azure.storage.cloudstorageaccount'
My Azure Devops pipeline:
pool:
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion#0
displayName: 'Install Python'
inputs:
versionSpec: '3.x'
addToPath: true
architecture: 'x64'
- task: AzureCLI#2
inputs:
azureSubscription: '$(AZURE_SUBSCRIPTION_NAME)'
addSpnToEnvironment: true
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(az account show --query='id' -o tsv)"
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]${servicePrincipalId}"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]${servicePrincipalKey}"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]${tenantId}"
- script: pip install ansible[azure]
displayName: 'Install Ansible'
- script: ansible-galaxy collection install azure.azcollection
displayName: 'Install Ansible Azure Collection'
- script: pip install -r https://raw.githubusercontent.com/ansible-collections/azure/dev/requirements-azure.txt
displayName: 'Install Azure modules needed'
- script: pip install azure-storage-blob azure-storage-file-share azure-storage-file-datalake azure-storage-queue
displayName: 'Install missing modules (to be sure to have the azure storage modules)'
- script: ansible-playbook -vvv -i inv site.yml
displayName: 'Run Ansible Playbook'
env:
AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
AZURE_SECRET: $(ARM_CLIENT_SECRET)
AZURE_TENANT: $(ARM_TENANT_ID)
AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
My playbook:
---
- name: config azure environment
hosts: localhost
connection: local
gather_facts: true
collections:
- azure.azcollection
vars_files:
- group_vars/common.yml
roles:
- roles/resourcegroup
and the role:
---
- name: create a resource group
azure_rm_resourcegroup:
name: "{{ app.name }}-{{ dict.resource_group }}"
location: "{{ azure.location }}"
state: present
According to the documentation (https://docs.ansible.com/ansible/latest/collections/azure/azcollection/azure_rm_resourcegroup_module.html) everything should be fine. So, what am I missing??
I have been googling for several hours now, but I did not find a working solution yet :-(
In the meantime. I have a working solution. I started from scratch, and created an Azure pipeline template without a role.
As an example I take my pipeline to create a docker container registry, but it works for everything you want to do in Azure using a playbook in a pipeline.
Just use this example to learn how I made it working. Hope it helps others who are struggling with the same problem.
- task: Bash#3
displayName: 'create vars file for docker registry playbook'
inputs:
targetType: 'inline'
workingDirectory: './playbooks'
script: |
touch vars.yml
echo 'azure:' > vars.yml
echo ' location: "${{ parameters.azure_location }}"' >> vars.yml
echo ' resourcegroup: "${{ parameters.resourcegroup_name }}"' >> vars.yml
echo ' containerregistry: "${{ parameters.containerregistry_name }}"' >> vars.yml
cat vars.yml
- template: steps/run_ansible.yml
parameters:
playbook: playbooks/setup_dockerhub.yml
varsfile: playbooks/vars.yml
The run_ansible step file:
parameters:
- name: playbook
type: string
- name: varsfile
type: string
steps:
- task: AzureCLI#2
displayName: 'install Azure CLI'
inputs:
#azureSubscription: '$(AZURE_SUBSCRIPTION_NAME)'
connectedServiceNameARM: 'ARM Outstanding24'
addSpnToEnvironment: true
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(az account show --query='id' -o tsv)"
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]${servicePrincipalId}"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]${servicePrincipalKey}"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]${tenantId}"
- script: sudo apt install -y python3-pip
displayName: 'install pip'
- script: sudo pip3 install --upgrade pip
displayName: 'ensure we have the latest version of pip3'
- script: pip3 install "ansible==2.9.17"
displayName: 'install ansible 2.9'
- script: pip3 install ansible[azure]
displayName: 'install ansible "azure modules"'
- script: 'ansible-playbook -v ${{ parameters.playbook }} --extra-vars #${{ parameters.varsfile }}'
displayName: 'run azure playbook'
env:
AZURE_CLIENT_ID: $(ARM_CLIENT_ID)
AZURE_SECRET: $(ARM_CLIENT_SECRET)
AZURE_TENANT: $(ARM_TENANT_ID)
AZURE_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
The playbook:
- name: setup docker registry in Azure
hosts: localhost
connection: local
gather_facts: false
collections:
- azure.azcollection
vars_files:
- vars.yml
tasks:
- name: ensure the resourcegroup exists
azure_rm_resourcegroup:
name: "{{ azure.resourcegroup }}"
location: "{{ azure.location }}"
state: present
- name: ensure docker registry exists
azure_rm_containerregistry:
name: "{{ azure.containerregistry }}"
location: "{{ azure.location }}"
resource_group: "{{ azure.resourcegroup }}"
sku: Basic
state: present

Azure Devops - Sign Mac OS .app and publish Artifact

I am trying to automate the process of signing a .app Binary file using Azure Devops CI pipelines. The current CI performs the following tasks:
CI builds a "setup" container containing the files to compile and pushes to our container registry
Runs a script against an Azure VM to pull this image down and compile against in an Ubuntu VM
The Ubuntu VM is required because the compiling process uses a GUI and a display on xhost is required for it to run.
Once it is done compiling we then zip the binary's and publish it to Azure Devops Artifacts.
What I wanted to do here is pull down the binary and then sign it and push it back up, but the pipeline just hangs indefinitely until it times out. My assumption is that it is waiting for some kind of prompt, and I have no idea how to pass it
name: $(Date:yyyyMMdd)$(Rev:.r)
trigger: none
pr: none
variables:
- group: MacOS
jobs:
- job: POC_Pipeline
pool:
vmImage: 'macOS-latest'
steps:
# - task: InstallAppleCertificate#2
# inputs:
# certSecureFile: '$(p12FileName)'
# certPwd: '$(p12Password)'
# keychain: 'temp'
# deleteCert: true
- task: DownloadSecureFile#1
name: AppleCertificate
displayName: 'Download Apple Certificate'
inputs:
secureFile: '$(p12FileName)'
- task: DownloadPackage#1
inputs:
packageType: 'upack'
feed: 'myfeed'
definition: 'mybinary'
version: '*' # Pulls latest
downloadPath: '$(System.ArtifactsDirectory)'
- script: 'security create-keychain -p password temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Create Keychain'
failOnStderr: true
- script: 'security unlock-keychain -p password temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Unlock Keychain'
failOnStderr: true
- script: 'security import $(AppleCertificate.secureFilePath) -k temp.keychain -P $(p12Password) -T /usr/bin/codesign'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Install Apple Certificate'
failOnStderr: true
- script: 'security find-certificate temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Find Certificate'
failOnStderr: true
- script: 'security find-identity -p codesigning -v keychain temp.keychain'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Find Identity'
failOnStderr: true
- script: 'security default-keychain -s "/Users/runner/Library/Keychains/temp.keychain-db"'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Set Default Keychain'
failOnStderr: true
- script: 'unzip -q myBinary.app.zip'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Unzip myBinary'
failOnStderr: true
- script: 'xattr -rc myBinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Extended Attributes'
failOnStderr: true
- script: 'sudo codesign -s Anasazi -f --deep myDinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Codesign Dragon.app'
failOnStderr: true
- script: 'codesign -dv myBinary.app'
workingDirectory: '$(System.ArtifactsDirectory)'
displayName: 'Verify Codesign myBinary.app'
failOnStderr: true
Any assistance or recommendations how we can sign the .app file in the CI would be appreciated.

Azure Pipeline- Copy files from one Repo to another Repo using YAML

There is a folder in one of the repositories (Source Repo) that I like to copy to another repository (Destination Repo) using Azure Pipeline (as they needed to be in sync)
so far I can Copy a folder in the same repository using:
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.Repository.LocalPath)\MyFolder\'
Contents: |
**
!**\obj\**
!**\bin\**
TargetFolder: '$(Build.Repository.LocalPath)\DestFolder'
flattenFolders: false
CleanTargetFolder: true
OverWrite: true
preserveTimestamp: true
this is how I connect to another repository:
resources:
repositories:
- repository: SourceRepo
type: git
name: MyCollection/SourceRepo
but I don't know how to get files from the source repo and place them in the Destination Repo
after a lot of searching, this is the answer:
resources:
repositories:
- repository: SourceRepo
type: git
name: MyCollection/SourceRepo
steps:
- checkout: SourceRepo
clean: true
- checkout: self
persistCredentials: true
clean: true
- task: DotNetCoreCLI#2
displayName: "restore DestRepo"
inputs:
command: 'restore'
projects: '$(Build.Repository.LocalPath)/DestRepo/**/*.csproj'
feedsToUse: 'select'
- task: DotNetCoreCLI#2
displayName: "build DestRepo"
inputs:
command: 'build'
projects: '$(Build.Repository.LocalPath)/DestRepo/DestRepo/**/*.csproj'
configuration: Release
# configurations for using git command
- task: CmdLine#2
inputs:
script: |
cd $(Agent.HomeDirectory)\externals\git\cmd
git config --global user.email ""
git config --global user.name "$(Build.RequestedFor)"
- task: CmdLine#2
displayName: checkout
inputs:
script: |
git -C RootRep checkout $(Build.SourceBranchName)
- task: CmdLine#2
displayName: pull
inputs:
script: |
git -C DestRepo pull
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.Repository.LocalPath)\SourceRepo\SourceFolder'
Contents: |
**
!**\obj\**
!**\bin\**
TargetFolder: '$(Build.Repository.LocalPath)\DestRepo\DestFolder'
flattenFolders: false
CleanTargetFolder: true
OverWrite: true
# preserveTimestamp: true
- task: CmdLine#2
displayName: add
inputs:
script: |
git -C DestRepo add --all
- task: CmdLine#2
displayName: commit
continueOnError: true
inputs:
script: |
git -C DestRepo commit -m "Azure Pipeline Repository Integration"
- task: CmdLine#2
displayName: push
inputs:
script: |
git -C DestRepo push -u origin $(Build.SourceBranchName)
I was trying to find some solution related to this problem, but instead of using a copy file task, I found a better way and we can use any number of repositories are resources in the build pipeline and we don't need to check out all these.
This is how my build pipeline looks like.
As you can see I have used two variables
$(System.AccessToken), this variable is available in Azure DevOps aka PAT(Personal Access Token)
$(Build.Repository.Uri) URL of the repository (this could be the URL of any repo in resources).

Azure pipelines read file from resources repository

I am referencing a repository in my azure-pipelines template like this:
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepoName
ref: MyRepoRef
I would like to know if it is possible to read the content of a file that is inside the referenced repository, inside this repository is another yaml that is being executed in the pipeline.
If we use another repo in the pipeline, we could read the content of a file in the referenced repository. We could refer this link for more details
In my test, project name is test, the referenced repository is test and current repository is sample, then I read the content of file pull_request_template.md
YAML build definition:
trigger: none
resources:
repositories:
- repository: test
type: git
name: test/test
ref: master
steps:
#checkout referenced repository
- checkout: test
#List SourcesDirectory files
- task: Bash#3
inputs:
targetType: 'inline'
script: 'ls ''$(Build.SourcesDirectory)'''
#Read the contents of the file pull_request_template.md
- task: PowerShell#2
inputs:
targetType: 'inline'
script: 'Get-Content -Path $(Build.SourcesDirectory)\pull_request_template.md'
Result:

Resources