we are setting up Policies for our Org. One need we have is to have a build and annotation at every Pull Request with Sonarcloud. Is there a way to create a common ci build pipeline which will run at every PR, checking out the respective repo, detect the type of project (or read a manifest file or so) do the code analytics and build, annotates the PR?
So in the Azure DevOps repo you can create a common branch policy for all the master branches. I tried to use a standalone yaml pipeline but it never started when I created the pr. Can someone help me on the right track? Do I need to create a resource in the yaml? Is there any variable I can use from the PR to detect the repo and the branches?
Just for everybody to understand, you can create common branch policies and individuals.
Thanks a lot
You will need to add all the repositories in the resources section in the yaml because of the known issue reported in this thread.
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
However, if you use a classic UI Pipeline in the Build Validation instead of the yaml pipeline. You donot need to add all the repositories in the resources section. But you need to skip the pipeline to sync the source: (Go pipeline edit page-->Click Get sources-->check Don't sync sources)
You can use the predefined variables to get the information about the pull request. See below:
$(Build.Repository.Name)
$(System.PullRequest.PullRequestId)
Then you can run the git commands in a script task to check out the pull request branch. See below example:
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
pool:
vmImage: windows-latest
steps:
- checkout: none
- powershell: |
git clone "https://$(System.AccessToken)#dev.azure.com/OrganizationName/$(System.TeamProject)/_git/$(Build.Repository.Name)"
cd "$(Build.Repository.Name)"
git fetch origin pull/$(System.PullRequest.PullRequestId)/merge:pr-$(System.PullRequest.PullRequestId)
# checkout pr branch
git checkout pr-$(System.PullRequest.PullRequestId)
Note: you need to grant build service account read permission for target repository
Related
OK, so the main question here is: How to Sync Github with Azure?
Now, I had main reference source: How to synchronize Azure Repos with External Git Repos
https://faun.pub/how-to-synchronize-azure-repos-with-external-git-repos-70ff92e51c63
And that is perfect match, but here is a catch, it is kind of abstract and you have to have work with Azure experience, like knowing what is YAML for and how to use it.
Long story short, it did not work. And then I found this gentleman's video: Merge From Github to Azure DevOps
https://www.youtube.com/watch?v=Kks1pCG51bI
That is super close, yet there still were several error and bugs ,like:
Cannot find path 'D:\a\1\s\copyrepo~' because it does not exist;
remote: Not Found fatal: repository 'https://github.com/***/' not found ##[warning]Git fetch failed with exit code 128, back off 5.443 seconds before retry.
error: unable to create file Filename too long
So, that is a bummer...
I mean, yes, you technically can create and synchronize GitHub repo with Azure, but you have to create new GitHub repo, with existing one there was: error: unable to create file Filename too long.
Please tell me what you think.
Try to use the following yaml file as I used to sync Github to Azure Repos:
name: Sync Azure with Github
variables:
REMOTE_ADDR: 'https://github.com/{user{/{repo.git}'
stages:
- stage: syncing Repos
displayName: syncing Repos
jobs:
- job: run_Git_Commands
displayName: run_Git_Commands
continueOnError: false
steps:
- checkout: self
clean: true
persistCredentials: true
displayName: run_commands
- bash: |
git checkout master
git remote add repoGithub $(REMOTE_ADDR)
git fetch repoGithub master
git reset --hard repoGithub/master
git pull --rebase repoGithub master
git push --force origin
I am very new in Azure Devops. All my repositories are in BitBucket. I have one project say A which has dependency in project B
In Visual Studio there are no build errors, but when I am building the CI pipeline for A I am getting an error saying "Project B not found"
The CI pipeline is created with repository of project A. How can I solve this?
Note: I am using dev.azure.com and selected "Use classic editor to create a pipeline without YAML" so I am not able to add yaml code. Can you please tell me how to do this?
Sounds like you need to check out multiple repositories in your pipeline for example like this:
resources:
repositories:
- repository: MyBitbucketRepoA
type: bitbucket
endpoint: MyBitbucketServiceConnection
name: MyBitbucketOrgOrUser/MyBitbucketRepoA
- repository: MyBitbucketRepoB
type: bitbucket
endpoint: MyBitbucketServiceConnection
name: MyBitbucketOrgOrUser/MyBitbucketRepoB
steps:
- checkout: self
- checkout: MyBitbucketRepoA
- checkout: MyBitbucketRepoB
I got a requirement to generate, archive and reuse the artifacts between two different repositories
Repository A: Compile Angular code and create a XLF file
Repository B: Use the 'XLF File' generated above and create a new XLF file
Repository A: Again use the newly generated XLF file to create the final output file
The activities mentioned above should be done using gitlab-ci.yml. I am not sure how to handle this using GitLab CI.
We can push the artifact from Repo A to Repo B. However, CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Ideally, you would not push a generated artifact to another Git source repository.
But a GitLab pipeline can retrieve an artifact produced by another one from its URL.
To avoid the back and forth, I would rather have 3 jobs instead of two
the first generates XLF file
the second curls/fetches that file, and use it to generate new XLF file
the third job curls/fetches that file, and complete the process.
How to release built artifacts back-and-forth from one to another repo on GitLab?
Repository A:
Compile Angular code and create a XLF file
Send a hook to repository B that it just compiled
just trigger: https://docs.gitlab.com/ee/ci/yaml/#trigger , works like a charm. It's even nicely visible in the gui.
or API https://docs.gitlab.com/ee/ci/triggers/
pass variables: PARENT_PIPELINE_ID: $CI_PIPELINE_ID to repository B so it can download artifacts from specific pipeline
Repository B:
Use the 'XLF File' generated above
use needs: https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-to-child-pipelines to download artifacts
or API: have personal access token from repository A https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html added to environment variables and use API to download artifacts https://docs.gitlab.com/ee/api/job_artifacts.html .
create a new XLF file
use trigger: or API to trigger repository A
but this time trigger different .gitlab-ci.yml file like: trigger: - project: repositoryA file: second_stage.gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/#trigger-child-pipeline-with-files-from-another-project
or use like variables: SECOND_STAGE: "true" and use a variable to differentiate
Repository A:
run pipeline from the file second_stage.gitlab-ci.yml
download artifacts from repository B - needs: or API
use the newly generated XLF file to create the final output file
Overall, what you need is rules: and needs: documentation. On older gitlab, it was done with API.
CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Don't wait. Let the API trigger it.
I tried the following approach and it worked fine or at least I was able to proceed
Due to some reason 'variables' along with CURL did not work as expected but I did not analyze the root cause
Repo A - Pipeline
trigger-repob: (Trigger Project B of Repo B)
stage: repob
trigger:
project: repob-namespace/projectb
branch: devops
test_job:
image: $CI_REGISTRY/$CI_PROJECT_PATH/base-image:latest
stage: test_pipeline
when: delayed
start_in: 2 minutes
needs: (Use artifacts from Repo B/Project B)
-
project: repob-namespace/projectb
job: buildprojectb
ref: devops
artifacts: true
script:
- do something here
Repo B Pipeline
buildprojectb:
image: php:7.4.11
stage: build
script:
- do something here
artifacts:
paths:
- outputs/*.xlf
So I'm trying to learn deployement with Azure Devops. I have this Angular app sitting in Gitlab which already has a CI/CD pipeline with jenkins to kubernetes cluster. So i was thinking to do the same with Azure Devops via YAML. Which is not possible according to Azure docs directly from gitlab.
So what i'm trying to do is create CI pipeline from github which takes checkout from gitlab UI repo and build it for deployement.
I have created a Repository Resource in my below pipeline YAMl file. Azure give me error saying:
Repository JpiPipeline references endpoint https://gitlab.com/myusername/myUiRepo.git which does not exist or is not authorized for use
trigger:
- master
resources:
repositories:
- repository: UiPipeline. #alias
type: git
name: repository_name
# ref: refs/heads/master # ref name to use; defaults to 'refs/heads/master'
endpoint: https://#gitlab.com/myusername/myUiRepo.git # <-- Is this possible
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-16.04'
continueOnError: true
steps:
- checkout: JpiPipeline
- script: echo "hello to my first Build"
Repository type gitlab is not support in YAML pipeline yet. The currently supported types are git, github, and bitbucket, see supported types.
The workaround to get the gitlab repo sources is to run git command inside the script tasks.
For below example Yaml pipeline:
- checkout: none to avoid checkout the github source.
Use git clone https://username:password#gitlab.com/useraccount/reponame.git to clone the gitlab repo inside a script task.
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-latest'
steps:
- checkout: none
- script: |
git clean -ffdx
git clone https://username:password#gitlab.com/useraccount/reponame.git
#if your password or username contain # replace it with %40
Your gitlab repo will be clone to folder $(system.defaultworkingdirectory)/reponame
Another workaround is to classic UI pipeline. Gitlab repo type is supported in Classic UI pipeline.
You can choose Use the classic editor to create a classic ui pipeline.
When you come to select source page. Choose other git and click Add connection to add your gitlab repo url. Then the pipeline will get the sources for your gitlab repo.
Could you please help me. How I can run the YAML pipeline from another Azure DevOps Git repository? I have two repositories, one for templates, one for code source. My code:
resources:
repositories:
- repository: CodeServerGit
name: BoxServer/CodeServerGit
type: git
#connection: CodeServerGit
ref: /v1/dev
# source: /v1/dev
I have an error
[error]Project file(s) matching the specified pattern were not found.
I see that I am in the wrong repository. Assembly goes in the repository in which the template is located, and does not connect to the repository in which the code.
Since what you want to link is Azure Devops git repos, please check below sample which work for me:
resources:
repositories:
- repository: {repos name}
name: {project name}/{repos name}
type: git
ref: master #branch name
jobs:
- template: azure-pipelines-1.yml#{repos name}
The above script is not very complicated, but there are many places should be noted.
(1) The project which store the template YAML file should be in the same organization.
(2) For the value of ref, if what you want is branch, just input the branch name here. The server will automatically add refs/heads before the value after the queue start.
If here you want to specify a tag, you must fill it with a completed format: refs/tags/{tag name}.
(3) Do not forget to specify the repos name after # when you trying to call one template
See the doc.