So I'm trying to learn deployement with Azure Devops. I have this Angular app sitting in Gitlab which already has a CI/CD pipeline with jenkins to kubernetes cluster. So i was thinking to do the same with Azure Devops via YAML. Which is not possible according to Azure docs directly from gitlab.
So what i'm trying to do is create CI pipeline from github which takes checkout from gitlab UI repo and build it for deployement.
I have created a Repository Resource in my below pipeline YAMl file. Azure give me error saying:
Repository JpiPipeline references endpoint https://gitlab.com/myusername/myUiRepo.git which does not exist or is not authorized for use
trigger:
- master
resources:
repositories:
- repository: UiPipeline. #alias
type: git
name: repository_name
# ref: refs/heads/master # ref name to use; defaults to 'refs/heads/master'
endpoint: https://#gitlab.com/myusername/myUiRepo.git # <-- Is this possible
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-16.04'
continueOnError: true
steps:
- checkout: JpiPipeline
- script: echo "hello to my first Build"
Repository type gitlab is not support in YAML pipeline yet. The currently supported types are git, github, and bitbucket, see supported types.
The workaround to get the gitlab repo sources is to run git command inside the script tasks.
For below example Yaml pipeline:
- checkout: none to avoid checkout the github source.
Use git clone https://username:password#gitlab.com/useraccount/reponame.git to clone the gitlab repo inside a script task.
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-latest'
steps:
- checkout: none
- script: |
git clean -ffdx
git clone https://username:password#gitlab.com/useraccount/reponame.git
#if your password or username contain # replace it with %40
Your gitlab repo will be clone to folder $(system.defaultworkingdirectory)/reponame
Another workaround is to classic UI pipeline. Gitlab repo type is supported in Classic UI pipeline.
You can choose Use the classic editor to create a classic ui pipeline.
When you come to select source page. Choose other git and click Add connection to add your gitlab repo url. Then the pipeline will get the sources for your gitlab repo.
Related
I have a repository in Github and want to integrate it into Azure-DevOps. I connected both the repositories in Github as well as Azure-devops.
When I commit some code into Github the changes are not getting updated automatically in Azure. Is there anyway that we can automatically pull the changes if there are any new changes in Github?
Any references/suggestions are much appreciated.
Update:
Azure DevOps doesn't have such a built-in feature to sync the Github repo to the DevOps repo now.
If you need the feature, you can upvote the feature request in this place:
https://developercommunity.visualstudio.com/t/automatically-sync-azure-devops-repository-with-gi/516057
When enough people request a new feature, Microsoft will include it in future product plans.
1, You need to use code/script to sync the repo and use the CI Trigger of the YAML pipeline to capture the changes in the Github repo.
trigger:
- <branch name>
pool:
vmImage: ubuntu-latest
variables:
- name: system.debug
value: true
steps:
- script: |
echo put the logic here
displayName: 'Push changes to DevOps repo'
The code you can refer to this page:
https://dileepveldi.medium.com/sync-azure-devops-repo-with-github-repo-35a958d7784e
2, Then after the above pipeline pushes the changes, you need to captrue the changes via the CI trigger of the YAML pipeline on DevOps side.
trigger:
- <branch name>
pool:
vmImage: ubuntu-latest
variables:
- name: system.debug
value: true
steps:
- script: |
echo xxx
displayName: 'Run a multi-line script'
Original Answer:
If you mean integrating Github repo and Azure DevOps pipeline, for example, you need continuous integration on main branch of your repo.
Then, follow the below steps.
1, For classic pipeline:
2, For YAML pipeline:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
If what you mean is not integrating Github repo and Azure DevOps pipeline, please clarify your requirements.
I am very new in Azure Devops. All my repositories are in BitBucket. I have one project say A which has dependency in project B
In Visual Studio there are no build errors, but when I am building the CI pipeline for A I am getting an error saying "Project B not found"
The CI pipeline is created with repository of project A. How can I solve this?
Note: I am using dev.azure.com and selected "Use classic editor to create a pipeline without YAML" so I am not able to add yaml code. Can you please tell me how to do this?
Sounds like you need to check out multiple repositories in your pipeline for example like this:
resources:
repositories:
- repository: MyBitbucketRepoA
type: bitbucket
endpoint: MyBitbucketServiceConnection
name: MyBitbucketOrgOrUser/MyBitbucketRepoA
- repository: MyBitbucketRepoB
type: bitbucket
endpoint: MyBitbucketServiceConnection
name: MyBitbucketOrgOrUser/MyBitbucketRepoB
steps:
- checkout: self
- checkout: MyBitbucketRepoA
- checkout: MyBitbucketRepoB
I have yaml file for the azure pipline in a repo. And I need to run powershell script from a different repo.
As far as I understood I can add side repo to resources section in yaml and then use task:ShellScript#2 with scriptPath parameter. But as I understood it works relatively for repo in which yaml is placed. And I'm not sure how can I access file from a different repo.
Yes, you have to use repository resource and checkout that repo as follows:
resources:
repositories:
- repository: devops
type: github
name: kmadof/devops-templates
endpoint: kmadof
steps:
- checkout: self
- checkout: devops
- task: ShellScript#2
inputs:
scriptPath: $(Agent.BuildDirectory)/devops/scripts/some-script.sh
we are setting up Policies for our Org. One need we have is to have a build and annotation at every Pull Request with Sonarcloud. Is there a way to create a common ci build pipeline which will run at every PR, checking out the respective repo, detect the type of project (or read a manifest file or so) do the code analytics and build, annotates the PR?
So in the Azure DevOps repo you can create a common branch policy for all the master branches. I tried to use a standalone yaml pipeline but it never started when I created the pr. Can someone help me on the right track? Do I need to create a resource in the yaml? Is there any variable I can use from the PR to detect the repo and the branches?
Just for everybody to understand, you can create common branch policies and individuals.
Thanks a lot
You will need to add all the repositories in the resources section in the yaml because of the known issue reported in this thread.
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
However, if you use a classic UI Pipeline in the Build Validation instead of the yaml pipeline. You donot need to add all the repositories in the resources section. But you need to skip the pipeline to sync the source: (Go pipeline edit page-->Click Get sources-->check Don't sync sources)
You can use the predefined variables to get the information about the pull request. See below:
$(Build.Repository.Name)
$(System.PullRequest.PullRequestId)
Then you can run the git commands in a script task to check out the pull request branch. See below example:
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
pool:
vmImage: windows-latest
steps:
- checkout: none
- powershell: |
git clone "https://$(System.AccessToken)#dev.azure.com/OrganizationName/$(System.TeamProject)/_git/$(Build.Repository.Name)"
cd "$(Build.Repository.Name)"
git fetch origin pull/$(System.PullRequest.PullRequestId)/merge:pr-$(System.PullRequest.PullRequestId)
# checkout pr branch
git checkout pr-$(System.PullRequest.PullRequestId)
Note: you need to grant build service account read permission for target repository
How could I rewrite this python script so that it can run within azure devops pipeline and export the dataframe as a csv to the devops repository. I'm able to achieve this locally but would like to achieve this remotely.
Put different, how can I export a pandas dataframe to devops repos folder as a csv file using an azure devops pipeline task. Below is the python script that needs to run as a pipeline task.
local_path in this case should be azure devops path.
from azureml.core import Workspace, Dataset
local_path = 'data/prepared.csv'
dataframe.to_csv(local_path)
⚠️You really should not do this. Azure pipelines are for building code, not for processing data. Assuming that you meant Azure DevOps Pipelines, opposed to Azure ML Pipelines.
Also you should not commit data to your repository.
If you still want to proceed, here is an example for what you try to achieve. Note that for the last line, i.e. git push, you need to give the agent permission to write the repository. See Run Git commands in a script for an approximate☹️ documentation on how to do that on your account.
trigger: none
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
persistCredentials: true
- task: UsePythonVersion#0
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- script: |
python your_data_generating_script.py
git config --global user.email "you#example.com"
git config --global user.name "Your Name"
git add data/prepared.csv
git commit -m'test commit'
git push origin HEAD:master
displayName: 'push data to master'