Azure devops VFS pipeline is doing sparse checkout instead of full clone on repo.?is there anyway we can do a complete repo clone?
You can tune your checkout process and achieve similar results using a different way.
First of all, you can clean the repo during the checkout process:
steps:
- checkout: self
clean: true
Next, you can check out more than one repo:
steps:
- checkout: self
- checkout: MyGitHubToolsRepo
- checkout: git://MyResourcesProject/resources
Moreover, you can place every into the different folder using "path" parameter (path will be relative to the agent's build directory).
steps:
- checkout: self
path: PutMyCodeHere
You can find other configuration properties here.
Related
OK, so the main question here is: How to Sync Github with Azure?
Now, I had main reference source: How to synchronize Azure Repos with External Git Repos
https://faun.pub/how-to-synchronize-azure-repos-with-external-git-repos-70ff92e51c63
And that is perfect match, but here is a catch, it is kind of abstract and you have to have work with Azure experience, like knowing what is YAML for and how to use it.
Long story short, it did not work. And then I found this gentleman's video: Merge From Github to Azure DevOps
https://www.youtube.com/watch?v=Kks1pCG51bI
That is super close, yet there still were several error and bugs ,like:
Cannot find path 'D:\a\1\s\copyrepo~' because it does not exist;
remote: Not Found fatal: repository 'https://github.com/***/' not found ##[warning]Git fetch failed with exit code 128, back off 5.443 seconds before retry.
error: unable to create file Filename too long
So, that is a bummer...
I mean, yes, you technically can create and synchronize GitHub repo with Azure, but you have to create new GitHub repo, with existing one there was: error: unable to create file Filename too long.
Please tell me what you think.
Try to use the following yaml file as I used to sync Github to Azure Repos:
name: Sync Azure with Github
variables:
REMOTE_ADDR: 'https://github.com/{user{/{repo.git}'
stages:
- stage: syncing Repos
displayName: syncing Repos
jobs:
- job: run_Git_Commands
displayName: run_Git_Commands
continueOnError: false
steps:
- checkout: self
clean: true
persistCredentials: true
displayName: run_commands
- bash: |
git checkout master
git remote add repoGithub $(REMOTE_ADDR)
git fetch repoGithub master
git reset --hard repoGithub/master
git pull --rebase repoGithub master
git push --force origin
I got a requirement to generate, archive and reuse the artifacts between two different repositories
Repository A: Compile Angular code and create a XLF file
Repository B: Use the 'XLF File' generated above and create a new XLF file
Repository A: Again use the newly generated XLF file to create the final output file
The activities mentioned above should be done using gitlab-ci.yml. I am not sure how to handle this using GitLab CI.
We can push the artifact from Repo A to Repo B. However, CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Ideally, you would not push a generated artifact to another Git source repository.
But a GitLab pipeline can retrieve an artifact produced by another one from its URL.
To avoid the back and forth, I would rather have 3 jobs instead of two
the first generates XLF file
the second curls/fetches that file, and use it to generate new XLF file
the third job curls/fetches that file, and complete the process.
How to release built artifacts back-and-forth from one to another repo on GitLab?
Repository A:
Compile Angular code and create a XLF file
Send a hook to repository B that it just compiled
just trigger: https://docs.gitlab.com/ee/ci/yaml/#trigger , works like a charm. It's even nicely visible in the gui.
or API https://docs.gitlab.com/ee/ci/triggers/
pass variables: PARENT_PIPELINE_ID: $CI_PIPELINE_ID to repository B so it can download artifacts from specific pipeline
Repository B:
Use the 'XLF File' generated above
use needs: https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-to-child-pipelines to download artifacts
or API: have personal access token from repository A https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html added to environment variables and use API to download artifacts https://docs.gitlab.com/ee/api/job_artifacts.html .
create a new XLF file
use trigger: or API to trigger repository A
but this time trigger different .gitlab-ci.yml file like: trigger: - project: repositoryA file: second_stage.gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/#trigger-child-pipeline-with-files-from-another-project
or use like variables: SECOND_STAGE: "true" and use a variable to differentiate
Repository A:
run pipeline from the file second_stage.gitlab-ci.yml
download artifacts from repository B - needs: or API
use the newly generated XLF file to create the final output file
Overall, what you need is rules: and needs: documentation. On older gitlab, it was done with API.
CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Don't wait. Let the API trigger it.
I tried the following approach and it worked fine or at least I was able to proceed
Due to some reason 'variables' along with CURL did not work as expected but I did not analyze the root cause
Repo A - Pipeline
trigger-repob: (Trigger Project B of Repo B)
stage: repob
trigger:
project: repob-namespace/projectb
branch: devops
test_job:
image: $CI_REGISTRY/$CI_PROJECT_PATH/base-image:latest
stage: test_pipeline
when: delayed
start_in: 2 minutes
needs: (Use artifacts from Repo B/Project B)
-
project: repob-namespace/projectb
job: buildprojectb
ref: devops
artifacts: true
script:
- do something here
Repo B Pipeline
buildprojectb:
image: php:7.4.11
stage: build
script:
- do something here
artifacts:
paths:
- outputs/*.xlf
I have yaml file for the azure pipline in a repo. And I need to run powershell script from a different repo.
As far as I understood I can add side repo to resources section in yaml and then use task:ShellScript#2 with scriptPath parameter. But as I understood it works relatively for repo in which yaml is placed. And I'm not sure how can I access file from a different repo.
Yes, you have to use repository resource and checkout that repo as follows:
resources:
repositories:
- repository: devops
type: github
name: kmadof/devops-templates
endpoint: kmadof
steps:
- checkout: self
- checkout: devops
- task: ShellScript#2
inputs:
scriptPath: $(Agent.BuildDirectory)/devops/scripts/some-script.sh
So I'm trying to learn deployement with Azure Devops. I have this Angular app sitting in Gitlab which already has a CI/CD pipeline with jenkins to kubernetes cluster. So i was thinking to do the same with Azure Devops via YAML. Which is not possible according to Azure docs directly from gitlab.
So what i'm trying to do is create CI pipeline from github which takes checkout from gitlab UI repo and build it for deployement.
I have created a Repository Resource in my below pipeline YAMl file. Azure give me error saying:
Repository JpiPipeline references endpoint https://gitlab.com/myusername/myUiRepo.git which does not exist or is not authorized for use
trigger:
- master
resources:
repositories:
- repository: UiPipeline. #alias
type: git
name: repository_name
# ref: refs/heads/master # ref name to use; defaults to 'refs/heads/master'
endpoint: https://#gitlab.com/myusername/myUiRepo.git # <-- Is this possible
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-16.04'
continueOnError: true
steps:
- checkout: JpiPipeline
- script: echo "hello to my first Build"
Repository type gitlab is not support in YAML pipeline yet. The currently supported types are git, github, and bitbucket, see supported types.
The workaround to get the gitlab repo sources is to run git command inside the script tasks.
For below example Yaml pipeline:
- checkout: none to avoid checkout the github source.
Use git clone https://username:password#gitlab.com/useraccount/reponame.git to clone the gitlab repo inside a script task.
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-latest'
steps:
- checkout: none
- script: |
git clean -ffdx
git clone https://username:password#gitlab.com/useraccount/reponame.git
#if your password or username contain # replace it with %40
Your gitlab repo will be clone to folder $(system.defaultworkingdirectory)/reponame
Another workaround is to classic UI pipeline. Gitlab repo type is supported in Classic UI pipeline.
You can choose Use the classic editor to create a classic ui pipeline.
When you come to select source page. Choose other git and click Add connection to add your gitlab repo url. Then the pipeline will get the sources for your gitlab repo.
I'm setting up several pipelines in Azure DevOps. To make my teams life easier, I'm using job templates.
These job templates are in a a proper repository, just for them.
For every pipeline I define the repository to get the templates from.
Some tasks in these templates run powershell code, and I want this code to be in a script file, to be reusable and stored in the same repo as the template.
When the pipelines runs, the template is embeded, it tries to locate the powershell script inside project repo actually being built/deployed.
How can i achieve this?
The workaround is to have inline code which I really don't want to have.
Any constructive answer will be very appreciated.
Thanks
After some digging I couldn't find any way to specify a script file as source to powershell task in a template.
Inside pipeline definition:
resources:
repositories:
- repository: templates
type: git
name: deploy-templates
variables:
artifactName: 'Trade Data ETL - $(Build.SourceBranchName)'
stages:
- stage: Build
displayName: Build
variables:
- group: DEV-Credential-Group
- group: COMMON-Settings-Group
jobs:
- template: ssis/pipelines/stage-build.yml#templates # Template reference
parameters:
artifactName: '$(artifactName)'
Inside template file:
- task: PowerShell#2
inputs:
filePath: ssis/pipelines/scripts/build-ssis-project.ps1
arguments: '-ProjectToBuild "tradedata-ldz-ssis/tradedata-ldz-ssis.dtproj'
pwsh: true
Update 2021
According to learn.microsoft.com, you can now also check out multiple repositories without custom scripting.
If you check out more than one repository, a separate folder containing the repository is created below $(Build.SourcesDirectory).
You can define multiple repositories like this:
resources:
repositories:
- repository: devops
type: git
name: DevOps
ref: main
- repository: infrastructure
type: git
name: Infrastructure
ref: main
And in the steps simply check them out as follows:
steps:
- checkout: self
- checkout: devops
- checkout: infrastructure
# List all available repositories
- script: ls
Original Answer
Currently the resources command only supports yml files in other repositories. However, you could simply checkout the repository in a task and then run the desired powershell script.
steps:
- task: PowerShell#2
inputs:
targetType: inline
script: |
git clone -b <your-desired-branch> https://azuredevops:$($env:token)#dev.azure.com/<your-organization>/<your-project>/_git/<your-repository> <target-folder-name>
./<target-folder-name>/foo.ps1
env:
token: $(System.AccessToken)
This script would checkout an arbitrary branch and execute a script foo.ps1 in the root of the target repository.
Call - checkout: templates inside the template file. This might only work when you insert a template but it successfully sees the repository resource and pulls it down.
You can copy the script files from source directory. Currently, you have not mentioned the root folder -
ssis/pipelines/scripts/build-ssis-project.ps1
Assuming, you are building on a repo where the powershell script resides -
Try -
- task: PowerShell#1
inputs:
scriptName: '$(ScriptsDir)/ssis/pipelines/scripts/build-ssis-project.ps1'
Pass the value of ScriptsDir where it could be the build source directory or build working directory