How to Sync Azure with Github, issues: Cannot find path 'D:\a\1\s\ because it does not exist; Repository not found fetch failed; Filename too long - azure

OK, so the main question here is: How to Sync Github with Azure?
Now, I had main reference source: How to synchronize Azure Repos with External Git Repos
https://faun.pub/how-to-synchronize-azure-repos-with-external-git-repos-70ff92e51c63
And that is perfect match, but here is a catch, it is kind of abstract and you have to have work with Azure experience, like knowing what is YAML for and how to use it.
Long story short, it did not work. And then I found this gentleman's video: Merge From Github to Azure DevOps
https://www.youtube.com/watch?v=Kks1pCG51bI
That is super close, yet there still were several error and bugs ,like:
Cannot find path 'D:\a\1\s\copyrepo~' because it does not exist;
remote: Not Found fatal: repository 'https://github.com/***/' not found ##[warning]Git fetch failed with exit code 128, back off 5.443 seconds before retry.
error: unable to create file Filename too long
So, that is a bummer...
I mean, yes, you technically can create and synchronize GitHub repo with Azure, but you have to create new GitHub repo, with existing one there was: error: unable to create file Filename too long.
Please tell me what you think.

Try to use the following yaml file as I used to sync Github to Azure Repos:
name: Sync Azure with Github
variables:
REMOTE_ADDR: 'https://github.com/{user{/{repo.git}'
stages:
- stage: syncing Repos
displayName: syncing Repos
jobs:
- job: run_Git_Commands
displayName: run_Git_Commands
continueOnError: false
steps:
- checkout: self
clean: true
persistCredentials: true
displayName: run_commands
- bash: |
git checkout master
git remote add repoGithub $(REMOTE_ADDR)
git fetch repoGithub master
git reset --hard repoGithub/master
git pull --rebase repoGithub master
git push --force origin

Related

How to release built artifacts back-and-forth from one to another repo on GitLab?

I got a requirement to generate, archive and reuse the artifacts between two different repositories
Repository A: Compile Angular code and create a XLF file
Repository B: Use the 'XLF File' generated above and create a new XLF file
Repository A: Again use the newly generated XLF file to create the final output file
The activities mentioned above should be done using gitlab-ci.yml. I am not sure how to handle this using GitLab CI.
We can push the artifact from Repo A to Repo B. However, CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Ideally, you would not push a generated artifact to another Git source repository.
But a GitLab pipeline can retrieve an artifact produced by another one from its URL.
To avoid the back and forth, I would rather have 3 jobs instead of two
the first generates XLF file
the second curls/fetches that file, and use it to generate new XLF file
the third job curls/fetches that file, and complete the process.
How to release built artifacts back-and-forth from one to another repo on GitLab?
Repository A:
Compile Angular code and create a XLF file
Send a hook to repository B that it just compiled
just trigger: https://docs.gitlab.com/ee/ci/yaml/#trigger , works like a charm. It's even nicely visible in the gui.
or API https://docs.gitlab.com/ee/ci/triggers/
pass variables: PARENT_PIPELINE_ID: $CI_PIPELINE_ID to repository B so it can download artifacts from specific pipeline
Repository B:
Use the 'XLF File' generated above
use needs: https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-to-child-pipelines to download artifacts
or API: have personal access token from repository A https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html added to environment variables and use API to download artifacts https://docs.gitlab.com/ee/api/job_artifacts.html .
create a new XLF file
use trigger: or API to trigger repository A
but this time trigger different .gitlab-ci.yml file like: trigger: - project: repositoryA file: second_stage.gitlab-ci.yml https://docs.gitlab.com/ee/ci/yaml/#trigger-child-pipeline-with-files-from-another-project
or use like variables: SECOND_STAGE: "true" and use a variable to differentiate
Repository A:
run pipeline from the file second_stage.gitlab-ci.yml
download artifacts from repository B - needs: or API
use the newly generated XLF file to create the final output file
Overall, what you need is rules: and needs: documentation. On older gitlab, it was done with API.
CI on Repo A should wait until Repo B pushes a new artifact to Repo A to complete the process
Don't wait. Let the API trigger it.
I tried the following approach and it worked fine or at least I was able to proceed
Due to some reason 'variables' along with CURL did not work as expected but I did not analyze the root cause
Repo A - Pipeline
trigger-repob: (Trigger Project B of Repo B)
stage: repob
trigger:
project: repob-namespace/projectb
branch: devops
test_job:
image: $CI_REGISTRY/$CI_PROJECT_PATH/base-image:latest
stage: test_pipeline
when: delayed
start_in: 2 minutes
needs: (Use artifacts from Repo B/Project B)
-
project: repob-namespace/projectb
job: buildprojectb
ref: devops
artifacts: true
script:
- do something here
Repo B Pipeline
buildprojectb:
image: php:7.4.11
stage: build
script:
- do something here
artifacts:
paths:
- outputs/*.xlf

Azure DevOps Common Branch Policy Build Pipeline for all Repositories

we are setting up Policies for our Org. One need we have is to have a build and annotation at every Pull Request with Sonarcloud. Is there a way to create a common ci build pipeline which will run at every PR, checking out the respective repo, detect the type of project (or read a manifest file or so) do the code analytics and build, annotates the PR?
So in the Azure DevOps repo you can create a common branch policy for all the master branches. I tried to use a standalone yaml pipeline but it never started when I created the pr. Can someone help me on the right track? Do I need to create a resource in the yaml? Is there any variable I can use from the PR to detect the repo and the branches?
Just for everybody to understand, you can create common branch policies and individuals.
Thanks a lot
You will need to add all the repositories in the resources section in the yaml because of the known issue reported in this thread.
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
However, if you use a classic UI Pipeline in the Build Validation instead of the yaml pipeline. You donot need to add all the repositories in the resources section. But you need to skip the pipeline to sync the source: (Go pipeline edit page-->Click Get sources-->check Don't sync sources)
You can use the predefined variables to get the information about the pull request. See below:
$(Build.Repository.Name)
$(System.PullRequest.PullRequestId)
Then you can run the git commands in a script task to check out the pull request branch. See below example:
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepo
pool:
vmImage: windows-latest
steps:
- checkout: none
- powershell: |
git clone "https://$(System.AccessToken)#dev.azure.com/OrganizationName/$(System.TeamProject)/_git/$(Build.Repository.Name)"
cd "$(Build.Repository.Name)"
git fetch origin pull/$(System.PullRequest.PullRequestId)/merge:pr-$(System.PullRequest.PullRequestId)
# checkout pr branch
git checkout pr-$(System.PullRequest.PullRequestId)
Note: you need to grant build service account read permission for target repository

Run Python Code within Azure Devops Pipeline and Export output to folder in Devops Repos ($(Build.SourcesDirectory))

How could I rewrite this python script so that it can run within azure devops pipeline and export the dataframe as a csv to the devops repository. I'm able to achieve this locally but would like to achieve this remotely.
Put different, how can I export a pandas dataframe to devops repos folder as a csv file using an azure devops pipeline task. Below is the python script that needs to run as a pipeline task.
local_path in this case should be azure devops path.
from azureml.core import Workspace, Dataset
local_path = 'data/prepared.csv'
dataframe.to_csv(local_path)
⚠️You really should not do this. Azure pipelines are for building code, not for processing data. Assuming that you meant Azure DevOps Pipelines, opposed to Azure ML Pipelines.
Also you should not commit data to your repository.
If you still want to proceed, here is an example for what you try to achieve. Note that for the last line, i.e. git push, you need to give the agent permission to write the repository. See Run Git commands in a script for an approximate☹️ documentation on how to do that on your account.
trigger: none
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
persistCredentials: true
- task: UsePythonVersion#0
inputs:
versionSpec: '3.8'
addToPath: true
architecture: 'x64'
- script: |
python your_data_generating_script.py
git config --global user.email "you#example.com"
git config --global user.name "Your Name"
git add data/prepared.csv
git commit -m'test commit'
git push origin HEAD:master
displayName: 'push data to master'

Is it possible to checkout Gitlab repository in YML which sits in Github?

So I'm trying to learn deployement with Azure Devops. I have this Angular app sitting in Gitlab which already has a CI/CD pipeline with jenkins to kubernetes cluster. So i was thinking to do the same with Azure Devops via YAML. Which is not possible according to Azure docs directly from gitlab.
So what i'm trying to do is create CI pipeline from github which takes checkout from gitlab UI repo and build it for deployement.
I have created a Repository Resource in my below pipeline YAMl file. Azure give me error saying:
Repository JpiPipeline references endpoint https://gitlab.com/myusername/myUiRepo.git which does not exist or is not authorized for use
trigger:
- master
resources:
repositories:
- repository: UiPipeline. #alias
type: git
name: repository_name
# ref: refs/heads/master # ref name to use; defaults to 'refs/heads/master'
endpoint: https://#gitlab.com/myusername/myUiRepo.git # <-- Is this possible
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-16.04'
continueOnError: true
steps:
- checkout: JpiPipeline
- script: echo "hello to my first Build"
Repository type gitlab is not support in YAML pipeline yet. The currently supported types are git, github, and bitbucket, see supported types.
The workaround to get the gitlab repo sources is to run git command inside the script tasks.
For below example Yaml pipeline:
- checkout: none to avoid checkout the github source.
Use git clone https://username:password#gitlab.com/useraccount/reponame.git to clone the gitlab repo inside a script task.
stages:
- stage: Checkout
jobs:
- job: Build
pool:
vmImage: 'Ubuntu-latest'
steps:
- checkout: none
- script: |
git clean -ffdx
git clone https://username:password#gitlab.com/useraccount/reponame.git
#if your password or username contain # replace it with %40
Your gitlab repo will be clone to folder $(system.defaultworkingdirectory)/reponame
Another workaround is to classic UI pipeline. Gitlab repo type is supported in Classic UI pipeline.
You can choose Use the classic editor to create a classic ui pipeline.
When you come to select source page. Choose other git and click Add connection to add your gitlab repo url. Then the pipeline will get the sources for your gitlab repo.

How to clone complete repo in YAML pipeline instead of sparse checkout?

Azure devops VFS pipeline is doing sparse checkout instead of full clone on repo.?is there anyway we can do a complete repo clone?
You can tune your checkout process and achieve similar results using a different way.
First of all, you can clean the repo during the checkout process:
steps:
- checkout: self
clean: true
Next, you can check out more than one repo:
steps:
- checkout: self
- checkout: MyGitHubToolsRepo
- checkout: git://MyResourcesProject/resources
Moreover, you can place every into the different folder using "path" parameter (path will be relative to the agent's build directory).
steps:
- checkout: self
path: PutMyCodeHere
You can find other configuration properties here.

Resources