I have a problem executing a pipeline in azure devops.
This task is part of my pipeline, as you can see it is an instruction to clone a repository.
- script: |
mkdir FOLDER_PUBLISH_AZURE
git clone https://$(AZURE_DEPLOY_USERNAME):$env:MY_AZURE_DEPLOY_PASSWORD#develroomlabs-events.scm.azurewebsites.net:443/develroomlabs-events.git FOLDER_PUBLISH_AZURE
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Download SCM repo for pipeline '
env:
MY_AZURE_DEPLOY_PASSWORD: $(AZURE_DEPLOY_PASSWORD)
The result I get is as follows:
But the pipeline variables are well added, I did a test by directly adding username and password in the YAML file, but the result was the same.
What could be happening? Am I doing something wrong?
I do this same task of cloning a repository from a bitbucket pipeline and I have no problems.
Try doing that in Windows environment with Microsoft-hosted windows agents instead of Micorosft-hosted Linux agents.
I can reproduce same issue when using Linux environment, but it actually works well in windows environment. I think this issue could be related to that Linux has different behavior when treating url format like https://$AppName:xxx#sAppName.scm.azurewebsites.net:443/AppName.git...
(Something looks like this ticket,the behavior should have something to do with Linux server itself)
Related
In my .gitlab-ci.yml i'm trying to deploy on merge request.
My pipeline works, script is executed , everything is ok and running , but i'm not able to read any predefined environment variables. My files looks like :
executeAutomationTests:
stage: check
only:
refs:
- merge_requests
script:
- echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
But $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME is not resolved. I need to know source branch for the merge_request in order to pull the code and make a deployment. I have tried other variables like : $CI_COMMIT_REF_NAME, $CI_JOB_STAGE , but non of them are resolved.
My GitLab version : GitLab Community Edition 13.4.2
The problem was that in the pipeline job monitor the variable wasn't resolved and that's confused me. It was looking like :
So, for anyone who is fighting with such things, keep that in mind and also, keep in mind differences in syntax between powershell , bash and so on ..
I'm working with SVN but I would like to move on to Git, and more specifically to Gitlab.
I have the following structure:
MyStructure/
customer/
client1/
delivery.sh
MyFiletoSend.sh
client2/
delivery.sh
MyFiletoSend2.sh
Currently, the "delivery.sh" will send the modifications (rsync) of the file "MyFiletoSend.sh" to the server "client1".
Can I run the "delivery.sh" via Gitlab automatically after/before the git push only on the files modified in this push?
Example:
I have a modification to make to the file "MyFiletoSend.sh" from client1/
I make my change
commit and push
Gitlab is running "delivery.sh" on my "client1/" file.
The file "MyFiletoSend.sh" is sent to the server of "client1" without touching "client2".
Yes, it is possible
but first of all you need to understand how gitlab ci works. Read this article https://docs.gitlab.com/ee/ci/yaml/
You will create a step in your pipeline that will do what you want after you push the code (in master or in any other branch/mr)
and about the job? you have to create one, you can use this code to help you
https://gist.github.com/hnlq715/6c222ba0fd868bae7e4dfd3af61bf26e
Assuming your delivery.sh scripts have all the rsync logic required, GitLab has built-in logic to detect changes in files and execute bash commands in response. You can create a separate job for each client, which can run in parallel in the same stage. This approach is also auditable in that it will clearly show you which clients got updated and with which version of the file.
update-client-1:
stage: update-clients
only:
changes:
# Detect change only in MyFiletoSend.sh:
- customer/client1/MyFiletoSend.sh
# Detect any change in the customer folder:
- customer/client1/*
script:
- cd customer/client1
- delivery.sh
update-client-2:
stage: update-clients
only:
changes:
- customer/client2/*
script:
- cd customer/client2
- delivery.sh
# repeat for all remaining clients
For more information: https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges
I am setting up my fork of a Github project with the azure_pipeline.yaml configuration.
This seems to work just fine for everyone else in the community but when I setup the pipeline it gives me the following exception:
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'compile_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'test_ci_build', step ''.)
A task is missing. The pipeline references a task called 'Cache'. This usually indicates the task isn't installed, and you may be able to install it from the Marketplace: https://marketplace.visualstudio.com. (Task version 2, job 'e2e_ci_build', step ''.)
Specifically,
Here is my Azure pipeline link
I am creating Flink CI build pipeline according to this instruction.
Which already have an azure-pipeline.yml in the repo
It uses the template to run the job parameterized in tools/azure-pipelines/jobs-template.yml
[UPDATE]
I modified the jobs-templates.yml and commented out all steps with Cache#2 and it runs fine.
Was able to get this working eventually.
Apparently for my Azure account. I am not allow to use Cache#2.
changing all lines with
- task: Cache#2
to
- task: CacheBeta#1
resolves all my problem.
By design, Azure DevOps does not automatically make all tasks available when you run a pipeline.
You have to add them manually as part of the pipeline.
'cache' is one such task.
I'm doing this in classic gui (non-yaml) mode as I find it easier to search for things
But what you do is in your pipeline add a new task and in the task search box type 'cache'.
This will bring up the task.
Click Add to include it in the pipeline.
For more information on this I would recommend reading:
https://learn.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops
I do not see you added cache task in your pipeline from your git repo.
It should look like the example here:
variables:
YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn
steps:
- task: Cache#2
inputs:
key: 'yarn | "$(Agent.OS)" | yarn.lock'
restoreKeys: |
yarn | "$(Agent.OS)"
yarn
path: $(YARN_CACHE_FOLDER)
displayName: Cache Yarn packages
- script: yarn --frozen-lockfile
Source
I was getting the same error A task is missing. The pipeline references a.. on one of azure task, PublishCucumberReport#1 . I resolved it by visiting https://marketplace.visualstudio.com/ and going to the task and then clicking on get free button, which installs it on your pipeline
I have a repository in Gitlab and what I wish to have is a setup in which when a specific file in a specific branch is changed, I want a script/job to be triggered, which will read and make operations based on this new version of the changed file.
That script can be in another machine and be accessed through SSH, or it can be inside the same repository and be executed somehow.
Is there any way to do this with Gitlab CI/CD?
Edit: I'm using GitLab Enterprise Edition 11.2.3-ee aadca99
You can use only/except changes to do this.
It has been introduced in Gitlab 11.4 and it works with files and directories within your repository, example :
docker build:
script: docker build -t my-image:$CI_COMMIT_REF_SLUG .
only:
changes:
- Dockerfile
- docker/scripts/*
- dockerfiles/**/*
- more_scripts/*.{rb,py,sh}
I have GitLab running on computer A, development environment (Visual studio Pro) on computer B and Windows Server on computer C.
I set up GitLab-Runner on computer C (Windows server). I also set up .gitlab-ci.yml file to perform build and run tests for ASP.NET Core application on every commit.
I don't know how can I get code on computer C (Windows server) so I can build it (dotnet msbuild /p:Configuration=Release "%SOLUTION%"). It bothers me that not a single example .gitlab-ci.yml I found on net, doesn't pull code form GitLab, before building application. Why?
Is this correct way to set-up CI/CD:
User create pull request (a new branch is created)
User writes code
User commit code to branch from computer B.
GitLab runner is started on computer C.
It needs to pull code from current branch (CI_COMMIT_REF_NAME)
Build, test, deploy ...
Should I use common git command to get the code, or is this something GitLab runner already do? Where is the code?
Why no-one pull code from GitLab in .gitlab-ci.yml?
Edited:
I get error
'"git"' is not recognized as an internal or external command
. Solution in my case was restart GitLab-Runner. Source.
#MilanVidakovic explain that source is automatically downloaded (which I didn't know).
I just have one remaining problem of how to get correct path to my .sln file.
Here is my complete .gitlab-ci.yml file:
variables:
SOLUTION: missing_path_to_solution #TODO
before_script:
- dotnet restore
stages:
- build
build:
stage: build
script:
- echo "Building %CI_COMMIT_REF_NAME% branch."
- dotnet msbuild /p:Configuration=Release "%SOLUTION%"
except:
- tags
I need to set correct variable for SOLUTION. My dir (where GitLab-Runner is located) currently holds this folder/files:
- config.toml
- gitlab-runner.exe
- builds/
- 7cab42e4/
- 0/
- web/ # I think this is project group in GitLab
- test/ # I think this is project name in GitLab
- .sln
- AND ALL OTHER PROJECT FILES #Based on first look
- testm.tmp
So, what are 7cab42e4, 0. Or better how to get correct path to my project structure? Is there any predefined variable?
Edited2:
Answer is CI_PROJECT_DIR.
I'm not sure I follow completely.
On every commit, Gitlab runner is fetching your repository to C:\gitlab-runner\builds.. on the local machine (Computer C), and builds/deploys or does whatever you've provided as an action for the stage.
Also, I don't see the need for building the source code again. If you're using Computer C for both runner and tests/acceptance, just let the runner do the building and add Artifacts item in your .gitlab-ci.yaml. Path defined in artifacts will retain your executables on Computer C, which you are then able to use for whatever purposes.
Hope it helps.
Edit after comment:
When you push to repository, Gitlab CI/CD automatically checks your root folder for .gitlab-ci.yaml file. If its there, the runner takes over, parses the file and starts executing jobs/stages.
As soon as the file itself is valid and contains proper jobs and stages, runner fetches the latest commit (automatically) and does whatever script item tells it to do.
To verify that everything works correctly, go to your Gitlab -> CI / CD -> Pipelines, and check out whats going on. You should see something like this:
Maybe it would be best if you posted your .yaml file, there could be a number of reasons your runner is not picking up the code. For instance, maybe your .yaml tags are not matching what runner is created to pick up etc.