Im using official terraform orb for CircleCi pipelines. I would like to specify path to my terraform code, but the error shows that path doesn't exist. But when I'm going into docker which run that job, that path is available. I don't understand why.
jobs:
- terraform/fmt:
checkout: true
path: 'project/terraform/'
context: terraform
We are on the same page. I have seen this error happening with me before and then without any changes on my side the pipeline started to work again. I think there is a problem on CircleCI
Related
I have been developing Terraform v0.14.7 to run via GitLab Pipeline. Everything's been going well until I tried upgrading Terraform to v0.15.5 (Upgrading above that is beyond my control at the moment). I followed the advice here and a how-to guide here.
When I run terraform version on my Terraform server, it shows the proper upgraded version, and I can successfully run terraform init each time just fine. However, when I run my pipeline, the version seems to flip between the pre, validate, and plan stages with no consistent pattern. I have it print terraform version at the start of each stage, and sometimes pre outputs 0.14.7, validate outputs 0.15.5, and then plan flips back to 0.14.7. Sometimes the order is different as well, such as pre outputting 0.15.5 and the other two stages 0.14.7. I can't figure out why it's doing this since I can't recreate the problem manually.
It's been very confusing trying to solve - I tried clearing the runner cache, removing the .terraform cache and .terraform.lock.hcl files, and switching up how I run the init (adding backend=false or lockfile=readonly flags). Clearing the cache and files allows for one single clean pipeline run, but the next run is always back to being mixed versions. Does anyone know what I'm doing wrong?
Edit: The closest documented issue I can find is this issue here, but a solution was never posted. I have also tried terraform refresh, init -migrate-state, and init -reconfigure with no luck.
Edit 2: I am running Terraform offline with no internet access, so upgrades need to be done manually in case that changes any answer.
Edit 3: I'm unable to paste the exact code but here's as much as I can give, not that the output at the beginning of each stage is Terraform has been successfully initialized!:
.gitlab-ci.yml:
before_script:
- |
export TF_VAR_pipeline=$CI_PIPELINE_URL
terraform -chdir="$WORKING_DIR" init -backend=false
pre:
stage: pre
script:
- which terraform
- terraform version
validate:
stage: validate
script:
- which terraform
- terraform version
- terraform -chdir="$WORKING_DIR" validate
plan:
stage: plan
script:
- which terraform
- terraform version
- terraform -chdir="$WORKING_DIR" plan -out=plan_file.txt
artifacts:
paths:
- $WORKING_DIR/plan_file.txt
when: on success
example pre output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.15.5
on linux_amd64
Saving cache
Creating cache default-3...
...
No URL provided, cache will be not up-loaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up file based variables
Job succeeded
example validate output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.15.5
on linux_amd64
$ terraform -chdir="$WORKING_DIR" validate
Success! The configuration is valid.
Saving cache
Creating cache default-3...
...
No URL provided, cache will be not up-loaded to shared cache server. Cache will be stored only locally.
Created cache
Cleaning up file based variables
Job succeeded
example plan output:
$ which terraform
/usr/bin/terraform
$ terraform version
Terraform v0.14.7
$ terraform -chdir="$WORKING_DIR" plan -out=plan_file
An execution plan has been generated and is shown below.
...
Unfortunately this issue was never solved. I believe it had something to do with the Gitlab cache but I couldn't figure it out. I ended up spinning up a new Terraform server instance and installing a fresh gitlab-runner, all seems to be working well now.
I have a gitlab job that does not seem to update the repository before being run. Sometimes it leaves some files in their old states and run the script... Any idea ?
For instance when I have a
packagePython:
stage: package
script:
- .\scripts\PackagePython.ps1
tags:
- myServer
cache:
paths:
- .\python\cache\
only:
changes:
- python/**/*
I finally managed to understand what was happening :
I realised that the gitlab-runner did not use exactly the same path for each run on my server, and my script assumed that it did... So I ended up pointing on a build made on the wrong path.
I guess if you think that it is not updating the repository (like I did) make sure you are not referencing hardcoded path/package in your scripts that could refer to previous versions !
In my .gitlab-ci.yml i'm trying to deploy on merge request.
My pipeline works, script is executed , everything is ok and running , but i'm not able to read any predefined environment variables. My files looks like :
executeAutomationTests:
stage: check
only:
refs:
- merge_requests
script:
- echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
But $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME is not resolved. I need to know source branch for the merge_request in order to pull the code and make a deployment. I have tried other variables like : $CI_COMMIT_REF_NAME, $CI_JOB_STAGE , but non of them are resolved.
My GitLab version : GitLab Community Edition 13.4.2
The problem was that in the pipeline job monitor the variable wasn't resolved and that's confused me. It was looking like :
So, for anyone who is fighting with such things, keep that in mind and also, keep in mind differences in syntax between powershell , bash and so on ..
I have a problem executing a pipeline in azure devops.
This task is part of my pipeline, as you can see it is an instruction to clone a repository.
- script: |
mkdir FOLDER_PUBLISH_AZURE
git clone https://$(AZURE_DEPLOY_USERNAME):$env:MY_AZURE_DEPLOY_PASSWORD#develroomlabs-events.scm.azurewebsites.net:443/develroomlabs-events.git FOLDER_PUBLISH_AZURE
workingDirectory: '$(System.DefaultWorkingDirectory)'
displayName: 'Download SCM repo for pipeline '
env:
MY_AZURE_DEPLOY_PASSWORD: $(AZURE_DEPLOY_PASSWORD)
The result I get is as follows:
But the pipeline variables are well added, I did a test by directly adding username and password in the YAML file, but the result was the same.
What could be happening? Am I doing something wrong?
I do this same task of cloning a repository from a bitbucket pipeline and I have no problems.
Try doing that in Windows environment with Microsoft-hosted windows agents instead of Micorosft-hosted Linux agents.
I can reproduce same issue when using Linux environment, but it actually works well in windows environment. I think this issue could be related to that Linux has different behavior when treating url format like https://$AppName:xxx#sAppName.scm.azurewebsites.net:443/AppName.git...
(Something looks like this ticket,the behavior should have something to do with Linux server itself)
A little background...
I have a small dotnet core application that is hosted on Azure and is being built and deployed using Azure DevOps Pipelines. Before we started using the DevOps Pipelines the CI was hooked up directly to Azure which compiled fine but took an actual lifetime to deploy, hence the decision to move.
However, the build pipeline no longer compiles or outputs the sass/css folder
Everything else works okay - I check in, the Build pipeline picks up my commits and has the following steps:
Restore [.NET Core]
Build [.NET Core]
Publish [.NET Core]
Publish Build Artifact
Part of step 3 (Publish) uses a Gulp task:
gulp.task('prod', function (callback) {
runSequence('clean','set-prod',
['icon-sprite', 'logo-sprite', 'images', 'sass', 'modernizr', 'mainjs', 'adminjs'],
callback);
});
And locally (and previously) this generated five folders:
icons
img
js
logos
css (now mysteriously missing in action)
Variations I've tried
I've tried deleting my local css folder and running the CLI dotnet publish exactly the same way the Pipeline does and that appears to work fine locally.
I've also stripped the sass task way back in case that was causing an issue somewhere in the pipeline, so that now looks like this:
return gulp.src('src/sass/style.scss')
.pipe(sass({outputStyle: 'compressed'}))
.pipe(gulp.dest('wwwroot/dist/css));
I can see all of the output in the console logs on the Pipeline and it successfully executes the sass task:
2019-01-02T14:43:51.3558593Z [14:43:51] Starting 'sass'...
2019-01-02T14:43:51.9284145Z [14:43:51] Finished 'sass' after 524 ms
There are no other errors or warnings in the build script and everything completes and fires off the Release pipeline (which copies the artifact up to the Azure site).
Speculation
I would expect an error somewhere... but nothing - all of the green ticks are downright cheerful... so I'm a little stumped at what may or may not be happening! I can only think that there must be some dependency or something missing in the Pipeline environment? Orrrrr maybe I'm missing a Pipeline step?
Any help or nudges or ideas would be greatly appreciated! Thank you for sticking it out through my small essay and for any help you can provide :)
Something I've done in this situation before is changing the Publish Build Artifact task to upload everything in the build folder. My guess is that right now the 'Path to Publish' value in that task is set to $(build.artifactStagingDirectory). Change it to $(build.SourcesDirectory). After running the build again you'll see that the entire build directory has was uploaded. This includes your source code and any other folders like you have on your local environment. From there you can figure out if the CSS folder is actually missing, or if it ended up in some other folder location.
If the folder ends up in a weird location you can either add a file copy task to move the CSS folder to the proper folder in $(build.artifactStagingDirectory) or make a change to the Gulp task. Whatever is better for your scenario.
Once you find the location, you can fix the Publish Build Artifact task.
I was having the exact same issue. I was able to get everything working locally without issue. gulp would generate the css folder just fine. dotnet publish -c release would do the same. However, when ran through the pipeline, no css folder.
The thing that I find the most strange, is that there is a sibling folder (scripts) that is used in the same way the css gulp task is used, but that folder makes it just fine. Here's my css task:
gulp.task('min', function() {
return gulp.src('wwwroot/css/**/*.css')
.pipe(cssnano({zindex:false}))
.pipe(gulp.dest('wwwroot/dist/css/'));
});
but, this task does works both locally and in the pipeline:
gulp.task('build-js', function() {
return gulp.src('wwwroot/scripts/**/*.js')
.pipe(concat('site.bundle.js'))
.pipe(uglify())
.pipe(gulp.dest('wwwroot/dist/scripts/'));
});
I ended up just giving up since this is legacy code anyways and settled on a workaround:
Add the Copy Files task right after your gulp task with the below configuration:
Or, if you like YAML:
steps:
- task: CopyFiles#2
displayName: 'Copy Files to: wwwroot/dist/css'
inputs:
SourceFolder: wwwroot/css
Contents: '*.css'
TargetFolder: wwwroot/dist/css