I'm using Gitlab CI/CD to deploy my .netcore application to AWS Beanstalk.
How can I update the appSettings.json values in my .netcore application when deploying to different environments using variables defined in my CI/CD pipeline settings?
Azure DevOps has a JSON variable substitution feature which I really liked.
GitHub Actions can also hook into this feature
How can I achieve this with Gitlab CI/CD?
I want to use this method of deployment because
I won't have to store sensitive production config values in my repository. Values will be updated by Masked variables setup in the CI/CD Pipeline.
I don't have to rebuild my artefacts every time I deploy to a new environment.
If this can't be done in Gitlabs, whats the recommended best practice?
thanks
I do something similar here with gitlab and the solution was to build a shell script that replaces some string from variable values before starting the deploy job
something like this
script:
- sed -i 's/STRING_TO_REPLACE/$GITLAB_VARIABLE/g' file.json
Just pay attention to escape json quotes correctly to make this possible
I use the same workaround. I filed an issue about this months ago:
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/78486
and I recently talked about this at the gitlab forum:
https://forum.gitlab.com/t/use-variables-per-branch-in-gitlab/43685
unfortunately there does not seem to be a nice/clean solution at the moment.
so I use the workarround:
1: use environment scopes for variables so the same variable can have different values (for test/prod environments)
2: define the environments and branches in the jobs in gitlab-ci.yml
3: create a bunch of sed one liners to do a search/replace..
call it ugly, call it low level but I searched for a nice/clean/gitlab native solution an did not find it.
Related
I have a great experience of pipeline creating automation (in case of huge amount of repos).
For example, a project has 20 similar repos with Java app (like a microservice) and a pipeline for each of them is differing only by repo url (and a few more minor attributes). The CI/CD process for each of them is the same.
So, we can create a separated devops-repo with declaration configuration for our services. Also we can create a single pipeline which will pull the devops repo and create all needed pipelines for each repo in the configuration (this operation is going to be executed only once in the beginning and in case if we want to change the devops-configuration)
I have implemented that using Jenkins. Now, I am going to do so using GitLab CI. But I can't get how is it possible.
Is it possible to create a pipeline from another one (dynamically)?
Any suggestions?
You can use include and put the generic pipeline in your devops repo.
In your java repos you can include the devops pipeline and set the variables which are specific for the respective java repo.
So the pipeline for your java repos can be as short as this:
include:
- project: 'your-group/devops-repository'
file: '.generic-ci.yml'
variables:
FOO: bar
Is there anyone out here who has implemented CI/CD with GitLab for Azure Snowflake? Is this even possible?
Our DB development is growing fast and it's turning out to be challenging experience to develop, maintain and deploy.
We have Visual Studio Code IDE which is now bound to a Git repository who's branches I would like to point to Prod, Dev and Test depending on commits to respective branches.? Also, is it even possible to have something like a Config.sql similar to SQLCMD in SQL Server or Application.properties in a Java Springboot project, where, one can maintain 3 different config files with environment specific variables whose values can be substituted dynamically depending on the BUILD step of the CICD pipeline? I want (at least now) to keep database and schema names as config variables which will be different on where one deploys.
Yes, you can install and configure snowsql tool as part of your CICD pipeline. Snowsql for Snowflake is the same as sqlcmd for SQL Server.
Installing snowsql
Configuring snowsql
There are tons of resources online on how to replace JSON configuration files in a release pipeline like this one. I configured this. It works. However, we have multiple integration tests which reach the database too. These tests are run during build time. I haven't seen any option yet to replace config values in the build pipeline. Does it exist? Or do I really have to use this custom task (see screenshot below)?
There is an out-of-the-box task since recently by Microsoft. It's called File Transform. It's currently in preview but it works really well! Haven't had any issues whatsoever with it and it works the same as you would configure it in the release pipeline. Would recommend this any day!
Below you can see my configuration.
There is no out-of-the-box task only to replace tokens/values in files (also in the release pipline the task is Azure App Service Deploy and not only for replace json configuration).
You need to use an external extension from here or write a PowerShell script for that.
I'd like to know if is it possible change the CI/CD variables assigned in repo settings by request. I know previously i can pass env variables by curl request to trigger the pipeline.
But now i have another situation which an automatic build and a docker image is pushed to a private registry happen when the master or dev branch suffers a merge. To do this, i use the CI/CD variables already setted.
And oftenly i will create a new registry, which it is done automatically in a bash script. After this what i'd like is setup the new keys accesses to the gitlab repo via request or other automatic way.
If someone could help me with some idea, thanks in advance.
If I am not mistaken this it what you are looking for:
Project-level Variables API
https://docs.gitlab.com/ee/api/project_level_variables.html
I can't find out how to access variables in a build-script provided by the gitlab-ci.yml-file.
I have tried to declare variables in two ways:
Private Variables in the Web-Interface of GitLab CI
Variable overrides/apennding in config.toml
I try to access them in my gitlab-ci.yml-files commands like that:
msbuild ci.msbuild [...] /p:Configuration=Release;NuGetOutputDir="$PACKAGE_SOURCE"
where $PACKAGE_SOURCE is the desired variable (PACKAGE_SOURCE) but it does not work (it does not seem to be replaced). Executing the same command manually works just as expected (replacing the variable name with its content)
Is there some other syntax required i am not aware of?
I have tried:
$PACKAGE_SOURCE
$(PACKAGE_SOURCE)
${PACKAGE_SOURCE}
PS: Verifying the runner raises no issues, if this matters.
I presume you are using Windows for your runner? I was having the same issue myself and couldn't even get the following to work:
script:
- echo $MySecret
However, reading the Gitlab documentation it has an entry for the syntax of environment variables in job scripts:
To access environment variables, use the syntax for your Runner’s shell
Which makes sense as most of the examples given are for bash runners. For my windows runner, it uses %variable%.
I changed my script to the following, which worked for me. (Confirmed by watching the build output.)
script:
- echo %MySecret%
If you're using the powershell for your runner, the syntax would be $env:MySecret
In addition to what was said in the answer marked as correct above, you should also check whether your CI variables in the gitlab settings are set as "protected". If so, you may not be able to use them in a branch that's not protected.
"You can protect a project, group or instance CI/CD variable so it is only passed to pipelines running on protected branches or protected tags." -> check it https://docs.gitlab.com/ee/ci/variables/index.html#protect-a-cicd-variable
Be aware that.. in certain cases SOME of the variables Gitlab CI CD offer are not always available..
In my case I wanted to use ${CI_COMMIT_BRANCH} but if you read the doc
https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines.