While deploying a build to multiple environments like UAT, Production. I want to replace one file config.uat.json or config.prod.json with config.json. Is there any option available? just like we have XML Transformation.
I am aware of Json Variable substitution but that doesn't serve my purpose as the variable list is long (almost 50 entries)
Thanks in Advance!
JSON variable substitution should be a good option, if you don't want to add variables one by one on Variables tab, you can update them with VSTS REST API, or add a powershell script to set the variables.
Otherwise, you may need to delete the config.uat.json or config.prod.json, and copy the config.json to the target machine.
Inside your Program.cs file, you can get an environment variable which will represent your current environment:
var builder = WebHost.CreateDefaultBuilder(args);
var currentEnv = builder.GetSetting("environnement");
using this currentEnv value, you will be able to load the file config.{currentEnv}.json
builder.AddJsonFile($"config.{currentEnv}.json", optional: false, reloadOnChange: true);
EDIT:
If you want to do this in powershell, you can make a transofmration of your configuration file with a default: appsettings.json containing keys, and appsettings.env.json containing overriding.
To transform your configuration, you can do something like this:
Param(
[Parameter(Mandatory=$true)][string]$SpecificConfig
)
$defaultConfig = "AppSettings.json";
$settingsContent = ConvertFrom-Json $defaultConfig;
$specificContent = ConvertFrom-Json $SpecificConfig;
# Do this on each <property> to override
if (![string]::IsNullOrEmpty($specificContent.<property>))
{
$settingsContent.<property> = $specificContent.<property>;
}
Write-Host $settingsContent > $defaultConfig;
Related
I have a Powershell script that will make a GET request to SonarQube's REST API to check if the Quality Gate passes or fails. If the Quality Gate fails, the pipeline will fail. I am able to make this work when only looking at the master branch however, I am trying to look at all branches and pull requests.
My pipeline Powershell script:
- job:
pool:
name: 'POEM-GBT-Agent'
variables:
- group: SonarQube
displayName: 'SonarQube API'
steps:
- checkout: none
- powershell: |
$token = [System.Text.Encoding]::UTF8.GetBytes("$(SONARQUBE_API_TOKEN)" + ":")
$base64 = [System.Convert]::ToBase64String($token)
$basicAuth = [string]::Format("Basic {0}", $base64)
$headers = #{ Authorization = $basicAuth }
if ($(System.PullRequest.PullRequestId)) {
$param = "pullRequest=$(System.PullRequest.PullRequestId)"
}
else {
$param = "branch=$env:$BRANCH_NAME"
}
$result = Invoke-RestMethod -Method Get -Uri https://sonarqube.tjx.com/api/qualitygates/project_status?projectKey=$(sonarProjectKey)"&"$param -Headers $headers
$result | ConvertTo-Json | Write-Host
if ($result.projectStatus.status -ne "OK") {
Write-Host "##vso[task.logissue type=error]Quality Gate Failed"
Write-Host "##vso[task.complete result=Failed]"
}
env:
BRANCH_NAME: replace('$(Build.SourceBranch)', 'refs/heads/', '')
This results in an error saying:
+ $param = "branch=$env:$BRANCH_NAME"
+ ~~~~~
Variable reference is not valid. ':' was not followed by a valid variable name
character. Consider using ${} to delimit the name.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : InvalidVariableReferenceWithDrive
After receiving this error I changed my conditional statement to:
if ($(System.PullRequest.PullRequestId)) {
$param = "pullRequest=$(System.PullRequest.PullRequestId)"
}
else {
$param = "branch=${env}:$BRANCH_NAME"
}
After changing my conditional I get this error:
System.PullRequest.PullRequestId : The term 'System.PullRequest.PullRequestId'
is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify
that the path is correct and try again.
At D:\GBT\agent\Workspace\_temp\0a756446-474a-4d58-94ff-ad25e38c3c7a.ps1:9
char:7
+ if ($(System.PullRequest.PullRequestId)) {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (System.PullRequest.PullRequestI
d:String) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : CommandNotFoundException
I am trying to set $param to pullRequest=1234 if the Git PR is affected, otherwise, I want to set $param to something like branch=feature/my-branch-name.
You have two problems here. I'll delve into each below.
The term 'System.PullRequest.PullRequestId' is not recognized
Under your job step, define some environment variables. I have limited experience with Azure DevOps (will reference as ADO from here on) but you will need two under your powershell environment variable definitions (you already have BRANCH_NAME but I will include it below):
- job:
steps:
- powershell: |
YOUR
CODE
HERE
env:
BRANCH_NAME: replace('$(Build.SourceBranch)', 'refs/heads/', '')
PULL_REQUEST_ID: $(System.PullRequest.PullRequestId)
You need to define PULL_REQUEST_ID as an env var because your rendered PowerShell script won't render $() as an inserted value from ADO. This is likely by design as $() is syntax used in other programming languages, including PowerShell. This is the crux of your issue where System.PullRequest.PullRequestId can't be found as a command; PowerShell literally tries to use that as a program name which it can't find.
Within your script, you can then reference the pull request ID with:
$env:PULL_REQUEST_ID
Variable reference is not valid. ':' was not followed by a valid variable name character
The issue is distinct from the issue above, but the solution is the same. Just reference the environment variable as:
$env:BRANCH_NAME
You don't need the extra $, as this just confuses the parser.
Now, you're not asking for this, but if you needed a conditional environment variable name (e.g. the name of the environment variable is sourced from some other ADO variable), you need to use a different syntax to access the environment variable, as the $ confuses the parser when accessed via the $env: syntax. Read on if you are curious about this.
If you did need to access an environment variable whose name is conditional
PowerShell doesn't allow special characters in variable names without specifying the variable name as a string. The notable exceptions are _ (no special meaning) and :, the latter of which is used as a separator for variables defined within a PSDrive. Environment variables are accessible via the Environment provider, which is accessed under the Env PSDrive. You can see this by running Get-PSDrive, there is an Environment provider exposed under the Env drive.
You can reference PSProvider variables in a few ways, the most common way is $drive:path (the variable name is technically considered a path node under the provider). So to reference the UserProfile variable, you can use:
$env:UserProfile # ====> Returns the path to your user profile directory
The problem with your code is you have the following:
$env:$VariableName
In this case the intent is to get an environment variable value whose name is based on another variable's value, but this confuses the parser for this syntax, as $ is now being translated as a literal portion of the variable name, which is invalid. Normally you would use the ${} syntax to avoid this... but you can't here, because the confused portion should be rendered as part of that same variable. In this case, you need to use an alternative approach to access the environment variable, go through the provider directly.
To use UserProfile as an example again:
$envVarName = 'UserProfile'
$envVarValue = (Get-ChildItem Env:/$envVarName).Value
This is the other way to get a value from a PSProvider; traverse its contents as a drive. This works somewhat like a filesystem, though Get-ChildItem will return different properties for different provider types than the FileInfo or DirectoryInfo objects you may be used to from the Filesystem provider. It's a bit cumbersome, but luckily it's not a scenario one often needs to account for.
Is there some special variable available in Terraform configuration files which would point to current file name?
I'd like to use it for description fields in various resources, so that someone seeing these resources in the systems would know where is the master definition for them.
e.g.
in myinfra.tf
resource "aws_iam_policy" "my_policy" {
name = "something-important"
description = "Managed by Terraform at ${HERE_I_WOULD_LIKE_TO_USE_THE_VARIABLE}"
policy = <<EOF
[...]
EOF
}
And I would hope the description becomes:
description = "Managed by Terraform at myinfra.tf"
I tried ${path.module} but that only gives "filesystem path of the module where the expression is placed", so pragmatically speaking - everything but the file name I want.
Here's what I can share. Use the data external resource to call an external script that would get the directory/file name and then return it back as a string or any other type that your resources require. Obviously it's not exactly what you wanted as you'll get the dir/file name indirectly but hopefully it helps for others or even yourself for use-cases.
We use that only for azurerm and for very complex integrations that are not yet supported with the current provider versions. I have have not tested it specifically for AWS but since it's a core Terraform resource provider, I'm guessing it might work across the board.
data "external" "cwd" {
program = ["./script.sh"]
query = {
cwd = "${path.cwd}"
}
}
resource "aws_iam_policy" "my_policy" {
name = "something-important"
description = "Managed by Terraform at ${data.external.dir_script.result.filename}"
policy = <<EOF
[...]
EOF
This is how my script looks like:
#!/bin/sh
#echo '{"cwd":"for_testing"}' | ./dir_name.sh | xargs
PIPED=`cat`
errPrint "INFO: Got PIPED data:\n$PIPED"
DIR=`jq -r .cwd <<< $PIPED`
cd $DIR
filename=`ls | grep \.tf$ | xargs`
errPrint "INFO: Returning this as STDOUT:${filename}"
echo "{\"name\":\"$filename\"}"
You need to be that the data from the script needs to return a valid JSON object.
The program must then produce a valid JSON object on stdout, which will be used to populate the result attribute exported to the rest of the Terraform configuration. This JSON object must again have all of its values as strings. On successful completion it must exit with status zero.
Unfortunately, like the others mentioned, there's no other way to get the current file name being 'applied'.
I think you might benefit from using something like yor from Bridge Crew.
From the project's README:
Yor is an open-source tool that helps add informative and consistent tags across infrastructure-as-code frameworks such as Terraform, CloudFormation, and Serverless.
Yor is built to run as a GitHub Action automatically adding consistent tagging logics to your IaC. Yor can also run as a pre-commit hook and a standalone CLI.
So basically, it updates your resources tags with things like:
tags = {
env = var.env
yor_trace = "912066a1-31a3-4a08-911b-0b06d9eac64e"
git_repo = "example"
git_org = "bridgecrewio"
git_file = "applyTag.md"
git_commit = "COMMITHASH"
git_modifiers = "bana/gandalf"
git_last_modified_at = "2021-01-08 00:00:00"
git_last_modified_by = "bana#bridgecrew.io"
}
Maybe that would be good enough to provide what you're trying to do?
As far as my testimony, I have not used yor since my tagging uses a different approach. Instead of having "raw" tags, we use a label module that builds the tags for us and then merges in local tags.
Just sharing this info FYI in case it helps.
I have the following use-case: I'm using a combination of the Azure DevOps pipelines and Terraform to synchronize our TAP for Grafana (v7.4). Intention is that we can tweak and tune our dashboards on Test, and push the changes to Acceptance (and Production) via the pipelines.
I've got one pipeline that pulls in the state of the Test environment and writes it to a set of json files (for the dashboards) and a single json array (for the folders).
The second pipeline should use these resources to synchronize the Acceptance environment.
This works flawlessly for the dashboards, but I'm hitting a snag putting the dashboards in the right folder dynamically. Here's my latest working code:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
}
The folder resources pushes the folders based on a variable list of names that I pass via variables. This generates the folders correctly.
The dashboard resource pushes the dashboards correctly, based on all dashboard files in the specified folder.
But now I'd like to make sure the dashboards end up in the right folder. The provider specifies that I need to do this based on the folder UID, which is generated when the folder is created. So I'd like to take the output from the grafana_folder resource and use it in the grafana_dashboard resource. I'm trying the following:
resource "grafana_folder" "folders" {
for_each = toset(var.grafana_folders)
title = each.key
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset(path.module, "../dashboards/*.json")
config_json = file("${path.module}/${each.key}")
folder = lookup(transpose(grafana_folder.folders), "Station_Details", "Station_Details")
depends_on = [grafana_folder.folders]
}
If I read the Grafana Provider github correctly, the grafana_folder resource should output a map of [uid, title]. So I figured if I transpose that map, and (by way of test) lookup a folder title that I know exists, I can test the concept.
This gives the following error:
on main.tf line 38, in resource "grafana_dashboard" "dashboards":
38: folder = lookup(transpose(grafana_folder.folders),
"Station_Details", "Station_Details")
Invalid value for "default" parameter: the default value must have the
same type as the map elements.
Both Uid and Title should be strings, so I'm obviously overlooking something.
Does anyone have an inkling where I'm going wrong and/or have suggestions on how I can do this (better)?
I think the problem this error is trying to report is that grafana_folder.folders is a map of objects, and so passing it to transpose doesn't really make sense but seems to be succeeding because Terraform has found some clever way to do automatic type conversions to produce some result, but then that result (due to the signature of transpose) is a map of lists rather than a map of strings, and so "Station_Details" (a string, rather than a list) isn't a valid fallback value for that lookup.
My limited familiarity with folders in Grafana leaves me unsure as to what to suggest instead, but I expect the final expression will look something like the following:
folder = grafana_folder.folders[SOMETHING].id
SOMETHING here will be an expression that allows you to know for a given dashboard which folder key it ought to belong to. I'm not seeing an answer to that from what you shared in your question, but just as a placeholder to make this a complete answer I'll suggest that one option would be to make a local map from dashboard filename to folder name:
locals {
# a local value probably isn't actually the right answer
# here, but I'm just showing it as a placeholder for one
# possible way to map from dashboard filename to folder
# name. These names should all be elements of
# var.grafana_folders in order for this to work.
dashboard_folders = {
"example1.json" = "example-folder"
"example2.json" = "example-folder"
"example3.json" = "another-folder"
}
}
resource "grafana_dashboard" "dashboards" {
for_each = fileset("${path.module}/dashboards", "*.json")
config_json = file("${path.module}/dashboards/${each.key}")
folder = grafana_folder.folders[local.dashboard_folders[each.key]].id
}
Although I am not familiar with DevOps best practices, I am trying to come up with a reliable and efficient method for managing multiple variables in production. The following represents my current approach:
/
|ENV_VAR.sh
|--/api1
|--/staging.api1
|--/api2
|--/staging.api2
Where:
ENV_VAR.sh
### API 1 variables ###
export API1_VAR_1=foo
export API1_VAR_2=foo2
export API1_STAG_VAR_1=foo_stag
export API1_VAR_2=foo2_stag2
### API 2 variables ###
export API2_VAR_1=foo
export API2_VAR_2=foo2
export API2_STAG_VAR_1=foo_stag
export API2_VAR_2=foo2_stag2
The API 1 and 2 are two nodejs-based apps running in the same server using a reverse-proxy configuration.
If nothing goes bad with the server (e.g. unexpected shutdown), I just have to (re)set the variables once in a while via SOURCE ENV_VAR.SH in order to make sure that new variables are defined.
Before proceeding with this approach, I would like to know whether it is correct at all, or if it has a big flaw.
If this approach is alright, how to automatically (re)source the environment variables from the package.json whenever a new version of any App is deployed? (just to guarantee that the variables are still defined)
Thanks in advance.
I like using Loren West's config package for these configuration parameters. I happen to like to extend it with the properties package: that way I don't have to put parameters in valid, comment-free, JSON format. JSON5 also helps solve the readability problem, but I haven't tried it.
Why do I like this?
It gives a structured way of dealing with development / test / staging / production environments. It keys off the ENV environment variable, which of course has values like development and production.
All properties files go into a single directory, typically ./config. Your production krewe can tell what they're looking at. default.properties, development.properties and production.properties are the names of typical files.
Most configuration parameters don't have to be secret, and therefore they can be committed to your repository.
Secrets (passwords, connection strings, API keys, etc) can be stored in local.properties files placed into ./config by your deployment system. (Mention local.properties in your .gitignore file.)
Secrets can also be loaded from environment variables, named in a file called ./config/custom_environment_variables.json.
It works nicely with pm2.
This is really easy to configure.
Your files:
default.properties (used when not overridden by another file)
[API1]
VAR_1 = foo
VAR_2 = foo2
[API2]
VAR_1 = foo
VAR_2 = foo_for_api2
staging.properties
[API1]
VAR_1 = foo_stag
VAR_2 = foo2_stag2
[API2]
VAR_1=foo_stag
VAR_2=foo2_stag2
custom_environment_variables.json
{
"API1" : {
"password": "API1_PASS"
},
"API2" : {
"password": "API2_PASS"
}
}
Your nodejs program:
const config = require( 'config' )
require( 'properties' )
const appConfig = config.get( 'API1' )
const var1 = appConfig.VAR_1
const password = appConfig.password
Then you run your program with API1_PASS=yaddablah nodejs program.js and you get all your configs.
How can I make the below logic work? My aim is to compare the value of custom fact $environment and the content of the file /etc/facter/facts.d/oldvalue.
If the custom fact $environment is not equal to the content of file /etc/facter/facts.d/oldvalue, then execute the following code.
exec {'catenvchange' :
command => "/bin/cat /root/oldvalue"}
if $environment != exec['catenvchange'] {#code#}
Exec resources do not work that way. In fact, no resource works that way, or any way remotely like that. Moreover, the directory /etc/facter/facts.d/ serves a special purpose, and your expectation for how it might be appropriate to use a file within is not consistent with that purpose.
What you describe wanting to do looks vaguely like setting up an external fact and testing its value. If you drop an executable script named /etc/facter/facts.d/anything by some means (manually, plugin sync, File resource, ...) then that script will be executed before each Puppet run as part of the process of gathering node facts. The standard output generated by the script would be parsed for key=value pairs, each defining a fact name and its value. The facts so designated, such as one named "last_environment" will be available during catalog building. You could then use it like so:
if $::environment != $::last_environment {
# ...
}
Update:
One way to use this mechanism to memorialize the value that a given fact, say $::environment, has on one run so that it can be read back on the next run would be to declare a File resource managing an external fact script. For example,
file { '/etc/facter/facts.d/oldvalues':
ensure => 'file',
owner => 'root',
group => 'root',
mode => '0755',
content => "#!/bin/bash\necho 'last_environment=${::environment}'\n"
}