I'd like to append a new key-value pair to an existing yaml-based structure by using terraform.
For example I have the following yaml file:
urls:
- id: site1
url: google.de
- id: site2
url: bing.com
- id: site3
url: duckduckgo.com
Now I want to add a key-value pair based on some conditions.
(It's not required to write to the file. The list will be used in a loop later on)
Expected:
urls:
- id: site1
url: google.de
secure: false
- id: site2
url: bing.com
secure: false
- id: site3
url: duckduckgo.com
secure: true
What I already tried:
locals {
x = tomap({
for k in keys(yamldecode(file("urls.yaml"))):
k => merge(
yamldecode(file("urls.yaml"))[k],
{ urls = { url = merge(yamldecode(file("urls.yaml"))[k].urls[0], { secure = false }) }}
)
})
}
Works for the first url but I wasn't able to loop over the urls to get an index.
Second approach:
locals {
x = tomap({
for k in keys(yamldecode(file("urls.yaml"))):
k => {
for l in keys(yamldecode(file("urls.yaml"))[k]):
l => l == "urls" ? <tbd> : yamldecode(file("urls.yaml"))[k][l]
}
})
}
But I wasn't able to merge or replace the structure at <tbd> if the key matches.
It always fails because of mismatches:
arguments must be maps or objects, got "tuple".
Any idea?
After a yamldecode function converts from YAMl formatted string to HCL2, the resulting type would be map(list(object)); for example:
{ "urls" = [{id="site1", url="google.de"}] }
That makes it a bit more clear how to add a key-value pair to the nested object with a for expression. We need to preserve the structure, keys, and values of the original, and add a single key value pair in the nested object.
# map constructor and iterate through yaml map
# key = "urls", urls is list of objects
{ for key, urls in yamldecode(file("urls.yaml")) : key => [
# inside list constructor now
# url_attrs is single object in list of objects
for url_attrs in urls : {
# inside object constructor now
# retain same key value pairs, and add a "secure" key value pair
id = url_attrs.id
url = url_attrs.url
secure = false
}
]
}
In HCL2 this results in (according to local testing)
{
urls = [
{
id = "site1"
secure = false
url = "google.de"
},
]
}
which is equivalent to:
urls:
- id: site1
url: google.de
secure: false
I notice the logic for the secure boolean is a placeholder and the example code always assigned false, so I have done the same above.
Related
I'm trying to create New Relic's service-level objects based on a yaml config file that provides the relevant configuration.
My yaml configuration:
slo:
targets:
- first_slo:
name: "My First SLO"
endpoints:
- path: /api/method1
method: GET
- path: /api/method2
method: PUT
objectives:
availability:
threshold: 99.9
- first_slo:
name: "My Second SLO"
endpoints:
- path: /api/method12
method: GET
- path: /api/method23
method: PUT
objectives:
availability:
threshold: 99.99
I want to iterate over this example configuration to build the object, but I'm struggling to form the right NRQL query using a nested iteration.
My terraform file:
resource "newrelic_service_level" "availability" {
for_each = var.config.slo.targets
guid = var.guid
name = "${each.value.name} - Availability"
description = "Proportion of requests that are served successfully."
events {
account_id = var.account_id
valid_events {
from = "Transaction"
where = "transactionType='Web' AND entityGuid = '${var.guid}' AND (OR_CONDITION_BETWEEN_ALL_THE_METHODS_AND_URIS)"
}
bad_events {
from = "Transaction"
where = "transactionType= 'Web' AND entityGuid = '${var.guid}' AND numeric(response.status) >= 500 AND (OR_CONDITION_BETWEEN_ALL_THE_METHODS_AND_URIS)"
}
}
objective {
target = each.value.objectives.availability.threshold
time_window {
rolling {
count = 7
unit = "DAY"
}
}
}
}
So basically what I'm trying to do here, is create a service level with an NRQL query that filters only for the specific combination of URI and method that are relevant for this specific target - the URI and methods that I have in my config file.
So for the first SLO, OR_CONDITION_BETWEEN_ALL_THE_METHODS_AND_URIS should translate to something like this:
(request.uri = '/api/method1' AND request.method = 'GET') OR (request.uri = '/api/method2' AND request.method = 'PUT')
My current solution would be to build the query manually and add it to the configurations for each SLO, but it is not readable and hard to maintain.
I would highly appreciate any suggestions on how to build the query dynamically.
You can certainly build that query with Terraform. Here's a wee .tf file that shows how you could do it:
locals {
config = yamldecode(file("${path.root}/vars.yaml"))
parsed = [for d in local.config.slo.targets : {
name : d["name"]
condition : join(" OR ", [for e in d["endpoints"] : "(request.uri = '${e["path"]}' AND request.method = '${e["method"]}')"])
}]
}
output "parsed" {
value = local.parsed
}
This expects your yaml file to be sitting next to it with name vars.yaml, and produces:
$ terraform plan
Changes to Outputs:
+ parsed = [
+ {
+ condition = "(request.uri = '/api/method1' AND request.method = 'GET') OR (request.uri = '/api/method2' AND request.method = 'PUT')"
+ name = "My First SLO"
},
+ {
+ condition = "(request.uri = '/api/method12' AND request.method = 'GET') OR (request.uri = '/api/method23' AND request.method = 'PUT')"
+ name = "My Second SLO"
},
]
For your module, you can just use the join(...) part in place of OR_CONDITION_BETWEEN_ALL_THE_METHODS_AND_URIS. Having it repeated should be fine (as long as you document it, naturally), but if you don't like the big long line, you can create a sub-module to encapsulate it. Or you could build the query string in a pre-processing locals block, possibly using the merge function to just add the query string alongside the rest of each target's config.
I've got the following variable in a module:
variable "container_registries" {
type = list(object({
name = string
addl_keys = list(string)
namespaces = set(string)
hostnames = list(string)
username = string
password = string
}))
default = []
}
I'm feeding the module variable as such:
container_registries = [
{
name : "server.example.com"
addl_keys : ["config.json"]
namespaces : ["flux-system", "tekton"]
hostnames : ["cr-lts.server.example.com", "cr-test.server.example.com"]
username : "foo"
password : "bar"
}
]
Now I need to create multiple Kubernetes Secrets, each in different namespaces - but with the same content. I need the Secrets in the flux-system and tekton namespace. I need the secret to look like this:
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
name: server.example.com
data:
.dockerconfigjson: eyJhdXRocyI6eyJjci1sdHMuc2VydmVyLmV4YW1wbGUuY29tIjp7ImF1dGgiOiJabTl2T21KaGNnbz0ifSwiY3ItdGVzdC5zZXJ2ZXIuZXhhbXBsZS5jb20iOnsiYXV0aCI6IlptOXZPbUpoY2dvPSJ9fX0K
config.json: eyJhdXRocyI6eyJjci1sdHMuc2VydmVyLmV4YW1wbGUuY29tIjp7ImF1dGgiOiJabTl2T21KaGNnbz0ifSwiY3ItdGVzdC5zZXJ2ZXIuZXhhbXBsZS5jb20iOnsiYXV0aCI6IlptOXZPbUpoY2dvPSJ9fX0K
Note that the Secret has two different keys, each with the same value. The .dockerconfigjson key is mandatory when the Secret type is set to kubernetes.io/dockerconfigjson, so it should always be included. The value is a base64 encoded JSON and the JSON looks as such:
{
"auths": {
"cr-lts.server.example.com": {
"auth": "Zm9vOmJhcgo="
},
"cr-test.server.example.com": {
"auth": "Zm9vOmJhcgo="
}
}
}
The value of auth is foo:bar (the username and password) in base64.
I've been trying and trying, but I am not getting any closer. All my attempts have felt like garbage 😰 How in the world can I achieve this with Terraform? 😅
Here's what I did to solve the problem, using #Kreetchy's answer as a base:
locals {
container_registries = toset(flatten([
for cr in var.container_registries : [
for ns in cr.namespaces : format("%s/%s", ns, cr.name)
]
]))
container_registry_data = {
for cr in var.container_registries : cr.name => {
for key in toset(concat([".dockerconfigjson"], cr.addl_keys)) : key => jsonencode({
auths = {
for hostname in cr.hostnames : hostname => {
auth = base64encode("${cr.username}:${cr.password}")
}
}
})
}
}
}
resource "kubernetes_secret" "container_registry" {
for_each = local.container_registries
metadata {
namespace = split("/", each.value)[0]
name = split("/", each.value)[1]
}
type = "kubernetes.io/dockerconfigjson"
data = local.container_registry_data[split("/", each.value)[1]]
}
Note the extra local.container_registries which is used for the loop in the resource. This local stores the namespace/name of the secret in the list. The drawback is that two secrets by the same name and different content can not be created in two namespaces. It's something I can live with :-)
I also renamed var.keys to var.addl_keys, because a key by the name of .dockerconfigjson always must exist in a kubernetes.io/dockerconfigjson kind of Secret.
#Kreetchy: Please feel free to copy the code above and put it into your answer and I'll mark it as accepted. Once done, I will edit my question to adapt the requirement to fit the answer :-)
You could achieve it this way in Terraform
locals {
secret_data = {
for registry in var.container_registries : {
for key in registry.keys : key => base64encode(jsonencode({
auths = {
for hostname in registry.hostnames : hostname => {
auth = base64encode("${registry.username}:${registry.password}")
}
}
}))
}
}
}
resource "kubernetes_secret" "example" {
for_each = flatten([for registry in var.container_registries : registry.namespaces])
metadata {
name = lookup(var.container_registries[each.key].name, each.value)
namespace = each.value
}
type = "kubernetes.io/dockerconfigjson"
data = {
for key, value in local.secret_data : key => value
}
}
This will handle a list of varying amount of entries in the container_registries variable, and create a separate kubernetes_secret resource for each namespace. The contents of the secrets will still be the encoded JSON, but this time generated dynamically based on the number of keys and hostnames specified in each entry in the container_registries variable.
This handles any number of keys, namespaces and hosts dynamically and will create separate kuberneteds_secret resource for each namespace. Content of secret is still encoded JSON, but generated dynamically based on number of keys and hostnames specified in container_registries
I'm trying to add values to an Azure App Configuration Store using Bicep. I have an issue where I add a label to a keyValue.
This is my module:
#description('Configuration Store Name')
param configurationStoreName string
#description('key prefix')
param prefix string
#description('key name')
param keyName string
#description('value')
param value string
#description('content type')
param contentType string = 'string'
#description('Deployment Environment')
param deploymentEnvironment string = 'dev'
resource configurationStore 'Microsoft.AppConfiguration/configurationStores#2021-10-01- preview' existing = {
name: configurationStoreName
}
resource configurationStoreValue 'Microsoft.AppConfiguration/configurationStores/keyValues#2021-10-01-preview' = {
name: '${prefix}:${keyName}'
parent: configurationStore
properties: {
contentType: contentType
value: value
tags: {
environment: deploymentEnvironment
}
}
}
There doesn't seem to be any way to add a label, which I want to do to enable filtering.
It can be done when creating KeyValues using the Azure Portal, therefore it should be possible using Bicep.
Am I missing something, or is this missing functionality from Bicep?
EDIT April 2022
The documentation has now been updated
The keyValues resource's name is a combination of key and label. The key and label are joined by the $ delimiter. The label is optional. In the above example, the keyValues resource with name myKey creates a key-value without a label.
Percent-encoding, also known as URL encoding, allows keys or labels to include characters that are not allowed in ARM template resource names. % is not an allowed character either, so ~ is used in its place. To correctly encode a name, follow these steps:
Apply URL encoding
Replace ~ with ~7E
Replace % with ~
For example, to create a key-value pair with key name AppName:DbEndpoint and label name Test, the resource name should be AppName~3ADbEndpoint$Test.
I tried this approach and it works:
#description('Configuration Store Name')
param configurationStoreName string
#description('key prefix')
param prefix string
#description('key name')
param keyName string
#description('value')
param value string
#description('label')
param label string
#description('content type')
param contentType string = 'string'
#description('Deployment Environment')
param deploymentEnvironment string = 'dev'
resource configurationStore 'Microsoft.AppConfiguration/configurationStores#2021-10-01-preview' existing = {
name: configurationStoreName
}
var keyValueName = empty(label) ? '${prefix}:${keyName}' : '${prefix}:${keyName}$${label}'
resource configurationStoreValue 'Microsoft.AppConfiguration/configurationStores/keyValues#2021-10-01-preview' = {
name: keyValueName
parent: configurationStore
properties: {
contentType: contentType
value: value
tags: {
environment: deploymentEnvironment
}
}
}
I have following values defined in hieradata in puppet
globals::myservers:
- fqdn: "host1.example.com"
port: 22
protocol: "ssh"
- fqdn: "host2.example.com"
port: 22
protocol: "ssh"
and I would like it to print the following values with above data
my_servers = host1.example.com host2.example.com
Your hiera data looks like you have an array called myservers with each element containing a hash.
# Build a =data structure the same as you have in hiera.
$myservers = [ {fqdn => 'host1', port => '22' }, {fqdn => 'host2', port => '22' } ]
# Use the map function to extract all the values for fqdn into an array.
$fqdnarray = $myservers.map |$item| { $item[fqdn] }
# use the join function to turn the array into a string.
$fqdnstring = join($fqdnarray,',')
# Print the output
notify { 'Hosts message':
message => "my_servers = ${fqdnstring}"
}
You should be able to drop the above code straight into a file, let's call it test.pp and then run puppet apply test.pp on a machine with a Puppet agent on to see what it does.
The first thing to do is to load the data into Puppet. The specified Hiera key is appropriate for automatically binding the data to a parameter $myservers of a class named globals:
class globals(
Array[Hash[String, String]] $myservers
) {
# ... do something with $myservers ...
}
If you do not already have such a class, however, then you can alternatively look up the data from within any other class or any defined type:
class mymodule::any_class {
$myservers = lookup('globals::myservers', Array[Hash[String, String]])
# ... do something with $myservers ...
}
Having obtained the data, the question is how to write an ERB template that iterates over it and formats it as desired. This is not hard, provided that you know some Ruby (that's what the 'R' stands for in ERB, after all). The wanted template might be something like this ...
my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>
Inside the template, the Ruby #myservers variable is bound to the $myservers Puppet variable from the local scope from which the template is evaluated. The scriptlet extracts the 'fqdn' members of all the array elements and joins their string representations together with space separators.
As for invoking the template evaluation, you have your choice of putting the template into a separate file and using the template() function, or putting it directly into your manifest and using inline_template(). For example,
$formatted_server_list = inline_template(
"my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>")
notify { "${formatted_server_list}": }
I'm struggling with setting dynamic private dns zone records for multiple private endpoints for same resource in Azure (ACR - Azure Container Registry).
So I have currently setup this simple example. Basically it simulate creation of ACR and building a map of records which need to be registered in the DNS so ACR could be used correclty. The thing here is that it is needed to somehow extract value from inner field of each resource/object (here it is endpoints variable) and then transform it correctly so it would match expected output. This is needed as in my case it is integral part of a module and it needs to be built dynamically based on values of real resources created.
Following example should help anyone who'd like to check this problem out and help a bit.
To verify if you've got correct result just type "terraform refresh" and the output should be printed.
terraform {
backend local {}
}
# simulates creation of multiple Azure Container Registry private endpoints
variable endpoints {
type = map
default = {
"registry" = {
custom_dns_configs = [
{
fqdn = "something.westeurope.data.azurecr.io"
ip_addresses = ["1.1.1.1",]
},
{
fqdn = "something.azurecr.io"
ip_addresses = ["1.1.1.2",]
}
]
},
"registry2" = {
custom_dns_configs = [
{
fqdn = "something.westeurope.data.azurecr.io"
ip_addresses = ["1.1.2.1",]
},
{
fqdn = "something.azurecr.io"
ip_addresses = ["1.1.2.2",]
},
]
},
#...
# "registryN" = {...}
}
}
# Question: How to write for block here to get out of it exactly what's in expected
# result having in mind that there can be multiple of "registry" blocks?
# Below line produce only single result which doesn't match expected result.
output result {
value = {for pe in var.endpoints:
pe.custom_dns_configs[0].fqdn => pe.custom_dns_configs[0].ip_addresses
}
}
# Expected result:
# result = {
# "something.westeurope.data.azurecr.io" = [
# "1.1.1.1",
# "1.1.2.1",
# ]
# "something.azurecr.io" = [
# "1.1.1.2",
# "1.1.2.2",
# ]
# }
Answering my own question:
output result {
value = {for k,v in {
for cdc in flatten(
[for pe in var.endpoints: pe.custom_dns_configs]
):
cdc.fqdn => cdc.ip_addresses...
}:
k => flatten(v)
}
}
Now few words of explanation:
[] and {} - [] produce a tuple which strip "registry" key from incomming map whereas {} would require to produce some kind of dynamic keys
[for pe in var.endpoints: pe.custom_dns_configs] just extract internal field from var.environments for each element of the map. Then flatten is used just to make thing simpler without the need to dig into different levels of nested lists
next for is to build new map for fqdn -> list(ip addresses)
the "..." after cdc.ip_addresses are required. That's notation to allow grouping values by keys. In this case we have at least 2 times same fqdn and by normal means terraform would complain that it cannot create such map when keys are non-uniqe. Adding those 3 dots there enable this grouping.
then top-most for block is used just to flatten whole value output.
Now the problem would be if we would still want to keep "registry", "registry2", "registryN" grouping. For that I haven't found solution yet.