I'm using the HTTP data source to retrieve data from an internal service. The service returns JSON data.
I can't interpolate the returned JSON data and look up data in it.
For example:
module A
data "http" "json_data" {
url = "http://myservice/jsondata"
# Optional request headers
request_headers {
"Accept" = "application/json"
}
}
output "json_data_key" {
value = "${lookup(data.http.json_data.body, "mykey")}"
}
main.tf
provider "aws" {
region = "${var.region}"
version = "~> 0.1"
}
module "moduleA" {
source = "../../../terraform-modules/moduleA"
}
resource "aws_instance" "example" {
ami = "ami-2757f631"
instance_type = "${module.moduleA.json_data_key}"
}
The lookup function will fail to extract the key within the JSON data.
Is there any way to decode the JSON data into a terraform map ?
variable "json" {
default = "{\"foo\": \"bar\"}"
}
data "external" "json" {
program = ["echo", "${var.json}"]
}
output "map" {
value = "${data.external.json.result}"
}
Not directly related to map convertion, but here's an additional sample with jsondecode if you got a multi-value secret (=JSON) in AWS SecretsManager & you want to use separate values from it in another service as I've struggled with this.
Retrieving the secret:
data "aws_secretsmanager_secret" "oauth_client" {
name = "oauth-client"
}
data "aws_secretsmanager_secret_version" "oauth_client" {
secret_id = data.aws_secretsmanager_secret.oauth_client.id
}
Using it at Lambda, as an example:
resource "aws_lambda_function" "lambda" {
[...]
environment {
variables = {
OAUTH_CLIENT_ID = jsondecode(data.aws_secretsmanager_secret_version.oauth_client.secret_string)["client_id"]
OAUTH_CLIENT_SECRET = jsondecode(data.aws_secretsmanager_secret_version.oauth_client.secret_string)["client_secret"]
}
}
}
Ok, so it seems to be that the way to do the is by using the external data, as it return a map from a json response.
https://www.terraform.io/docs/providers/external/data_source.html
terraform version v0.10.6
Since 0.12 version of the Terraform you can use jsondecode function to decode json into a Terraform map. More details on: https://www.terraform.io/docs/configuration/functions/jsondecode.html
example from the page above:
> jsondecode("{\"hello\": \"world\"}")
{
"hello" = "world"
}
> jsondecode("true")
true
Related
I've got the following variable in a module:
variable "container_registries" {
type = list(object({
name = string
addl_keys = list(string)
namespaces = set(string)
hostnames = list(string)
username = string
password = string
}))
default = []
}
I'm feeding the module variable as such:
container_registries = [
{
name : "server.example.com"
addl_keys : ["config.json"]
namespaces : ["flux-system", "tekton"]
hostnames : ["cr-lts.server.example.com", "cr-test.server.example.com"]
username : "foo"
password : "bar"
}
]
Now I need to create multiple Kubernetes Secrets, each in different namespaces - but with the same content. I need the Secrets in the flux-system and tekton namespace. I need the secret to look like this:
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
name: server.example.com
data:
.dockerconfigjson: eyJhdXRocyI6eyJjci1sdHMuc2VydmVyLmV4YW1wbGUuY29tIjp7ImF1dGgiOiJabTl2T21KaGNnbz0ifSwiY3ItdGVzdC5zZXJ2ZXIuZXhhbXBsZS5jb20iOnsiYXV0aCI6IlptOXZPbUpoY2dvPSJ9fX0K
config.json: eyJhdXRocyI6eyJjci1sdHMuc2VydmVyLmV4YW1wbGUuY29tIjp7ImF1dGgiOiJabTl2T21KaGNnbz0ifSwiY3ItdGVzdC5zZXJ2ZXIuZXhhbXBsZS5jb20iOnsiYXV0aCI6IlptOXZPbUpoY2dvPSJ9fX0K
Note that the Secret has two different keys, each with the same value. The .dockerconfigjson key is mandatory when the Secret type is set to kubernetes.io/dockerconfigjson, so it should always be included. The value is a base64 encoded JSON and the JSON looks as such:
{
"auths": {
"cr-lts.server.example.com": {
"auth": "Zm9vOmJhcgo="
},
"cr-test.server.example.com": {
"auth": "Zm9vOmJhcgo="
}
}
}
The value of auth is foo:bar (the username and password) in base64.
I've been trying and trying, but I am not getting any closer. All my attempts have felt like garbage 😰 How in the world can I achieve this with Terraform? 😅
Here's what I did to solve the problem, using #Kreetchy's answer as a base:
locals {
container_registries = toset(flatten([
for cr in var.container_registries : [
for ns in cr.namespaces : format("%s/%s", ns, cr.name)
]
]))
container_registry_data = {
for cr in var.container_registries : cr.name => {
for key in toset(concat([".dockerconfigjson"], cr.addl_keys)) : key => jsonencode({
auths = {
for hostname in cr.hostnames : hostname => {
auth = base64encode("${cr.username}:${cr.password}")
}
}
})
}
}
}
resource "kubernetes_secret" "container_registry" {
for_each = local.container_registries
metadata {
namespace = split("/", each.value)[0]
name = split("/", each.value)[1]
}
type = "kubernetes.io/dockerconfigjson"
data = local.container_registry_data[split("/", each.value)[1]]
}
Note the extra local.container_registries which is used for the loop in the resource. This local stores the namespace/name of the secret in the list. The drawback is that two secrets by the same name and different content can not be created in two namespaces. It's something I can live with :-)
I also renamed var.keys to var.addl_keys, because a key by the name of .dockerconfigjson always must exist in a kubernetes.io/dockerconfigjson kind of Secret.
#Kreetchy: Please feel free to copy the code above and put it into your answer and I'll mark it as accepted. Once done, I will edit my question to adapt the requirement to fit the answer :-)
You could achieve it this way in Terraform
locals {
secret_data = {
for registry in var.container_registries : {
for key in registry.keys : key => base64encode(jsonencode({
auths = {
for hostname in registry.hostnames : hostname => {
auth = base64encode("${registry.username}:${registry.password}")
}
}
}))
}
}
}
resource "kubernetes_secret" "example" {
for_each = flatten([for registry in var.container_registries : registry.namespaces])
metadata {
name = lookup(var.container_registries[each.key].name, each.value)
namespace = each.value
}
type = "kubernetes.io/dockerconfigjson"
data = {
for key, value in local.secret_data : key => value
}
}
This will handle a list of varying amount of entries in the container_registries variable, and create a separate kubernetes_secret resource for each namespace. The contents of the secrets will still be the encoded JSON, but this time generated dynamically based on the number of keys and hostnames specified in each entry in the container_registries variable.
This handles any number of keys, namespaces and hosts dynamically and will create separate kuberneteds_secret resource for each namespace. Content of secret is still encoded JSON, but generated dynamically based on number of keys and hostnames specified in container_registries
I'm trying to create the following block dynamically based on a list of strings
env {
name = "SECRET_ENV_VAR"
value_from {
secret_key_ref {
name = google_secret_manager_secret.secret.secret_id
key = "1"
}
}
}
Based off documentation: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_service#example-usage---cloud-run-service-secret-environment-variables
I would like to dynamically add Secrets, and have defined the following dynamic block:
dynamic "env" {
for_each = toset(var.secrets)
content {
name = each.value
value_from {
secret_key_ref {
name = each.value
key = "1"
}
}
}
}
Where secrets is a variable of type list(string)
However, this throws an error: Blocks of type "value_from" are not expected here.
I'm not sure what I'm missing, or where I have incorrectly specified the value_from block.
Could someone point me in the right direction for fixing this up?
UPDATE;
I have also tried to implement this variable as a map, as per the suggestion in the comments on this post. (https://www.terraform.io/docs/language/expressions/dynamic-blocks.html#multi-level-nested-block-structures)
dynamic "env" {
for_each = var.secrets
content {
name = each.key
dynamic "value_from" {
for_each = env.value.name
secret_key_ref {
name = value_from.value.name
key = value_from.value.version
}
}
}
}
However, this also gives the same error. Blocks of type "value_from" are not expected here.
In this example, the secrets variable is defined as a list(any) with this value:
secrets = [
{
name = "SECRET"
version = "1"
}
]
You have to upgrade your gcp provider. Support for secrets in google_cloud_run_service was added in v3.67.0. Current version is v4.1.0, which means that you must be using very old gcp provider.
In the end, I solved this by changing the variable type to a map(any):
secrets = {
"SECRET" = "1"
}
This allowed me to create the "dynamic" env block, without needing to implement the nested dynamic block.
Example: https://www.terraform.io/docs/providers/kubernetes/r/service_account.html
We see this:
resource "kubernetes_service_account" "example" {
metadata {
name = "terraform-example"
}
secret {
name = "${kubernetes_secret.example.metadata.0.name}"
}
}
Metadata is not a list, why does the secret/name value reference metadata using .0?
The provider has defined the metadata block type as being represented internally as a list of objects. This is common in today's Terraform providers because Terraform versions prior to Terraform v0.12 required a provider to choose between only lists and sets as the backing data type for nested block types.
Terraform v0.12 and later do support having a singleton block represented as a single object, but providers that predated the Terraform v0.12 release tend to still use the list representation for backward-compatibility. (Most providers are, at the time of writing, still compatible with both Terraform v0.11 and v0.12 in their new releases.)
The new kubernetes provider (still under development at the time I'm writing this) was built specifically for Terraform v0.12 and later, so it is able to more directly reflect the underlying Kubernetes schema, including treating metadata as a single object. The announcement post about the new provider version includes this example:
resource "kubernetes_manifest" "example_crd" {
provider = kubernetes-alpha
manifest = {
apiVersion = "apiextensions.k8s.io/v1"
kind = "CustomResourceDefinition"
metadata = {
name = "testcrds.hashicorp.com"
labels = {
app = "test"
}
}
spec = {
group = "hashicorp.com"
names = {
kind = "TestCrd"
plural = "testcrds"
}
scope = "Namespaced"
versions = [
{
name = "v1"
served = true
storage = true
schema = {
openAPIV3Schema = {
type = "object"
properties = {
data = {
type = "string"
}
refs = {
type = "number"
}
}
}
}
}
]
}
}
}
The metadata name for this new provider can be accessed in a more intuitive way, because metadata is a single object:
kubernetes_manifest.example_crd.manifest.metadata.name
I am trying to access the Kubernetes_secret data.token attribute in terraform, but I keep on getting the error
Resource 'data.kubernetes_secret.misp_whitelist_secret' does not have attribute 'data.token' for variable 'data.kubernetes_secret.misp_whitelist_secret.data.token'
Whats the way to resolve this issue?
resource "kubernetes_service_account" "misp_whitelist_sa" {
metadata {
name = "misp-whitelist-sa"
}
}
data "kubernetes_secret" "misp_whitelist_secret" {
metadata {
name = "${kubernetes_service_account.misp_whitelist_sa.default_secret_name}"
namespace = "${kubernetes_service_account.misp_whitelist_sa.metadata.0.namespace}"
}
depends_on = [
"kubernetes_service_account.misp_whitelist_sa",
]
}
And I'm trying to access the data.token inside the terraform google_cloud_function resource
resource "google_cloudfunctions_function" "misp_whitelist_function" {
name = "${var.cluster}-misp-whitelist"
....<additional data> .....
environment_variables = {
CLUSTER = "${var.cluster}"
PROJECT = "${var.project}"
AUTH = "${data.kubernetes_secret.misp_whitelist_secret.data.token}"
}
}
The correct way to access the data secret key is:
AUTH = "${data.kubernetes_secret.misp_whitelist_secret.data["token"]}"
Ok banged my head against a wall here for a really long time. The other answer is correct, but skips a crucial step.
You need to make sure that the secret declares the correct type (and also maybe specify the annotation?)
resource "kubernetes_secret" "vault" {
metadata {
name = "vault-token"
annotations = {
"kubernetes.io/service-account.name" = "vault"
}
}
type = "kubernetes.io/service-account-token" // THIS!
}
Then, once you have the proper type specified, you can use the token
output "token" {
value = kubernetes_secret.vault.data.token
}
The following code gives me a list of all EC2 instances that are part of my cluster:
data "aws_instances" "emrMaster" {
instance_tags {
Name = "emr-cluster-name"
}
}
But when I try to narrow the list using the AWS generated tag for the master instance, I get the same list.
data "aws_instances" "emrMaster" {
instance_tags {
Name = "emr-cluster-name"
"aws:elasticmapreduce:instance-group-role" = "MASTER"
}
}
If I remove the quotes on the key name, I get a illegal character error due to the colons in the key name.
data "aws_instances" "emrMaster" {
instance_tags {
Name = "emr-cluster-name"
aws:elasticmapreduce:instance-group-role = "MASTER"
}
}
Is there a better way to do this, am I doing something wrong, or have I uncovered a bug in Terraform?
I am using Terraform v0.11.7
I am looking to capture this value so I can build specific cloudwatch alerts for the master instance that are different from the Core instances.
For this purpose better use aws_instance (not aws_instances)
resource "aws_emr_cluster" "emr-cluster" {
....
}
data "aws_instance" "master" {
filter {
name = "tag:Name"
values = ["${aws_emr_cluster.emr-cluster.name}"]
}
filter {
name = "tag:aws:elasticmapreduce:instance-group-role"
values = ["MASTER"]
}
}
And then just use output:
output "master_id" {
value = "${data.aws_instance.master.id}"
}