Converting json to yaml in Terraform - terraform

How can we generate an yaml out from a json
{
"users":[
{
"name":"rock",
"age":33,
"city":"paris"
},
{
"name":"austin",
"age":45,
"city":"greece"
}
]
}
Expected output with a additional field with random password
users:
- key: 'user[0].name'
value: rock
- key: 'user[0].age'
value: '33'
- key: 'user[0].city'
value: paris
- key: 'user[0].password'
value: '5]L+J7rA*<7+:PO6'
- key: 'user[1].name'
value: austin
- key: 'user[1].age'
value: '45'
- key: 'user[1].city'
value: greece
- key: 'user[1].password'
value: P=x&385YGMI0?!Is
The files are used only on a local machine.

A straight conversion from JSON to YAML would involve just passing the result of jsondecode to yamlencode.
However, the meat of your question seems to not be about conversion between serialization formats but rather about transforming from a typical nested data structure into a flattened structure using a compound key syntax.
Transforming from a multi-level structure into a single-level structure is a good job for flatten:
locals {
users = flatten([
for idx, u in local.users : [
for k, v in u : {
key = "user[${idx}].${v}"
value = v
}
]
])
}
The result would be a Terraform value corresponding to the data structure you illustrated in YAML:
[
{
key = "user[0].name"
value = "rock"
},
{
key = "user[0].age"
value = 33
},
# ...etc...
]
You can then pass the result to yamlencode to produce the YAML serialization of that data structure.

You can use file function of terraform to read the content of your json.
You can then use jsondecode to decode/represent the string as json.
The last step is to use , yamlencode function to encode returned json representation in yaml format.
Please see below terraform config.
locals {
test = yamlencode(jsondecode(file("${path.module}/test.json")))
}
output "test" {
value = local.test
}
The result from the above tf config is as below
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
test = "users":
- "age": 33
"city": "paris"
"name": "rock"
- "age": 45
"city": "greece"
"name": "austin"

locals {
users = jsondecode(file("${path.module}/user.json"))
}
resource "random_string" "password" {
count = length(local.users.users)
length = 16
special = true
}
output "pass" {
value = random_string.password
}
locals {
user_flat = yamlencode(flatten([for i,j in local.users.users:[
for k,v in j : {
key = "user.[${i}].${k}"
"value" = v
}
]
]
)
)
out = yamlencode({"users"= local.user_flat})
}
output "Desired_output" {
value = local.out
}
Try this code.
Outputs:
Desired_output = <<EOT
"users": |
- "key": "user.[0].age"
"value": 23
- "key": "user.[0].city"
"value": "barcelona"
- "key": "user.[0].name"
"value": "john"
- "key": "user.[1].age"
"value": 29
- "key": "user.[1].city"
"value": "london"
- "key": "user.[1].name"
"value": "bob"
EOT
pass = [
{
"id" = "XEXj*svXm#d&_%Fn"
"keepers" = tomap(null) /* of string */
"length" = 16
"lower" = true
"min_lower" = 0
"min_numeric" = 0
"min_special" = 0
"min_upper" = 0
"number" = true
"override_special" = tostring(null)
"result" = "XEXj*svXm#d&_%Fn"
"special" = true
"upper" = true
},
{
"id" = "Ews%V%Xk[YBd]D_M"
"keepers" = tomap(null) /* of string */
"length" = 16
"lower" = true
"min_lower" = 0
"min_numeric" = 0
"min_special" = 0
"min_upper" = 0
"number" = true
"override_special" = tostring(null)
"result" = "Ews%V%Xk[YBd]D_M"
"special" = true
"upper" = true
},
]

Related

Loading values dynamically from Terraform into a map

I'm trying to load some external data from a json file into Terraform to merge into an appSettings map
It's loading it in as a tuple - and no matter what conversion I do, I can't get a map out of it:
Call to function "merge" failed: arguments must be maps or objects, got "tuple".
Json File
[
{
"appCode": "value",
"containerName": "value",
"databaseName": "value",
"referer": "bvalue",
"shortCode": "value",
"user": "value"
},
{
"appCode": "value",
"containerName": "value",
"databaseName": "value",
"referer": "value",
"shortCode": "value",
"user": "value"
}
]
Locals:
customerSettings = jsondecode(file("vars/${var.environment}.json"))
customerAppSettingsFromJson = {
for index, externalCustomer in local.customerSettings :
externalCustomer => {
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__AppCode" = "${externalCustomer.appCode}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__ContainerName" = "${externalCustomer.containerName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__DatabaseName" = "${externalCustomer.databaseName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__Referer" = "${externalCustomer.databaseName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__ShortCode" = "${externalCustomer.shortCode}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__User" = "${externalCustomer.user}"
}
}
Main.tf appSettings block, has inferred ones, ones from vars and ones from json
app_settings = merge({}, var.app_settings, local.customerAppSettingsFromJson)
You were very close to the solution, but here is how to convert the list of objects into a map:
customerSettings = jsondecode(file("vars/${var.environment}.json"))
customerAppSettingsFromJson = {
for index, externalCustomer in local.customerSettings :
index => {
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__AppCode" = "${externalCustomer.appCode}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__ContainerName" = "${externalCustomer.containerName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__DatabaseName" = "${externalCustomer.databaseName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__Referer" = "${externalCustomer.databaseName}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__ShortCode" = "${externalCustomer.shortCode}"
"DynamicCosmosDbSettings__CosmosHostSettings__${index}__User" = "${externalCustomer.user}"
}
}

Terraform: nested for loop from yaml

I am trying to run nested for loop on terraform.
I have the following Yaml file:
Employees:
- Department:
- Dev:
- name: "danielf"
role: developer
details:
email : danielf#example.com
firstname : daniel
lastname : folsik
- name: "johnb"
role: developer
details:
email : johnb#example.com
firstname : john
lastname : belk
- Ops:
- name: "benol"
role: devops
details:
email : benol#example.com
firstname : ben
lastname : olkin
- name: "pauld"
role: devops
details:
email : pauld#example.com
firstname : paul
lastname : dempler
I am using locals to get the yaml data:
locals {
ou_config = yamldecode(file("employees.yaml"))
}
I want to run into the list of objects on "Dev" and "Ops" lists using for_each.
for example, I want to run on the "Dev" list to get the following list of objects in the first iteration:
[
{
key = "email"
value = "danielf#example.com"
},
{
key = "firstname"
value = "daniel"
},
{
key = "lastname"
value = "folskin"
}
]
The next run on the for_each will be:
[
{
key = "email"
value = "johnb#example.com"
},
{
key = "firstname"
value = "john"
},
{
key = "lastname"
value = "belk"
}
]
etc.
How can I do it on terraform?
If I understand correctly, all you are trying to extract is the details portion of that yaml file ...
Here is what I would do to get all:
locals {
ou_config = yamldecode(file("employees.yaml"))
expanded_names = flatten([
for e in local.ou_config.Employees : [
for d in e.Department : [
for key, person in d : [
for key, value in person : [
value.details
]
]
]
]
])
}
output "test" {
value = local.expanded_names
}
And if we want to filter we add an if key == "Dev"
locals {
ou_config = yamldecode(file("employees.yaml"))
expanded_names = flatten([
for e in local.ou_config.Employees : [
for d in e.Department : [
for key, person in d : [
for key, value in person : [
value.details
]
] if key == "Dev"
]
]
])
}
output "test" {
value = local.expanded_names
}
A terraform plan on that will look like:
Changes to Outputs:
+ test = [
+ {
+ email = "danielf#example.com"
+ firstname = "daniel"
+ lastname = "folsik"
},
+ {
+ email = "johnb#example.com"
+ firstname = "john"
+ lastname = "belk"
},
]
That format should be easier to loop in the final resource than the key value you suggested

Is it possible to perform nested iterations in HCL resulting in a flat list without calling flatten?

Is it possible with HCL to have nested iterations returning a flat list(map) without resorting to flatten?
I have this:
locals {
mappings = flatten([
for record_type in var.record_types : [
for host in var.hosts : {
type = record_type,
host = host
}
]
])
}
I would like to remove the need for flatten like this:
locals {
mappings = [
for record_type in var.record_types :
for host in var.hosts : {
type = record_type,
host = host
}
]
}
But it seems like each for .. in must return data.
One alternative I could think of to only have a single for-loop is using setproduct():
variable "record_types" {
default = ["type1", "type2"]
}
variable "hosts" {
default = ["host1", "host2"]
}
locals {
mappings = [
for i in setproduct(var.record_types, var.hosts) : {
type = i[0],
host = i[1],
}
]
}
output "mappings" {
value = local.mappings
}
after terraform apply resulting in:
Outputs:
mappings = [
{
"host" = "host1"
"type" = "type1"
},
{
"host" = "host2"
"type" = "type1"
},
{
"host" = "host1"
"type" = "type2"
},
{
"host" = "host2"
"type" = "type2"
},
]
Of course, the two variables need to be independent sets here.
If you want to support duplicates or have dependent inputs, flatten() with two loops is the way.

Power Query Expression error: A value of type "Record" cannot converted into type "Text"

I have the issue that I'm not able to execute the following code. The syntax seems to be okay, but when I try to execute it, I get the response, that:
Expression.Error: We cannot convert a value of type Record to type "Text".
Details:
Value=[Record]
Type=[Type]
let
body="{
""page"": ""1"",
""pageSize"": ""100"",
""requestParams"": {
""deviceUids"": [
""xxx-yyy-xxx-yyyy-xxxx"",
""yyy-xxx-yyy-xxxx-yyyy"",
""aaa-bbb-aaa-bbbb-aaaa"",
""ccc-ddd-ccc-dddd-cccc""
],
""entityColumns"": [
{
""entityId"": ""144"",
""joinColumnName"": ""device_uid"",
""columnName"": ""device_random_date""
}
],
""columnNames"": [
""ts"",
""device_uid"",
""1"",
""32"",
""55"",
""203"",
""204""
],
""startUnixTsMs"": ""1583413637000"",
""endUnixTsMs"": ""1583413640000"",
""columnFilters"": [
{
""filterType"": ""eq"",
""columnName"": ""55"",
""value"": ""1234""
}
],
""sortOrder"": [
{
""column"": ""ts"",
""order"": ""DESC""
},
{
""column"": ""55"",
""order"": ""ASC""
}
],
""entityFilters"": [
{
""entityId"": ""144"",
""entityEntryIds"": [
""12345-221-232-1231-123456""
]
}
]
}
}",
Parsed_JSON = Json.Document(body),
BuildQueryString = Uri.BuildQueryString(Parsed_JSON),
Quelle = Json.Document(Web.Contents("http://localhost:8101/device-data-reader-api/read-paginated/xxx-xxx-yyyy-yyyy", [Headers=[#"Content-Type"="application/json"], Content = Text.ToBinary(BuildQueryString)]))
in
Quelle
I tried to remove the quotes of the numbers, but this leads to the same issue, as system complains it cannot convert numbers into text.
I need the body which needs to be handed over with the request in order to do a POST request. What I'm doing wrong?
Since you seem to want to send this as application/json, I think you would change this bit in your code:
Content = Text.ToBinary(BuildQueryString)
to:
Content = Text.ToBinary(body)
and then you'd also get rid of the lines below (since you don't need them):
Parsed_JSON = Json.Document(body),
BuildQueryString = Uri.BuildQueryString(Parsed_JSON),
I don't think you would need Uri.BuildQueryString unless you wanted to send as application/x-www-form-urlencoded (i.e. URL encoded key-value pairs).
Unrelated: If it helps, you can build the structure in M and then use JSON.FromValue to turn the structure into bytes which can be put directly into the POST body. Untested example is below.
let
body = [
page = "1",
pageSize = "100",
requestParams = [
deviceUids = {
"xxx-yyy-xxx-yyyy-xxxx",
"yyy-xxx-yyy-xxxx-yyyy",
"aaa-bbb-aaa-bbbb-aaaa",
"ccc-ddd-ccc-dddd-cccc"
},
entityColumns = {
[
entityId = "144",
joinColumnName = "device_uid",
columnName = "device_random_date"
]
},
columnNames = {
"ts",
"device_uid",
"1",
"32",
"55",
"203",
"204"
},
startUnixTsMs = "1583413637000",
endUnixTsMs = "1583413640000",
columnFilters = {
[
filterType = "eq",
columnName = "55",
value = "1234"
]
},
sortOrder = {
[
column = "ts",
order = "DESC"
],
[
column = "55",
order = "ASC"
]
},
entityFilters = {
[
entityId = "144",
entityEntryIds = {
"12345-221-232-1231-123456"
}
]
}
]
],
Quelle = Json.Document(
Web.Contents(
"http://localhost:8101/device-data-reader-api/read-paginated/xxx-xxx-yyyy-yyyy",
[
Headers = [#"Content-Type" = "application/json"],
Content = Json.FromValue(body)
]
)
)
in
Quelle
It might look a little weird (since M uses [] instead of {}, {} instead of [] and = instead of :), but just mentioning in case it helps.

any smart way to convert nodejs object to a string

The problem is coming from this discussion
https://github.com/hashicorp/terraform/issues/11036#issuecomment-357334325
Terraform doesn't really understand multiple list, so I have to convert it to below format
locals {
test = [{
a = "a1"
b = "b1"
},{
a= "a2"
b = "b2"
}]
}
So suppose I have below object
[
{
"name": "a",
"value": "foo"
},
{
"name": "b",
"value": "bar"
},
{
"name": "c",
"value": "boo"
},
{
"name": "d",
"value": "far"
}
]
I'd like to convert to string as below:
[ { name = "a", value = "foo" }, { name = "b", value = "bar" }, { name = "c", value = "boo" }, { name = "d", value = "far" } ]
The code I am currently working is to go through each key and export it with new format.
$ cat a.js
var array1 = [{"name":"a","value":"foo"},{"name":"b","value":"bar"},{"name":"c","value":"boo"},{"name":"d","value":"far"}]
array1.forEach(function(element) {
Object.keys(element).forEach(function(key) {
console.log(key);
console.log(element[key]);
});
});
$ node a.js
name
a
value
foo
name
b
value
bar
name
c
value
boo
name
d
value
far
Any smart way to convert it, more than go through each key and export the key and value with nominated format?
Regex replacing will do the job:
var array1 = [{"name":"a","value":"foo"},{"name":"b","value":"bar"},{"name":"c","value":"boo"},{"name":"d","value":"far"}]
a = JSON.stringify(array1);
a = a.replace(/\[|{|\]|}|"/g, "")
a = a.replace(/:|,/g, "\n");
console.log(a);
Updates
Finally I go with #MrfksIV 's solution.
The limit is, there should be no space in any values.
var array1 = [{"name":"a","value":"foo"},{"name":"b","value":"bar"},{"name":"c","value":"boo"},{"name":"d","value":"far"}]
a = JSON.stringify(array1);
a = a.replace(/"/g, "")
a = a.replace(/:/g, "=")
console.log(a);
output is
[{name=a,value=foo},{name=b,value=bar},{name=c,value=boo},{name=d,value=far}]

Resources