Terraform loop through JSON array to create iam users - terraform

I have a simple json file containing a list of users and groups. From this list, I would like to create the users in AWS IAM but my for_each or merging syntax is wrong.
When running terraform plan, I get the following error:
Error: Error in function call
│
│ on locals.tf line 3, in locals:
│ 3: json_data = merge([for f in local.json_files : jsondecode(file("${path.module}/input/${f}"))]...)
│ ├────────────────
│ │ local.json_files is set of string with 1 element
│ │ path.module is "."
│
│ Call to function "merge" failed: arguments must be maps or objects, got "tuple".
How do I properly loop through the list (tuple) of objects in the JSON file?
JSON File sample:
[
{ "name": "user1", "groups": ["Admins", "DevOps"], "policies": [] },
{ "name": "user2", "groups": ["DevOps"], "policies": [] }
]
Terraform Code:
locals {
json_files = fileset("${path.module}/input/", "*.json")
json_data = merge([for f in local.json_files : jsondecode(file("${path.module}/input/${f}"))]...)
}
resource "aws_iam_user" "create_new_users" {
for_each = local.json_data
name = each.name
}
As a side note, I did manage to get the service to work by changing the JSON file to the following structure, but prefer to use the former:
{
"user1": {"groups": ["Admins","DevOps"],"policies": []},
"user2": {"groups": ["DevOps"],"policies": []}
}
and updating the aws_iam_user resource to:
resource "aws_iam_user" "create_new_users" {
for_each = local.json_data
name = each.key
}

The JSON document you showed is using a JSON array, which corresponds with the tuple type in Terraform, so it doesn't make sense to use merge for that result -- merge is for merging together maps, which would correspond most closely with an object in JSON. (and indeed, that's why your second example with an object containing a property with each user worked).
For sequence-like types (lists and tuples) there is a similar function concat which will append them together to produce a single longer sequence containing all of the items in the order given. You could use that function instead of merge to get a single list of all of the users as a starting point:
locals {
json_files = fileset("${path.module}/input/", "*.json")
json_data = concat([for f in local.json_files : jsondecode(file("${path.module}/input/${f}"))]...)
}
The resource for_each argument wants a mapping type though, so you'll need to do one more step to project this list of objects into a map of objects using the name attribute values as the keys:
resource "aws_iam_user" "create_new_users" {
for_each = { for u in local.json_data : u.name => u }
name = each.value.name
}
This will cause Terraform to identify each instance of the resource by the object's "name" property, and so with the sample input file you showed this will declare two instances of this resource with the following addresses:
aws_iam_user.create_new_users["user1"]
aws_iam_user.create_new_users["user2"]
(Note that it's unusual to name a Terraform resource using a verb. Terraform doesn't understand English grammar of course, so it doesn't really matter what you name it, but it's more typical to use a noun because this is only describing that a set of users should exist; you'll use this same object later to describe updating or destroying these objects. If this JSON document just represents all of your users then a more typical name might be aws_iam_user.all, since the resource type already says that these are users -- so there's no need to restate that -- and so all that's left to say is which users these are.)

Related

Terraform Lifecycle Ignore changes

I am trying to apply lifecycle ignore_changes rule against parameter in resource resource "aws_servicecatalog_provisioned_product" as shown below.
resource "aws_servicecatalog_provisioned_product" "example" {
name = "example"
product_name = "Example product"
provisioning_artifact_name = "Example version"
provisioning_parameters {
key = "foo"
value = "bar"
}
provisioning_parameters {
key = "key2"
value = lookup(var.parameter_group, "key2", "test2")
}
provisioning_parameters {
key = "key3"
value = "test3"
}
tags = {
foo = "bar"
}
lifecycle {
ignore_changes = [
tags["foo"],
aws_servicecatalog_provisioned_product.provisioning_parameters.example["key2"]
]
}
}
variable parameter_group {
description = "Parameters map required for modules.
type = map(any)
default = {}
}
when i am running the plan i am getting below error
│ Error: Unsupported attribute
│
│ on modules/example_provision/main.tf line 28, in resource "aws_servicecatalog_provisioned_product" "example":
│ 28: aws_servicecatalog_provisioned_product.provisioning_parameters.example["key2"]
│
│ This object has no argument, nested block, or exported attribute named "aws_servicecatalog_provisioned_product".
I would like to ignore the changes made to this parameter value. The Ignore on tags is working fine but as soon as i add my second line which is aws_servicecatalog_provisioned_product.provisioning_parameters.example["key2"] the error starts to come in.
looking for suggestion/help here :)
Regards
ignore_changes can only ignore changes to the configuration of the same resource where it's declared, and so you only need to name the argument you wish to ignore and not the resource type or resource name:
lifecycle {
ignore_changes = [
tags["foo"],
provisioning_parameters,
]
}
The provisioning_parameters block type doesn't seem to be represented as a mapping (the provisioning_parameter blocks don't have labels in their headers) so you won't be able to refer to a specific block by its name.
However, the provider does declare it as being a list of objects and so I think you will be able to ignore a specific item by its index, where the indices are assigned in order of declaration. Therefore in your current example the one with key = "key2" will have index 1, due to being the second block where Terraform counts up from zero:
lifecycle {
ignore_changes = [
tags["foo"],
provisioning_parameters[1],
]
}

jsondecode fails when using for_each to pass variables to module

I'm trying to use for_each with a terraform module creating datadog synthetic tests. The object names in an s3 bucket are listed and passed as the set for the for_each. The module reads the content of each file using the each.value passed in by the calling module as the key. I hardcoded the s3 object key value in the module during testing and it was working. When I attempt to call the module from main.tf, passing in the key name dynamically from the set it fails with the below error.
│ Error: Error in function call
│
│ on modules\Synthetics\trial2.tf line 7, in locals:
│ 7: servicedef = jsondecode(data.aws_s3_object.tdjson.body)
│ ├────────────────
│ │ data.aws_s3_object.tdjson.body is ""
│
│ Call to function "jsondecode" failed: EOF.
main.tf
data "aws_s3_objects" "serviceList" {
bucket = "bucketname"
}
module "API_test" {
for_each = toset(data.aws_s3_objects.serviceList.keys)
source = "./modules/Synthetics"
S3key = each.value
}
module
data "aws_s3_object" "tdjson" {
bucket = "bucketname"
key = var.S3key
}
locals {
servicedef = jsondecode(data.aws_s3_object.tdjson.body)
Keys = [for k,v in local.servicedef.Endpoints: k]
}
Any clues as to what's wrong here?
Thanks
Check out the note on the aws_s3_object data source:
The content of an object (body field) is available only for objects which have a human-readable Content-Type (text/* and application/json). This is to prevent printing unsafe characters and potentially downloading large amount of data which would be thrown away in favour of metadata.
Since it's successfully getting the data source (not throwing an error), but the body is empty, this is very likely to be your issue. Make sure that your S3 object has the Content-Type metadata set to application/json. Here's a Stack Overflow question/answer on how to do that via the CLI; you can also do it via the AWS console, API, or Terraform (if you created the object via Terraform).
EDIT: I found the other issue. Check out the syntax for using for_each with toset:
resource "aws_iam_user" "the-accounts" {
for_each = toset( ["Todd", "James", "Alice", "Dottie"] )
name = each.key
}
The important bit is that you should be using each.key instead of each.value.

Terraform: Reference locals values inside the creation block

Is it possible to reference another locals value inside the creation of a locals value?
The example below was the smallest and simplest example I could come up with.
variable "size" {
default = 3
}
variable "infrastructure_version" {
default = 1
}
locals {
values = {
for n in range(var.size) : n => {
name = "instance_${n + 1}"
full_name = "test_${name}_v${var.infrastructure_version}"
}
}
}
When trying to access name within the for loop inside the locals block i get the following error:
│ Error: Invalid reference
│
│ on instances.tf line 13, in locals:
│ 13: full_name = "test_${name}_v${var.infrastructure_version}"
│
│ A reference to a resource type must be followed by at least one attribute access, specifying the resource name.
Other attempts:
(These were desperate attempts with no real likelihood of succeeding)
local.values[n].name which gives Error: Self-referencing local value
n.name which gives Error: Unsupported attribute
self.name which gives Error: Invalid "self" reference
Anyone know if this is possible? Or am I stuck repeating the creation of name inside full_name as well?
full_name = "test_instance_${n + 1}_v${var.infrastructure_version}"
Instead of doing variable interpolation every time a value should be used, it's possible to create a local module that acts as a function.
Where you can use local variables and reuse previously created variables.
Short example below, it's better suited when used for more complex and larger applications due to the amount of overhead.
main.tf:
From the main module I import the local module that will serve as a function
module "instance_conf" {
source = "./modules"
count = var.size
index = count.index
infra = var.infrastructure_version
}
locals {
values = {for idx, val in module.instance_conf: idx => val}
}
I send in index and infra to the module as input. The variable definitions in the other module must be matching those, here you could also provide descriptions if needed.
modules/func.tf:
variable "index" {
type = number
}
variable "infra" {
type = number
}
locals {
name = "instance_${var.index + 1}"
}
output "name" {
value = local.name
}
output "full_name" {
value = "test_${local.name}_v${var.infra}"
}
To get the desired output to the main module, calculate values either in the locals block or directly in the output block. When importing this module the values will be available as a list of maps, for each count.index in the main module. count = var.size
The list could look like this:
[
{
name: "instance_1",
full_name: "test_instance_1_v1"
},
{
name: "instance_2",
full_name: "test_instance_2_v1"
},
...
]
So in order to use the module output as previous with for_each I converted the list of map objects, to a map with the index of each map object as the key for that object.
locals {
values = {for idx, val in module.instance_conf: idx => val}
}
And now when using local.values it will look like this:
{
"1": {
name: "instance_1",
full_name: "test_instance_1_v1"
},
"2": {
name: "instance_2",
full_name: "test_instance_2_v1"
},
...
}
The project structure now looks like this:
.
├── main.tf
├── modules
│   └── values_function.tf
Hopefully this helps someone else out. When variable interpolation and recreation of a value every time it is used, is not an acceptable answer. Mainly because of the maintainability factor.
Your last attempt is correct. You can't make it different and it works:
full_name = "test_instance_${n + 1}_v${var.infrastructure_version}"

How to skip for_each loop if key does not exist in terraform

My problem statement is simple but I am not able to find a solution anywhere on the internet.
I have users list as locals:
// users
locals {
allUsers = {
dev_user_1 = {
Name = "user1"
Email = "user1#abc.com"
GitHub = "user1" # github username
Team = "Dev"
}
devops_user_2 = {
Name = "user2"
Email = "user2#abc.com"
GitHub = "user2" # github username
Team = "DevOps"
}
product_user_3 = {
Name = "user3"
Email = "user3#abc.com"
Team = "Product"
}
}
}
These are the local tags that are being used for purposes of creating access to internal tools such as Github, Monitoring tools, etc.
Now, for the 2 users who belong to the Dev and DevOps team, they need access to Github ORG, while, the product user only needs access to some dashboards but not to Github, hence, the tag is missing.
How can I loop over the terraform resource github_membership to skip this product user (or simply anyone who does not have tag key GitHub?)
I am trying the following code, but no luck
// Send GitHub invite
resource "github_membership" "xyzTeam" {
for_each = local.allUsers
username = each.value.GitHub
role = "member"
}
Errors:
╷
│ Error: Unsupported attribute
│
│ on users.tf line 12, in resource "github_membership" "xyzTeam":
│ 12: username = each.value.GitHub
│ ├────────────────
│ │ each.value is object with 3 attributes
│
│ This object does not have an attribute named "GitHub".
What I did to solve this issue?
Set GitHub key for everyone but it's value as null.
Error:
╷
│ Error: "username": required field is not set
│
│ with github_membership.xyzTeam["user3"],
│ on users.tf line 10, in resource "github_membership" "xyzTeam":
│ 10: resource "github_membership" "devops" {
│
╵
If I left the value empty, errors:
Error: PATCH https://api.github.com/user/memberships/orgs/XYZ: 422 You can only update an organization membership's state to 'active'. []
for k, v in local.allUsers : k => v if v != ""
Same error because it tries to create the user with empty value still, and fails ultimately.
I cannot think of anything else. If someone can help to create separate locals from these existing locals, which creates the list of locals that grep the GitHub values, that hack would be super helpful.
You had the right idea with your third attempt, but the conditional logic in the for expression is slightly off. You need to use the can function instead:
{ for user, attributes in local.allUsers : user => attributes if can(attributes.GitHub) }
If the nested map contains a Github key, then can(attributes.Github) returns true, and the map constructor will contain the key-value pair. With this algorithm, you can construct a new map from the old map with the entries removed that do not contain a Github key in the nested map value.

Dynamic resources for_each output in terraform module

Terraform v1.0.0
Provider: aws v3.49.0
I created dynamic AWS subnets resources with a for_each from a module.
The resources creation is working fine, however being able to output dynamically created resources is not working and cannot find proper documentation for it.
The subnet module is
resource "aws_subnet" "generic" {
vpc_id = var.vpc_id
cidr_block = var.cidr_block
map_public_ip_on_launch = var.public_ip_on_launch
tags = {
Name = var.subnet_tag_name
Environment = var.subnet_environment
}
}
With simple module output defined
output "subnet_id" {
value = aws_subnet.generic.id
}
Then from root module, I am creating a for_each loop over a list variable to create multiple dynamic resources from the module
module "subnets" {
source = "../modules/networking/subnet"
for_each = var.subnets
vpc_id = "vpc-09d6d4c17544f3a49"
cidr_block = each.value["cidr_block"]
public_ip_on_launch = var.public_ip_on_launch
subnet_environment = var.subnet_environment
subnet_tag_name = each.value["subnet_tag_name"]
}
When I run this without defining outputs in the root module, things get created normally. The problem comes when I try to define the outputs
output "subnets" {
value = module.subnets.*.id
description = "Imported VPC ID"
}
It comes up with this error
│ Error: Unsupported attribute
│
│ on output.tf line 2, in output "subnets":
│ 2: value = module.subnets.*.id
│
│ This object does not have an attribute named "id".
I tried different output definitions. Would appreciate guidance on how to properly define outputs of instances dynamically created with a for_each module.
Per the Terraform documentation, the "splat" operator (*) can only be used with lists, and since you're using for_each your output will be a map.
You need to use map/list comprehension to achieve what you want.
For an output that is a map of key/value pairs (note that I've changed the output description to something that makes more sense):
output "subnets" {
value = {
for k, v in module.subnets:
k => v.subnet_id
}
description = "Subnet IDs"
}
For a list that only contains the subnet IDs:
output "subnets" {
value = [
for k, v in module.subnets:
v.subnet_id
]
description = "Subnet IDs"
}

Resources