Terraform registry AWS VPC example terraform-aws-vpc/examples/complete-vpc/main.tf has the code below which seems to me a circular dependency.
data "aws_security_group" "default" {
name = "default"
vpc_id = module.vpc.vpc_id
}
module "vpc" {
source = "../../"
name = "complete-example"
...
# VPC endpoint for SSM
enable_ssm_endpoint = true
ssm_endpoint_private_dns_enabled = true
ssm_endpoint_security_group_ids = [data.aws_security_group.default.id] # <-----
...
data.aws_security_group.default refers to "module.vpc.vpc_id" and module.vpc refers to "data.aws_security_group.default.id".
Please explain why this does not cause an error and how come module.vpc can refer to data.aws_security_group.default.id?
In the Terraform language, a module creates a separate namespace but it is not a node in the dependency graph. Instead, each of the module's Input Variables and Output Values are separate nodes in the dependency graph.
For that reason, this configuration contains the following dependencies:
The data.aws_security_group.default resource depends on module.vpc.vpc_id, which is specifically the output "vpc_id" block in that module, not the module as a whole.
The vpc module's variable "ssm_endpoint_security_group_ids" variable depends on the data.aws_security_group.default resource.
We can't see the inside of the vpc module in your question here, but the above is okay as long as there is no dependency connection between output "vpc_id" and variable "ssm_endpoint_security_group_ids" inside the module.
I'm assuming that such a connection does not exist, and so the evaluation order of objects here would be something like this:
aws_vpc.example in module.vpc is created (I just made up a name for this because it's not included in your question)
The output "vpc_id" in module.vpc is evaluated, referring to module.vpc.aws_vpc.example, and producing module.vpc.vpc_id.
data.aws_security_group.default in the root module is read, using the value of module.vpc.vpc_id.
The variable "ssm_endpoint_security_group_ids" for module.vpc is evaluated, referring to data.aws_security_group.default.
aws_vpc_endpoint.example in module.vpc is created, including a reference to var.ssm_endpoint_security_group_ids.
Notice that in all of the above I'm talking about objects in modules, not modules themselves. The modules serve only to create separate namespaces for objects, and then the separate objects themselves (which includes individual variable and output blocks) are what participate in the dependency graph.
Normally this design detail isn't visible: Terraform normally just uses it to potentially optimize concurrency by beginning work on part of a module before the whole module is ready to process. In some interesting cases like this though, you can also intentionally exploit this design so that an operation for the calling module can be explicitly sandwiched between two operations for the child module.
Another reason why we might make use of this capability is when two modules naturally depend on one another, such as in an experimental module I built that hides some of the tricky details of setting up VPC peering connections:
locals {
vpc_nets = {
us-west-2 = module.vpc_usw2
us-east-1 = module.vpc_use1
}
}
module "peering_usw2" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-east-1 = module.peering_use1.outgoing_connection_ids
}
providers = {
aws = aws.usw2
}
}
module "peering_use1" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-west-2 = module.peering_usw2.outgoing_connection_ids
}
providers = {
aws = aws.use1
}
}
(the above is just a relevant snippet from an example in the module repository.)
In the above case, the peering-mesh module is carefully designed to allow this mutual referencing, internally deciding for each pair of regional VPCs which one will be the peering initiator and which one will be the peering accepter. The outgoing_connection_ids output refers only to the aws_vpc_peering_connection resource and the aws_vpc_peering_connection_accepter refers only to var.other_region_connections, and so the result is a bunch of concurrent operations to create aws_vpc_peering_connection resources, followed by a bunch of concurrent operations to create aws_vpc_peering_connection_accepter resources.
Related
My Terraform code is broadly architected like so:
module "firewall_hub" {
# This creates the Azure Firewall resource
source = "/path/to/module/a"
# attribute = value...
}
module "firewall_spoke" {
# This creates, amongst other things, firewall rule sets
source = "/path/to/module/b"
hub = module.firewall_hub
# attribute = value...
}
module "another_firewall_spoke" {
# This creates, amongst other things, firewall rule sets
source = "/path/to/module/c"
hub = module.firewall_hub
# attribute = value...
}
i.e., The Azure Firewall resource is created in module.firewall_hub, which is used as an input into module.firewall_spoke and module.another_firewall_spoke that create their necessary resources and inject firewall rule sets into the Firewall resource. Importantly, the rule sets are mutually exclusive between spoke modules and designed such that their priorities don't overlap.
When I try to deploy this code (either build or destroy), Azure throws an error:
Error: deleting Application Rule Collection "XXX" from Firewall "XXX (Resource Group "XXX"): network.AzureFirewallsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status= Code="AnotherOperationInProgress" Message="Another operation on this or dependent resource is in progress. To retrieve status of the operation use uri: https://management.azure.com/subscriptions/XXX" Details=[]
My working hypothesis is that one cannot make multiple create/update/delete requests of firewall rule sets against the same firewall simultaneously, even if the rule sets are mutually exclusive. Indeed, if you wait a minute-or-so after the failed deployment and restart it -- without changing any Terraform code or manually updating resources in Azure -- it will happily carry on without error and complete successfully.
To test my assumption, I tried to workaround this by forcing serialisation of modules:
module "another_firewall_spoke" {
# This creates, amongst other things, firewall rule sets
source = "/path/to/module/c"
hub = module.firewall_hub
# attribute = value...
depends_on = [module.firewall_spoke]
}
However, unfortunately, this is not possible with the way my modules are written:
Providers cannot be configured within modules using count, for_each or depends_on.
Short of rewriting my modules (not an option), is it possible to get around this race condition -- if that's the problem -- or would you consider it a bug with the azurerm provider (i.e., it should recognise that API error response and wait its turn, up to some timeout)?
(Terraform v1.1.7, azurerm v2.96.0)
Following #silent's tip-off to this answer, I was able to resolve the race using the method described therein.
Something like this:
module "firewall_hub" {
# This creates the Azure Firewall resource
source = "/path/to/module/a"
# attribute = value...
}
module "firewall_spoke" {
# This creates, amongst other things, firewall rule sets
# Has an output "blockers" containing resources that cannot be deployed concurrently
source = "/path/to/module/b"
hub = module.firewall_hub
# attribute = value...
}
module "another_firewall_spoke" {
# This creates, amongst other things, firewall rule sets
source = "/path/to/module/c"
hub = module.firewall_hub
waits_for = module.firewall_spoke.blockers
# attribute = value...
}
So the trick is for your modules to export an output that contains a list of all the dependent resources that need to be deployed first. This can then be an input to subsequent modules, that is threaded through to the actual resources that require a depends_on value.
That is, in the depths of my module, resources have:
resource "some_resource" "foo" {
# attribute = value...
depends_on = [var.waits_for]
}
There are two important notes to bear in mind when using this method:
The wait_for variable in your module must have type any; list(any) doesn't work, as Terraform interprets this as a homogeneous list (which it most likely won't be).
Weirdly, imo, the depends_on clause requires you to explicitly use a list literal (i.e., [var.waits_for] rather than just var.waits_for), even if the variable you are threading through is a list. This doesn't type check in my head, but apparently Terraform is not only fine with it, but it expects it!
New to terraform. Trying to create api gateway resources with loop. Below is the structure of API
/test
|
/testchild
POST
So, I want to create 2 resources with for loop. Below is the code snippets
ag.tf
resource "aws_api_gateway_rest_api" "demo" {
name = "demo-api"
description = "This is demo"
endpoint_configuration {
types = ["REGIONAL"]
}
}
resource "aws_api_gateway_resource" "res" {
for_each = var.resources
rest_api_id = aws_api_gateway_rest_api.demo.id
parent_id = each.value.parent ? aws_api_gateway_resource.res[each.value.parent_resource].id : var.parent_id
path_part = each.key
depends_on = [
aws_api_gateway_rest_api.demo.id
]
}
vars.tf
variable "resources" {
default = {
"test" = {
"parent" : false
},
"testchild" = {
"parent" : true,
"parent_resource" : "test"
}
}
}
Getting below error while terraform plan command.
Error: Cycle: aws_api_gateway_resource.res["testchild"],
aws_api_gateway_resource.res["test"]
Can someone guide me how to reference the id of created resource in the loop ?
TIA
Dependencies in Terraform are between whole resource blocks (in the Terraform sense), not between individual instances of those resources, because Terraform must build the dependency graph before evaluating the count or for_each expressions in order to establish which instances exist. All actions on instances of a particular resource conceptually happen concurrently, although in practice Terraform might serialize them (in no specific order) to limit the number of simultaneous actions.
Therefore in effect you've made this resource refer to itself, and hence Terraform reported a dependency cycle. The Terraform language is not a good fit for this sort of recursive data structure because Terraform requires all structures to be finite. This is one of the tradeoffs of the Terraform's declarative DSL compared to general-purpose languages.
Since the API Gateway API model is inherently a recursive data structure, unfortunately it's typically necessary to write it out as explicit resources in order to correctly represent the dependences between them, because it wouldn't be valid to create all of the aws_api_gateway_resource objects simultaneously.
Fortunately, the aws_api_gateway_rest_api resource type has an alternative method for creating all of the sub-objects of a rest API, by providing an OpenAPI definition to its body argument. In that case, creating the entire API structure is a single operation as far as Terraform is concerned, and then API Gateway internally deals with the various separate operations to represent all of those constructs.
The OpenAPI structure is also easier to work with in Terraform because it's has flat mapping from URL templates to resource definitions, rather than a recursive structure with resources nested inside other resources. That makes it practical to, where needed, dynamically construct the paths object using a Terraform for expression if you need to accept the definition of the resources and their methods in a different shape than the OpenAPI schema expects.
I have a Terraform module that manages AWS GuardDuty.
In the module, an aws_guardduty_detector resource is declared. The resource allows no specification of region, although I need to configure one of these resources for each region in a list. The region used needs to be declared by the provider, apparently(?).
Lack of module for_each seems to be part of the problem, or, at least, module for_each, if it existed, might let me declare the whole module, once for each region.
Thus, I wonder, is it possible to somehow declare a provider, for each region in a list?
Or, short of writing a shell script wrapper, or doing code generation, is there any other clean way to solve this problem that I might not have thought of?
To support similar processes I have found two approaches to this problem
Declare multiple AWS providers in the Terraform module.
Write the module to use a single provider, and then have a separate .tfvars file for each region you want to execute against.
For the first option, it can get messy having multiple AWS providers in one file. You must give each an alias and then each time you create a resource you must set the provider property on the resource so that Terraform knows which region provider to execute against. Also, if the provider for one of the regions can not initialize, maybe the region is down, then the entire script will not run, until you remove it or the region is back up.
For the second option, you can write the Terraform for what resources you need to set up and then just run the module multiple times, once for each regional .tfvars file.
prod-us-east-1.tfvars
prod-us-west-1.tfvars
prod-eu-west-2.tfvars
My preference is to use the second option as the module is simpler and less duplication. The only duplication is in the .tfvars files and should be more manageable.
EDIT: Added some sample .tfvars
prod-us-east-1.tfvars:
region = "us-east-1"
account_id = "0000000000"
tags = {
env = "prod"
}
dynamodb_read_capacity = 100
dynamodb_write_capacity = 50
prod-us-west-1.tfvars:
region = "us-west-1"
account_id = "0000000000"
tags = {
env = "prod"
}
dynamodb_read_capacity = 100
dynamodb_write_capacity = 50
We put whatever variables might need to be changed for the service or feature based on environment and/or region. For instance in a testing environment, the dynamodb capacity may be lower than in the production environment.
terraform has adjusted its authorization
in main.tf [for sql config] I now have:
resource "google_sql_database_instance" "master" {
name = "${random_id.id.hex}-master"
region = "${var.region}"
database_version = "POSTGRES_9_6"
# allow direct access from work machines
ip_configuration {
authorized_networks = "${var.authorized_networks}"
require_ssl = "${var.sql_require_ssl}"
ipv4_enabled = true
}
}
where
in variables.tf I have
variable "authorized_networks" {
description = "The networks that can connect to cloudsql"
type = "list"
default = [
{
name = "work"
value = "xxx.xxx.xx.xxx/32"
}
]
}
where xxx.xxx.xx.xxx is the ip address I would like to allow. However, I prefer not to put this in my variables.tf file, but rather in a non-source controlled .tfvars file.
for variables that have a simple value, this is easy, but it is not clear to me how to do it with the nested structure. Replacing xxx.xxx.xx.xxx by a variable [e.g. var.work_ip] leads to an error
variables may not be used here
any insights?
If you omit the default argument in your main configuration altogether, you will mark variable "authorized_networks" as a required input variable, which Terraform will then check to ensure that it is set by the caller.
If this is a root module variable, then you can provide the value for it in a .tfvars file using the following syntax:
authorized_networks = [
{
name = "work"
value = "xxx.xxx.xx.xxx/32"
}
]
If this file is being generated programmatically by some wrapping automation around Terraform, you can also write it into a .tfvars.json file and use JSON syntax, which is often easier to construct robustly in other languages:
{
"authorized_networks": [
{
"name": "work",
"value": "xxx.xxx.xx.xxx/32"
}
]
}
You can either specify this file explicitly on the command line using the -var-file option, or you can give it a name ending in .auto.tfvars or .auto.tfvars.json in the current working directory when you run Terraform and Terraform will then find and load it automatically.
A common reason to keep something out of version control is because it's a dynamic setting configured elsewhere in the broader system rather than a value fixed in version control. If that is true here, then an alternative strategy is to save that setting in a configuration data store that Terraform is able to access via data sources and then write your Terraform configuration to retrieve that setting directly from the place where it is published.
For example, if the network you are modelling here were a Google Cloud Platform subnetwork, and it has either a fixed name or one that can be derived systematically in Terraform, you could retrieve this setting using the google_compute_subnetwork data source:
data "google_compute_subnetwork" "work" {
name = "work"
}
Elsewhere in configuration, you can then use data.google_compute_subnetwork.work.ip_cidr_range to access the CIDR block definition for this network.
The major Terraform providers have a wide variety of data sources like this, including ones that retrieve specific first-class objects from the target platform and also more generic ones that access configuration stores like AWS Systems Manager Parameter Store or HashiCorp Consul. Accessing the necessary information directly or publishing it "online" in a configuration store can be helpful in a larger system to efficiently integrate subsystems.
I'm trying to create a module in Terraform that can be instantiated multiple times with different variable inputs. Within the module, how do I reference resources when their names depend on an input variable? I'm trying to do it via the bracket syntax ("${aws_ecs_task_definition[var.name].arn}") but I just guessed at that.
(Caveat: I might be going about this in completely the wrong way)
Here's my module's (simplified) main.tf file:
variable "name" {}
resource "aws_ecs_service" "${var.name}" {
name = "${var.name}_service"
cluster = ""
task_definition = "${aws_ecs_task_definition[var.name].arn}"
desired_count = 1
}
resource "aws_ecs_task_definition" "${var.name}" {
family = "ecs-family-${var.name}"
container_definitions = "${template_file[var.name].rendered}"
}
resource "template_file" "${var.name}_task" {
template = "${file("task-definition.json")}"
vars {
name = "${var.name}"
}
}
I'm getting the following error:
Error loading Terraform: Error downloading modules: module foo: Error loading .terraform/modules/af13a92c4edda294822b341862422ba5/main.tf: Error reading config for aws_ecs_service[${var.name}]: parse error: syntax error
I was fundamentally misunderstanding how modules worked.
Terraform does not support interpolation in resource names (see the relevant issues), but that doesn't matter in my case, because the resources of each instance of a module are in the instance's namespace. I was worried about resource names colliding, but the module system already handles that.
The picture below shows what is going on.
The terraform documentation does not make their use of "NAME" clear versus the "name" values that are used for the actual resources created by the infrastructure vender (like, AWS or Google Cloud).
Additionally, it isn't always "name=, but sometimes, say, "endpoint= or even "resource_group_name= or whatever.
And there are a couple of ways to generate multiple "name" values -- using count, variables, etc., or inside tfvar files and running terraform apply -var-file=foo.tfvars