Is it a bad practice to wrap your terraform root modules (dir where you run terraform plan) in another module?
for example:
dev/base
main.tf
dev/databases
main.tf
dev/base/main.tf
module "data-science" { << inside this module i would setup s3 buckets, iam roles, etc with publicly available tf modules
...
...
}
dev/databases/main.tf
module "databases" { << inside this module i would setup mysql, postgres with publicly available tf modules
...
...
}
My reason for this is that I would like to be able to create other environments with as little copying/pasting/modifying of the root modules code as much as possible, wrapping the contents of each root module in another module seems to keep this as dry as possible.
Is this a good/bad idea?
Update:
To clarify, this what what I may see this in a root module
dev/main.tf
module "ec2-instance" {
cpu = 1
mem = 1
}
module "iam-role" {
}
module "security-group" {
}
data "iam-policy {
}
You often times end up with multiple modules, data sources, etc. this could grow large.
Say I want to replicate this setup in another environment. I end up copying dev/main.tf to another environment and just searching through the main.tf and making modifications to things to make it fit the environment.
to keep it more dry and more easily deployable, would this make sense. wrapping all the modules above into a parent module.
dev/main.tf
module "my-root-module-wrapper" {
ec2_cpu = 1
ec2_cpu = 2
}
Now if I want to deploy my a new environment, i'm just copying the calling of my root module wrapper that calls all the other terraform. this seems cleaner.
Thought is that all terraform that would otherwise go into a root module instead should now go in "my-root-module-wrapper" instead. making it easier and more DRY to deploy to different environments.
Related
I've found that I can access a local coming from my root Terraform module in its children Terraform modules.
I thought that a local is scoped to the very module it's declared in.
See: https://developer.hashicorp.com/terraform/language/values/locals#using-local-values
A local value can only be accessed in expressions within the module where it was declared.
Seems like the documentation says locals shouldn't be visible outside their module. At my current level of Terraform knowledge I can't foresee what may be wrong with seeing locals of a root module in its children.
Does Terraform locals visibility scope span children (called) modules?
Why is that?
Is it intentional (by design) that a root local is visible in children modules?
Details added later:
Terraform version I use 1.1.5
My sample project:
.
├── childmodulecaller.tf
├── main.tf
└── child
└── some.tf
main.tf
locals {
a = 1
}
childmodulecaller.tf
locals {
b = 2
}
module "child" {
for_each = toset(try(local.a + local.b == 3, false) ? ["name"] : [])
source = "./child"
}
some.tf
resource "local_file" "a_file" {
filename = "${path.module}/file1"
content = "foo!"
}
Now I see that my question was based on a wrongly interpreted observation.
Not sure if it is still of any value but leaving it explained.
Perhaps it can help someone else to understand the same and avoid the confusion I experienced and explained in my corrected answer.
Each module has an entirely distinct namespace from others in the configuration.
The only way values can pass from one module to another is using input variables (from caller to callee) or output values (from callee to caller).
Local values from one module are never automatically visible in another module.
EDIT: Corrected answer
After reviewing my sample Terraform project code I see that my finding was wrong. The local a from main.tf I access in childmodulecaller.tf is actuallly accessed in a module block but still in the scope of my root module (I understand that is because childmodulecaller.tf is directly in the root module config dir). So I confused a module block in a calling parent with the child module called.
My experiments like changing child/some.tf the following way:
resource "local_file" "a_file" {
filename = "${path.module}/file1"
content = "foo!"
}
output "outa" {
value = local.a
}
output "outb" {
value = local.b
}
cause Error: Reference to undeclared local value
on terraform validate issued (similarly to what Mark B already mentioned in question comments for Terraform version 1.3.0)
So no, Terraform locals scope don't span children (called) modules.
Initial wrong answer:
I think I've understood why locals are visible in children modules.
It's because children (called) modules are included into the configuration of root (parent) module.
To call a module means to include the contents of that module into the configuration with specific values for its input variables.
https://developer.hashicorp.com/terraform/language/modules/syntax#calling-a-child-module
So yes, it's by design and not a bug. Just it may be not clear from locals documentation. As root (parent) module's locals visible in children module parts of configuration which are essentially also parts of the root (parent) modules being included into the root (parent) module.
I am trying to create dependency between multiple sub modules which should be able to create the resource individually as well as should be able to create the resource if they are dependent on each other.
basically i am trying to create multiple VMs, and based on the ip addresses and vip ip address returned as the output i want to create the lbaas pool and lbaas pool members.
i have kept the project structure as below
- Root_Folder
- main.tf // create all the vm's
- output.tf
- variable.tf
- calling_module.tf
- modules
- lbaas-pool
- lbaas-pool.tf // create lbaas pool
- variable.tf
- output.tf
- lbaas-pool-members
- lbaas-pool-members.tf // create lbaas pool member
- variable.tf
- output.tf
calling_module.tf contains the reference to the lbaas-pool module and lbaas-pool-members, as these 2 modules are dependent on the output of the resource generated by main.tf file.
It is giving below error:
A managed resource has not been declared.
As the resource has not been generated yet, and while running terraform plan and apply command is trying to load the resource object which has not been created. Not sure with his structure declare the module implicit dependency between the resources so the module can work individually as well as when required the complete stack.
Expected behaviour:
main.tf output parameters should be create the dependency automatically in the terraform version 0.14 but seems like that is not the case from the above error.
Let's say you have a module that takes an instance ID as an input, so in modules/lbaas-pool you have this inside variable.tf
variable "instance_id" {
type = string
}
Now let's say you define that instance resource in main.tf:
resource "aws_instance" "my_instance" {
...
}
Then to pass that resource to any modules defined in calling_module.tf (or in any other .tf file in the same folder as main.tf), you would do so like this:
module "lbaas-pool" {
src="modules/lbaas-pool"
instance_id = aws_instance.my_instance.id
...
}
Notice how there is no output defined at all here. Any output at the root level is for exposing outputs to the command line console, not for sending things to child modules.
Also notice how there is no data source defined here. You are not writing a script that will run in a specific order, you are writing templates that tell Terraform what you want your final infrastructure to look like. Terraform reads all that, creates a dependency graph, and then deploys everything in the order it determines. At the time of running terraform plan or apply anything you reference via a data source has to already exist. Terraform doesn't create everything in the root module, then load the submodule and create everything there, it creates things in whatever order is necessary based on the dependency graph.
I'm trying to create a mono repository to host multiple small packages that I plan to deploy on npm. I use Rollup for the bundling part.
After many hours watching what others do, searching on the internet and experimenting, I've reached a point where I'm stuck and little push in the right direction would be very much appreciated.
I've created a minimalist demo project that I've hosted on GitHub so it's easy to experiment with. You can find it here:
https://github.com/Stnaire/my-lib
In the repository you'll find 3 packages in the packages directory:
config: contains a SharedConfiguration service
container: a wrapper around Inversify to have a statically accessible container with helper methods
storage: a package containing a StorageService (normally for writing in the local storage, cookies, etc) and a VarHolder helper which is just a memory storage.
In each package there is a package.json (defining the npm package parameters) and a tsconfig.json for the build.
What I'm trying to do is the have a npm package for each of the packages, each allowing for the following types of usages:
With a bundler in a TypeScript environment
import { SharedConfiguration } from '#my-lib/config';
// or ideally:
import { SharedConfiguration } from '#my-lib/config/shared-configuration';
// this way only this dependency is included, not the whole `config` pacakge
With a bundler in a JavaScript environment
var SharedConfiguration = require('#my-lib/config');
// Or like above, ideally:
var SharedConfiguration = require('#my-lib/config/shared-configuration');
By including the output JS file in the browser
<script src="my-lib-config.umd.js"></script>
<script>
MyLibConfig.SharedConfiguration.get(...);
</script>
What I tried
I've created two branches in the demo repository, corresponding to two strategies.
First strategy: create aliases (branch detailed-modules-resolution)
In the tsconfig.json, I do:
{
"paths": {
"#my-lib/config/*": ["packages/config/src/*"],
"#my-lib/container/*": ["packages/container/src/*"],
"#my-lib/storage/*": ["packages/storage/src/*"]
}
}
This way I can import precisely what I need:
import { SharedConfiguration } from '#my-lib/config/shared-configuration';
And because the aliases also correspond to the folder structure in node_modules, it should work for the end user with a bundler as well.
But this way I get warnings when building as UMD:
No name was provided for external module '#my-lib/config/shared-configuration' in output.globals – guessing 'sharedConfiguration'
No name was provided for external module '#my-lib/container/container' in output.globals – guessing 'container'
Creating a global alias for each import is out of question, thus the second strategy.
Second strategy: centralize all public exports (branch centralized-imports)
So the idea is simply to export everthing the package wants to expose to other packages in the index.ts:
// packages/config/index.ts
export * from './shared-configuration';
Then in the build script, I do this:
// scripts/build/builds.js
/**
* Define the mapping of externals.
*/
export const Globals = {
'inversify': 'Inversify'
};
for (const build of Object.keys(Builds)) {
Globals[`#my-lib/${Builds[build].package}`] = Builds[build].moduleName;
}
Wihch creates the following object:
{
'inversify': 'Inversify',
'#my-lib/config': 'MyLibConfig',
'#my-lib/container': 'MyLibContainer',
'#my-lib/storage': 'MyLibStorage',
}
This way umd builds are working.
But there is two main drawbacks:
You have very little control on what you import. You import a whole package or nothing. And if the package depends on other packages you can quickly import thousands of lines for a little service.
This creates circular dependencies, as in my example project.
SharedConfiguration uses VarHolder which is in the #my-lib/storage. But this package also contain a StorageService which uses SharedConfiguration, creating a circular dependency
because all the imports are based on the index.ts: #my-lib/config => #my-lib/storage => #my-lib/config.
I thought about using one strategy or the other depending if I build in umd or not, but it feels wrong.
It must be simplier way to handle all of this.
Thank you very much for reading all this.
I'm struggling with Terragrunt (I'm still quite new).
I can describe my problem even using pure Terragrunt repo examples:
Looking here(https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/tree/master/prod/us-east-1/prod/webserver-cluster) we can see terragrunt.hcl that imports a module asg-elb-service taken from particular URL (also terragrunt example)
Now my point is that everything is fine untill module solves all my needs. But using mentioned example let's say that I want to add something on top of this module (e.g listener rule for ALB or anything) - then I would like to rely on module outputs and as we can check "used" module exposes those: outputs (https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example/blob/master/asg-elb-service/outputs.tf)
But how even if I add tf file inside my structure - continuing my example, it would be something like:
I'm just not able to anyhow "interpolate" and get access to those outputs from module :(
terragrunt is a thin wrapper that just provides some extra tools for configuration. terragrunt is used to make management of multiple terraform modules easier, it takes care about remote state and so on. But it does not extend terraform modules by adding some functionality on top of it.
Coming back to your example, common approach is to create a new terraform module, probably on top of the existing one and add missing functionality there. You should consider terraform module as a function that does particular job on a certain level of abstraction. With that said, it's completely valid to create modules that use another modules. Consider following example: you need to provision an infrastructure that can send Slack notifications if AWS CloudWatch alarm is triggered. To simplify it a little bit, let's imagine, that Alarm is already created. The missing part is a Lambda function that will send notification, SNS topic that will trigger Lambda function.
This is something, that can be created using terraform module, but under the hood it will most probably rely on another terraform modules (one that provisions Lambda and another one that provisions SNS topic). Those "internal" modules are on another level of abstraction and you still can reuse them in other cases individually. Pseudo code might look like this:
module "sns_topic" {
source = "git::https://github.com/..."
name = "trigger_lambda_to_send_notification_to_slack"
}
module "labmda_function" {
source = "git::https://github.com/..."
name = "SendMessageToSlack"
...
}
# Invoke Lambda by SNS
resource "aws_sns_topic_subscription" "sns_subscriptions" {
endpoint = module.labmda_function.lambda_endpoint # this is how you reference module output
protocol = "lambda"
topic_arn = module.sns_topic.sns_topic_arn
}
And then, you can simply use this module in terragrunt.
Terraform registry AWS VPC example terraform-aws-vpc/examples/complete-vpc/main.tf has the code below which seems to me a circular dependency.
data "aws_security_group" "default" {
name = "default"
vpc_id = module.vpc.vpc_id
}
module "vpc" {
source = "../../"
name = "complete-example"
...
# VPC endpoint for SSM
enable_ssm_endpoint = true
ssm_endpoint_private_dns_enabled = true
ssm_endpoint_security_group_ids = [data.aws_security_group.default.id] # <-----
...
data.aws_security_group.default refers to "module.vpc.vpc_id" and module.vpc refers to "data.aws_security_group.default.id".
Please explain why this does not cause an error and how come module.vpc can refer to data.aws_security_group.default.id?
In the Terraform language, a module creates a separate namespace but it is not a node in the dependency graph. Instead, each of the module's Input Variables and Output Values are separate nodes in the dependency graph.
For that reason, this configuration contains the following dependencies:
The data.aws_security_group.default resource depends on module.vpc.vpc_id, which is specifically the output "vpc_id" block in that module, not the module as a whole.
The vpc module's variable "ssm_endpoint_security_group_ids" variable depends on the data.aws_security_group.default resource.
We can't see the inside of the vpc module in your question here, but the above is okay as long as there is no dependency connection between output "vpc_id" and variable "ssm_endpoint_security_group_ids" inside the module.
I'm assuming that such a connection does not exist, and so the evaluation order of objects here would be something like this:
aws_vpc.example in module.vpc is created (I just made up a name for this because it's not included in your question)
The output "vpc_id" in module.vpc is evaluated, referring to module.vpc.aws_vpc.example, and producing module.vpc.vpc_id.
data.aws_security_group.default in the root module is read, using the value of module.vpc.vpc_id.
The variable "ssm_endpoint_security_group_ids" for module.vpc is evaluated, referring to data.aws_security_group.default.
aws_vpc_endpoint.example in module.vpc is created, including a reference to var.ssm_endpoint_security_group_ids.
Notice that in all of the above I'm talking about objects in modules, not modules themselves. The modules serve only to create separate namespaces for objects, and then the separate objects themselves (which includes individual variable and output blocks) are what participate in the dependency graph.
Normally this design detail isn't visible: Terraform normally just uses it to potentially optimize concurrency by beginning work on part of a module before the whole module is ready to process. In some interesting cases like this though, you can also intentionally exploit this design so that an operation for the calling module can be explicitly sandwiched between two operations for the child module.
Another reason why we might make use of this capability is when two modules naturally depend on one another, such as in an experimental module I built that hides some of the tricky details of setting up VPC peering connections:
locals {
vpc_nets = {
us-west-2 = module.vpc_usw2
us-east-1 = module.vpc_use1
}
}
module "peering_usw2" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-east-1 = module.peering_use1.outgoing_connection_ids
}
providers = {
aws = aws.usw2
}
}
module "peering_use1" {
source = "../../modules/peering-mesh"
region_vpc_networks = local.vpc_nets
other_region_connections = {
us-west-2 = module.peering_usw2.outgoing_connection_ids
}
providers = {
aws = aws.use1
}
}
(the above is just a relevant snippet from an example in the module repository.)
In the above case, the peering-mesh module is carefully designed to allow this mutual referencing, internally deciding for each pair of regional VPCs which one will be the peering initiator and which one will be the peering accepter. The outgoing_connection_ids output refers only to the aws_vpc_peering_connection resource and the aws_vpc_peering_connection_accepter refers only to var.other_region_connections, and so the result is a bunch of concurrent operations to create aws_vpc_peering_connection resources, followed by a bunch of concurrent operations to create aws_vpc_peering_connection_accepter resources.