We are working on creating various terraform modules for Azure cloud in our organization. I have a basic doubt on using these modules.
Lets say we have a module created for creating resource groups. When we write a module for storage container, Would it be better to use the resource group module inside the storage module itself or would it be better to let the user terraform script handle it specifying multiple module resource. Eg,
module resourcegroup {
…
}
module storage {
}
Thanks,
Hound
What you're considering here is a design tradeoff rather than a question with a universal answer. With that said, the Terraform documentation section Module Composition recommends that you use only one level of module nesting where possible, and then have the root module connect the outputs from one module into the inputs of another.
One situation where you might decide to go against that advice and create multiple levels of nesting is when you want to write a module which intentionally constrains or raises the level of abstraction of another module written by someone else. Modules shared on Terraform Registry are often very general in order to serve various different use-cases, but those modules might also encapsulate some design best-practices for the system in question and so you might choose to wrap one or more of those general modules in a more specific module that more directly meets your use-case, and hopefully in turn make your "wrapper module" easier to use.
However, it's always important to keep in mind that although Terraform modules can in some sense encapsulate complexity, in the case of Terraform they can't truly hide that complexity the way we might expect for libraries in general-purpose languages, because the maintainer of the root module is ultimately responsible for understanding the full consequences of applying a plan, which involves reviewing all of the proposed changes even to resources encapsulated in nested modules.
Related
I'm using Terraform to deploy Azure resources and now want to deploy across multiple regions.
I'm finding even with Modules I'm repeating code, once for each region.
How should I be writing code for multi region? I can't find any best practices
You could create a list variable and put your regions inside.
Then you could create a for loop and create the resource for each region. This approach works only when you really want to have each resource in each region.
It really depends on your resources. Some resources are reasonably maintained as multi-region within a single module, but this is rare. This would be a case where a module specifically addresses resources in multiple regions, with some kind of unifying logic for those resources. Since regions are typically very independent by design, this is typically an anti-pattern.
Often, it is more sane to use an infrastructure module (or root module, which means the same thing) per region. Some methodologies would have you use a different directory for each region, and again per environment. Yes, you're repeating yourself, but not that much. Your root modules should usually be pretty small and opinionated, serving as a hub for modules and top-level resources to be called.
Yes, you should keep your code DRY, but don't get carried away with it. Some duplication for the sake of organizing resources is totally acceptable.
In the cases where this is truly a problem (large root modules, and/or many regions across many environments), there are tools that can handle this effectively for you. Terragrunt is a fairly effective one, and can template your root modules (including their backend configuration) via a single code location, which is then callable via fairly small files. This can help to deduplicate a codebase like the one I just described.
You may also design your infrastructure modules to be re-usable by defining variables for regional and environmental variances between deployments. Backend configuration is also configurable during Terraform runtime via CLI or environment variable settings. Between these two, you can create infrastructure modules that are capable of being applied in arbitrary environments and regions. I like this better than Terragrunt's approach, because it's much simpler.
How you call these re-usable modules is up to your orchestration implementation, be that a CI/CD system, Kubernetes, Terraform Enterprise/Cloud, whatever.
Hopefully that helps you to make a decision.
I have 3 environments to manage via Terraform: dev, staging, prod.
An example of use case is below:
create a "common" service account for each environment (sa-xxx#dev + sa-xxx#staging + sa-xxx#prod)
create a "dev-specific" role for this sa-xxx#dev SA
create a "staging-specific" role for this sa-xxx#staging SA
create a "prod-specific" role for this sa-xxx#prod SA
How can I easily manage common & specific resources for each environment?
Terraform is very simple if all environments are equals, but for specificities it looks more complicated. The goal is have a structural way to manage it, and then to avoid:
code duplication in 3 distinct folders
"count" conditions in each tf resource definition
It should be possible for Terraform to look into current root folder UNION dev/staging/prod folder (depending on the environment).
The need is very simple but implementation seems so difficult.
Thanks for help ! :)
This is a pretty broad question and so it's hard to answer specifically, but one general answer to this question is to make use of shared modules as a means for sharing code between your separate configurations.
The Module Composition guide describes some different patterns that might help you in your goal. The idea would be to make each of your configurations share modules wherever it makes sense for them to do so but to also potentially use different modules -- or the same modules but with different relationships/cardinalities -- so that your configurations can represent both what is the same and what is different between each of them.
One way would be to put shared resources in a common configuration managed in a remote state. Then in other configurations, you can refer to the shared, remote state using terraform_remote_state data source.
Global class?
I'd like to use a module in 2 other modules. And use defaults.
Then update the module after application is initialized (and connected to DB).
How this can be achieved?
Example use case:
Logger module is started with default configuration. It will fetch custom one from the database after database is connected.
Database module is using same logger (using default configuration until it gets configuration from that same database).
In many other languages I could create a class, then use instances of it and finally update class (not the instance) with new configuration. Updated values will be shared across the instances.
Some ideas that came into my mind:
Maybe I am thinking about it wrong way?
Can I use global variables?
I can use a local shared resource (file for example) to trigger change after startup is completed and connections are established/configuration fetched.
Another problem: How to avoid strong coupling between the modules?
Maybe I am thinking about it wrong way?
Right or wrong isn't really black and white here. It's more about benefits of modularity.
Can I use global variables?
You can, but you probably shouldn't.
Modularity in nodejs offers all sorts of benefits. Using a global variable creates a global environment dependency that breaks some of the fundamental tenets of modularity.
Instead, it generally makes more sense to create a single module that encapsulates the shared instance that you wish to use. When that module is initialized, it creates the shared instance and stores it locally in its own module level variable. Then, when other modules require() or import this module, it exports that shared instance. In this way you both retain the modularity and all the benefits of it and you get a common, shared instance that everyone who wants to can use.
The only downside? One line of code is required in any module that wants to use the shared resource to import that shared resource. That one line of code helps you retain all the benefits of modularity while still getting access to a shared resource.
I can use a local shared resource (file for example) to trigger change after startup is completed and connections are established/configuration fetched.
It isn't clear what you mean by this. Any modular, shared resource (without
globals) as described above can capture a configuration and preserve that configuration change.
How to avoid strong coupling between the modules?
This is indeed one of the reasons to avoid globals as it creates strong coupling. Any module that exports or shares (in any way) some shared resource creates some level of coupling. The code using the shared resource has to know what the interface is to the shared resource and that cannot be avoided in order to use it. You can often take advantage of existing interfaces (like eventEmitters) in order to avoid reinventing a lot of new interface, but the caller still needs to know what common interface is being used and how.
We have just started a new project in our company which would help dev teams and operators to be able to provision cloud infrastructure as self-service. We plan to go with terraform and publishing modules based on business requirements, enterprise and security compliance (naming convention, forbidden values, etc).
Our new Cloud architect suggest we create the following structure.
Resources Modules
Wrappers around single terraform resource. 1:1 mapping
Eg: resource azurerm_vnet would have a module that wrap all the input and output variables and the resource.
Core Modules
Small modules which use resources modules.
At this level we would setup some compliance requirements, enforcing values, etc.
Eg. Module which create a storage account but some values are enforced (https only, no anonymous, etc), setup diagnostic settings with defaults, allowed locations etc
Modules
More complex infrastructure that would use core modules.
First, Core Modules and Modules for me would make sense since they contains a set of requirements/resources/business cases.
But I think having the Resources modules is a bad idea.
The architect reasons were:
Avoid duplication of terraform default resources
Since all other modules would use the resource if you need to block a property, refactor or enforce policy it would be at a single place.
This is the way he did in previous jobs
He talked with someone at Hashicorp which approved that structure.
Why I dislike the idea:
it doesn't avoid duplication of code, instead of duplicating a terraform resource you are duplicating modules and you get useless information in the state.
Hard to maintain, we are a small team, we would not be able to keep up with all the changes coming from terraform and providers.
It doesn't help for refactoring
I think policies/compliance should be enforced at another level. Core/Modules is a good start but we could leverage tools like terraform-compliance, Azure Policy, etc.
Terraform suggest otherwise
We had a debate on this and the discussion ended with me giving up because I felt I hadn't enough experience to challenge further.
What would be your take on this?
From Puppet Best Practices:
The Puppet Labs documentation describes modules as self-contained bundles of code and data.
Ok it's clear.
A single module can easily manage a single application.
So, puppetlabs-apache manages Apache only, puppetlabs-mysql manages MySQL only.
.... So, my module my_company-mediawiki manages Mediawiki only (i suppose... with database and virtual host... because a module is self-contained bundles of code and data).
Modules are most effective when the serve a single purpose, limit dependencies, and concern themselves only with managing system state relating to their named purpose.
But my_company-mediawiki needs to depend by:
puppetlabs-mysql: to create database;
puppetlabs-apache: to manage a virtual host.
And... from a rapid search I understand that many modules refer to other modules.
But...
They provide complete functionality without creating dependencies on any other modules, and can be combined as needed to build different application stacks.
Ok, a good module is self-contained and has no dependencies.
So I have to necessarily use the pattern roles and profiles to accomplish these best practices? Or I'm confused...
The Puppet documentation's description of modules as self-contained is more aspirational than definitive. Don't read too much into it, or into others' echoes of it. Modules are quite simply Puppet's next level of code organization above classes and defined types, incorporating also plug-ins and owned data.
Plenty of low-level modules indeed have no cross-module dependencies, but such dependencies inescapably arise when you start forming aggregations at a level between that and whole node configurations. There is nothing inherently wrong with that. The Roles & Profiles pattern is a good way to structure such aggregations, but it is not the only way, and in any case it does not avoid cross-module dependencies because role and profile classes, like any other, should themselves belong to modules.