Organising folders in terraform - terraform

I have a terraform setup in which i am creating resources in aws, i am using s3, ec2 and also kubernetes. For kubernetes i have more than 5 .tf files. I have created a folder called kube-aws and placed the .tf files there. Right now i have a setup like below
scripts/
|
s3.tf
ec2.tf
kube-aws/
|
web-deploy.tf
web-service.tf
app-deploy.tf
app-service.tf
Is this a right approach, will terraform pick the .tf files from kubw-aws folder as well? or should i do anything else to make this work.

The resources in kube-aws directory will not be included when scripts is your working directory. The scripts directory is considered the root module in this instance (see Modules documentation):
The .tf files in your working directory when you run terraform plan or terraform apply together form the root module.
You have two options to include kube-aws resources:
Move them up to the scripts directory.
Create a module block in one of the scripts/*.tf files and pass in required variables.
For example, in, say, s3.tf:
module "kube_aws" {
source = "./kube-aws"
// pass in your variables here
}
The choice you make is entirely up to you but the guidance in when to write a module is pretty persuasive:
We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead.
I would recommend option 1 above, i.e. move your .tf files into a single directory, at least until you clear about when to use a module and how to best structure them. I would also highly recommend getting acquainted with the official (and excellent) documentation on Modules and Module Composition, as well as looking at example modules in Terraform Registry and their associated source code (links to source can be found on module pages).

Related

Optimize retrieval of multiple git-sourced terraform modules from the same repo

We have terraform code in a git repo that references custom modules in another private repo:
module "myModule" {
source = "git::https://mygiturl//some/module"
bla bla bla...
}
When we reference multiple modules that live in the same git repo, terraform init will go and clone the same git repo repeatedly for every module reference. In the end, it takes minutes to do something that would take seconds if the same repo were not cloned repeatedly into different folders.
What options do we have for optimizing the module retrieval for speed?
The terraform init command does include an optimization where it tries to recognize if a module has a module package address that matches a module that was already installed, and if so it will try to copy the existing content already cached on local disk rather than retrieving the content over the network a second time.
In order for that to work though, all of the modules must have the same package address. The "package address" is the part of the address which tells Terraform what "package" (repository, archive) it should download, as opposed to which directory inside that package it should look in to find the module's .tf files.
If you are specifying particular subdirectories inside a single repository then you are presumably already using the Modules in Package Sub-directories syntax where the package name is separated from the subdirectory path using a pair of slashes //, giving a source address like this:
module "example" {
source = "git::https://example.com/foo/bar.git//path/to/directory"
}
In the above, the package address is git::https://example.com/foo/bar.git and the subdirectory path is path/to/directory. It's the package address portion that needs to match across multiple module calls in order for Terraform to detect this opportunity for optimization.
Another option, if your goal is to have everything in a single repository anyway, is to use only relative paths starting with ../ and ./ in your module source addresses.
When you specify a local path, Terraform understands it as referring to another directory within the same module package as the caller, and so Terraform doesn't need to download anything else or create any local copies in order to create a unique directory for that call.
This approach does assume that you want to have everything in a single repository. If you have a hybrid approach where some modules are isolated into separate repositories but others are kept together in a large repository then that is a design pattern that Terraform's module installer is not designed to support well.
If the installer optimization isn't sufficient and you cannot use a single repository for everything then the only remaining option would be to split your modules across multiple smaller packages. A Git repository is one example of a "package", but you can also potentially add a level of indirection by adding a CI process to your repository which packages up the modules into separate packages and publishes those packages somewhere else that Terraform can install from, such as .zip files in an Amazon S3 bucket.
Terraform does not offer a way to share the same local directory between multiple module packages because modules are sometimes written in a way that causes them to modify their own source directory during execution (not a recommended pattern, but still possible) and in that case the module is likely to misbehave if multiple instances of it are trying to work in the same directory.

need clarity on advantages of terragrunt instead of tfvars file

I'm new to terragrunt and terraform. Right now i'm little confused about why to use terragrunt instead of tfvars file.
My scenario is for one of the application we create multiple resources like webapps, azure sql, storage accounts. This application has multiple environments like dev, qa, prod and also it is deployed in multiple regions like weu, eus, wus2. I tried to parameterise this with tfvars files.
My directory structure will look like:
app1/storageaccount.tf
/webapp.tf
/sqldb.tf
/main.tf
/variables.tf
/dev.tfvars
/prod-weu.tfvars
/dev-weu.tfvars
/qa-eus.tfvars
.......
I use tfvars file to pass environment, location variables etc. I can even keep state file names, subscription names as variables. So here i'm trying to understand how terragrunt will be more helpful instead of tfvars.
I can have only 1 tfvar file for each environment/region instead of multiple hcl files which are created for terragrunt.
Could you please shed some light here?
thanks,
Santosh

Using variables when specifying location of the module in Terraform

I am trying to run this code:
locals {
terraform_modules_git = "git::ssh://....#vs-ssh.visualstudio.com/v3/...../terraform-modules"
terraform_modules_module = "resource_group?ref=v15.0.0"
}
module "MyModuleCall" {
source = "${local.terraform_modules_git}/${local.terraform_modules_module}"
}
My goal was to consolidate all Tag references in one place and not duplicate long string with the name of the repo with all my modules numerous times.
And I get this error:
Error: Variables not allowed
on main.tf line 12, in module "MyModuleCall":
12: source = "${local.terraform_modules_git}/${local.terraform_modules_module}"
Variables may not be used here.
Does anybody know why they have put this limitation? What is wrong with using variables?
Does anybody see any work around?
You can't dynamically generate source. You must explicitly hardcode it, as explained in the docs:
This value must be a literal string with no template sequences; arbitrary expressions are not allowed.
Sadly, I'm not aware of any workaround, except pre-processing templates before using them. The pre-processing would just find and replace the source with what you want.
Dependencies in Terraform are handled statically before executing the program, because the runtime needs to have access to all of the involved code (in Terraform's case, both modules and providers) before it can create a runtime context in order to execute any code. This is similar to most other programming languages, where you'd typically install dependencies using a separate command like pip install or npm install or go get before you can run the main program. In Terraform's case, the dependency installer is terraform init, and "running the program" means running terraform plan or terraform apply.
For this reason, Terraform cannot and does not permit dynamically-constructed module or provider source addresses. If you need to abstract the physical location and access method of your modules from the address specified in calling modules then one option is to use a module registry to tell Terraform how to map a local address like yourcompany.example.com/yourteam/resource-group/azure to the more complex Git URL that Terraform will actually use to fetch it.
However, in practice most teams prefer to specify their Git URLs directly because it results in a simpler overall system, albeit at the expense of it being harder to move your modules to a new location at a later date. A compromise between these two is to use a hosted service which provides Terraform Registry services, such as Terraform Cloud, but of course that comes at the expense of introducing another possible point of failure into your overall system.

How can I use terraform to setup in multi regions?

I have a terraform configurations file without modules and it is hosted in production .
Currently I am trying to deploy it in another region using provider alias and modules .But when I plan the same , it says that I need to destroy my previous configuration and recreate it via modules
When I modularise the files are tehy treated by terraform as a new resource ?I have around 35 reources , it says 35 to destroy and 35 to create .In the process of modularising I removed the .terraform folder under the root and initialised it in the main module
Is this the expected behaviour?
Yes, it is the expected behaviour because the resources have now different Terraform identifiers. When Terraform runs plan or apply, it looks in the status file and cannot find the new identifiers listed there.
You can solve this situation using the terraform state mv command. I have used it successfully, but it is a tedious task and mistakes are easy. I recommend to make additional backups of your state file via -backup-out option and checking the plan after each run.
I would suggest that you look into terragrunt to make your code more dry. Once you've integrated with Terragrunt you can start using a hierarchy like this:
--root
terragrunt.hcl
|
--dev_account
account.hcl
|
--us-east-2
region.hcl
|
--services
--us-east-1
--us-west-2
--uat_account
|
--us-east-2
|
--services
--us-east-1
--us-west-2
--prod_account
|
--us-east-2
|
--services
--us-east-1
--us-west-2
Terragrunt will allow you to have one module and with a hierarchy similar to this, you'll be able to parse the account.hcl, and region.hcls to deploy to different regions without having to re-enter this information.
Terragrunt allows you to run an apply-all if you choose to apply all child terraform configurations, though I wouldn't suggest this for production.

Terraform: possible to use an enterprise github repo with sub-directories as module source?

Is there a way to use an enterprise Github repo with sub-directories as a Terraform module source? Reading the Module source doc, I'm attempting to use the Generic Git repository type, however, it does not appear as though its possible to access sub-directories using this method. E.g.,
module "awesome" {
source = "git::https://github.enterprise.com/org/repo.git//path//to//module"
# Note: I've also tried without the "forced source type."
}
Unfortunately terraform get downloads the entire repository contents.
I can't say that I would expect this to work (given that its not explicitly supported) but perhaps someone out there has figured it out. If not, perhaps it could be a feature request.

Resources