I have several lambdas which either get triggered by messages from queues or through api gateway, have different storage types and so on.
Each of these components seat in their respective repos but over all and together they are part of the same architecture.
I am attempting to structure my terraform templates but one point of concern for me was the fact that some of these lambdas have shared resources, for example storage tables or s3 buckets, so I was wondering if it would be a good idea to only have a main.tf file in each lambda's repo that only creates the lambda itself and not any of its other dependencies, this way I could redeploy the lambda through ci/cd without worrying about the other components, and I would place all the other parts of the architecture that are more or less long lasting in a central repo and only run them when necessary through this repos dedicated ci/cd pipeline. I was also thinking of having a tfvar file that has all the shared resource names.
Is this a valid approach? What are the downsides? What are the alternatives?
You are on the right track. You can have a module for lambdas that accepts those common resources as variables. You can either hardcode the resources your terraform import then, and pass them to your lambda:
On a modules/lambda folder
resource "aws_lambda_function" "this" {
"""
My lambda stuff
"""
}
On your main folder main.tf
module "lambda1"{
s3_bucket = "${var.common_bucket}"
iam_role = "${var.common_role}"
subnets = "${var.private_subnets}"
....
}
module "lambda2"{
s3_bucket = "${var.common_bucket}"
iam_role = "${var.common_role}"
subnets = "${var.private_subnets}"
....
}
module "lambda3"{
s3_bucket = "${var.common_bucket}"
iam_role = "${var.common_role}"
subnets = "${var.private_subnets}"
....
}
If you want yo import the common resources, e.g iam the iam policy for the rule, just do something like this
resource "aws_iam_policy" "my_policy" {
}
Then run the command terraform import aws_iam_policy.my_policy <policy arn>
Related
I'm not sure that there is a right answer for this and it will vary per scenario but I am curious because there isn't much documentation that I can find for a code first azure bicep infrastructure. Most examples you find show how to make a resource within a resource group, or using a module to define scope and deploy to another resource group, but what if you're trying to do more?
Let's do the following scenario: using 2 subscriptions(1 for prod, 1 for dev & qa) with 20 resource groups each containing multiple difference resources and you want to manage this within a CI/CD pipeline, plus the 3 environments: prod, qa, and dev. How would you go about this? I can think of a few scenarios but don't necessarily, but nothing sticks out as the best way to do it, maybe I'm missing something.
CI/CD portion:
Let's assume:
az account set --subscription(set our sub)
az group create --name --location (create resource group if it doesn't exist)
az deployment group create --name --resource-group --template-file --parameters(read from our files to deploy to a resource group)
You could pass an array of resource groups to loop through to create the resource group if it doesn't exist.
You could have the resource group list in a parameters file that you read from and do the same thing as above.
You could create a step for every resource group and the resources inside of it.(seems excessive?)
Bicep Portion:
Bicep restrictions: to specify scope(a resource group in our scenario) we'd have to have use modules dealing with multiple resource groups or have a step for each resource group and have a main.bicep file for the different resource groups/resources.
You could create a folder structure for each resource group and the resources inside of it with a main.bicep but that would mean you have a lot of extra deploy steps(seems excessive?).
You could have 1 main.bicep file and have a folder structure that uses a lot of modules to specify your scope while reading the resource group, resource variables etc using an environment parameters.json file.
You could create a folder for each environment, have folders with each environment then create each resource group and resources inside of it not using a parameters.json but using params in each file instead since they would be specific for each environment.
1 final issue:
Lastly let's say you want to add a step before the deployment of resources to use bicep what-if to check what resources will be updated or deleted(this is pretty important!). Last I checked there was an issue where what-if does not work for bicep modules so you wouldn't get the luxury of knowing what changes would be made prior to a deployment with the what-if. That is a pretty big safety net you'd be losing, so would you want to scratch the module strategy all together?
What would be the best way to tackle something like this while keeping it readable for average non experts to be able to hop in and work on it? I would lean towards making a folder structure using modules and reading from an environment parameters.json but I'm not convinced that's the best way, especially if what-if isn't fully working for bicep modules.
IMO this does depend a lot on the scenario, topology, permissions, etc. The way I would start thinking about this is that you want an "environment" that will vary a bit between dev/test and prod. That env has multiple resourceGroups and a dedicated subscription for each env.
In this case, I would use a single bicep "project" (e.g. main.bicep with modules) and change the deployment using parameter files (for dev/test vs. prod). The project would lay down everything needed for the environment (think greenfield). The main.bicep file is a subscription scoped deployment that will create the RGs and all the resources needed. Oversimplified example:
targetScope = 'subscription'
param sqlAdminUsername string'
param keyVaultResourceGroup
param keyVaultName string
param keyVaultSecretName string
param location string = deployment().location
resource kv 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
scope: resourceGroup(subscription().subscriptionId, keyVaultResourceGroup)
name: keyVaultName
}
resource sqlResourceGroup 'Microsoft.Resources/resourceGroups#2021-04-01' = {
name: 'shared-sql'
location: location
}
resource webResourceGroup 'Microsoft.Resources/resourceGroups#2021-04-01' = {
name: 'shared-web'
location: location
}
module sqlDeployment 'modules/shared-sql.bicep' = {
scope: resourceGroup(sqlResourceGroup.name)
name: 'sqlDeployment'
params: {
sqlAdminUsername: sqlAdminUsername
sqlAdminPassword: kv.getSecret(keyVaultSecretName)
location: location
}
}
module webDeployment 'modules/shared-web.bicep' = {
scope: resourceGroup(webResourceGroup.name)
name: 'webDeployment'
params: {
location: location
}
}
A single template + modules that creates the RGs, creates a SQL Server (via module) and an app service plan with an admin website (also via module). You can then parameterize whatever you want to for each environment.
re: what-if - what if will skip evaluation of a module if that module has a parameter that is an output of another module. If you don't pass outputs between modules then the module will be evaluated by what-if. The sample above does not pass outputs - often you don't need to do this because the information output was known by the parent (i.e. main.bicep) but sometimes you can't avoid it - ymmv.
Once you have the template designed in such a way, the pipeline is really straightforward - just deploy the template to the desired subscription.
That help?
I'm building CI/CD pipeline using GitHub Actions and Terraform. I have a main.tf file like below, which I'm calling from GitHub action for multiple environments. I'm using https://github.com/hashicorp/setup-terraform to interact with Terraform in GitHub actions. I have MyService component and I'm deploying to DEV, UAT and PROD environments. I would like to reuse main.tf for all of the environments and dynamically set workspace name like so: MyService-DEV, MyService-UAT, MyService-PROD. Usage of variables is not allowed in the terraform/cloud block. I'm using HashiCorp cloud to store state.
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 2.0"
}
}
cloud {
organization = "tf-organization"
workspaces {
name = "MyService-${env.envname}" #<==not allowed to use variables
}
}
}
Update
I finally managed to get this up and running with helpful comments. Here are my findings:
TF_WORKSPACE needs to be defined upfront like: service-dev
I didn't get tags to work the way I want when running in automation. If I define a tag in cloud.workspaces.tags as 'service' then there is no way to set a second tag like 'dev' dynamically. Both of the tags are needed to during init ['service', 'dev'] in order for TF to select workspace service-dev automatically.
I ended up using tfe provider in order to set up workspaces(with tags) automatically. In the end I still needed to set TF_WORKSPACE=service-dev
It doesn't make sense to refer to terraform.workspace as part of the workspaces block inside a cloud block, because that block defines which remote workspaces Terraform will use and therefore dictates what final value terraform.workspace will have in the rest of your configuration.
To declare that your Terraform configuration belongs to more than one workspace in Terraform Cloud, you can assign each of those workspaces the tag "MyService" and then use the tags argument instead of the name argument:
cloud {
organization = "tf-organization"
workspaces {
tags = ["MyService"]
}
}
If you assign that tag to hypothetical MyService-dev and MyService-prod workspaces in Terraform Cloud and then initialize with the configuration above, Terraform will present those two workspaces for selection using the terraform workspace commands when working in this directory.
terraform.workspace will then appear as either MyService-dev or MyService-prod, depending on which one you have selected.
I'm trying to launch AWS Config and rules in each region in my account. Right now in my root main.tf I create an AWS provider in a single region and call my AWS Config module from my modules directory. This is fine for creating one module, but I would be hoping to have a regions list that I could iterate over to create AWS Config rules in each
I have tried creating individual modules with region as a parameter, but I do not know if 10+ different modules is effective. It seems using a for loop would be more effective, but I cant find any examples for this behavior.
provider "aws" {
region = "${var.aws_region}"
}
module "config" {
source = "./modules/config"
...
}
My goal is to use my config modules over all and create them in all regions. us-east-1, us-east-2, us-west1-, etc etc
I believe you're trying to dynamically pass in a list of regions to a module's provider in order to provision resources across regions in a single module. This is not possible at the moment.
Here is the ticket to upvote and follow: https://github.com/hashicorp/terraform/issues/24476
I’m using terraform to spin up Aws-DMS. To spin up DMS, we need subnet groups, dms replication task, dms endpoints, dms replication instance. I’ve configured everything using terraform documentation. My question is how will terraform know which task to be completed first to spin up other dependency tasks?
Do we need to declare it somewhere in terraform or is terraform intelligent enough to run accordingly?
Terraform uses references in the configuration to infer ordering.
Consider the following example:
resource "aws_s3_bucket" "example" {
bucket = "terraform-dependencies-example"
acl = "private"
}
resource "aws_s3_bucket_object" "example" {
bucket = aws_s3_bucket.example.bucket # reference to aws_s3_bucket.example
key = "example"
content = "example"
}
In the above example, the aws_s3_bucket_object.example resource contains an expression that refers to aws_s3_bucket.example.bucket, and so Terraform can infer that aws_s3_bucket.example must be created before aws_s3_bucket_object.example.
These implicit dependencies created by references are the primary way to create ordering in Terraform. In some rare circumstances we need to represent dependencies that cannot be inferred by expressions, and so for those exceptional circumstances only we can add additional explicit dependencies using the depends_on meta argument.
One situation where that can occur is AWS IAM policies, where the graph created naturally by references will tend to have the following shape:
Due to AWS IAM's data model, we must first create a role and then assign a policy to it as a separate step, but the objects assuming that role (in this case, an AWS Lambda function just for example) only take a reference to the role itself, not to the policy. With the dependencies created implicitly by references then, the Lambda function could potentially be created before its role has the access it needs, causing errors if the function tries to take any actions before the policy is assigned.
To address this, we can use depends_on in the aws_lambda_function resource block to force that extra dependency and thus create the correct execution order:
resource "aws_iam_role" "example" {
# ...
}
resource "aws_iam_role_policy" "example" {
# ...
}
resource "aws_lambda_function" "exmaple" {
depends_on = [aws_iam_role_policy.example]
}
For more information on resource dependencies in Terraform, see Resource Dependencies in the Terraform documentation.
Terraform will automatically create the resources in an order that all dependencies can be fulfilled.
E.g.: If you set a security group id in your DMS definition as "${aws_security_group.my_sg.id}", Terraform recognizes this dependency and created the security group prior to the DMS resource.
I am new to Terraform so not even sure something like this is possible. As an example, lets say I have a template that deploys an Azure resource group and a key vault in it. And then lets say I have another template that deploys a virtual machine into the same resource group. Is it possible to do a destroy with the virtual machine template without destroying the key vault and resource group? We are trying to compartmentalize the parts of a large solution without having to put it all in a single template and we want to be able to manage each piece separately without affecting other pieces.
On a related note...we are storing state files in an Azure storage account. If we breakup our deployment into multiple compartmentalized deployments...should each deployment have its own state file or should they all use the same state file?
For larger systems it is common to split infrastructure across multiple separate configurations and apply each of them separately. This is a separate idea from (and complimentary to) using shared modules: modules allow a number of different configurations to have their own separate "copy" of a particular set of infrastructure, while the patterns described below allow an object managed by one configuration to be passed by reference to another.
If some configurations will depend on the results of other configurations, it's necessary to store these results in some data store that can be written to by its producer and read from by its consumer. In an environment where the Terraform state is stored remotely and readable broadly, the terraform_remote_state data source is a common way to get started:
data "terraform_remote_state" "resource_group" {
# The settings here should match the "backend" settings in the
# configuration that manages the network resources.
backend = "s3"
config {
bucket = "mycompany-terraform-states"
region = "us-east-1"
key = "azure-resource-group/terraform.tfstate"
}
}
resource "azurerm_virtual_machine" "example" {
resource_group_name = "${data.terraform_remote_state.resource_group.resource_group_name}"
# ... etc ...
}
The resource_group_name attribute exported by the terraform_remote_state data source in this example assumes that a value of that name was exposed by the configuration that manages the resource group using an output.
This decouples the two configurations so that they have an entirely separate lifecycle. You first terraform apply in the configuration that creates the resource group, and then terraform apply in the configuration that contains the terraform_remote_state data resource shown above. You can then apply that latter configuration as many times as you like without risk to the shared resource group or key vault.
While the terraform_remote_state data source is quick to get started with for any organization already using remote state (which is recommended), some organizations prefer to decouple configurations further by introducing an intermediate data store like Consul, which then allows data to be passed between configurations more explicitly.
To do this, the "producing" configuration (the one that manages your resource group) publishes the necessary information about what it created into Consul at a well-known location, using the consul_key_prefix resource:
resource "consul_key_prefix" "example" {
path_prefix = "shared/resource_group/"
subkeys = {
name = "${azurerm_resource_group.example.name}"
id = "${azurerm_resource_group.example.id}"
}
resource "consul_key_prefix" "example" {
path_prefix = "shared/key_vault/"
subkeys = {
name = "${azurerm_key_vault.example.name}"
id = "${azurerm_key_vault.example.id}"
uri = "${azurerm_key_vault.example.uri}"
}
}
The separate configuration(s) that use the centrally-managed resource group and key vault would then read it using the consul_keys data source:
data "consul_keys" "example" {
key {
name = "resource_group_name"
path = "shared/resource_group/name"
}
key {
name = "key_vault_name"
path = "shared/key_vault/name"
}
key {
name = "key_vault_uri"
path = "shared/key_vault/uri"
}
}
resource "azurerm_virtual_machine" "example" {
resource_group_name = "${data.consul_keys.example.var.resource_group_name}"
# ... etc ...
}
In return for the additional complexity of running another service to store these intermediate values, the two configurations now know nothing about each other apart from the agreed-upon naming scheme for keys within Consul, which gives flexibility if, for example, in future you decide to refactor these Terraform configurations so that the key vault has its own separate configuration too. Using a generic data store like Consul also potentially makes this data available to the applications themselves, e.g. via consul-template.
Consul is just one example of a data store that happens to already be well-supported in Terraform. It's also possible to achieve similar results using any other data store that Terraform can both read and write. For example, you could even store values in TXT records in a DNS zone and use the DNS provider to read, as an "outside the box" solution that avoids running an additional service.
As usual, there is a tradeoff to be made here between simplicity (with "everything in one configuration" being the simplest possible) and flexibility (with a separate configuration store), so you'll need to evaluate which of these approaches is the best fit for your situation.
As some additional context: I've documented a pattern I used successfully for a moderate-complexity system. In that case we used a mixture of Consul and DNS to create an "environment" abstraction that allowed us to deploy the same applications separately for a staging environment, production, etc. The exact technologies used are less important than the pattern, though. That approach won't apply exactly to all other situations, but hopefully there are some ideas in there to help others think about how to best make use of Terraform in their environment.
You can destroy specific resources using terraform destroy -target path.to.resource. Docs
Different parts of a large solution can be split up into modules, these modules do not even have to be part of the same codebase and can be referenced remotely. Depending on your solution you may want to break up your deployments into modules and reference them from a "master" state file that contains everything.