How do I manage remote state for different environments? I originally wanted to use variables in my remote state definations but realized I cannot use variables like:
provider "aws" {
region = "ap-southeast-1"
}
terraform {
backend "s3" {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
data "terraform_remote_state" "s3_state" {
backend = "s3"
config {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
But realised I cannot use variables in this case? I can hardcode the bucket name but the bucket may not be the same across environments
You will want to use what Terraform calls workspaces. Here is the documentation: https://www.terraform.io/docs/state/workspaces.html
So way you have a piece of state called: MyStateKey
When you use workspaces it will append the workspace name to the end of the existing key. For example if you created a workspace called "dev" then the key in the remote state would be "MyStateKey:dev".
I would suggest you use some conventions to make it easier like using the "default" workspace as production, with additional workspaces named after your other environments. Then when you run terraform you can set the workspace or use the TF_WORKSPACE environment variable to set it.
Related
I'm running terraform v0.14.8 with a non-default terraform workspace.
I have an Atlantis server that handles my plans and applies.
When I clone my repo locally and run my plans I get errors about my datasource. I don't quite understand why as I don't get these errors on my Atlantis server which I believe performs the same operations. The Atlantis server also uses tf v0.14.8.
My terraform:
data "terraform_remote_state" "route53" {
backend = "s3"
config = {
key = "web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
Before I run my local plan, i switch the workspace
terraform workspace select web
# in addition I also tried
export TF_WORKSPACE=web
My plan
teraform plan
...
Error: Unable to find remote state
on provider.tf line 46, in data "terraform_remote_state" "route53":
46: data "terraform_remote_state" "route53" {
No stored state was found for the given workspace in the given backend.
I could easily edit my "key" with env: and things will work, but I'm trying to figure out how to do this without making that adjustment, seeing that my Atlantis server just works.
data "terraform_remote_state" "route53" {
backend = "s3"
config = {
key = "env:/web/web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
Your question seems to imply some confusion over which of these backends the web workspace is selected for. Running terraform workspace select web selects the web workspace from the backend of the current configuration (the directory where you are running Terraform), but I suspect your intent is to select the web backend from the backend you've configured in data "terraform_remote_state" instead.
If so, you can do that by setting the workspace argument in the data source configuration:
data "terraform_remote_state" "route53" {
backend = "s3"
workspace = "web"
config = {
key = "web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
}
I'm using Terraform cloud. I would like to take advantage of using AWS Tags with my resources. I want to tag each resource defined in Terraform with the current GIT Branch Name. That way I can separate dev from production.
Terraform has a list of Environmental Variables that do reference the GIT Branch Name with their service in the cloud as:
TFC_CONFIGURATION_VERSION_GIT_BRANCH - This is the name of the branch that the associated Terraform configuration version was ingressed from (e.g. master).
How can I reference the TFC_CONFIGURATION_VERSION_GIT_BRANCH environmental variable in the following resource for an example VPC?
resource "aws_vpc" "example_vpc" {
cidr_block = "10.0.0.0/16"
tags = {
product = var.product
stage = var.TFC_CONFIGURATION_VERSION_GIT_BRANCH
}
}
reference: https://www.terraform.io/docs/language/values/variables.html#environment-variables
I figured it out! Wish the documentation was clearer on this with the cloud.
You will have to set a empty variable. I defined mine in variables.tf as:
variable "TFC_CONFIGURATION_VERSION_GIT_BRANCH" {
type = string
default = ""
}
Per the documentation I linked in question. TFC_CONFIGURATION_VERSION_GIT_BRANCH is injected automatically into the environmental variables with each cloud run. Defining the full name of the environmental variable as the variable worked.
resource "aws_vpc" "example_vpc" {
cidr_block = "10.0.0.0/16"
tags = {
product = var.product
stage = var.TFC_CONFIGURATION_VERSION_GIT_BRANCH
}
}
Then the plan output was successful in the cloud:
Terraform will perform the following actions:
# aws_vpc.example_vpc will be updated in-place
~ resource "aws_vpc" "example_vpc" {
id = "vpc-0b19679e6464b8481"
~ tags = {
~ "stage" = "None" -> "develop"
# (1 unchanged element hidden)
}
# (14 unchanged attributes hidden)
}
Terraform Cloud serves as a remote execution environment for Terraform CLI (amongst other things) and so when you configure a workspace in Terraform Cloud many of the settings are about the context where Terraform CLI will run, and how Terraform Cloud will run it.
Part of that configuration model is the idea of environment variables, which correspond with the same environment variables you might set in your shell when running Terraform CLI locally. As with local Terraform, those environment variables are not directly usable from your configuration but are instead settings for other systems that Terraform and providers will interact with, such as the AWS_ACCESS_KEY_ID environment variable conventionally used by AWS software as a way to statically configure the access key identifier to use.
Terraform Cloud also allows you to set "Terraform Variables", and those correspond with Input Variables in the Terraform language. These are the settings for your Terraform configuration itself, as opposed to other software it will interact with, and so you can refer to these by declaring them using variable blocks and then using expressions like var.example elsewhere in the root module. Internally, Terraform Cloud is passing the configured values to Terraform CLI by generating a file called terraform.tfvars, which Terraform CLI looks for as a default source of variable values.
Both of these types of variables are useful for different purposes, and so most Terraform workspaces include a mixture of environment variables for configuring external systems and "Terraform variables" for configuring the current Terraform configuration itself.
For the benefit of folks using Terraform CLI outside of Terraform Cloud, Terraform CLI actually also offers a way to set Input Variables using environment variables, and technically you can do that within Terraform Cloud too because it's ultimately just running Terraform with environment variables set in the same way as you might locally. That's not the intended way to use Terraform Cloud, and so I'm mentioning it only for completeness because the terminology overlap here might be confusing.
Terraform cloud workspace variables can be set as category "terraform" or as category "env". In a remote execution setup, you are unable to reference workspace vars defined as "env" from your terraform code. Instead, those vars will be automatically injected into the execution environment. To be able to reference a workspace variable in terraform code, set the variable type to "terraform" but do not check the "HCL" tick. Defining a blank variable is still needed. Hope this helps someone! Unfortunately, the documentation is very unclear about this.
Terraform Cloud: Create a workspace variable
key: example_variable
value: example_value
category: terraform
Implementation: Configuration (.tf) file
Declare a blank variable
Reference your variable
# Declaring a blank variable
variable "example_variable" {}
# Referencing a variable
block "name" {
input = var.example_variable
Snowflake CI/CD Example
CI/CD pipeline for Snowflake with GitHub Actions and Terraform configured similar to Snowflake's quick start guide
Implemented a private key instead of a user password
variable "snowflake_private_key" {}
terraform {
required_providers {
snowflake = {
source = "chanzuckerberg/snowflake"
version = "0.25.17"
}
}
backend "remote" {
organization = "terraform-snowflake"
workspaces {
name = "gh-actions"
}
}
}
provider "snowflake" {
username = "snowflake-username"
account = "snowflake-account"
region = "snowflake-region"
private_key = var.snowflake_private_key
role = "SYSADMIN"
}
Terraform Cloud Workspaces allow me to define variables, but I'm unable to find a way to share variables across more than one workspace.
In my example I have, lets say, two workspaces:
Database
Application
In both cases I'll be using the same AzureRM credentials for connectivity. The following are common values used by the workspaces to connect to my Azure subscription:
provider "azurerm" {
subscription_id = "00000000-0000-0000-0000-000000000000"
client_id = "00000000-0000-0000-0000-000000000000"
client_secret = "00000000000000000000000000000000"
tenant_id = "00000000-0000-0000-0000-000000000000"
}
It wouldn't make sense to duplicate values (in my case I'll have probably 10 workspaces).
Is there a way to do this?
Or the correct approach is to define "database" and "application" as a Module, and then use Workspaces (DEV, QA, PROD) to orchestrate them?
In Terraform Cloud, the Workspace object is currently the least granular location where you can specify variable values directly. There is no built in mechanism to share variable values between workspaces.
However, one way to approach this would be to manage Terraform Cloud with Terraform itself. The tfe provider (named after Terraform Enterprise for historical reasons, since it was built before Terraform Cloud launched) will allow Terraform to manage Terraform Cloud workspaces and their associated variables.
variable "workspaces" {
type = set(string)
}
variable "common_environment_variables" {
type = map(string)
}
provider "tfe" {
hostname = "app.terraform.io" # Terraform Cloud
}
resource "tfe_workspace" "example" {
for_each = var.workspaces
organization = "your-organization-name"
name = each.key
}
resource "tfe_variable" "example" {
# We'll need one tfe_variable instance for each
# combination of workspace and environment variable,
# so this one has a more complicated for_each expression.
for_each = {
for pair in setproduct(var.workspaces, keys(var.common_environment_variables)) : "${pair[0]}/${pair[1]}" => {
workspace_name = pair[0]
workspace_id = tfe_workspace.example[pair[0]].id
name = pair[1]
value = var.common_environment_variables[pair[1]]
}
}
workspace_id = each.value.workspace_id
category = "env"
key = each.value.name
value = each.value.value
sensitive = true
}
With the above configuration, you can set var.workspaces to contain the names of the workspaces you want Terraform to manage and var.common_environment_variables to the environment variables you want to set for all of them.
Note that for setting credentials on a provider the recommended approach is to set them in environment variables rather than Terraform variables, because that then makes the Terraform configuration itself agnostic to how those credentials are obtained. You could potentially apply the same Terraform configuration locally (outside of Terraform Cloud) using the integration with Azure CLI auth, while the Terraform Cloud execution environment would often use a service principal.
Therefore to provide the credentials in the Terraform Cloud environment you'd put the following environment variables in var.common_environment_variables:
ARM_CLIENT_ID
ARM_TENANT_ID
ARM_SUBSCRIPTION_ID
ARM_CLIENT_SECRET
If you use Terraform Cloud itself to run operations on this workspace managing Terraform Cloud (naturally, you'd need to set this one up manually to bootstrap, rather than having it self-manage) then you can configure var.common_environment_variables as a sensitive variable on that workspace.
If you instead set it via Terraform variables passed into the provider "azurerm" block (as you indicated in your example) then you force any person or system running the configuration to directly populate those variables, forcing them to use a service principal vs. one of the other mechanisms and preventing Terraform from automatically picking up credentials set using az login. The Terraform configuration should generally only describe what Terraform is managing, not settings related to who is running Terraform or where Terraform is being run.
Note though that the state for the Terraform Cloud self-management workspace will include
a copy of those credentials as is normal for objects Terraform is managing, so the permissions on this workspace should be set appropriately to restrict access to it.
You can now use variable sets to reuse variable across multiple workspaces
I am trying to configure a Terraform enterprise workspace in Jenkins on the fly. To do this, I need to be able to set the remote backend workspace name in my main.tf dynamically. Like this:
# Using a single workspace:
terraform {
backend "remote" {
hostname = "app.xxx.xxx.com"
organization = "YYYY"
# new workspace variable
workspaces {
name = "${var.workspace_name}"
}
}
}
Now when I run:
terraform init -backend-config="workspace_name=testtest"
I get:
Error loading backend config: 1 error(s) occurred:
* terraform.backend: configuration cannot contain interpolations
The backend configuration is loaded by Terraform extremely early, before
the core of Terraform can be initialized. This is necessary because the backend
dictates the behavior of that core. The core is what handles interpolation
processing. Because of this, interpolations cannot be used in backend
configuration.
If you'd like to parameterize backend configuration, we recommend using
partial configuration with the "-backend-config" flag to "terraform init".
Is what I want to do possible with terraform?
You cann't put any variables "${var.workspace_name}" or interpolations into the Backend Remote State Store.
However, you can create a file beside with your Backend values, it could look like this into the main.tf file:
# Terraform backend State-Sotre
terraform {
backend "s3" {}
}
and into a dev.backend.tfvars for instance:
bucket = "BUCKET_NAME"
encrypt = true
key = "BUCKET_KEY"
dynamodb_table = "DYNAMODB_NAME"
region = "AWS_REGION"
role_arn = "IAM_ROLE_ARN"
You can use partial configuration for s3 Backend as well.
Hope it'll help.
Hey I found the correct way to do this:
While the syntax is a little tricky, the remote backend supports partial backend initialization. What this means is that the configuration can contain a backend block like this:
terraform {
backend "remote" { }
}
And then Terraform can be initialized with a dynamically set backend configuration like this (replacing ORG and WORKSPACE with appropriate values):
terraform init -backend-config "organization=ORG" -backend-config 'workspaces=[{name="WORKSPACE"}]'
My team relies heavily on S3 remote state from within Terraform. We use the -backend-config feature of the CLI to specify the S3 configuration when initializing projects, so our actual terraform code looks like:
terraform {
backend "s3" {}
}
The above works great as long as all the S3 attributes are specified on the CLI with -backend-config.
We would like to use a similar strategy for referencing these states elsewhere in our configurations. Since the parameters for the backend are dynamic and specified on the CLI, we are looking to do the same.
data "terraform_remote_state" "dns" {
backend = "s3"
config {
key = "configurations/production/dns/terraform.tfstate"
}
}
In the above example, we've omitted the required region and bucket parameters, which of course causes plan/apply to fail (with not a valid region:).
Is there a method by which we can specify the region and bucket for remote state references from the CLI instead of hard-coding them?
The backend block is rather special because it gets processed so early in Terraform's workflow, and thus it doesn't have access to normal Terraform features such as variables. That's why it has its own special mechanism for configuring it.
The terraform_remote_state data source, on the other hand, is just a regular data source and so any normal interpolation strategy can be used with it. To pass settings from the CLI, for example, you could use variables:
variable "dns_state_region" {
}
variable "dns_state_key" {
}
data "terraform_remote_state" "dns" {
backend = "s3"
config {
region = "${var.dns_state_region}"
key = "${var.dns_state_key}"
}
}
You can then pass these to the terraform plan command:
$ terraform plan \
-var="dns_state_region=us-west-1" \
-var="dns_state_key=configurations/production/dns/terraform.tfstate"