I am trying to configure Terraform so it uses environment variables for AWS Secrets.
terraform.tfvars:
access_key = "${var.TF_VAR_AWS_AK}"
secret_key = "${var.TF_VAR_AWS_SK}"
aws_region = "eu-north-1"
main.tf:
provider "aws" {
region = "${var.aws_region}"
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
}
In console(it's on Windows 10):
set TF_VAR_AWS_AK = asd12345asd12345
set TF_VAR_AWS_SK = asd12345asd12345
terraform plan
Error messages:
Error: Variables not allowed
on terraform.tfvars line 1:
1: access_key = "${var.TF_VAR_AWS_AK}"
Variables may not be used here.
Error: Variables not allowed
on terraform.tfvars line 2:
2: secret_key = "${var.TF_VAR_AWS_SK}"
Variables may not be used here.
Not sure where the problem is. TF docs say it is possible to use env vars for secrets.
To configure providers and backends with environment variables, you don't need to write anything special in the configuration at all. Instead, you can just set the conventional environment variables related to the provider in question.
For example, you seem to be using AWS in which case you can use either the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY environment variables or you can populate a credentials file, the same as for the AWS SDK. You can then skip all of the declaration of variables and just reduce your provider block as follows:
provider "aws" {
region = "${var.aws_region}"
}
Terraform's AWS provider supports the same set of credentials sources that the AWS CLI does without any Terraform-specific configuration. That is the recommended way to configure credentials for the AWS provider, because then you only need to set up your AWS credentials once and you can use both AWS SDK, Terraform, and any other software that interacts with AWS and supports its conventions.
There's more information on the AWS provider authentication options in the AWS provider documentation.
As described here in the documentation : https://www.terraform.io/docs/configuration/variables.html#environment-variables
The environment variable names must be TF_VAR_<yourtfvariablename>.
With a terraform variable like this :
variable "aws_region" {
type = string
}
Your environment variable name must be TF_VARS_aws_region
There is actually no way to use environment variables directly in terraform. (ex: region = env.AWS_REGION) you must use TF_VAR to use env vars.
Related
I am using Terraform to integrate with GitHub Actions and AWS.
I want Terraform to setup AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY in GitHub repository secrets for GitHub Actions using the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that I've set in Terraform Cloud.
I want to write something like below.
resource "github_actions_secret" "example_secret" {
repository = "example_repository"
secret_name = "AWS_SECRET_ACCESS_KEY"
plaintext_value = env.AWS_SECRET_ACCESS_KEY // <-- this will not work
}
Terraform Cloud will automatically export any environment variables to populate your shell environment before running any Terraform.
This is equivalent to running export TF_VAR_foo=bar if you wanted to set foo to bar.
To then actually use these variables you must define a variable block as normal and then access it as usual as well.
So in your case if you have defined an AWS_SECRET_ACCESS_KEY environment variable in Terraform Cloud then you could do this:
variable "AWS_SECRET_ACCESS_KEY" {}
resource "github_actions_secret" "example_secret" {
repository = "example_repository"
secret_name = "AWS_SECRET_ACCESS_KEY"
plaintext_value = var.aws_secret_access_key
}
You have to export those env variables yourself before you run your TF apply, as shown here. This is what TF cloud is doing:
Terraform Cloud performs Terraform runs on disposable Linux worker VMs using a POSIX-compatible shell. Before running Terraform, Terraform Cloud populates the shell with environment variables using the export command.
I'm using Terraform cloud. I would like to take advantage of using AWS Tags with my resources. I want to tag each resource defined in Terraform with the current GIT Branch Name. That way I can separate dev from production.
Terraform has a list of Environmental Variables that do reference the GIT Branch Name with their service in the cloud as:
TFC_CONFIGURATION_VERSION_GIT_BRANCH - This is the name of the branch that the associated Terraform configuration version was ingressed from (e.g. master).
How can I reference the TFC_CONFIGURATION_VERSION_GIT_BRANCH environmental variable in the following resource for an example VPC?
resource "aws_vpc" "example_vpc" {
cidr_block = "10.0.0.0/16"
tags = {
product = var.product
stage = var.TFC_CONFIGURATION_VERSION_GIT_BRANCH
}
}
reference: https://www.terraform.io/docs/language/values/variables.html#environment-variables
I figured it out! Wish the documentation was clearer on this with the cloud.
You will have to set a empty variable. I defined mine in variables.tf as:
variable "TFC_CONFIGURATION_VERSION_GIT_BRANCH" {
type = string
default = ""
}
Per the documentation I linked in question. TFC_CONFIGURATION_VERSION_GIT_BRANCH is injected automatically into the environmental variables with each cloud run. Defining the full name of the environmental variable as the variable worked.
resource "aws_vpc" "example_vpc" {
cidr_block = "10.0.0.0/16"
tags = {
product = var.product
stage = var.TFC_CONFIGURATION_VERSION_GIT_BRANCH
}
}
Then the plan output was successful in the cloud:
Terraform will perform the following actions:
# aws_vpc.example_vpc will be updated in-place
~ resource "aws_vpc" "example_vpc" {
id = "vpc-0b19679e6464b8481"
~ tags = {
~ "stage" = "None" -> "develop"
# (1 unchanged element hidden)
}
# (14 unchanged attributes hidden)
}
Terraform Cloud serves as a remote execution environment for Terraform CLI (amongst other things) and so when you configure a workspace in Terraform Cloud many of the settings are about the context where Terraform CLI will run, and how Terraform Cloud will run it.
Part of that configuration model is the idea of environment variables, which correspond with the same environment variables you might set in your shell when running Terraform CLI locally. As with local Terraform, those environment variables are not directly usable from your configuration but are instead settings for other systems that Terraform and providers will interact with, such as the AWS_ACCESS_KEY_ID environment variable conventionally used by AWS software as a way to statically configure the access key identifier to use.
Terraform Cloud also allows you to set "Terraform Variables", and those correspond with Input Variables in the Terraform language. These are the settings for your Terraform configuration itself, as opposed to other software it will interact with, and so you can refer to these by declaring them using variable blocks and then using expressions like var.example elsewhere in the root module. Internally, Terraform Cloud is passing the configured values to Terraform CLI by generating a file called terraform.tfvars, which Terraform CLI looks for as a default source of variable values.
Both of these types of variables are useful for different purposes, and so most Terraform workspaces include a mixture of environment variables for configuring external systems and "Terraform variables" for configuring the current Terraform configuration itself.
For the benefit of folks using Terraform CLI outside of Terraform Cloud, Terraform CLI actually also offers a way to set Input Variables using environment variables, and technically you can do that within Terraform Cloud too because it's ultimately just running Terraform with environment variables set in the same way as you might locally. That's not the intended way to use Terraform Cloud, and so I'm mentioning it only for completeness because the terminology overlap here might be confusing.
Terraform cloud workspace variables can be set as category "terraform" or as category "env". In a remote execution setup, you are unable to reference workspace vars defined as "env" from your terraform code. Instead, those vars will be automatically injected into the execution environment. To be able to reference a workspace variable in terraform code, set the variable type to "terraform" but do not check the "HCL" tick. Defining a blank variable is still needed. Hope this helps someone! Unfortunately, the documentation is very unclear about this.
Terraform Cloud: Create a workspace variable
key: example_variable
value: example_value
category: terraform
Implementation: Configuration (.tf) file
Declare a blank variable
Reference your variable
# Declaring a blank variable
variable "example_variable" {}
# Referencing a variable
block "name" {
input = var.example_variable
Snowflake CI/CD Example
CI/CD pipeline for Snowflake with GitHub Actions and Terraform configured similar to Snowflake's quick start guide
Implemented a private key instead of a user password
variable "snowflake_private_key" {}
terraform {
required_providers {
snowflake = {
source = "chanzuckerberg/snowflake"
version = "0.25.17"
}
}
backend "remote" {
organization = "terraform-snowflake"
workspaces {
name = "gh-actions"
}
}
}
provider "snowflake" {
username = "snowflake-username"
account = "snowflake-account"
region = "snowflake-region"
private_key = var.snowflake_private_key
role = "SYSADMIN"
}
Terraform Cloud Workspaces allow me to define variables, but I'm unable to find a way to share variables across more than one workspace.
In my example I have, lets say, two workspaces:
Database
Application
In both cases I'll be using the same AzureRM credentials for connectivity. The following are common values used by the workspaces to connect to my Azure subscription:
provider "azurerm" {
subscription_id = "00000000-0000-0000-0000-000000000000"
client_id = "00000000-0000-0000-0000-000000000000"
client_secret = "00000000000000000000000000000000"
tenant_id = "00000000-0000-0000-0000-000000000000"
}
It wouldn't make sense to duplicate values (in my case I'll have probably 10 workspaces).
Is there a way to do this?
Or the correct approach is to define "database" and "application" as a Module, and then use Workspaces (DEV, QA, PROD) to orchestrate them?
In Terraform Cloud, the Workspace object is currently the least granular location where you can specify variable values directly. There is no built in mechanism to share variable values between workspaces.
However, one way to approach this would be to manage Terraform Cloud with Terraform itself. The tfe provider (named after Terraform Enterprise for historical reasons, since it was built before Terraform Cloud launched) will allow Terraform to manage Terraform Cloud workspaces and their associated variables.
variable "workspaces" {
type = set(string)
}
variable "common_environment_variables" {
type = map(string)
}
provider "tfe" {
hostname = "app.terraform.io" # Terraform Cloud
}
resource "tfe_workspace" "example" {
for_each = var.workspaces
organization = "your-organization-name"
name = each.key
}
resource "tfe_variable" "example" {
# We'll need one tfe_variable instance for each
# combination of workspace and environment variable,
# so this one has a more complicated for_each expression.
for_each = {
for pair in setproduct(var.workspaces, keys(var.common_environment_variables)) : "${pair[0]}/${pair[1]}" => {
workspace_name = pair[0]
workspace_id = tfe_workspace.example[pair[0]].id
name = pair[1]
value = var.common_environment_variables[pair[1]]
}
}
workspace_id = each.value.workspace_id
category = "env"
key = each.value.name
value = each.value.value
sensitive = true
}
With the above configuration, you can set var.workspaces to contain the names of the workspaces you want Terraform to manage and var.common_environment_variables to the environment variables you want to set for all of them.
Note that for setting credentials on a provider the recommended approach is to set them in environment variables rather than Terraform variables, because that then makes the Terraform configuration itself agnostic to how those credentials are obtained. You could potentially apply the same Terraform configuration locally (outside of Terraform Cloud) using the integration with Azure CLI auth, while the Terraform Cloud execution environment would often use a service principal.
Therefore to provide the credentials in the Terraform Cloud environment you'd put the following environment variables in var.common_environment_variables:
ARM_CLIENT_ID
ARM_TENANT_ID
ARM_SUBSCRIPTION_ID
ARM_CLIENT_SECRET
If you use Terraform Cloud itself to run operations on this workspace managing Terraform Cloud (naturally, you'd need to set this one up manually to bootstrap, rather than having it self-manage) then you can configure var.common_environment_variables as a sensitive variable on that workspace.
If you instead set it via Terraform variables passed into the provider "azurerm" block (as you indicated in your example) then you force any person or system running the configuration to directly populate those variables, forcing them to use a service principal vs. one of the other mechanisms and preventing Terraform from automatically picking up credentials set using az login. The Terraform configuration should generally only describe what Terraform is managing, not settings related to who is running Terraform or where Terraform is being run.
Note though that the state for the Terraform Cloud self-management workspace will include
a copy of those credentials as is normal for objects Terraform is managing, so the permissions on this workspace should be set appropriately to restrict access to it.
You can now use variable sets to reuse variable across multiple workspaces
I have a $HOME/.aws/credentials file like this:
[config1]
aws_access_key_id=accessKeyId1
aws_secret_access_key=secretAccesskey1
[config2]
aws_access_key_id=accessKeyId2
aws_secret_access_key=secretAccesskey2
So I was expecting that with this configuration, terraform will choose the second credentials:
terraform {
backend "s3" {
bucket = "myBucket"
region = "eu-central-1"
key = "path/to/terraform.tfstate"
encrypt = true
}
}
provider "aws" {
profile = "config2"
region = "eu-central-1"
}
But when I try terraform init it says it hasn't found any valid credentials:
Initializing the backend...
Error: No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider
As as workaround, I changed config2 by default in my credentials file and I removed the profile field from the provider block so it works but I really need to use something like the first approach. What am I missing here?
Unfortunately you also need to provide the IAM credential configuration to the backend configuration as well as your AWS provider configuration.
The S3 backend configuration takes the same parameters here as the AWS provider so you can specify the backend configuration like this:
terraform {
backend "s3" {
bucket = "myBucket"
region = "eu-central-1"
key = "path/to/terraform.tfstate"
encrypt = true
profile = "config2"
}
}
provider "aws" {
profile = "config2"
region = "eu-central-1"
}
There's a few reasons behind this needing to be done separately. One of the reasons would be that you can independently use different IAM credentials, accounts and regions for the S3 bucket and the resources you will be managing with the AWS provider. You might also want to use S3 as a backend even if you are creating resources in another cloud provider or not using a cloud provider at all, Terraform can manage resources in a lot of places that don't have a way to store Terraform state. The main reason though is that the backends are actually managed by the core Terraform binary rather than the provider binaries and the backend initialisation happens before pretty much anything else.
How do I manage remote state for different environments? I originally wanted to use variables in my remote state definations but realized I cannot use variables like:
provider "aws" {
region = "ap-southeast-1"
}
terraform {
backend "s3" {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
data "terraform_remote_state" "s3_state" {
backend = "s3"
config {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
But realised I cannot use variables in this case? I can hardcode the bucket name but the bucket may not be the same across environments
You will want to use what Terraform calls workspaces. Here is the documentation: https://www.terraform.io/docs/state/workspaces.html
So way you have a piece of state called: MyStateKey
When you use workspaces it will append the workspace name to the end of the existing key. For example if you created a workspace called "dev" then the key in the remote state would be "MyStateKey:dev".
I would suggest you use some conventions to make it easier like using the "default" workspace as production, with additional workspaces named after your other environments. Then when you run terraform you can set the workspace or use the TF_WORKSPACE environment variable to set it.