I'm trying to create a mechanism so that I use terraform backend to upload the state to a S3, so that my teammate can use my terraform state to resume my work. This is my setup:
terraform {
backend "s3" {
bucket = "username-terraform-state"
key = "billow/terraform.tfstate"
region = "us-west-2"
dynamodb_table = "username-terraform-state-test-locks"
encrypt = true
}
}
data "terraform_remote_state" "network" {
backend = "s3"
config = {
bucket = "username-terraform-state"
key = "billow/terraform.tfstate"
region = var.region
}
}
With this setup, I have two folders in the S3 bucket. One is billow/ with a terraform.tfstate file. There's another folder env:/remote_s3/billow/ (remote_s3 is the name of my terraform workspace) with another terraform.tfstate. Both of them are updated also when I execute a terraform import command.
What I want is when I create a new workspace, I would be able to pull the state file from existing folder in the bucket, and continue the project. The step I took was placing the same .tf file in the directory and run terraform init, terraform refresh and then terraform state pull to pull the state file. However, this only pulls an empty state file, and I would need to re-import all the resources again.
So here are my two questions:
Why are there two folders in the bucket? I thought with my backend setup there should be only one of them.
What should I do to make it so that when I set up a new terraform workspace, I would be able to import the whole state file from my previously saved terraform state?
Thanks!
Related
I am trying the terraform operations using azure as remote backend. So far, I have been able to store the state remotely in azure. I am now unable to retrieve the state from remote, as I do not store the .tf files or .tfstate files locally. How to fetch the same from remote? This is the code I have:
terraform init -backend-config=config.backend.tfbackend
terraform state pull # fails due to no configuration file existing locally
This is my config.backend.tfbackend
backend "azurerm" {
resource_group_name = "rg098"
storage_account_name = "tstate6565"
container_name = "tstate"
key = "test5176"
}
I'm running terraform v0.14.8 with a non-default terraform workspace.
I have an Atlantis server that handles my plans and applies.
When I clone my repo locally and run my plans I get errors about my datasource. I don't quite understand why as I don't get these errors on my Atlantis server which I believe performs the same operations. The Atlantis server also uses tf v0.14.8.
My terraform:
data "terraform_remote_state" "route53" {
backend = "s3"
config = {
key = "web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
Before I run my local plan, i switch the workspace
terraform workspace select web
# in addition I also tried
export TF_WORKSPACE=web
My plan
teraform plan
...
Error: Unable to find remote state
on provider.tf line 46, in data "terraform_remote_state" "route53":
46: data "terraform_remote_state" "route53" {
No stored state was found for the given workspace in the given backend.
I could easily edit my "key" with env: and things will work, but I'm trying to figure out how to do this without making that adjustment, seeing that my Atlantis server just works.
data "terraform_remote_state" "route53" {
backend = "s3"
config = {
key = "env:/web/web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
Your question seems to imply some confusion over which of these backends the web workspace is selected for. Running terraform workspace select web selects the web workspace from the backend of the current configuration (the directory where you are running Terraform), but I suspect your intent is to select the web backend from the backend you've configured in data "terraform_remote_state" instead.
If so, you can do that by setting the workspace argument in the data source configuration:
data "terraform_remote_state" "route53" {
backend = "s3"
workspace = "web"
config = {
key = "web/terraform.tfstate"
region = "us-west-2"
bucket = "prod-terraform"
role_arn = "arn:aws:iam::xxxxxxxxxx:role/atlantis"
}
}
I am using terraform 0.12.9 and state is saved on s3 bucket. I'd like to list all resources by terraform state list. Based on this document, https://www.terraform.io/docs/commands/state/list.html, it says -state=path - Path to the state file. Defaults to "terraform.tfstate". Ignored when remote state is used.. How can I pass the state file if it is on remote s3 bucket?
You need to configure the tfstate bucket path in your terraform.tf file:
terraform {
backend "s3" {
bucket = "bucket_name"
key = "my/key/location/terraform.tfstate"
region = "bucket region"
}
}
and later you need to run terraform init so that terraform would fetch the state from the remote bucket
I'm defining a remote state data resource (gcp backend) in terraform. When I plan, the state file, for the remote state, is created if it does not exist previously, even when I'm not referencing the state in other resources.
Terraform v0.11.14
So when I plan for env dev:
data "terraform_remote_state" "example" {
backend = "gcs"
workspace = "dev-us-east1"
config {
bucket = "bucket"
prefix = "global/projects/example-project"
}
}
and the file in gcs bucket/global/projects/example-project/dev-us-east1 does not exist, then it's created as an empty state.
I expected kind of a state not found error, but instead, the remote state is created with an empty content.
How do I manage remote state for different environments? I originally wanted to use variables in my remote state definations but realized I cannot use variables like:
provider "aws" {
region = "ap-southeast-1"
}
terraform {
backend "s3" {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
data "terraform_remote_state" "s3_state" {
backend = "s3"
config {
bucket = "${var.state_bucket}"
key = "${var.state_key}"
region = "ap-southeast-1"
}
}
But realised I cannot use variables in this case? I can hardcode the bucket name but the bucket may not be the same across environments
You will want to use what Terraform calls workspaces. Here is the documentation: https://www.terraform.io/docs/state/workspaces.html
So way you have a piece of state called: MyStateKey
When you use workspaces it will append the workspace name to the end of the existing key. For example if you created a workspace called "dev" then the key in the remote state would be "MyStateKey:dev".
I would suggest you use some conventions to make it easier like using the "default" workspace as production, with additional workspaces named after your other environments. Then when you run terraform you can set the workspace or use the TF_WORKSPACE environment variable to set it.