I used to use multiple .sh files that ran different "terraform remote config" commands to switch between state files in buckets in different Google Cloud projects for different environments (dev, test and prod).
With version 0.9.0, I understand that this now goes into a a .tf file:
terraform {
backend "gcs" {
bucket = "terraform-state-test"
path = "terraform.tfstate"
project = "cloud-test"
}
}
In version 0.9.0 there is now also the State Environment ("terraform env"):
resource "google_container_cluster" "container_cluster" {
initial_node_count = "${terraform.env == "prod" ? 5 : 1}"
}
But how should I now manage multiple environments in the same directory structure with the new backend configuration?
At the time of this writing, not all of the remote backends in Terraform have been updated to support state environments. For those that have, each backend has its own conventions for how to represent the now-multiple states in the data store.
As of version 0.9.2, the "consul", "s3" and "local" backends have been updated. The "gcs" backend has not yet, but once it has the procedure described here will apply to that too.
There's initially a "default" environment, but if you never run terraform apply with this environment selected then you can ignore it and name your environments whatever you want.
To create a new environment called "production" and switch to it:
terraform workspace new production
This will establish an entirely separate state on the backend, so terraform plan should show that all of the resources need to be created fresh.
You can switch between already-existing environments like this:
terraform workspace select production
Before 0.9, many teams (including yours, it sounds like) wrote wrapper scripts to simulate this behavior. It's likely that these scripts didn't follow exactly the same naming conventions in the storage backends, so some manual work will be required to migrate.
One way to do that migration is to start off using the "local" backend, which stores state in a local directory called terraform.state.d. While working locally you can create the environments you want and then carefully overwrite the empty state files in the local directory with the existing state files from your previous scripted solution. Once all of the local environments have appropriate states in place you can then change the backend block in the config to the appropriate remote backend and run terraform init to trigger a migration of all of the local environments into the new backend.
After this, the terraform workspace select command will begin switching between the remote environments rather than the local ones.
If your chosen remote backend doesn't yet support environments, it's best to continue with a scripted solution for the time being. This means replacing terraform remote config in your existing wrapper script with a use of the partial configuration pattern to pass environment-specific configuration into terraform init.
Related
I recreated a state file for an existing infrastructure using a third party tool i.e. terraformer. Now I want o move the .tfstate to another azurerm back-end and manage from there.
If I just copy the file i.e. mystate.tfstate from local to inside storage account container with the same file name/key as in the backend configurations will it work or do I need to do something else to achieve it?
I don't want to risk the state file or infrastructure by trying to do something that isn't sure to work as I expect.
Terraform has some automatic migration behavior built in to terraform init.
Based on your description, it sounds like so far you've been using local state storage, and so the latest state snapshot is in a .tfstate file on your local system and you probably don't have a backend block in your configuration yet, since local storage is the default.
Before beginning this process, I suggest first making a copy of your state file in a safe place so that you can experiment more confidently. This process should not risk your existing state file, but it can't hurt to be careful if you've invested significant work in constructing that state file already.
Next, add a backend "azurerm" block to tell Terraform it should use that backend. Refer to the documentation to see which settings you'll need to set and what other preparation steps you may need to make first, such as establishing a new storage container.
If you've been using local state then you will presumably have a terraform.tfstate file in your current working directory, which Terraform will check for in the next step. If you've renamed that file at any point so far, you'll need to rename it back to terraform.tfstate to match the expectations of Terraform's local state storage implementation.
If you now run terraform init, Terraform should notice the following two things:
You have a backend block but the current working directory doesn't currently have an initialized backend connection.
You have an existing terraform.tfstate file in your working directory.
With those two things being true, Terraform will propose to migrate your state from the local backend to the azurerm backend. You can follow the steps it proposes and answer the prompts that appear, after which you should find the same state snapshot stored in your configured Azure storage container.
Once you've confirmed that the object is present in Azure storage, you can delete the terraform.tfstate file, since Terraform will no longer refer to it.
I don't want to risk the state file or infrastructure by trying to do
something that isnt sure to work as I expect.
Make a backup of the state file first, and then you won't be risking the state file.
As long as you aren't running an apply command, you won't be risking the infrastructure. And even if you are running an apply command, you will be able to review the plan before proceeding.
So just (always) backup your state file, and always review a plan before applying it, and there is no risk.
Enviornment Isolation: Dirs v. Workspaces v. Modules
The Terraform docs Separate Development and Production Environments seem to take two major approaches for handling a "dev/test/stage" type of CI enviornment, i.e.
Directory seperation - Seems messy especially when you potentially have multiple repos
Workspaces + Different Var Files
Except when you lookup workspaces it seems to imply workspaces are NOT a correct solution for isolating enviornments.
In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams. In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls. Named workspaces are not a suitable isolation mechanism for this scenario.
Instead, use one or more re-usable modules to represent the common elements, and then represent each instance as a separate configuration that instantiates those common elements in the context of a different backend. In that case, the root module of each configuration will consist only of a backend configuration and a small number of module blocks whose arguments describe any small differences between the deployments.
I would also like to consider using remote state -- e.g. azurerm backend
Best Practice Questions
When the docs refer to using a "re-usable" module, what would this look like if say I had na existing configuration folder? Would I still need to create a sepreate folder for dev/test/stage?
When using remote backends, should the state file be shared across repos by default or separated by repo and enviornment?
e.g.
terraform {
backend "azurerm" {
storage_account_name = "tfstorageaccount"
container_name = "tfstate"
key = "${var.enviornment}.terraform.tfstate"
}
}
vs.
terraform {
backend "azurerm" {
storage_account_name = "tfstorageaccount"
container_name = "tfstate"
key = "cache_cluster_${var.enviornment}.terraform.tfstate"
}
}
When the docs refer to using a "re-usable" module, what would this look like if say I had na existing configuration folder? Would I still need to create a sepreate folder for dev/test/stage?
A re-usable module for your infrastructure would essentially encapsulate the part of your infrastructure that is common to all your "dev/test/stage" environments. So no, you wouldn't have any "dev/test/stage" folders in there.
If, for example, you have an infrastructure that consists of a Kubernetes cluster and a MySQL database, you could have two modules - a 'compute' module that handles the k8s cluster, and a 'storage' module that would handle the DB. These modules go into a /modules subfolder. Your root module (main.tf file in the root of your repo) would then instantiate these modules and pass the appropriate input variables to customize them for each of the "dev/test/stage" environments.
Normally it would be a bit more complex:
Any shared VPC or firewall config might go into a networking module.
Any service accounts that you might automatically create might go into a credentials or iam module.
Any DNS mappings for API endpoints might go into a dns module.
You can then easily pass in variables to customize the behavior for "dev/test/stage" as needed.
When using remote backends, should the state file be shared across repos by default or separated by repo and enviornment?
Going off the Terraform docs and their recommended separation:
In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls.
You would not share tfstorageaccount. Now take this with a grain of salt and determine your own needs - essentially what you need to take into account is the security and data integrity implications of sharing backends/credentials. For example:
How sensitive is your state? If you have sensitive variables being output to your state, then you might not want your "production" state sitting in the same security perimeter as your "test" state.
Will you ever need to wipe your state or perform destructive actions? If, for example, your state storage provider only versions by folder, then you probably don't want your "dev/test/stage" states sitting next to each other.
Background:
I have a shared module called "releases". releases contains the following resources:
aws_s3_bucket.my_deployment_bucket
aws_iam_role.my_role
aws_iam_role_policy.my_role_policy
aws_iam_instance_profile.my_instance_profile
These resources are used by ec2 instances belonging to an ASG to pull code deployments when they provision themselves. The release resources are created once and will rarely/never change. This module is one of a handful used inside an environment-specific project (qa-static) that has it's own tfstate file in AWS.
Fast Forward: It's now time to create a "prd-static" project. This project wants to re-use the environment agnostic AWS resources defined in the releases module. prd-static is basically a copy of qa with beefed up configuration for the database and cache server, etc.
The Problem:
prd-static sees the environment-agnostic AWS resources defined in the "releases" module as new resources that don't exist in AWS yet. An init and plan call shows that it wants to create these from scratch. It makes sense to me since prd-static has it's own tfstate - and tfstate is essentially the system-of-record - that terraform doesn't know that no changes should be applied. But, ideally terraform would use AWS as the source of truth for existing resources and their configuration.
If we try to apply the plan as is, the prd-static project just bombs out with an Entity Already Exists error. Leading me to this post:
what is the best way to solve EntityAlreadyExists error in terraform?
^-- logically I could import these resources into the tfstate file for prd-static and be on my merry way. Then, both projects know about the resources and in theory would only apply updates if the configuration had changed. I was able to import the bucket and the role and then re-run the plan.
Now terraform wants to delete the s3 bucket and recreate the role. That's weird - and not at all what I wanted to do.
TLDR: It appears that while modules like to be shared, modules that create single re-usable resources (like an S3 bucket) really don't want to be shared. It looks like I need to pull the environment-agnostic static resources module into it's own project with it's own tfstate that can be used independently rather than try and share the releases module across environments. Environment-specific stuff that depend on the release resources can reference them via their outputs in my build-process.
Should I be able to define a resource in a module, like an S3 bucket where the same instance is used across terraform projects that each have their own tfstate file (remote state in S3). Because I cannot.
If I really shouldn't be able to do this is the correct approach to extract the single instance stuff into its own project and depend on the outputs?
I want to run a script using terraform inside an existing instance on any cloud which is pre-created .The instance was created manually , is there any way to push my script to this instance and run it using terraform ?
if yes ,then How can i connect to the instance using terraform and push my script and run it ?
I believe ansible is a better option to achieve this easily.
Refer the example give here -
https://docs.ansible.com/ansible/latest/modules/script_module.html
Create a .tf file and describe your already existing resource (e.g. VM) there
Import existing thing using terraform import
If this is a VM then add your script to remote machine using file provisioner and run it using remote-exec - both steps are described in Terraform file, no manual changes needed
Run terraform plan to see if expected changes are ok, then terraform apply if plan was fine
Terraform's core mission is to create, update, and destroy long-lived infrastructure objects. It is not generally concerned with the software running in the compute instances it deploys. Instead, it generally expects each object it is deploying to behave as a sort of specialized "appliance", either by being a managed service provided by your cloud vendor or because you've prepared your own machine image outside of Terraform that is designed to launch the relevant workload immediately when the system boots. Terraform then just provides the system with any configuration information required to find and interact with the surrounding infrastructure.
A less-ideal way to work with Terraform is to use its provisioners feature to do late customization of an image just after it's created, but that's considered to be a last resort because Terraform's lifecycle is not designed to include strong support for such a workflow, and it will tend to require a lot more coupling between your main system and its orchestration layer.
Terraform has no mechanism intended for pushing arbitrary files into existing virtual machines. If your virtual machines need ongoing configuration maintenence after they've been created (by Terraform or otherwise) then that's a use-case for traditional configuration management software such as Ansible, Chef, Puppet, etc, rather than for Terraform.
If one has two AWS accounts, one for development and one for live (for example) I am aware that one can use terraform workspaces to manage the state of each environment.
However, if i switch workspace from "dev" to "live" is there a way to tell terraform it should now be applying the state to the live account rather than the test one?
One way I thought of, which is error prone, would be swap my secret.auto.tfvars file each time i switch workspace since I presume when running with a different access key (the one belonging to the "live" account) the AWS provider will then be applying to that account. However, it'd be very easy to swap workspace and have the wrong credentials present which would run the changes against the wrong environment.
I'm looking for a way to almost link a workspace with an account id in AWS.
I did find this https://github.com/hashicorp/terraform/issues/13700 but it refers to the deprecated env command, this comment looked somewhat promising in particular
Update
I have found some information on GitHub where I left this comment as a reply to an earlier comment which recommended considering modules instead of workspaces and actually indicates that workspaces aren't well suited to this task. If anyone can provide information on how modules could be used to solve this issue of maintaining multiple versions of the "same" infrastructure concurrently I'd be keen to see how this improves upon the workspace concept.
Here's how you could use Terraform modules to structure your live vs dev environments that point to different AWS accounts, but the environments both have/use the same Terraform code.
This is one (of many) ways that you could structure your dirs; you could even put the modules into their own Git repo, but I'm going to try not to confuse things too much. In this example, you have a simple app that has 1 EC2 instance and 1 RDS database. You write whatever Terraform code you need in the modules/*/ subdirs, making sure to parameterize whatever attributes are different across environments.
Then in your dev/ and live/ dirs, main.tf should be the same, while provider.tf and terraform.tfvars reflect environment-specific info. main.tf would call the modules and pass in the env-specific params.
modules/
|-- ec2_instance/
|-- rds_db/
dev/
|-- main.tf # --> uses the 2 modules
|-- provider.tf # --> has info about dev AWS account
|-- terraform.tfvars # --> has dev-specific values
live/
|-- main.tf # --> uses the 2 modules
|-- provider.tf # --> has info about live/prod AWS account
|-- terraform.tfvars # --> has prod-specific values
When you need to plan/apply either env, you drop into the appropriate dir and run your TF commands there.
As for why this is preferred over using Terraform Workspaces, the TF docs explains it well:
In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production) or different internal teams. In this case, the backend used for each deployment often belongs to that deployment, with different credentials and access controls. Named workspaces are not a suitable isolation mechanism for this scenario.
Instead, use one or more re-usable modules to represent the common elements, and then represent each instance as a separate configuration that instantiates those common elements in the context of a different backend. In that case, the root module of each configuration will consist only of a backend configuration and a small number of module blocks whose arguments describe any small differences between the deployments.
BTW> Terraform merely changed the env subcommand to workspace when they decided that 'env' was a bit too confusing.
Hope this helps!
Terraform workspaces hold the state information. They connect to user accounts based on the way in which AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are used in the environment. In order to link any given workspace to an AWS account they'd have to store user credentials in some kind of way, which they understandably don't. Therefore, I would not expect to see workspaces ever directly support that.
But, to almost link a workspace to an account you just need to automatically switch AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY each time the workspace is switched. You can do this by writing a wrapper around terraform which:
Passes all commands on to the real terraform unless it finds
workspace select in the command line.
Upon finding workspace select in the command line it parses out the
workspace name.
Export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the
AWS account which you want to link to the workspace
Finish by passing on the workspace command to the real terraform
This would load in the correct credentials each time
terraform workspace select <WORKSPACE>
was used