Context: I'm developing a TF Provider using Terraform SDKv2.
I've noticed there's prevent_destroy attribute that should protect against accidental deletions of important resources (that should only be deleted in exceptional cases).
However it looks like it won't prevent from issues where the resource is accidentally removed from the TF configuration or some other silly developer error, since the prevent_destroy attribute will be lost as well and not applied.
And then we noticed there's deletion_protection attribute for aws_rds_cluster resource of AWS TF Provider. Is it a thing or there're any other built in mechanism to address it?
For example, GCP's Best practices for using Terraform mentions:
resource "google_sql_database_instance" "main" {
name = "primary-instance"
settings {
tier = "D0"
}
lifecycle {
prevent_destroy = true
}
}
so it sounds like prevent_destroy might be good enough. That said, there's deletion_protection attribute for sql_database_instance resource too. What's even more interesting, for google_bigquery_table deletion_protection attribute defaults to true and there's a note:
: On newer versions of the provider, you must explicitly set deletion_protection=false (and run terraform apply to write the field to state) in order to destroy an instance. It is recommended to not set this field (or set it to true) until you're ready to destroy.
Related
I am fairly new to terraform. I am trying to understand what is the right way to spin up resources using terraform. I have come across two different ways to do it.
Using terraform modules from terraform registry:
module "vpc" {
source = "terraform-google-modules/network/google"
version = "~> 6.0"
project_id = var.project_id
network_name = var.network_name
routing_mode = var.routing_mode
auto_create_subnetworks = var.auto_create_subnetworks
delete_default_internet_gateway_routes = var.delete_default_internet_gateway_routes
mtu = var.mtu
subnets = var.subnets
firewall_rules = var.firewall_rules
}
Using resources
resource "google_compute_network" "network" {
name = var.network_name
auto_create_subnetworks = var.auto_create_subnetworks
routing_mode = var.routing_mode
project = var.project_id
delete_default_routes_on_create = var.delete_default_internet_gateway_routes
mtu = var.mtu
}
Both these approaches work fine and they give me desired output but I would like to know which one should be used when? When should I use Terraform Module directly and when should I use resource?
As you may read in the documentation, modules are a way of organising Terraform scripts, it makes them more reusable, sharable and publishable.
In general, when working with Terraform you want to have an infrastructure that's portable and that can be transferred and deployed on multiple environments.
It would have been harder to maintain a complex infrastructure where you only use resources that will be sitting in the same codebase.
modules allows you to group one, to multiple resources, so you can break your infrastructure code to reusable and configurable portions of code, where you can enforce certain defaults. These modules can be used in the main infrastructure codebase making it easily readable and maintainable since it abstracts the details of each part of the infrastructure.
And with this abstraction, you can afford using different patterns to deploy your infrastructure on different environments (example: 'dev', 'staging', and 'prod') since you may only pass different (and less) values for your variables, or your can just duplicate your infrastructure folder (dev-infra) for example to create another (stg-infra), the cost of this code duplication is cheaper than duplicating many files in case of using only ressources.
TLDR; as you shared in your example:
module "vpc" {
source = "terraform-google-modules/network/google"
version = "~> 6.0"
project_id = var.project_id
network_name = var.network_name
routing_mode = var.routing_mode
auto_create_subnetworks = var.auto_create_subnetworks
delete_default_internet_gateway_routes = var.delete_default_internet_gateway_routes
mtu = var.mtu
subnets = var.subnets
firewall_rules = var.firewall_rules
}
Does more than creating the VPC on GCP. As you can see in their source-code, it is composed on many other modules, in which they use the google_compute_network resources along others to setup networks, firewall rules, subnets and more.
You may have a very complex VPC network that can be done only by using the module that abstracts the usage of the resources. That same VPC setup might be re-used for other GCP projects for different environments. Or you can even build a module on top of this if you want to enforce some configurations, like you hardcode auto_create_subnetworks = true.
In some organisations, it is recommended to use only the organisation's approved terraform module, where they apply certain patterns, and set some defaults/best-practices.
See more:
https://developer.hashicorp.com/terraform/tutorials/modules/module#what-are-modules-for
https://www.terraform-best-practices.com/key-concepts#composition
https://www.terraform-best-practices.com/examples/terraform
https://developer.hashicorp.com/terraform/language/modules
Hope this helps!
Context: I'm working on a new TF Provider using SDKv2.
I'm adding a new data plane resource which has a very weird API. Namely, there're some sensitive attributes (that are specific to this resource so they can't be set under provider block -- think about DataDog / Slack API secrets that this resource needs to interact with under the hood) I need to pass on creation that are not necessary later on (for example, even for update operation). My minimal code sample:
resource "foo" "bar" {
name = "abc"
sensitive_creds = {
"datadog_api_secret" = "abc..."
// might pass "slack_api_secret" instead
}
...
}
How can I implement it in Terraform to avoid state drifts etc?
So far I can see 3 options:
Make a user pass it first, don't save "sensitive_creds" to TF state. Make a user set it to sensitive_creds = {} to avoid a state drift for the next terraform plan run.
Make a user pass it first, don't save "sensitive_creds" to TF state. Make a user add ignore_changes = [sensitive_creds] } to their Terraform configuration.
Save "sensitive_creds" to TF state and live with it since users are likely to encrypt TF state anyways.
The most typical compromise is for the provider to still save the user's specified value to the state during create and then to leave it unchanged in the "read" operation that would normally update the state to match the remote system.
The result of this compromise is that Terraform can still detect when the user has intentionally changed the secret value in the configuration, but Terraform will not be able to detect changes made to the value outside of Terraform.
This is essentially your option 3. The Terraform provider protocol requires that the values saved to the state after create exactly match anything the user has specified in the configuration, so your first two options would violate the expected protocol and thus be declared invalid by Terraform Core.
Since you are using SDKv2, you can potentially "get away with it" because Terraform Core permits that older SDK to violate some of the rules as a pragmatic way to deal with the fact that SDKv2 was designed for older versions of Terraform and therefore doesn't implement the type system correctly, but Terraform Core will still emit warnings into its own logs noting that your provider produced an invalid result, and there may be error messages raised against downstream resources if they have configuration derived from the value of your sensitive_creds argument.
I am going through a terraform guide, where the author is spinning up a docker setup using the docker_image and docker_container resources.
In the sample code the main.tf file includes both the required_providers and the provider blocks, as follows:
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
}
}
}
provider "docker" {}
Why are they both needed?
Shouldn't terraform be able to understand the need for a docker provider, only by this line?
provider "docker" {}
When considering Terraform providers there are two related notions to think about: the provider itself, and a configuration for the provider.
As an analogy, the provider kreuzwerker/docker here is a bit like a class you're importing from another library, giving it the local name docker. I'll use a pseudo-JavaScript syntax just to make this a bit more concrete:
var docker = require("kreuzwerker/docker");
However, all we have here so far is the class itself. In order to use it we need to create an instance of it, which in Terraform's vernacular is called a "configuration". Again, using pseudo-JavaScript syntax:
var dockerInstance = new docker({});
Terraform's syntax here is decidedly less explicit than this pseudo-JavaScript form, but we can make the distinction more visible by adding a second instance of the provider to the configuration, which in Terraform we do by assigning it a configuration "alias":
provider "docker" {
alias = "example"
host = "ssh://user#remote-host:22"
}
This is like creating a second instance of the provider "class" in our pseudo-JavaScript example:
var dockerInstance2 = new docker({
host: 'ssh://user#remote-host:22'
});
Another variant that shows the distinction is when a module inherits a provider configuration from its calling module. In that case, it's as if the calling module were implicitly passing the provider configuration (instance) into the module, but the child module still needs to import the provider "class" so Terraform can see that we're talking about kreuzwerker/docker as opposed to any other provider that might have the name "docker".
Terraform has some automatic "magic" behaviors that try to make simpler cases implicit, but unfortunately that comes at the cost of making it harder to understand what's going on when things get more complicated. Providers and provider configurations are a particularly hard example of this, because providers have been in the Terraform language for a long time and the current incarnation of the language is trying to stay broadly backward-compatible with the simple uses while still allowing for the newer features like having third-party providers installable from multiple namespaces.
The particularly confusing assumption here is that if you don't declare a particular provider Terraform will create an implicit required_providers declaration assuming that you mean a provider in the hashicorp/ namespace, which makes it seem as though required_providers is only for third-party providers. In fact though, that is largely a backward-compatibility mechanism and so I'd suggest always writing out the required_providers entries, even for the providers in the hashicorp/ namespace, so that less-experienced readers don't need to know about this special backward-compatibility behavior. In your case though, the provider you're using is in a third-party namespace anyway and so the required_providers entry is mandatory.
The source needs to be provided since this isn't one of the "official" HashiCorp providers. There could be multiple providers with the name "docker" in the provider registry, so providing the source is needed in order to tell Terraform exactly which provider to download.
I am working with Terraform V11 and AWS provider; I am looking for a way to prevent destroying few resources during the destroy phase. So I used the following approach.
lifecycle {
prevent_destroy = true
}
When I run a "terraform plan" I get the following error.
the plan would destroy this resource, but it currently has
lifecycle.preven_destroy set to true. to avoid this error and continue with the plan.
either disable or adjust the scope.
All that I am looking for is a way to avoid destroying one of the resources and its dependencies during the destroy command.
AFAIK This feature is not yet supported
You need to remove that resource from state file and then reimport it
terraform plan | grep <resource> | grep id
terraform state rm <resource>
terraform destroy
terraform import <resource> <ID>
The easiest way to do this would be to comment out all of the the resources that you want to destroy and then do a terraform apply.
I've found the most practical way to manage this is through a combination of variables that allow the resource in question to be conditionally created or not on via the use of count, alongside having all other resources depend on the associated Data Source instead of the conditionally created resource.
A good example of this is a Route 53 Hosted Zone which can be a pain to destroy and recreate if you manage your domain outside of AWS and need to update your nameservers, waiting for DNS propagation each time you spin it up.
1. By specifying some variable
variable "should_create_r53_hosted_zone" {
type = bool
description = "Determines whether or not a new hosted zone should be created on apply."
}
2. you can use it alongside count on the resource to conditionally create it.
resource "aws_route53_zone" "new" {
count = var.should_create_r53_hosted_zone ? 1 : 0
name = "my.domain.com"
}
3. Then, by following up with a call to the associated Data Source
data "aws_route53_zone" "existing" {
name = "my.domain.com"
depends_on = [
aws_route53_zone.new
]
}
4. you can give all other resources consistent access to the resource's attributes regardless of whether or not your flag has been set.
resource "aws_route53_record" "rds_reader_endpoint" {
zone_id = data.aws_route53_zone.existing.zone_id
# ...
}
This approach is only slightly better than commenting / uncommenting resources during apply, but at least gives some consistent, documented way of working around it.
Ok, so I have three .tf-files: main.tf where I state azure as provider, resources.tf where all the my resources are claimed, and variables.tf.
I use variables.tf to store keys used by resources.tf.
However, I want to use variables stored in my variable file to fill in the fields in the backend scope like this:
main.tf:
provider "azurerm" {
version = "=1.5.0"
}
terraform {
backend "azurerm" {
storage_account_name = "${var.sa_name}"
container_name = "${var.c_name}"
key = "${var.key}"
access_key = "${var.access_key}"
}
}
Variables stored in variables.tf like this:
variable "sa_name" {
default = "myStorageAccount"
}
variable "c_name" {
default = "tfstate"
}
variable "key" {
default = "codelab.microsoft.tfstate"
}
variable "access_key" {
default = "weoghwoep489ug40gu ... "
}
I got this when running terraform init:
terraform.backend: configuration cannot contain interpolations
The backend configuration is loaded by Terraform extremely early,
before the core of Terraform can be initialized. This is necessary
because the backend dictates the behavior of that core. The core is
what handles interpolation processing. Because of this, interpolations
cannot be used in backend configuration.
If you'd like to parameterize backend configuration, we recommend
using partial configuration with the "-backend-config" flag to
"terraform init".
Is there a way of solving this? I really want all my keys/secrets in the same file... and not one key in the main which I preferably want to push to git.
Terraform doesn't care much about filenames: it just loads all .tf files in the current directory and processes them. Names like main.tf, variables.tf, and outputs.tf are useful conventions to make it easier for developers to navigate the code, but they won't have much impact on Terraform's behavior.
The reason you're seeing the error is that you're trying to use variables in a backend configuration. Unfortunately, Terraform does not allow any interpolation (any ${...}) in backends. Quoting from the documentation:
Only one backend may be specified and the configuration may not contain interpolations. Terraform will validate this.
So, you have to either hard-code all the values in your backend, or provide a partial configuration and fill in the rest of the configuration via CLI params using an outside tool (e.g., Terragrunt).
There are some important limitations on backend configuration:
A configuration can only provide one backend block.
A backend block cannot refer to named values (like input variables, locals, or data source attributes).
Terraform backends configurations one can see at below link:
https://www.terraform.io/docs/configuration/backend.html