I am using Terraform for creating resources within Azure and attaching current time-stamp for enforcing unique names for resources like storage account, key vault etc in the .tf file.
Problem is, for existing resources, when you expect Terraform apply to do nothing, these resources are being destroyed and recreated since the time-stamp has changed between last execution and the current one!
Wondering what's the best strategy to enforce uniqueness, dynamically.
The initial texts for these resources come from Azure Pipeline variables and I'm appending current time-stamp for adding uniqueness.
According to Cloud Adoption framework you could define a convention like the one shown below.
Terraform example
name = "${local.prefix}-${var.location}-app-plan"
https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming
I also suggest you to look the naming azure plugin which can help you create unique names across resources
https://registry.terraform.io/modules/Azure/naming/azurerm/latest
Related
I have a lot of resources in my Azure subscription. I want to manage them with terraform, so I want to import them using terraform import. I import every resource manually one-by-one. Then I run terraform plan and check that there are no changes to be made reported i.e. that the current infrastructure matches the configuration.
Does this mean that if I were to manually delete some of the resources via Azure portal or cli, I would be able to recreate them wit terraform apply perfectly so that they would have exactly the same configuration as before and would operate in exactly the same way?
In general Terraform cannot guarantee that destroying an object and recreating it will produce an exactly equivalent object.
It is possible for that to work, but it requires a number of things to be true, including:
Your configuration specifies the values for resource arguments exactly as they are in the remote API. For example, if a particular resource type has a case-insensitive (but case-preserving) name then a provider will typically ignore differences in case when planning changes but it will use exactly the case you wrote in the configuration, potentially selecting a different name.
The resource type does not include any "write-only" arguments. Some resource types have arguments that are used only by the provider itself and so they don't get saved as part of the object in the remote API even though they are saved in the Terraform state. terraform import therefore cannot re-populate those into the state, because there is nowhere else to read them from except the Terraform state.
The provider doesn't have any situations where it treats an omitted argument as "ignore the value in the remote system" instead of "unset the value in the remote system". Some providers make special exceptions for certain arguments where leaving them unset allows them to "drift" in the remote API without Terraform repairing them, but if you are using any resource types which behave in that way then the value stored in the remote system will be lost when you delete the remote object and Terraform won't be able to restore that value because it's not represented in your Terraform configuration.
The hashicorp/azurerm provider in particular has many examples of situation 3 in the above list. For example, if you have an azurerm_virtual_network resource which does not include any subnet blocks then the provider will not propose to delete any existing subnets, even though the configuration says that there should be no subnets. However, if you delete the virtual network and then ask Terraform to recreate it then the Terraform configuration has no record of what subnets were supposed to exist and so it will propose to create a network with no subnets at all.
I have a Resource Group where many of the resources are managed via terraform. However, in recent days, I have noticed there are some resources created manually by the users. Is there an easy way to figure out or make a list of resources that are not managed via terraform?
Terraform only tracks the objects that are connected to Terraform resources, so I think you would need to approach this analysis by asking the following questions:
What are all of the objects in the resource group? Call that set of resources A.
What are all of the objects tracked by Terraform? Call that set of resources B.
What is left if we subtract set B from set A? That's your result.
The main challenge here is that set of resources A is expressed only in Azure's terms and set of resources B is expressed in Terraform's terms, so it will likely take some custom analysis to get them both converted into a form where it's possible to decide question 3 above.
Fortunately, the hashicorp/azurerm provider is pretty consistent in putting the fully-qualified Azure resource ID in the id field of most resource types, and so if you use terraform show -json to get a JSON representation of what's saved in the state, and then use some scripting of your own to extract just the id attribute from any resource whose type name starts with azurerm_, that should hopefully give you a reasonable answer for set of resources B as a set of resource IDs.
Then you can hopefully use the Azure API to retrieve all of the objects belonging to your resource group and take the id of each result to get set of resources A.
There is no ready-to-run answer to this, because Terraform is intentionally designed not to interact with objects it isn't managing unless explicitly asked to using terraform import, but hopefully the above is enough building blocks to construct a solution which is sufficient for your needs.
I'm using Terraform to create some resources. One of the side effects of creating the resource is the creation of another resource (let's call this B). The issue is that I can't access B to edit it in terraform because terraform considers it as "out of the state". I can't also import B in the state before the terraform apply is started because B does not exist.
Is there any solution to add (import) a remote resource to the state while running the apply command?
I'm thinking about this as a general question, if there was no solution I can also share the details of the resources I'm creating.
More details:
When I create a "Storage Account" on Azure using Terraform and enable static_website, Azure automatically creates a storage_container named $web. I need to edit one of the attributes of the $web container but Terraform tells me it is not in the current state and needs to be imported. Storage Account is A, Container is B
Unfortunately I do not have an answer to your specific question of importing a resource during an apply. The fundamental premise of Terraform is that it manages resources from creation. Therefore, you need to have a (in this case, azurerm_storage_container) resource declared, before you can import the current state of that resource into your state.
In an ideal world you would be able to explicitly create the container first and specify that the storage account uses that, but a quick look in the docs does not suggest that is an option (and I think is something you have already tried). If it is not exposed in Terraform, that is likely because it is not exposed by the Azure API (Disclaimer: not an Azure user)
The only (bad) answer I can think to suggest, is that you define an azurerm_storage_container data resource in your code, dependent on the the azurerm_storage_account resource, that will be able to pull back the details of the created container. You could then potentially have a null_resource that calls a local-exec provisioner that can fire a CLI command, using the params taken from the data resource to allow you to use the Azure CLI tools to edit the container.
I really hope someone else can come along with a better answer tho :|
This question is not how to import and it's not what's the purpose of tfstate. It's what's the purpose of importing a pre-existing resource, esp. compared to just referencing the ID of the existing resource?
Terraform has the feature of terraform import. HashiCorp describes the purpose of this as:
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
This is a great way to slowly transition infrastructure to Terraform, or to be able to be confident that you can use Terraform in the future if it potentially doesn't support every feature you need today.
I read the article about the purpose of Terraform state. It does make sense to me to track Terraform state with .tfstate files when those files are mappings back to the configurations in .tf files.
But it's still unclear to me what the purpose of a standalone .tfstate file is when it only maps to an empty resource block. If there is a resource not in terraform yet, I would typically do one of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so, why would you use that approach?
The only way to make changes to an imported resource (that only has an empty resource block in the .tf file and detailed state in .tfstate) is to make manual changes and then re-import into .tfstate`, right? And if so, then what's the point of tracking the state of that resource in terraform?
I'm sure there's a good reasons. Just want to understand this deeper! Thanks!
But it's still unclear to me what the purpose of a standalone .tfstate
file is when it only maps to an empty resource block.
You wouldn't use a standalone .tfstate file. You would be using the same .tfstate file that all your other resources are in.
If there is a resource not in terraform yet, I would typically do one
of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for
terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so,
why would you use that approach?
Consider the case where you have a production database with terrabytes of data already load in it, and users actively performing actions that query that database 24 hours a day. Your option 1 would require some down time, possibly a lot of down time, because you would have to deal with backing up and restoring terrabytes of data. Your option 2 would never let you manage changes to your database server via Terraform. That's what the Terraform import feature solves. It lets Terraform take "full control" of resources that already exist, without having to recreate them.
I agree that if a system outage is not an issue, and if recreating a resource isn't going to take much time, using option 1 is the way to go. Option 2 is only for resources that you never want to fully manage in Terraform, which is really a separate issue from the one Terraform import solves.
When importing a resource with terraform import it is necessary to write the configuration block to manage it with Terraform. On the same page you linked it states:
The current implementation of Terraform import can only import resources into the state. It does not generate configuration. A future version of Terraform will also generate configuration.
Because of this, prior to running terraform import it is necessary to
write manually a resource configuration block for the resource, to
which the imported object will be mapped.
So to bring preexisting resources under Terraform management, you first write the resource block for it in a .tf file. Next you use terraform import to map the resource to this resource block in your .tfstate. The next time you run terraform plan, Terraform will determine what changes (if any) will need to be made upon the next terraform apply based on the resource block and the actual state of the resource.
EDIT
The "why" of terraform import is to manage resources that are previously unknown to Terraform. As you alluded to in your second bullet point, if you want metadata from a resource but do not want to change the configuration of the resource, you would use a data block and reference that in dependent resources.
When you want to manage the configuration of a resource that was provisioned outside of Terraform you use terraform import. If you tear down the resource there may be data loss or service downtime until you re-deploy with Terraform, but if you use terraform import the resource will be preserved.
The import process can be started with an empty resource block, but the attributes need to be filled out to describe the resource. You will get the benefits of terraform plan after importing, which can help you find the discrepancies between the resource block and the actual state of the resource. Once the two match up, you can continue to make additional changes to the resource like any other resource in Terraform.
Terraform state file is your source of truth for your cloud infrastructure. Terraform uses local state to create plans and make changes to the infrastructure. Before any terraform operation, terraform does a refresh to update the state with the real infrastructure.
I have an Azure terraform configuration. It sets up resource groups, key vaults, passwords, etc ...
When I destroy it terraform does the same in reverse - deleting secrets, access polices, key vaults and the last are resource groups.
But, if the resource groups are to be destroyed anyway, it makes sense just to destroy them first - all the child resources will be deleted automatically. But the azurerm provider does not do it this way.
What am I missing here? And if my understanding is correct, is there a way to implement it (without altering the provider, that is) ?
In Terraform's model, each resource is distinct. Although Terraform can see the dependencies you've defined or implied between them, it doesn't actually understand that e.g. a key vault is a child object of a resource group and so the key vault might be deleted as a side-effect of deleting the resource group.
With that said, unfortunately there is no built-in way in Terraform today to achieve the result you are looking for.
A manual approximation of the idea would be to use terraform state rm to tell Terraform to "forget" about each of the objects (that is, they will still exist in Azure but Terraform will have no record of them) that will eventually be destroyed as a side-effect of deleting the resource group anyway, and then running terraform destroy will only delete the resource group, because Terraform will believe that none of the other objects exist yet anyway. However, that is of course a very manual approach that, without some careful scripting, would likely take longer than just letting the Azure provider work through all of the objects in dependency order.
There is an exploratory issue in the Terraform repository that covers this use-case (disclaimer: I'm the author of that issue), but the Terraform team isn't actively working on that at the time I write this, because efforts are focused elsewhere. The current set of use-cases captured there doesn't quite capture your idea here of having Terraform recognize when it can skip certain destroy operations, so you might choose to share some details about your use-case on that issue to help inform potential future design efforts.
Terraform is built this way, it wouldn't traverse the graph and understand that if the resource group is deleted - anything inside resource group will be deleted as well. which isn't even true in some cases. So I would say it doesn't make sense to do that.
Only real time when this is annoying - when you are testing. for that time you can create a script that would initiate resource group deletion and clear local state, for example