Our terraform remote state file in Azure has been completely destroyed and we're now faced with the challenge of recreating the state file from scratch. My option is to use the Terraform import command, using the following simple syntax:
terraform import <Terraform Resource Name>.<Resource Label> <Azure Resource ID>
To import the existing resource group for example, I will create the following configuration in a main.tf file.
provider "azurerm" {
version="1.39.0"
}
# create resource group
resource "azurerm_resource_group" "rg"{
name = "rg-terraform"
location = "uksouth"
}
Now, the problem I have is as follows:
When the existing Azure resources were originally created, they were assigned names that used an extremely complex naming convention, with some characters even being randomly generated. To further compound matters, they were all unique and there are hundreds of them. All would have been rosy if they were assigned a simplistic name like "main", as is used commonly in a lot of Terraform examples, but unfortunately, that's not the case.
My question therefore, is this:
When putting together my main.tf configuration file to be used for the Import, is it an absolute requirement that my "Resource Label" (given in my Import command) has to match the original "Resource Label" name from when the resource was created?
If it is a mandatory requirement, is there any way I could retrieve the original "Resource Label" from Azure in the same way that I can for instance obtain the "Azure Resource ID" from the Azure Portal or even an Az CLI query?
How can I ensure any child resources such as Subnets are included in the Import, without having to trawl manually through the Azure Portal to identify each one of them?
No, absolutely not. Choose whatever you want.
No, Azure generally does not know about this label, it is something terraform internal.
Unfortunately you need to import each and every resource manually and separately.
Have you made absolutely sure the current state file is lost? The storage location was not versioned? Does no developer still have a local copy of the state file laying around?
Related
I'm using Terraform to create some resources. One of the side effects of creating the resource is the creation of another resource (let's call this B). The issue is that I can't access B to edit it in terraform because terraform considers it as "out of the state". I can't also import B in the state before the terraform apply is started because B does not exist.
Is there any solution to add (import) a remote resource to the state while running the apply command?
I'm thinking about this as a general question, if there was no solution I can also share the details of the resources I'm creating.
More details:
When I create a "Storage Account" on Azure using Terraform and enable static_website, Azure automatically creates a storage_container named $web. I need to edit one of the attributes of the $web container but Terraform tells me it is not in the current state and needs to be imported. Storage Account is A, Container is B
Unfortunately I do not have an answer to your specific question of importing a resource during an apply. The fundamental premise of Terraform is that it manages resources from creation. Therefore, you need to have a (in this case, azurerm_storage_container) resource declared, before you can import the current state of that resource into your state.
In an ideal world you would be able to explicitly create the container first and specify that the storage account uses that, but a quick look in the docs does not suggest that is an option (and I think is something you have already tried). If it is not exposed in Terraform, that is likely because it is not exposed by the Azure API (Disclaimer: not an Azure user)
The only (bad) answer I can think to suggest, is that you define an azurerm_storage_container data resource in your code, dependent on the the azurerm_storage_account resource, that will be able to pull back the details of the created container. You could then potentially have a null_resource that calls a local-exec provisioner that can fire a CLI command, using the params taken from the data resource to allow you to use the Azure CLI tools to edit the container.
I really hope someone else can come along with a better answer tho :|
I am currently writing a Terraform file which should create all required resources in Azure in a specific resource group. As I do not have the permissions to create resource groups I imported an existing resource group following this link.
Now the problem is that when I run terraform destroy, the system also wants to delete the resource group as it was included as a resource.
Is there a simple way I can tell Terraform to destroy all created resources except the imported resource group? Or another way to approach this problem?
I tried the lifecycle variable prevent_destroy but it prevents any resource from being destroyed.
The solution presented here does not really work for me as I do not want to use multiple commands as this adds the risk of human error. What I am basically looking for is a way to label the resource group in the Terraform file in a way that it is not deleted.
If you want to refer to a resource but not have Terraform manage the lifecycle of it (either at all or just in that specific Terraform workspace) then you can use data sources.
In your case you could replace the azure_resource_group resource with the azure_resource_group data source.
This question is not how to import and it's not what's the purpose of tfstate. It's what's the purpose of importing a pre-existing resource, esp. compared to just referencing the ID of the existing resource?
Terraform has the feature of terraform import. HashiCorp describes the purpose of this as:
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
This is a great way to slowly transition infrastructure to Terraform, or to be able to be confident that you can use Terraform in the future if it potentially doesn't support every feature you need today.
I read the article about the purpose of Terraform state. It does make sense to me to track Terraform state with .tfstate files when those files are mappings back to the configurations in .tf files.
But it's still unclear to me what the purpose of a standalone .tfstate file is when it only maps to an empty resource block. If there is a resource not in terraform yet, I would typically do one of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so, why would you use that approach?
The only way to make changes to an imported resource (that only has an empty resource block in the .tf file and detailed state in .tfstate) is to make manual changes and then re-import into .tfstate`, right? And if so, then what's the point of tracking the state of that resource in terraform?
I'm sure there's a good reasons. Just want to understand this deeper! Thanks!
But it's still unclear to me what the purpose of a standalone .tfstate
file is when it only maps to an empty resource block.
You wouldn't use a standalone .tfstate file. You would be using the same .tfstate file that all your other resources are in.
If there is a resource not in terraform yet, I would typically do one
of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for
terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so,
why would you use that approach?
Consider the case where you have a production database with terrabytes of data already load in it, and users actively performing actions that query that database 24 hours a day. Your option 1 would require some down time, possibly a lot of down time, because you would have to deal with backing up and restoring terrabytes of data. Your option 2 would never let you manage changes to your database server via Terraform. That's what the Terraform import feature solves. It lets Terraform take "full control" of resources that already exist, without having to recreate them.
I agree that if a system outage is not an issue, and if recreating a resource isn't going to take much time, using option 1 is the way to go. Option 2 is only for resources that you never want to fully manage in Terraform, which is really a separate issue from the one Terraform import solves.
When importing a resource with terraform import it is necessary to write the configuration block to manage it with Terraform. On the same page you linked it states:
The current implementation of Terraform import can only import resources into the state. It does not generate configuration. A future version of Terraform will also generate configuration.
Because of this, prior to running terraform import it is necessary to
write manually a resource configuration block for the resource, to
which the imported object will be mapped.
So to bring preexisting resources under Terraform management, you first write the resource block for it in a .tf file. Next you use terraform import to map the resource to this resource block in your .tfstate. The next time you run terraform plan, Terraform will determine what changes (if any) will need to be made upon the next terraform apply based on the resource block and the actual state of the resource.
EDIT
The "why" of terraform import is to manage resources that are previously unknown to Terraform. As you alluded to in your second bullet point, if you want metadata from a resource but do not want to change the configuration of the resource, you would use a data block and reference that in dependent resources.
When you want to manage the configuration of a resource that was provisioned outside of Terraform you use terraform import. If you tear down the resource there may be data loss or service downtime until you re-deploy with Terraform, but if you use terraform import the resource will be preserved.
The import process can be started with an empty resource block, but the attributes need to be filled out to describe the resource. You will get the benefits of terraform plan after importing, which can help you find the discrepancies between the resource block and the actual state of the resource. Once the two match up, you can continue to make additional changes to the resource like any other resource in Terraform.
Terraform state file is your source of truth for your cloud infrastructure. Terraform uses local state to create plans and make changes to the infrastructure. Before any terraform operation, terraform does a refresh to update the state with the real infrastructure.
When Terraform run task executes in azure devops release pipeline I get an error "A resource with the ID already exists".
The resource exists in Azure but why it is complaining about the resource if this already exists. This should ignore this part. Please help what I need to add in my code that will fix this error!
Am I just using this bugging terraform tool for deploying azure resource? Terraform help is terrible!!!
resource "azurerm_resource_group" "test_project" {
name = "${var.project_name}-${var.environment}-rg"
location = "${var.location}"
tags = {
application = "${var.project_name}"
}
}
Terraform is designed to allow you to manage only a subset of your infrastructure with a particular Terraform configuration, in case either some objects are managed by another tool or in case you've decomposed your infrastructure to be managed by many separate configurations that cooperate to produce the desired result.
As part of that design, Terraform makes a distinction between an object existing in the remote system and that object being managed by the current Terraform configuration. Where technical constraints of an underlying API allow it, Terraform providers will avoid implicitly taking ownership of something that was not created by that specific Terraform configuration. The error message you saw here is the Azure provider's implementation of that, where it pre-checks to make sure the name you give it is unique so that it won't overwrite (and thus take implicit ownership of) an object created elsewhere.
To proceed here you have two main options, depending on your intended goal:
If this object was formerly managed by some other system and you now want to manage it exclusively with this Terraform configuration, you can tell Terraform to associate the existing object with the resource block you've written and thus behave as if that object were originally created by that resource block:
terraform import azurerm_resource_group.test_project /subscriptions/YOUR-SUBSCRIPTION-ID/resourceGroups/PROJECTNAME-ENVIRONMENTNAME-rg
After you run terraform import you must ensure that whatever was previously managing that object will no longer associate with it. This object is now owned by this Terraform configuration and must not be changed by any other system.
If this object is managed by some other system and you wish to continue managing it that way then you can instead use a data block to retrieve information about that existing object to use elsewhere in your configuration without Terraform taking ownership:
data "azurerm_resource_group" "example" {
name = "${var.project_name}-${var.environment}-rg"
}
If you needed the resource group's location name elsewhere in your module, for example, you could use data.azurerm_resource_group.example.location to access it. If you wanted to make any later changes to this resource group, you would continue to do that using whichever other system is considered the owner of it in your environment.
The main difference between these two approaches is how Terraform will record the object in state snapshots. terraform import causes Terraform to create a binding between the resource configuration you wrote and the remote object whose id you gave on the command line, which is henceforth indistinguishable to Terraform from it having created that object and recorded the binding itself in the first place. For a data resource, Terraform just reads the data about the existing object and saves a cache of it in the state so it can determine if the value has changed on a future run; it will never plan to make any modifications to an object used with a data block.
Try to delete the .terraform local folder to clean the cache, then run terraform init again and retry running the pipeline.
For my future self:
Today I stumbled across this same problem, because I renamed some resources, and terraform could not track them. I found out about terraform state mv ... which gives you the ability to rename resources in your state file, so that it can track remote resources. Really useful.
I have a hosted zone in the main.tf:
provider "aws" {
region = "us-east-1"
}
resource "aws_route53_zone" "zone" {
}
I then can import an existing resource and use its parameters in other resources:
terraform import aws_route53_zone.zone <ZoneId>
Inspecting the state file I see the parameters are all there, including the domain name. But when I want to apply it it says that name is not found:
Error: aws_route53_zone.zone: "name": required field is not set
I don't want to specify the name in the .tf file as it would decrease the portability of my .tf, but specifying a placeholder would change the hosted zone itself.
Is there a way to ignore parameters for imported resources or specify them as "leave as-is"?
I could add a variable and populate that from the state file for every terraform calls, but I'm hoping for something simpler.
When you import a resource, Terraform doesn't (yet) automatically generate Terraform code for you and instead you must write the resource and then check the plan.
Normally the pattern would be to create a skeleton resource like you've done, import the resource and then fill out any required fields, run a plan, then adjust your resource configuration so that it doesn't make any unwanted changes.
From then on Terraform will be able to manage the resource as normal, applying any changes you make to the configuration or reverting changes done outside of Terraform back to how they are done in your Terraform code.