Initially resources in our authentication provider were created manually through the provider web console. It worked and things went to production this way. The problem is that the configuration is increasing in complexity and I'd like to manage it through terraform files instead of continuing through the provider Web console (no backup, no way to recreate everything easily , etc.)
I initially thought of modelling my configuration with data block for the existing resources and use new resources block for the new resources we need to create. Then I wanted to migrate from the data blocks to terraform managed resources (aka resource block). Is it possible through moved block or something else? Is it possible to do it without having to recreate a new managed resource and destroy the data resource which is most likely to cause down time or disruption to the end-user?
In order to manage the resources which were initially created manually or out of terraform scope by any means, Terraform cli offers import as a native solution by Hashicorp.
Every resource has its own way of importing syntax (starting with terraform import ) which you can find at the bottom of any terraform resource definition.
As an example:
Azurerm windows_virtual_machine Import
terraform import azurerm_windows_virtual_machine.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1/providers/Microsoft.Compute/virtualMachines/machine1
Downside of Native import: You have to import all resources one by one and sometimes just for one resource(solution) you have to make multiple import calls
as an example for a windows virtual machine, you might import
azurerm_virtual_machine_extension
azurerm_managed_disk
azurerm_virtual_machine_data_disk_attachment
as separate. It strongly depends on how would you like them to manage them at the end.
BUT
There are few open-source tools available that help If you have lots of resources that you want to bring under terraform management in a lot easier and faster way.
If you working with Azure resources then aztfy is the recommended tool as it is natively from Azure.
It does generate the terraform code, additionally, it has a feature where you can import the azure resource group, it automatically imports and generates config for the resources that the resource group is holding. Not to mention but the tool gives you a nice terminal-based-UI experience.
For other hyperscalers, there are two choices.
terracognita: can generate modules too as per their docs.
terraformer: Developed by Google people but not an official product.
Related
I'm using Terraform to create some resources. One of the side effects of creating the resource is the creation of another resource (let's call this B). The issue is that I can't access B to edit it in terraform because terraform considers it as "out of the state". I can't also import B in the state before the terraform apply is started because B does not exist.
Is there any solution to add (import) a remote resource to the state while running the apply command?
I'm thinking about this as a general question, if there was no solution I can also share the details of the resources I'm creating.
More details:
When I create a "Storage Account" on Azure using Terraform and enable static_website, Azure automatically creates a storage_container named $web. I need to edit one of the attributes of the $web container but Terraform tells me it is not in the current state and needs to be imported. Storage Account is A, Container is B
Unfortunately I do not have an answer to your specific question of importing a resource during an apply. The fundamental premise of Terraform is that it manages resources from creation. Therefore, you need to have a (in this case, azurerm_storage_container) resource declared, before you can import the current state of that resource into your state.
In an ideal world you would be able to explicitly create the container first and specify that the storage account uses that, but a quick look in the docs does not suggest that is an option (and I think is something you have already tried). If it is not exposed in Terraform, that is likely because it is not exposed by the Azure API (Disclaimer: not an Azure user)
The only (bad) answer I can think to suggest, is that you define an azurerm_storage_container data resource in your code, dependent on the the azurerm_storage_account resource, that will be able to pull back the details of the created container. You could then potentially have a null_resource that calls a local-exec provisioner that can fire a CLI command, using the params taken from the data resource to allow you to use the Azure CLI tools to edit the container.
I really hope someone else can come along with a better answer tho :|
We are migrating from doing Azure deployments via ARM to using Terraform across dev teams. We're looking at ways to migrate the current deployments into TF state, as there are a large number of resource groups across many subscriptions, each with a number of resources. Most existing deployments use monolithic ARM templates (thousands of lines in some cases) with a number of dependencies, so deploying from scratch is not an option at the current time for some teams.
Trying to use tf import 1-by-1 is problematic because each of these resource groups has multiple environments, so with the number of things to move it'll be a huge undertaking. There's also the difficuly that is looking up Azure resource ID's for each one.
To do an import in a more straightforward and batch-like way, we've looked at terraformer but that has some shortcomings as well. Not all Azure resources are supported, and while we could contribute to the code to add them there seems to be a bigger problem. Terraformer wants to interrogate all deployed resources for the subscription, then filter by what you're asking it to do after. The number of resources deployed seems to overwhelm the program and it starts spitting errors. We've also seen Az2TF but that stopped development with the azurerm provider 1.44.
Has anyone done a large tf migration like this that could point us in another direction?
You can store into multiple tfstates
One environnement, one tfstate
### Import - ENV1 Prod
terraform import -var-file="environment/prod/env_terraform.tfvars" -state=tfstates/prod/terraform.tfstate azurerm_monitor_autoscale_setting.VMSS_autoscale /subscriptions/XXXX/resourceGroups/my_resourceGroupsPROD/providers/Microsoft.insights/autoscalesettings/my_autoscale
### Import - ENV2 No Prod
terraform import -var-file="environment/nopro/env_terraform.tfvars" -state=tfstates/noprod/terraform.tfstate azurerm_monitor_autoscale_setting.VMSS_autoscale /subscriptions/XXXX/resourceGroups/my_resourceGroupsNOPROD/providers/Microsoft.insights/autoscalesettings/my_autoscale
Then, when you apply, you must specify the file
### Apply - ENV2 No Prod
terraform apply -var-file="environment/noprod/env_terraform.tfvars" -var="action=charge" -state=tfstates/noprod/terraform.tfstate
If I have existing cloudflare settings (using web console), how to import/download existing cloudflare resource(s),e.g DNS records using terraform?
So, I could use it as a starting point to modify/update it.
Thanks.
Each Terraform resource has an example of importing existing data. For example, the record import
$ terraform import cloudflare_record.default ae36f999674d196762efcc5abb06b345/d41d8cd98f00b204e9800998ecf8427e
You can use cf-terraforming tool to terraforming existing CF resources.
Your steps will look like this:
Describe a Terraform configuration that contains the resources you want to manage with Terraform.
Initialize the created configuration using terraform init and after (this is convenient) quickly get the addresses of the resources that you need for import using terraform plan.
Next, you need to get the IDs of existing resources, they will also be required for import. For the Cloudflare Zone resource, this is easy to do, just refer to this instruction, for other resources, for example, for Cloudflare record you will either need to use cf-terraforming or Cloudflare API.
Import your resources using the terraform import RESOURCE_ADDRESS RESOURCE_ID command.
Another way is to delete your resources and recreate them with Terraform instead of importing them. This can make your infrastructure unavailable for some time.
I highly recommend that you read these guides:
https://developers.cloudflare.com/terraform/advanced-topics/import-cloudflare-resources/.
https://github.com/alex-feel/terraform-cloudflare-zone/wiki/Migration (it is written for a module, but is equally applicable to "raw" resources).
When I run terraform apply -auto-approve I get the following error:
Error: A resource with the ID "/subscriptions/.../resourceGroups/RG-SCUSTFStorage" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.
I underestand that I need to run terraform import to import the resource to my worksapce. The problem is that I need to specify the resource id for all of the missing resources one at a time.
Is any way to have terraform import import all of the "already exists " resources automatically witout entering the resource IDs one at a time?
Unfortunately, you only can import the existing resources one by one with the resource IDs manually:
The import command doesn’t automatically generate the configuration to
manage the infrastructure, though. Because of this, importing existing
infrastructure into Terraform is a multi-step process.
More details here. I will suggest you use remote state storage for all the Terraform scripts before deployment. If you do not have the state file that contains all the deployed resources, then you only can import them one by one.
If you are looking to import Azure resources then aztfy is the recommended tool as it is natively from Azure.
It does generate the terraform code, additionally, it has a feature where you can import the azure resource group, it automatically imports and generates config for the resources that the resource group is holding.
Not to mention but the tool gives you a nice terminal-based-UI experience.
For other hyperscalers, there are two choices.
terracognita : can generate modules too as per their docs.
terraformer : Developed by Google people but not official product.
There isn't a native way in Terraform to import already existing resources, however, there are a couple of tools available that allow you to not only import the resources but also generate Terraform code for them if it doesn't already exist.
For Azure, the best tool to use is Azure-built AZtfy. A tool to bring your existing Azure resources under the management of Terraform.
Another tool to that can be used to import Azure resources is Google Cloud Terraformer which supports Azure.
This question is not how to import and it's not what's the purpose of tfstate. It's what's the purpose of importing a pre-existing resource, esp. compared to just referencing the ID of the existing resource?
Terraform has the feature of terraform import. HashiCorp describes the purpose of this as:
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
This is a great way to slowly transition infrastructure to Terraform, or to be able to be confident that you can use Terraform in the future if it potentially doesn't support every feature you need today.
I read the article about the purpose of Terraform state. It does make sense to me to track Terraform state with .tfstate files when those files are mappings back to the configurations in .tf files.
But it's still unclear to me what the purpose of a standalone .tfstate file is when it only maps to an empty resource block. If there is a resource not in terraform yet, I would typically do one of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so, why would you use that approach?
The only way to make changes to an imported resource (that only has an empty resource block in the .tf file and detailed state in .tfstate) is to make manual changes and then re-import into .tfstate`, right? And if so, then what's the point of tracking the state of that resource in terraform?
I'm sure there's a good reasons. Just want to understand this deeper! Thanks!
But it's still unclear to me what the purpose of a standalone .tfstate
file is when it only maps to an empty resource block.
You wouldn't use a standalone .tfstate file. You would be using the same .tfstate file that all your other resources are in.
If there is a resource not in terraform yet, I would typically do one
of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for
terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so,
why would you use that approach?
Consider the case where you have a production database with terrabytes of data already load in it, and users actively performing actions that query that database 24 hours a day. Your option 1 would require some down time, possibly a lot of down time, because you would have to deal with backing up and restoring terrabytes of data. Your option 2 would never let you manage changes to your database server via Terraform. That's what the Terraform import feature solves. It lets Terraform take "full control" of resources that already exist, without having to recreate them.
I agree that if a system outage is not an issue, and if recreating a resource isn't going to take much time, using option 1 is the way to go. Option 2 is only for resources that you never want to fully manage in Terraform, which is really a separate issue from the one Terraform import solves.
When importing a resource with terraform import it is necessary to write the configuration block to manage it with Terraform. On the same page you linked it states:
The current implementation of Terraform import can only import resources into the state. It does not generate configuration. A future version of Terraform will also generate configuration.
Because of this, prior to running terraform import it is necessary to
write manually a resource configuration block for the resource, to
which the imported object will be mapped.
So to bring preexisting resources under Terraform management, you first write the resource block for it in a .tf file. Next you use terraform import to map the resource to this resource block in your .tfstate. The next time you run terraform plan, Terraform will determine what changes (if any) will need to be made upon the next terraform apply based on the resource block and the actual state of the resource.
EDIT
The "why" of terraform import is to manage resources that are previously unknown to Terraform. As you alluded to in your second bullet point, if you want metadata from a resource but do not want to change the configuration of the resource, you would use a data block and reference that in dependent resources.
When you want to manage the configuration of a resource that was provisioned outside of Terraform you use terraform import. If you tear down the resource there may be data loss or service downtime until you re-deploy with Terraform, but if you use terraform import the resource will be preserved.
The import process can be started with an empty resource block, but the attributes need to be filled out to describe the resource. You will get the benefits of terraform plan after importing, which can help you find the discrepancies between the resource block and the actual state of the resource. Once the two match up, you can continue to make additional changes to the resource like any other resource in Terraform.
Terraform state file is your source of truth for your cloud infrastructure. Terraform uses local state to create plans and make changes to the infrastructure. Before any terraform operation, terraform does a refresh to update the state with the real infrastructure.