I have an existing AWS infrastructure which has more than 1000 subnets. We are to start infrastructure automation using Terraform and use it against the existing infra for further infra deployments on top of the existing one.
Is there an easy method to import 1000+ subnets in such a way that after I import, it should understand the existing resources and then apply (add/delete) on top of the existing infra?
I did try terraformer (https://github.com/GoogleCloudPlatform/terraformer/blob/master/docs/aws.md). However, when the statefile doesn't seem to recognise the same.
Related
Initially resources in our authentication provider were created manually through the provider web console. It worked and things went to production this way. The problem is that the configuration is increasing in complexity and I'd like to manage it through terraform files instead of continuing through the provider Web console (no backup, no way to recreate everything easily , etc.)
I initially thought of modelling my configuration with data block for the existing resources and use new resources block for the new resources we need to create. Then I wanted to migrate from the data blocks to terraform managed resources (aka resource block). Is it possible through moved block or something else? Is it possible to do it without having to recreate a new managed resource and destroy the data resource which is most likely to cause down time or disruption to the end-user?
In order to manage the resources which were initially created manually or out of terraform scope by any means, Terraform cli offers import as a native solution by Hashicorp.
Every resource has its own way of importing syntax (starting with terraform import ) which you can find at the bottom of any terraform resource definition.
As an example:
Azurerm windows_virtual_machine Import
terraform import azurerm_windows_virtual_machine.example /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1/providers/Microsoft.Compute/virtualMachines/machine1
Downside of Native import: You have to import all resources one by one and sometimes just for one resource(solution) you have to make multiple import calls
as an example for a windows virtual machine, you might import
azurerm_virtual_machine_extension
azurerm_managed_disk
azurerm_virtual_machine_data_disk_attachment
as separate. It strongly depends on how would you like them to manage them at the end.
BUT
There are few open-source tools available that help If you have lots of resources that you want to bring under terraform management in a lot easier and faster way.
If you working with Azure resources then aztfy is the recommended tool as it is natively from Azure.
It does generate the terraform code, additionally, it has a feature where you can import the azure resource group, it automatically imports and generates config for the resources that the resource group is holding. Not to mention but the tool gives you a nice terminal-based-UI experience.
For other hyperscalers, there are two choices.
terracognita: can generate modules too as per their docs.
terraformer: Developed by Google people but not an official product.
This would appear to be a fairly simple and basic scenario but I'm frankly at a loss on how to get around this using Terraform and would appreciate any suggestions.
The issue is this. In Azure, I have a number of resource groups, each containing a number of resources, including virtual networks, subnets, storage accounts, etc. What I would now like to do is add new resources to one or two of the resource groups. Typical example, I would like to provision a new virtual machine in each of the resource groups.
Now, so far all of the documentation and blogs I seem to come across only provide guidance on how to create resources whereby you also create a new resource group, vnet, subnet, from scratch. This is definitely not what I wish to do.
All I'm looking to do is get Terraform to add a single virtual machine to an existing resource group, going on to configure it to connect to existing networking resources such as a VNet, Subnet, etc. Any ideas?
I tested for ECS by destroying the launch configuration.
terraform destroy -target module.ecs.module.ec2_alb.aws_launch_configuration.launchcfg
I recreated the launch configuration and it worked:
terraform plan -target=module.ecs.module.ec2_alb.aws_launch_configuration
terraform apply -target=module.ecs.module.ec2_alb.aws_launch_configuration
Also, you can go read more on Terraform target here: https://learn.hashicorp.com/tutorials/terraform/resource-targeting
If you just want to be able to reference your existing resources in your TF script, you normally would use data sources in TF to fetch their information.
So for resource group, you would use data source azurerm_resource_group, for vnet there is azurerm_virtual_network and so forth.
These data sources would allow you to only reference and get details of existing resources, not to manage them in your TF script. Thus if you would like to actually manage these resources using TF (modify, delete, etc), you would have to import them first to TF.
We are migrating from doing Azure deployments via ARM to using Terraform across dev teams. We're looking at ways to migrate the current deployments into TF state, as there are a large number of resource groups across many subscriptions, each with a number of resources. Most existing deployments use monolithic ARM templates (thousands of lines in some cases) with a number of dependencies, so deploying from scratch is not an option at the current time for some teams.
Trying to use tf import 1-by-1 is problematic because each of these resource groups has multiple environments, so with the number of things to move it'll be a huge undertaking. There's also the difficuly that is looking up Azure resource ID's for each one.
To do an import in a more straightforward and batch-like way, we've looked at terraformer but that has some shortcomings as well. Not all Azure resources are supported, and while we could contribute to the code to add them there seems to be a bigger problem. Terraformer wants to interrogate all deployed resources for the subscription, then filter by what you're asking it to do after. The number of resources deployed seems to overwhelm the program and it starts spitting errors. We've also seen Az2TF but that stopped development with the azurerm provider 1.44.
Has anyone done a large tf migration like this that could point us in another direction?
You can store into multiple tfstates
One environnement, one tfstate
### Import - ENV1 Prod
terraform import -var-file="environment/prod/env_terraform.tfvars" -state=tfstates/prod/terraform.tfstate azurerm_monitor_autoscale_setting.VMSS_autoscale /subscriptions/XXXX/resourceGroups/my_resourceGroupsPROD/providers/Microsoft.insights/autoscalesettings/my_autoscale
### Import - ENV2 No Prod
terraform import -var-file="environment/nopro/env_terraform.tfvars" -state=tfstates/noprod/terraform.tfstate azurerm_monitor_autoscale_setting.VMSS_autoscale /subscriptions/XXXX/resourceGroups/my_resourceGroupsNOPROD/providers/Microsoft.insights/autoscalesettings/my_autoscale
Then, when you apply, you must specify the file
### Apply - ENV2 No Prod
terraform apply -var-file="environment/noprod/env_terraform.tfvars" -var="action=charge" -state=tfstates/noprod/terraform.tfstate
If I have existing cloudflare settings (using web console), how to import/download existing cloudflare resource(s),e.g DNS records using terraform?
So, I could use it as a starting point to modify/update it.
Thanks.
Each Terraform resource has an example of importing existing data. For example, the record import
$ terraform import cloudflare_record.default ae36f999674d196762efcc5abb06b345/d41d8cd98f00b204e9800998ecf8427e
You can use cf-terraforming tool to terraforming existing CF resources.
Your steps will look like this:
Describe a Terraform configuration that contains the resources you want to manage with Terraform.
Initialize the created configuration using terraform init and after (this is convenient) quickly get the addresses of the resources that you need for import using terraform plan.
Next, you need to get the IDs of existing resources, they will also be required for import. For the Cloudflare Zone resource, this is easy to do, just refer to this instruction, for other resources, for example, for Cloudflare record you will either need to use cf-terraforming or Cloudflare API.
Import your resources using the terraform import RESOURCE_ADDRESS RESOURCE_ID command.
Another way is to delete your resources and recreate them with Terraform instead of importing them. This can make your infrastructure unavailable for some time.
I highly recommend that you read these guides:
https://developers.cloudflare.com/terraform/advanced-topics/import-cloudflare-resources/.
https://github.com/alex-feel/terraform-cloudflare-zone/wiki/Migration (it is written for a module, but is equally applicable to "raw" resources).
This question is not how to import and it's not what's the purpose of tfstate. It's what's the purpose of importing a pre-existing resource, esp. compared to just referencing the ID of the existing resource?
Terraform has the feature of terraform import. HashiCorp describes the purpose of this as:
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
This is a great way to slowly transition infrastructure to Terraform, or to be able to be confident that you can use Terraform in the future if it potentially doesn't support every feature you need today.
I read the article about the purpose of Terraform state. It does make sense to me to track Terraform state with .tfstate files when those files are mappings back to the configurations in .tf files.
But it's still unclear to me what the purpose of a standalone .tfstate file is when it only maps to an empty resource block. If there is a resource not in terraform yet, I would typically do one of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so, why would you use that approach?
The only way to make changes to an imported resource (that only has an empty resource block in the .tf file and detailed state in .tfstate) is to make manual changes and then re-import into .tfstate`, right? And if so, then what's the point of tracking the state of that resource in terraform?
I'm sure there's a good reasons. Just want to understand this deeper! Thanks!
But it's still unclear to me what the purpose of a standalone .tfstate
file is when it only maps to an empty resource block.
You wouldn't use a standalone .tfstate file. You would be using the same .tfstate file that all your other resources are in.
If there is a resource not in terraform yet, I would typically do one
of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for
terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so,
why would you use that approach?
Consider the case where you have a production database with terrabytes of data already load in it, and users actively performing actions that query that database 24 hours a day. Your option 1 would require some down time, possibly a lot of down time, because you would have to deal with backing up and restoring terrabytes of data. Your option 2 would never let you manage changes to your database server via Terraform. That's what the Terraform import feature solves. It lets Terraform take "full control" of resources that already exist, without having to recreate them.
I agree that if a system outage is not an issue, and if recreating a resource isn't going to take much time, using option 1 is the way to go. Option 2 is only for resources that you never want to fully manage in Terraform, which is really a separate issue from the one Terraform import solves.
When importing a resource with terraform import it is necessary to write the configuration block to manage it with Terraform. On the same page you linked it states:
The current implementation of Terraform import can only import resources into the state. It does not generate configuration. A future version of Terraform will also generate configuration.
Because of this, prior to running terraform import it is necessary to
write manually a resource configuration block for the resource, to
which the imported object will be mapped.
So to bring preexisting resources under Terraform management, you first write the resource block for it in a .tf file. Next you use terraform import to map the resource to this resource block in your .tfstate. The next time you run terraform plan, Terraform will determine what changes (if any) will need to be made upon the next terraform apply based on the resource block and the actual state of the resource.
EDIT
The "why" of terraform import is to manage resources that are previously unknown to Terraform. As you alluded to in your second bullet point, if you want metadata from a resource but do not want to change the configuration of the resource, you would use a data block and reference that in dependent resources.
When you want to manage the configuration of a resource that was provisioned outside of Terraform you use terraform import. If you tear down the resource there may be data loss or service downtime until you re-deploy with Terraform, but if you use terraform import the resource will be preserved.
The import process can be started with an empty resource block, but the attributes need to be filled out to describe the resource. You will get the benefits of terraform plan after importing, which can help you find the discrepancies between the resource block and the actual state of the resource. Once the two match up, you can continue to make additional changes to the resource like any other resource in Terraform.
Terraform state file is your source of truth for your cloud infrastructure. Terraform uses local state to create plans and make changes to the infrastructure. Before any terraform operation, terraform does a refresh to update the state with the real infrastructure.