I'm currently learning terraform and I come across the command terraform refresh. It seems that it syncs the terraform.tfstate file to changes I did manually (I tried changing EC2 instance type). I found out that terraform plan can identify the drift between current and desired state without updating the tfstate file. Also, running terraform apply automatically updates the tfstate file.
So I was thinking, if there are any drifts detected during terraform plan, I will just update the terraform code to account for them and let terraform apply update the tfstate file. Is there any reason to use terraform refresh independently?
P.S.
I'm using terraform v0.15.0
You're correct that terraform refresh is used to update your terraform state file to match the present state--which can drift if resources have been edited outside of terraform.
terraform refresh itself is deprecated, with a note that it can be unsafe in certain situations. The documentation suggests using terraform apply -refresh-only as an alternative, since it prompts for the user to confirm the changes prior to them being persisted.
As to your question of "when is this used?". In my experience, which primarily uses terraform for AWS deployment, we almost never actually run a refresh operation. Terraform automatically checks current state as part of the terraform plan / terraform apply cycle. This may or may not be specific to the AWS provider.
The one scenario where I could see it being important to refresh the state is when the statefile is used as a datasource via a data remote_state_data block. Specifically, if you have intentionally modified the resource and cannot (or haven't yet) updated the terraform markup to reflect the change. In that scenario other terraform modules are reading values from your statefile (as opposed to from the resources themselves)--if your resource and statefile are out of sync then consumers of the statefile would receive inaccurate data.
However in most cases you want your resources to match their terraform representation--so you would terraform apply to bring the resources and state back in alignment with your terraform module.
Related
Currently, I am using "Mongey/kafka" provider and now I have to switch to "confluentinc/confluent" provider with my existing terraform pipeline.
How can I do this ?
Steps currently following to switch the provider
Changing the provider in main.tf file and running following command to replace provider
terraform state replace-provider Mongey/kafka confluentinc/confluent
and after that I run
terraform init command to install the new provider
But after that when I am running
terraform plan
it is giving "no schema available for module.iddn_news_cms_kafka_topics.kafka_acl.topic_writer[13] while reading state; this is a bug in terraform and should be reported" error.
Is there any way, I will change the terraform provider without disturbing the existing resources created using terraform pipeline ?
The terraform state replace-provider command is intended for switching between providers that are in some way equivalent to one another, such as the hashicorp/google and hashicorp/google-beta providers, or when someone forks a provider into their own namespace but remains compatible with the original provider.
Mongey/kafka and confluentinc/confluent do both have resource types that seem to represent the same concepts in the remote system:
Mongey/kafka
confluentinc/confluent
kafka_acl
confluent_kafka_acl
kafka_quota
confluent_kafka_client_quota
kafka_topic
confluent_kafka_topic
However, despite representing the same concepts in the remote system these resource types have different names and incompatible schemas, so there is no way to migrate directly between them. Terraform has no way to understand which resource types in one provider match with resource types in another, or to understand how to map attributes from one of the resource types onto corresponding attributes of the other.
Instead, I think the best thing to do here would be to ask Terraform to "forget" the objects and then re-import them into the new resource types:
terraform state rm kafka_acl.example to ask Terraform to forget about the remote object associated with kafka_acl.example. There is no undo for this action.
terraform import confluent_kafka_acl.example OBJECT-ID to bind the OBJECT-ID (as described in the documentation) to confluent_kafka_acl.example.
I suggest practicing this in a non-production environment first so that you can be confident about the behavior of each of these commands, and learn how to translate from whatever ID format the Mongey/kafka provider uses into whatever import ID format the confluentinc/confluent provider uses to describe the same objects.
I know that when I do terraform apply it does not deploy a resource if the previous deployment within the same terraform state, it would not re-create it .
But I want to do something different:
Create a resource if it is not created by someone else.
But if the resource is already there and even it is not in the terraform state, do not generate an error and have refrence to its name.
Is there any known pattern to do this?
By design Terraform providers will typically not automatically "adopt" existing objects as now being managed by Terraform, because to do so would potentially lead to costly mistakes if you inadvertently bind a remote object to a Terraform resource and then run terraform destroy without realizing what is going to be destroyed.
Instead, you must bind existing objects to your Terraform resources using the terraform import command, telling Terraform explicitly that you intend it to become the sole manager of that object.
Let's say I created an app service using azurerm_app_service with the bare minimum arguments. If I later go into Azure Portal and start creating IP restrictions and later run Terraform apply, it seems that Terraform doesn't detect drift and try to undo all of the IP restrictions. Is that working as intended? Is there Terraform documentation that says this is intended?
This behavior works in our favor, because it tells us that Terraform is OK with configuration drift so long as the argument or argument object is not defined in the configuration script. We intend to let Azure Portal users what is OK and not OK to modify in the portal without worrying about Terraform resetting it all.
Update: Accepted answer below and also received more feedback from Hashicorp's forum, https://discuss.hashicorp.com/t/terraform-not-detecting-drift-on-resources-intended-or-not/17547/4?u=terraform-man
It is going to depend on how the resource is implemented, the schema and what is changing. Typically with azurerm provider it should compute those changes even if they are optional. I would double check your terraform config and the portal resource you are modify are the same. You might need to run a refresh. In other words it is very common for changes in the portal to show up as drift when you run terraform.
This is called Desired State Vs Current State.
If you create any resource using terraform and manually modify it, Terraform reverts it after next apply.
Before Terraform Apply command, it fetches the current state into the statefile and then compares the current state vs desired state.
If there is any manual change, it will make necessary changes to make it as Desired state.
This question is not how to import and it's not what's the purpose of tfstate. It's what's the purpose of importing a pre-existing resource, esp. compared to just referencing the ID of the existing resource?
Terraform has the feature of terraform import. HashiCorp describes the purpose of this as:
Terraform is able to import existing infrastructure. This allows you take resources you've created by some other means and bring it under Terraform management.
This is a great way to slowly transition infrastructure to Terraform, or to be able to be confident that you can use Terraform in the future if it potentially doesn't support every feature you need today.
I read the article about the purpose of Terraform state. It does make sense to me to track Terraform state with .tfstate files when those files are mappings back to the configurations in .tf files.
But it's still unclear to me what the purpose of a standalone .tfstate file is when it only maps to an empty resource block. If there is a resource not in terraform yet, I would typically do one of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so, why would you use that approach?
The only way to make changes to an imported resource (that only has an empty resource block in the .tf file and detailed state in .tfstate) is to make manual changes and then re-import into .tfstate`, right? And if so, then what's the point of tracking the state of that resource in terraform?
I'm sure there's a good reasons. Just want to understand this deeper! Thanks!
But it's still unclear to me what the purpose of a standalone .tfstate
file is when it only maps to an empty resource block.
You wouldn't use a standalone .tfstate file. You would be using the same .tfstate file that all your other resources are in.
If there is a resource not in terraform yet, I would typically do one
of two things:
put the resource in terraform, tear down the resource manually and re-deploy the resource with terraform, or...
keep the resource un-templated, reference its resource ID as a parameter and get its metadata via a data element for
terraform-managed resources that rely on it.
Is terraform import an alternative to those two approaches? And if so,
why would you use that approach?
Consider the case where you have a production database with terrabytes of data already load in it, and users actively performing actions that query that database 24 hours a day. Your option 1 would require some down time, possibly a lot of down time, because you would have to deal with backing up and restoring terrabytes of data. Your option 2 would never let you manage changes to your database server via Terraform. That's what the Terraform import feature solves. It lets Terraform take "full control" of resources that already exist, without having to recreate them.
I agree that if a system outage is not an issue, and if recreating a resource isn't going to take much time, using option 1 is the way to go. Option 2 is only for resources that you never want to fully manage in Terraform, which is really a separate issue from the one Terraform import solves.
When importing a resource with terraform import it is necessary to write the configuration block to manage it with Terraform. On the same page you linked it states:
The current implementation of Terraform import can only import resources into the state. It does not generate configuration. A future version of Terraform will also generate configuration.
Because of this, prior to running terraform import it is necessary to
write manually a resource configuration block for the resource, to
which the imported object will be mapped.
So to bring preexisting resources under Terraform management, you first write the resource block for it in a .tf file. Next you use terraform import to map the resource to this resource block in your .tfstate. The next time you run terraform plan, Terraform will determine what changes (if any) will need to be made upon the next terraform apply based on the resource block and the actual state of the resource.
EDIT
The "why" of terraform import is to manage resources that are previously unknown to Terraform. As you alluded to in your second bullet point, if you want metadata from a resource but do not want to change the configuration of the resource, you would use a data block and reference that in dependent resources.
When you want to manage the configuration of a resource that was provisioned outside of Terraform you use terraform import. If you tear down the resource there may be data loss or service downtime until you re-deploy with Terraform, but if you use terraform import the resource will be preserved.
The import process can be started with an empty resource block, but the attributes need to be filled out to describe the resource. You will get the benefits of terraform plan after importing, which can help you find the discrepancies between the resource block and the actual state of the resource. Once the two match up, you can continue to make additional changes to the resource like any other resource in Terraform.
Terraform state file is your source of truth for your cloud infrastructure. Terraform uses local state to create plans and make changes to the infrastructure. Before any terraform operation, terraform does a refresh to update the state with the real infrastructure.
Suppose I run terraform apply without the refresh flag and with no plan mentioned.
Will terraform refresh the state of what I have defined as IaC against the state of what is already deployed? Or will it only look at my current state file and compare that with what I updated as IaC.
The documentation does not mention the default refresh behavior of terraform.
Terraform will automatically refresh the state before running a command that would rely on it (such as plan, apply, destroy).
You can see this yourself by looking at the output from running these commands:
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.aws_region.current: Refreshing state...
------------------------------------------------------------------------
No changes. Infrastructure is up-to-date.
This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.
terraform apply will check for new state always. Any terraform action will always first refresh the state by default. Ideally, you should always do a terraform plan -out <planName>.plan and then apply this plan on the environment using terraform apply <planName>.plan. This ensures consistency and you exactly know what is going to happen in the environment.
In cases where the plan is old and there have been changes to the environment meanwhile, terraform will detect that and will give you an error - forcing you to plan and apply again.