I am new to terraform. Can someone please explain
why do we need to save .tfstate file in local or remote storage,
when terraform apply always refreshes the state file with new infrastructure.
Thanks in advance.
The state file tracks the resources that Terraform is managing, whether it created them or imported them. Terraform's refresh only detects drift in managed resources and won't detect if you have created new resources outside of the state file.
If you lose the state you will end up with orphaned resources that are not being managed by Terraform. If, for some reason, you are okay with that or you have some other way of sharing state with other team members/CI and backing it up then you're fine.
Of course, using Terraform's remote state neatly solves those things so you should use it if you care about any of those things or think you might need to in the future (you probably will).
I will add a more developer-oriented perspective to help understand.
Think about you are using yarn or npm to do a NodeJS app, package.json is like your tf files, while yarn.lock or package-lock.json.
Dont take literally though, as terraform state file has physical underlying implications.
Related
I recreated a state file for an existing infrastructure using a third party tool i.e. terraformer. Now I want o move the .tfstate to another azurerm back-end and manage from there.
If I just copy the file i.e. mystate.tfstate from local to inside storage account container with the same file name/key as in the backend configurations will it work or do I need to do something else to achieve it?
I don't want to risk the state file or infrastructure by trying to do something that isn't sure to work as I expect.
Terraform has some automatic migration behavior built in to terraform init.
Based on your description, it sounds like so far you've been using local state storage, and so the latest state snapshot is in a .tfstate file on your local system and you probably don't have a backend block in your configuration yet, since local storage is the default.
Before beginning this process, I suggest first making a copy of your state file in a safe place so that you can experiment more confidently. This process should not risk your existing state file, but it can't hurt to be careful if you've invested significant work in constructing that state file already.
Next, add a backend "azurerm" block to tell Terraform it should use that backend. Refer to the documentation to see which settings you'll need to set and what other preparation steps you may need to make first, such as establishing a new storage container.
If you've been using local state then you will presumably have a terraform.tfstate file in your current working directory, which Terraform will check for in the next step. If you've renamed that file at any point so far, you'll need to rename it back to terraform.tfstate to match the expectations of Terraform's local state storage implementation.
If you now run terraform init, Terraform should notice the following two things:
You have a backend block but the current working directory doesn't currently have an initialized backend connection.
You have an existing terraform.tfstate file in your working directory.
With those two things being true, Terraform will propose to migrate your state from the local backend to the azurerm backend. You can follow the steps it proposes and answer the prompts that appear, after which you should find the same state snapshot stored in your configured Azure storage container.
Once you've confirmed that the object is present in Azure storage, you can delete the terraform.tfstate file, since Terraform will no longer refer to it.
I don't want to risk the state file or infrastructure by trying to do
something that isnt sure to work as I expect.
Make a backup of the state file first, and then you won't be risking the state file.
As long as you aren't running an apply command, you won't be risking the infrastructure. And even if you are running an apply command, you will be able to review the plan before proceeding.
So just (always) backup your state file, and always review a plan before applying it, and there is no risk.
Does Terraform allow storing the state file in SVN? If this is not directly supported by Terraform, do any third party/ open source options exist?
When using Terraform it's not typical to store the Terraform state in the same version control repository as the configuration that drives it, because the expected workflow to use version control with Terraform is to review and commit the proposed changes first, and only then apply the changes to your real infrastructure from your main branch.
To understand why it might help to think about the relationship between an application's main code and its database. We don't typically store the main database for a web application in the version control repository along with the code, because the way we interact with the two are different: many developers can be concurrently working on and proposing changes to the application source code, and our version control system is often able to merge together those separate proposals to reduce collisions, but for the application's central database it's more common to use locks so that two writers don't try to change the same data (or interconnected data) at the same time.
In this way, Terraform state is roughly analogous to Terraform's "backend database". When using Terraform in a team setting then, you'd typically select one of the backends that stores state remotely and supports locking, and then anyone working with that particular Terraform configuration will find that Terraform will take out a lock before making any remote system modifications, hold that lock throughout its work, and then write the newly-updated state to the backend before releasing the lock.
Although you specifically asked about Subversion, my suggestions here are intended to apply to all version control systems. Version control is a good place to keep the source code for your Terraform modules, but it's not a good place to keep your Terraform state.
I'm new to this terraform world and I've been assigned into the task of creating many configurations to azure with it.
I'm developing a main.tf script (which creates some resources, like resource group, vnets, kubernetes cluster, app services, etc.) and while coding it and executing
Terraform apply, it seems to only apply what changed doing in fact updates.
Then we deleted the resource group the script created and a colegue of mine had to run the same script with terraform creating a resource group with another name since i didn't had a required permission, after that, if i run the command Terraform apply it fails and gives errors, that say that the resource cannot be created because it already exists.
After reading some documentation i found that it might be because of the state
https://www.terraform.io/docs/state/index.html
Is the update of a script something that only works for each session of terraform?
Even doing a Terraform refresh doesn't seem to work.
Or probably I'm just mistaking and there is no way to update some resources.
EDIT: for some reason the state file that was on the storage only had a few things, the solution was to delete everything and create again.
For the new resources, there is nothing more, the Terraform script helps you create the resources you set in the script.
For the existing resources, when you make changes in the script that you already deployed via the Terraform, then it will check the state file to make sure what changes the resources should update. If there is no state file ( or you delete it), then it will deploy the Terraform script directly, but if any resources you want to deploy already exists, then it will fail due to the existing resources. And the command terraform refresh just updates the last state of the resources in the Terraform script that you already deployed. If the deployment failed and the state file has no resources in it, then refresh is not useful.
If someone else ran terraform apply for you because you didn't have access, and now you want to modify that terraform and run it yourself, you need to get the state file that was generated when that other person ran it. You absolutely have to maintain the Terraform state file somewhere, so that it can be accessed on subsequent runs. You should really configure a Terraform backend, instead of using local state files.
You need to be aware that Terraform stores everything it does in the state file, and refers to that file before every run. A terraform refresh only tells Terraform to refresh the state of the things that are in the state file, it doesn't rebuild the state file from scratch. Understanding Terraform state files is so fundamental to the use of Terraform that you really need to understand this before using it.
I am trying to find my way in terraform , i am going through the documentation and got confused about what is exactly will happen if a resource has been deleted manually from the configuration file then we ran the apply command on the modified configuration file please ?
my understanding that the state file will still have the deleted resource as well it will still be actually running on the cloud platform so terraform apply will not perform any action, but i am not sure.
Appreciate if you help to clear my understanding please
Also another relevant point please what if a resource was changed manually from the cloud console for example and we tried to do any action from terraform on that resouces , what will happen ?
Thanks a lot ,
First, some background from the documentation at https://www.terraform.io/intro/index.html
Terraform generates an execution plan describing what it will do to reach the desired state, and then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what changed and create incremental execution plans which can be applied.
The state mentioned gets saved when resources are modified. When you add a resource, it will get created and the state will be updated to reflect this. The same is for removing of resources. When you remove a resource in Terraform by deleting the code or template file, the resource will be removed and state updated to to reflect the removed resource (emphasis mine to illustrate the answer).
The second question of changing resources that drift from the state is a little more involved. When you create a plan against a resource that may have change, the provider will usually refresh the resources in state to compare them and show you what the changes to be made will be (ie. trying to change the resource to the declared state in the code).
I usually run all my Terraform scripts through Bastion server and all my code including the tf statefile resides on the same server. There happened this incident where my machine accidentally went down (hard reboot) and somehow the root filesystem got corrupted. Now my statefile is gone but my resources still exist and are running. I don't want to again run terraform apply to recreate the whole environment with a downtime. What's the best way to recover from this mess and what can be done so that this doesn't get repeated in future.
I have already taken a look at terraform refresh and terraform import. But are there any better ways to do this ?
and all my code including the tf statefile resides on the same server.
As you don't have .backup file, I'm not sure if you can recover the statefile smoothly in terraform way, do let me know if you find a way :) . However you can take few step which will help you come out from situation like this.
The best practice is keep all your statefiles in some remote storage like S3 or Blob and configure your backend accordingly so that each time you destroy or create a new stack, it will always contact the statefile remotely.
On top of it, you can take the advantage of terraform workspace to avoid the mess of statefile in multi environment scenario. Also consider creating a plan for backtracking and versioning of previous deployments.
terraform plan -var-file "" -out "" -target=module.<blue/green>
what can be done so that this doesn't get repeated in future.
Terraform blue-green deployment is the answer to your question. We implemented this model quite a while and it's running smoothly. The whole idea is modularity and reusability, same templates is working for 5 different component with different architecture without any downtime(The core template remains same and variable files is different).
We are taking advantage of Terraform module. We have two module called blue and green, you can name anything. At any given point of time either blue or green will be taking traffic. If we have some changes to deploy we will bring the alternative stack based on state output( targeted module based on terraform state), auto validate it then move the traffic to the new stack and destroy the old one.
Here is an article you can keep as reference but this exactly doesn't reflect what we do nevertheless good to start with.
Please see this blog post, which, unfortunately, illustrates import being the only solution.
If you are still unable to recover the terraform state. You can create a blueprint of terraform configuration as well as state for a specific aws resources using terraforming But it requires some manual effort to edit the state for managing the resources back. You can have this state file, run terraform plan and compare its output with your infrastructure. It is good to have remote state especially using any object stores like aws s3 or key value store like consul. It has support for locking the state when multiple transactions happened at a same time. Backing up process is also quite simple.