Get state from terratest and run manually terraform destroy? - terraform

I make test with Terratest but I set wrong timeout and now test failed and resources are still there. I can remove them manually but better way will be using terraform destroy.
Is there way how get state or save them during running test via Terratest?

Terratest essentially just runs terraform commands with the parameters you give it in your local folder structure.
So, you should just be able to change directories to the module run by terratest (i.e. the TerraformDir) and run typical terraform commands (e.g. terraform plan, terraform destroy).
If you're asking about having the terratest do the destroy automatically for you, you can do that by passing along the same options to terraform.Destroy as in this example. The defer is recommended so that destruction happens after you have validated your resources.

I had the same problem. But in the output, it tells you the temporary folder where it's putting everything to run. It also had the state file. I was able to run a terraform destroy from that temp folder!

We have a couple approaches to tackle this issue:
Increase the Go test timeout by adding flag go test -timeout 1h
Increase the task and job timeout in the pipeline.
Use the test_structure "framework" from TerraTest with skippable Stages (env SKIP_<name of stage>). With the test_structure you can defer the destroy and will *always run.
Custom post cleanup script step in the build that run always, unless the destroy is skipped. The custom script will reset peerings, private endpoints and even clear state files.
*Not always, in case of a bigger issue, a timeout for example.

Related

Is there a terraform resource which can force an intermediate update of the remote statefile, before continuing?

I set up terraform to use a backend to remotely store the statefile. That works fine.
My project take several minutes for the full terraform apply to complete. During development, sometimes one of the later stages hangs (seemingly) eternally. I need the outputs in order to manually connect to the servers and inspect what is broken. However, the statefile does not get written until the terraform process completes. So there are no outputs available during the first terraform apply.
Is there a way to make terraform update the statefile intermediately, while it is still busy applying things?
I know I could solve this by separating the process into multiple modules, and apply each one after the other. But I am looking for a solution where I can still apply all at once.
When you run
terraform plan
you get the outputs also. What you can do is save that file before applying -
terraform plan -out tf.plan
Then you apply.
You can look into this file to find the changes.
Remember, you won't find the output data that were to be showed after apply, like a thing that does not exist yet.
Best wishes.

Destroy resource on trigger Terraform

I'm trying to decide whether to use Ansible or Terraform for a project. This project is doing scientific computations on cloud servers. I want to be able to say "here's an input file, go grab a computer with 16vCPUs. When everything's done, copy the output file back and destroy the resource". Is there a way for me to tell terraform "destroy/take action on this resource when you detect a command is done or some api is hit"?
Is there a way for me to tell terraform "destroy/take action on this
resource when you detect a command is done or some api is hit"?
If it's detectable via API hit then you can use local-exec Provisioner and use curl to hit the api endpoint in loop (in a wrapping shell script), polling the status of the API to deduce if the task is done or not. If you do so, make sure to give appropriate timeout value to this provisioner.
For the cleanup sequence (post API status is affirmative), declare the dependency explicitly to ensure Terraform is able to follow the execution accordingly.
Having explained all this, your end to end objective is best suited with a combination of IaC + Config mgmt tool. If I had to do it, I'd have preferred to use Terraform + Ansible combination in following outline
A shell script invoking Terraform to provision the 16vCPUs resource with all necessary depdendency
From within Terraform, invoke local-exec provisioner to launch ansible for appropriately configuring the instance, copying the file and launching remote execution etc.
Let the control come back to shell script which will start polling the status of the task based on API polling using curl
As and when shell script detects the task is done, it'd have copied the remote file locally (this step can be made better but that's a topic for another discussion)
Invoke Terraform again, with destroy this time, to tear down entire environment.
I'd further recommend to have it tied up in a jenkins workflow (if you are using jenkins already) giving control to other user's and still having full visibility and auditing in place.

Is `terraform init` compulsory before every `terraform plan`?

Just wondering if terraform init is compulsory before every terraform plan?
i.e. if I've already done a terraform init but are about to do a second terraform plan based on some changed Terraform code would you need to do a second terraform init?
It depends.
Depending on exactly what the Terraform code is you've changed, you may need to re-run init. For instance, if you've made changes to a configured backend, you'll need to rerun terraform init to re-initialise with those changes. If the changes are to non-terraform configuration parts of your script, terraform plan and terraform apply should be fine to use by themselves.
One further note is that if you're using modules, and you make a change in a module, you will need to re-run terraform get -update to get those changes before running plan or apply.
Agree with all the above answers, but something to add here is that it is safe to run terraform init many times even if nothing changed, its not going to affect anything.
terraform init running state depends on what has been changes, in case there are changes in plugins or your backend configs.
This command is always safe to run multiple times, to bring the working directory up to date with changes in the configuration. Though subsequent runs may give errors, this command will never delete your existing configuration or state.
So, you can run init every time you run terraform plan to keep things up to date.
In case there are no change then you skip it.
However, if multiple people are working on the project & you are storing state somewhere then always run terraform init before running terraform plan.
it depends on the situation.terraform init command is used to initialize a working directory containing Terraform configuration files. If you weren't changed the terraform configurations(key words you can say) you don't need to issue a terraform init. Instead you can terraform plan and terraform apply. Usually terraform tell, if terraform need be initialize by a messege

How to test terraform templates other than trial and error

I'm creating cloud resources using Terraform. Each resource is expected to be in a particular desired state after provisioning. For example, when I create a Google Cloud Bucket, I would like certain permissions to be applied automatically. So, my plan contains necessary code for this but I wanted to make sure that this works all the time regardless before I apply. Is there any testing tool/library that can help here?
Yes, I had the same thinking before. Currently, I use several ways to reduce the risk when I apply a new terraform change.
They can't guarantee a 100% successful terraform apply, but will fix the most issues before you apply it.
Validate terraform configuration files.
Terraform has the validate function for starting. But it is not smart enough to go through subfolders. I create a small shell function and add in CI/CD pipeline to run it automatically before terraform apply.
validate() {
modules=$(find . -type f -name "*.tf" -exec dirname {} \;|sort -u)
for m in ${modules}
do
(terraform validate "$m" && echo "√ $m") || exit 1
done
}
Of course, do terraform fmt before you submit your change is not bad idea.
terraform plan
#Martin Atkins explained it already, and terraform.io has details about this command.
run automation test kitchen.
That's a test Kitchen plugin for testing Terraform configurations
https://github.com/newcontext-oss/kitchen-terraform
That's an integration test. The test will run in separate VPC with as more as test cases you added. Add the automation test in CI/CD pipeline as well to trigger an automation test every time when you raise merge request to master branch. Apply the change only after getting the test passed.
The terraform plan command is intended to give a preview of what changes Terraform will make when the plan is applied, which is the closest we can get to testing a Terraform configuration without touching the "real" API.
For situations where that isn't enough, it's common to deploy the same config multiple times with different states, thus allowing one to be used as a "staging" environment to test changes without affecting the primary environment. The State Environments feature added in Terraform 0.9 can make this easier, since the multiple environment states can be managed directly with Terraform CLI commands.
When it comes to automated testing of the result, there is currently no full solution to this integrated into Terraform, but there are some building blocks that could be useful to assist in writing tests in a separate programming language.
Terraform produces state files in JSON format that can, in principle, be used by external programs to extract certain data about what Terraform created. While this format is not yet considered officially stable, in practice it changes infrequently enough that people have successfully integrated with it, accepting that they might need to make adjustments as they upgrade Terraform.
What strategy is appropriate here will depend a lot on what exactly you want to test. For example:
In an environment that's spinning up virtual servers, tools like Serverspec can be used to run tests from the perspective of these servers. This can either be run separately from Terraform using some out-of-band process, or as part of the Terraform apply using the remote-exec provisioner. This allows verification of questions like "can the server reach the database?", but is not suitable for questions such as "is the instance's security group restrictive enough?", since robustly checking that requires accessing data from outside of the instance itself.
It's possible to write tests using an existing test framework (such as RSpec for Ruby, unittest for Python, etc) which gather relevant resource ids or addresses from the Terraform state file and then use the relevant platform's SDK to retrieve data about the resources and assert that they are set up as expected. This is a more general form of the previous idea, running the tests from the perspective of a host outside of the infrastructure under test, and can thus collect a broader set of data to make assertions on.
For more modest needs, one can choose to trust that the Terraform state is an accurate representation of reality (a valid assumption in many cases) and simply assert directly on that. This is most appropriate for simple "lint-like" cases, such as verifying that the correct resource tagging scheme is being followed for cost-allocation purposes.
There is some more discussion about this in a relevant Terraform Github issue.
In the latest versions of Terraform it is strongly recommended to use a remote backend for any non-toy application, but that means that the state data is not directly available on local disk. However, a snapshot of it can be retrieved from the remote backend using the terraform state pull command, which prints the JSON-formatted state data to stdout so it can be captured and parsed by a calling program.
We recently open sourced Terratest, our swiss army knife for testing infrastructure code.
Today, you're probably testing all your infrastructure code manually by deploying, validating, and undeploying. Terratest helps you automate this process:
Write tests in Go.
Use helpers in Terratest to execute your real IaC tools (e.g., Terraform, Packer, etc.) to deploy real infrastructure (e.g., servers) in a real environment (e.g., AWS). Note that this environment would be a separate "sandbox" account and not prod!
Use helpers in Terratest to validate that the infrastructure works correctly in that environment by making HTTP requests, API calls, SSH connections, etc.
Use helpers in Terratest to undeploy everything at the end of the test.
Here's an example test for some Terraform code:
terraformOptions := &terraform.Options {
// The path to where your Terraform code is located
TerraformDir: "../examples/terraform-basic-example",
}
// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
terraform.InitAndApply(t, terraformOptions)
// At the end of the test, run `terraform destroy` to clean up any resources that were created
defer terraform.Destroy(t, terraformOptions)
// Run `terraform output` to get the value of an output variable
instanceUrl := terraform.Output(t, terraformOptions, "instance_url")
// Verify that we get back a 200 OK with the expected text
// It can take a minute or so for the Instance to boot up, so retry a few times
expected := "Hello, World"
maxRetries := 15
timeBetweenRetries := 5 * time.Second
http_helper.HttpGetWithRetry(t, instanceUrl, 200, expected, maxRetries, timeBetweenRetries)
These are integration tests, and depending on what you're testing, can take 5 - 50 minutes. It's not fast (though using Docker and test stages, you can speed some things up), and you'll have to work to make the tests reliable, but it is well worth the time.
Check out the Terratest repo for docs and lots of examples of various types of infrastructure code and the corresponding tests for them.

Is terraform destroy needed before terraform apply?

Is terraform destroy needed before terraform apply? If not, what is a workflow you follow when updating existing infrastructure and how do you decide if destroy is needed?
That would be pretty non-standard, in my opinion. Terraform destroy is only used in cases where you want to completely wipe your infrastructure. One of the biggest features of terraform is that it can do an intelligent delta of your desired infrastructure and your existing infrastructure and only make the changes needed. By performing a refresh, plan and apply you can ensure that terraform:
refresh - Has an up-to-date understanding of your current infrastructure. This is important in case anything was changed manually, outside of your terraform script.
plan - Prepares a list for you to review of what terraform intends to modify, or delete (or leave alone).
apply - Performs the changes laid out in the plan.
By executing these 3 commands in sequence terraform will only perform the changes necessary, in the order required, to bring your environments in line with any changes to your terraform file.
Where I find destroy to be useful is in non-production environments or in cases where you are performing a restructure that's so invasive that starting from scratch would ensure a safer build.
*There are also edge cases where terraform may fail to understand the correct order of operations (do I modify a security group first or a security group rule?), or it will find itself in a dependency cycle and will be unable to perform an operation. In those cases, however, running destroy is a nuclear solution. In general, I would perform the problem change manually (via command line, or AWS Console, if I'm in AWS), to nudge it along and then run a refresh, plan, apply sequence to get back on track.
No terraform destroy is not needed before terraform apply.
Your Terraform configuration (*.tf and *.tfvars files) describes the desired state of your infrastructure. It says "this is how I want my infrastructure to be."
You use the terraform tool to plan and apply changes to get your infrastructure into the desired state you have described. You can make those changes incrementally without destroying anything.
A typical workflow might be:
make changes to .tf and .tfvars files
refresh state
plan changes
review the planned changes
apply those changes
If you wanted to completely destroy that infrastructure you'd use terraform plan -destroy to see what Terraform intends to destroy. If you are happy with that you'd then use terraform destroy to destroy it.
Typically, destroy is rarely used, unless you are provisioning infrastructure for a temporary purpose (e.g., builds) or testing your ability to provision from a clean slate with different parameters. Even then, you could use a count parameter on resources to temporarily provision resources by increasing the count, then decreasing it again when no longer needed.
More comments after #mwielbut's answer.
Instead of option apply + destroy, you need to run terraform with option taint + apply
Normally we don't need run terraform destroy at all. It is a really dangerous option, especially for a production environment.
with option plan and apply, it is good enough to update the infrastructure with code.
But if you do need to destroy some resources and re-build something which is already created, you can use the option of taint, which is the right answer for your question, it is so important and missed in #mwielbut's answer.
The terraform taint command manually marks a Terraform-managed resource as tainted, forcing it to be destroyed and recreated on the next apply.
This command will not modify infrastructure but does modify the state file in order to mark a resource as tainted. Once a resource is marked as tainted, the next plan will show that the resource will be destroyed and recreated and the next apply will implement this change.
Refer:
command taint:
https://www.terraform.io/docs/commands/taint.html
a sample of option taint:
https://www.terraform.io/docs/modules/usage.html
Terraform destroy destroys all the resources and it is not required if you want to apply incremental changes. Destroy should be only used if you want to destroy the whole infrastructure.
No need to use the destroy command before apply. as long as you are in testing period you can use destroy command or destroy the complete infra you can use destroy command
You can use below flow
terraform init
terraform plan
terraform apply
if you made any manual changes that needs to update in your state file, use below command to update the state file.
Terrafrom refresh
You don't need to run to terraform destroy . If you have made any changes to you infrastructure, [added/ removed a resource], on next terraform plan & terraform apply, the changes will be reflected automatically
Terraform apply always refreshes the Terraform state, so if you change anything, it auto recognizes the changes, lets say you've updated your NSG rules, added new VM, deleted old VM, so when you run terraform apply again, your old state gets updated with the new state where you've Added/Updated/Deleted.
If you use terraform destroy, it just kills the entire state and you'll be back to the new state if you are running terraform apply.
You need to use terraform destroy only if you think you just want to bring down your infrastructure and you don't really need it.
For minor - major changes like Adding Components, Updating Rules, Deleting other things, you can use plan and apply without any problem.
Simply NO.
You don't need to run terraform apply before terraform destroyو Your terraform (.tf) files describe the state of your infrastructure.
terraform apply always refresh your infrastructure. And it identifies the state of infrastructure and updates it.
terraform destroy only use is to bring down and completely wipe down your infrastructure. (You have to think twice before using it) you can use terraform plan and terraform refresh to ensure the state of the infrastructure.
You could always manually destroy your instances, after only running your terraform apply. Then when you run terraform apply it will create brand new instances without the terraform destroy.
No! you don't need to run terraform destroy when you need a modification of resources! This is the beauty of Infra-as-Code.
Here are some more details on Terraform init, plan, apply and destroy -
terraform init command is used to initialize a working directory containing Terraform configuration files. This is the first command that should be run after writing a new Terraform configuration or cloning an existing one from version control. It is safe to run this command multiple times.
terraform plan command creates an execution plan. By default, creating a plan consists of:
a) Reading the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date.
b) Comparing the current configuration to the prior state and noting any differences.
c) Proposing a set of change actions that should, if applied, make the remote objects match the configuration.
terraform apply command executes the actions proposed in a Terraform plan. (you can do an apply without plan however it's not a best practice)
terraform destroy command is a convenient way to destroy all remote objects managed by a particular Terraform configuration.
Core Terraform workflows:
The core Terraform workflow has five steps:
Write - Author infrastructure as code.
Terraform init - it’ll automatically download and install partner and community provider directly to the local disk so that it can be used by other commands Plugin_Installation, Backend_Initialization, ChildModule_Installation and Community and third party plugin
Terraform plan - Preview changes before applying.
Terraform Apply - Provision reproducible infrastructure.
Terraform destroy - It will destroy your infrastructure.
No need for terraform destroy, as it will just destroy all the resources created.
You just need to provide the backend configuration in your tf file.
Backend configuration is the configuration in order to retrieve terraform state files.
Terraform apply first time will create your cloud infrastructure, this will update your state file also.
And next apply terraform will compare what new/update resources are to be done with what is already there using state file and will deploy accordingly.

Resources