Is there anyway to ignore changes to resource output? Or tell terraform to not refresh it?
A terraform resource I'm using returns a state_info output (map of string) that can be modified by processes outside of Terraform. I want to ignore these changes. Is this possible.
resource "aiven_vpc_peering_connection" "this" {
lifecycle {
ignore_changes = [
state_info
]
}
}
state_info is getting set to null outside of Terraform. I'm using state_info in other terraform resources. It's failing with aiven_vpc_peering_connection.this.state_info is empty map of string on subsequent terraform plans I run
The ignore_changes mechanism instructs Terraform to disregard a particular argument when it's comparing values in the configuration with values in the prior state snapshot, so it doesn't have any effect for attributes that are only saved in the prior state due to them not being explicitly configurable.
It sounds like what you want is instead to have Terraform disregard a particular argument when it's updating the prior state to match remote objects (the "refresh" step), so that the result would end up being a mixture of new content from the remote API and content previously saved in the state. Terraform has no mechanism to achieve that: the values stored in the state after refreshing are exactly what the provider returned. This guarantee can be important for some resource types because retaining an old value for one argument while allowing others to change could make the result inconsistent, if e.g. the same information is presented in multiple different ways.
The closest you can get to what you described is to use the value as exported by the upstream resource and then specify ignore_changes on the resource where you ultimately use that value, telling Terraform to ignore the changes in the upstream object when comparing the downstream object with its configuration.
Related
I don't fully understand how Terraform handles external changes. Let's take an example:
resource "aws_instance" "ec2-test" {
ami = "ami-0d71ea30463e0ff8d"
instance_type = "t2.micro"
}
1: security group modification
The default security group has been manually replaced by another one. Terraform detects the change:
❯ terraform plan --refresh-only
aws_instance.ec2-test: Refreshing state... [id=i-5297abcc6001ce9a8]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:
# aws_instance.ec2-test has changed
~ resource "aws_instance" "ec2-test" {
id = "i-5297abcc6001ce9a8"
~ security_groups = [
- "default",
+ "test",
]
tags = {}
~ vpc_security_group_ids = [
+ "sg-8231be9a95a4b1886",
- "sg-f2fc3af19c4adefe0",
]
# (28 unchanged attributes hidden)
# (7 unchanged blocks hidden)
}
No change planned:
❯ terraform plan
aws_instance.ec2-test: Refreshing state... [id=i-5297abcc6001ce9a8]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
It seems normal as we did not set the security_groups argument in the resource block (the desired state is aligned with the current state).
2: IAM instance profile added
An IAM role has been manually attached to the instance. Terraform also detects the change:
❯ terraform plan --refresh-only
aws_instance.ec2-test: Refreshing state... [id=i-5297abcc6001ce9a8]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:
# aws_instance.ec2-test has changed
~ resource "aws_instance" "ec2-test" {
+ iam_instance_profile = "test"
id = "i-5297abcc6001ce9a8"
tags = {}
# (30 unchanged attributes hidden)
# (7 unchanged blocks hidden)
}
This is a refresh-only plan, so Terraform will not take any actions to undo these. If you were expecting these changes then you can apply this plan to record the updated values in the Terraform state without
changing any remote objects.
However, Terraform also plans to revert the change:
❯ terraform plan
aws_instance.ec2-test: Refreshing state... [id=i-5297abcc6001ce9a8]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_instance.ec2-test will be updated in-place
~ resource "aws_instance" "ec2-test" {
- iam_instance_profile = "test" -> null
id = "i-5297abcc6001ce9a8"
tags = {}
# (30 unchanged attributes hidden)
# (7 unchanged blocks hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
I tried to figure out why these two changes don't produce the same effect. This article highlights differences depending on the argument default values: https://nedinthecloud.com/2021/12/23/terraform-apply-when-external-change-happens/
But the security_groups and iam_instance_profile arguments seems similar (optional with no default value), so why Terraform is handling these two cases differently?
(tested with Terraform v1.2.2, hashicorp/aws 4.21.0)
The handling of these situations unfortunately depends a lot on decisions made by the provider developer, since it's the provider's responsibility to decide how to reconcile any differences between the configuration and the prior state. (The "prior state" is what Terraform calls the state that results from running the "refresh" steps to synchronize with the remote system).
Terraform Core takes the values you've defined in the configuration (if any optional arguments are unset, Terraform Core uses null to represent that) and the values from the prior state and sends both of them to the provider to implement the planning step. The provider can then do whatever logic it wants as long as the planned new value for each attribute is consistent with the input. "Consistent" means that one of the following conditions is true:
The planned value is equal to the value set in the configuration.
This is the most straightforward situation to follow, but there are various reasons why a provider might not do this, which I'll discuss later.
The planned value is equal to the value stored in the prior state.
This represents situations where the value in the prior state is functionally equivalent to the value in the configuration but not exactly equal, such as if the remote system treats a particular string as case insensitive and the two values differ only in case.
The provider indicated in its schema that this is a value that can be decided by the remote system, such as an object ID that's generated by the remote system during the apply step, and the corresponding value in the configuration was null to represent the argument not being set at all.
In this case the provider gets to choose whichever value it wants, because the configuration says nothing about the attribute and thus the remote system has authority on what the value is.
From what you've described, it sounds like in your first example the provider used approach number 3, while in the second example the provider used approach number 1.
Since I am not the developer of this provider I cannot say for certain why the developers made the decisions they did here, but one common reason why a provider developer might choose option three is for situations where a particular value can potentially be set by multiple different resource types, in which case the provider might be designed to treat an absent argument in the configuration as meaning "keep whatever the remote system already has", whereas a non-null argument in the configuration would mean "set the remote system to use this given value".
For iam_instance_profile it seems like the provider considers null to be a valid configuration value for that argument and uses it to represent the EC2 instance having no associated instance profile at all. For vpc_security_groups and security_groups though, leaving the argument set to null in the configuration (or omitting it, which is equivalent) the provider treats that as "keep whatever the remote system has", and so Terraform just acknowledges the change but doesn't propose to undo it.
Based on my knowledge about EC2, I can guess that the reason here is probably that the underlying EC2 API has two different ways to set security groups: you can either use the legacy EC2-Classic style of specifying a security group by name (the security_groups argument in the provider), or the new EC2-VPC style of specifying it by ID (the vpc_security_group_ids argument in the provider). Whichever of the two you choose, the remote system will presumably populate the other one automatically and therefore without this special exception in the provider it would be impossible for any configuration to converge unless you set both security_groups and vpc_security_group_ids and set them to both refer to the same security groups. To avoid that, I think the provider just lets whichever one of the two you left unset automatically track the remote system, which has the side-effect the provider cannot automatically "fix" changes made outside of Terraform unless you set at least one of them so the provider can see what the correct value ought to be.
Terraform's ability to reconcile changes in the remote system by resetting back to match the configuration is a "best effort" mechanism because in many cases that requirement comes into conflict with other requirements, and provider developers must therefore decide on a case-by-case basis what to prioritize. Although Terraform does try its best to tell you about changes outside of Terraform and to propose fixing them where possible, the only certain way to keep your Terraform configuration and your remote system synchronized is to prevent anyone from making changes outside of Terraform, for example using IAM policies in AWS.
I'm trying to generate json from terraform modules using terraform-config-inspect (https://github.com/hashicorp/terraform-config-inspect).
Note: Started with terraform-docs but then found what it uses underneath and it's terraform-config-inspect library.
The problem is that I want to go beyond what terraform-config-inspect provides out of box at the moment:
As an example, I want to get the name of aws_ssm_parameter resource.
For example, I have resource like this:
resource "aws_ssm_parameter" "service_security_group_id" {
name = "/${var.deployment}/shared/${var.service_name}/security_group_id"
type = "String"
value = aws_security_group.service.id
overwrite = "true"
tags = var.tags
}
and I would like to extract the value of the name parameter but by default it does not output this parameter. I tried to hack the code by modifying resource schema and other parts but ended up in getting empty string instead of name value or error because it contains parts like ${var.deployment}.
When I set it to plain string then my modified code returns what I expect
"aws_ssm_parameter.service_security_group_id": {
"mode": "managed",
"type": "aws_ssm_parameter",
"name": "service_security_group_id",
"value": "/test-env/shared/my-service/security_group_id",
"provider": {
"name": "aws"
}
}
but in normal case it fails with the following error
{
"severity": "error",
"summary": "Unsuitable value type",
"detail": "Unsuitable value: value must be known",
...
}
I know that I could build something totally custom for my specific use case but I hope there is something that could be re-used :)
So the questions are:
Is it somehow possible to take the real raw value from terraform resource so I could get "/${var.deployment}/shared/${var.service_name}/security_group_id" in json output?
Maybe some other tool out there?
Thanks in advance!
Input Variables in Terraform are a planning option and so to resolve them fully requires creating a Terraform plan. If you are able to create a Terraform plan then you can find the resolved values in the JSON serialization of the plan, using steps like the following:
terraform plan -out=tfplan (optionally include -var=... and -var-file=... if you need to set particular values for those variables.
terraform show -json tfplan to get a JSON representation of the plan.
Alternatively, if you've already applied the configuration you want to analyse then you can get similar information from the JSON representation of the latest state snapshot:
terraform show -json to get a JSON representation of the latest state snapshot.
As you've seen, terraform-config-inspect is only for static analysis of the top-level declarations and so it contains no functionality for evaluating expressions.
In order to properly evaluate expressions here without creating a Terraform plan or reading from a Terraform state snapshot would require reimplementing the Terraform Core runtime, at least to some extent. However, for this particular expression (which only relies on input variable values) you could potentially use the HCL API directly with some hard-coded placeholder values for those variables in order to get a value for that argument, derived from whatever you happen to have set var.deployment and var.service_name to in the hcl.EvalContext you construct yourself.
I want to initially create a resource using Terraform, but if the resource gets later deleted outside of TF - e.g. manually by a user - I do not want terraform to re-create it. Is this possible?
In my case the resource is a blob on an Azure Blob storage. I tried using ignore_changes = all but that didn't help. Every time I ran terraform apply, it would recreate the blob.
resource "azurerm_storage_blob" "test" {
name = "myfile.txt"
storage_account_name = azurerm_storage_account.deployment.name
storage_container_name = azurerm_storage_container.deployment.name
type = "Block"
source_content = "test"
lifecycle {
ignore_changes = all
}
}
The requirement you've stated is not supported by Terraform directly. To achieve it you will need to either implement something completely outside of Terraform or use Terraform as part of some custom scripting written by you to perform a few separate Terraform steps.
If you want to implement it by wrapping Terraform then I will describe one possible way to do it, although there are various other variants of this that would get a similar effect.
My idea for implementing it would be to implement a sort of "bootstrapping mode" which your custom script can enable only for initial creation, but then for subsequent work you would not use the bootstrapping mode. Bootstrapping mode would be a combination of an input variable to activate it and an extra step after using it.
variable "bootstrap" {
type = bool
default = false
description = "Do not use this directly. Only for use by the bootstrap script."
}
resource "azurerm_storage_blob" "test" {
count = var.bootstrap ? 1 : 0
name = "myfile.txt"
storage_account_name = azurerm_storage_account.deployment.name
storage_container_name = azurerm_storage_container.deployment.name
type = "Block"
source_content = "test"
}
This alone would not be sufficient because normally if you were to run Terraform once with -var="bootstrap=true" and then again without it Terraform would plan to destroy the blob, after noticing it's no longer present in the configuration.
So to make this work we need a special bootstrap script which wraps Terraform like this:
terraform apply -var="bootstrap=true"
terraform state rm azurerm_storage_blob.test
That second terraform state rm command above tells Terraform to forget about the object it currently has bound to azurerm_storage_blob.test. That means that the object will continue to exist but Terraform will have no record of it, and so will behave as if it doesn't exist.
If you run the bootstrap script then, you will have the blob existing but with Terraform unaware of it. You can therefore then run terraform apply as normal (without setting the bootstrap variable) and Terraform will both ignore the object previously created and not plan to create a new one, because it will now have count = 0.
This is not a typical use-case for Terraform, so I would recommend to consider other possible solutions to meet your use-case, but I hope the above is useful as part of that design work.
If you have a resource defined in terraform configuration then terraform will always try to create it. I can't imagine what is your setup, but maybe you want to take the blob creation to a CLI script and run terraform and the script in desired order.
I have a Terraform config that looks like this:
resource "random_string" "foo" {
length = 31
special = false
}
resource "aws_ssm_parameter" "bar" {
name = "baz"
type = "SecureString"
value = random_string.foo.result
lifecycle {
ignore_changes = [value]
}
}
The idea is that on the first terraform apply the bar resource will be stored in baz in SSM based on the value of foo, and then on subsequent calls to apply I'll be able to reference aws_ssm_parameter.bar.value, however what I see is that it works on the first run, stores the newly created random value, and then on subsequent runs aws_ssm_parameter.bar.value is empty.
If I create a aws_ssm_parameter data source that can pull the value correctly, but it doesn't work on the first apply when it doesn't exist yet. How can I modify this config so I can get the value stored in baz in SSM and work for creating the value in the same config?
(Sorry not enough karma to comment)
To fix the chicken-egg problem, you could add depends_on = [aws_ssm_parameter.bar] to a data resource, but this introduces some awkwardness (especially if you need to call destroy often in your workflow). It's not particularly recommended (see here).
It doesn't really make sense that it's returning empty, though, so I wonder if you've hit a different bug. Does the value actually get posted to SSM (i.e. can you see it when you run aws ssm get-paramter ...)?
Edit- I just tested your example code above with:
output "bar" {
value = aws_ssm_parameter.bar.value
}
and it seems to work fine. Maybe you need to update tf or plugins?
Oh, I forgot about this question, but turns out I did figure out the problem.
The issue was that I was creating the ssm parameter inside a module that was being used in another module. The problem was because I didn't output anything related to this parameter, so it seemed to get dropped from state by Terraform on subsequent replans after it was created. Exposing it as output on the module fixed the issue.
I am using Terraform version 0.12. I have a requirement to skip resource creation if resource with the same name already exists.
I did the following for this :
Read the list of custom images,
data "ibm_is_images" "custom_images" {
}
Check if image already exists,
locals {
custom_vsi_image = contains([for x in data.ibm_is_images.custom_images.images: "true" if x.visibility == "private" && x.name == var.vnf_vpc_image_name], "true")
}
output "abc" {
value="${local.custom_vsi_image}"
}
Create only if image exists is false.
resource "ibm_is_image" "custom_image" {
count = "${local.custom_vsi_image == true ? 0 : 1}"
depends_on = ["data.ibm_is_images.custom_images"]
href = "${local.image_url}"
name = "${var.vnf_vpc_image_name}"
operating_system = "centos-7-amd64"
timeouts {
create = "30m"
delete = "10m"
}
}
This works fine for the first time with "terraform apply". It finds that the image did not exists, so it creates image.
When I run "terraform apply" for the second time. It is deleting the resource "custom_image" that is created above. Any idea why it is deleting the resource, when it is run for the 2nd time ?
Also, how to create a resource based on some condition(like only when it does not exists) ?
In Terraform, you're required to decide explicitly what system is responsible for the management of a particular object, and conversely which systems are just consuming an existing object. There is no way to make that decision dynamically, because that would make the result non-deterministic and -- for objects managed by Terraform -- make it unclear which configuration's terraform destroy would destroy the object.
Indeed, that non-determinism is why you're seeing Terraform in your situation flop between trying to create and then trying to delete the resource: you've told Terraform to only manage that object if it doesn't already exist, and so the first time you run Terraform after it exists Terraform will see that the object is no longer managed and so it will plan to destroy it.
If you goal is to manage everything with Terraform, an important design task is to decide how object dependencies flow within and between Terraform configurations. In your case, it seems like there is a producer/consumer relationship between a system that manages images (which may or may not be a Terraform configuration) and one or more Terraform configurations that consume existing images.
If the images are managed by Terraform then that suggests either that your main Terraform configuration should assume the image does not exist and unconditionally create it -- if your decision is that the image is owned by the same system as what consumes it -- or it should assume that the image does already exist and retrieve the information about it using a data block.
A possible solution here is to write a separate Terraform configuration that manages the image and then only apply that configuration in situations where that object isn't expected to already exist. Then your configuration that consumes the existing image can just assume it exists without caring about whether it was created by the other Terraform configuration or not.
There's a longer overview of this situation in the Terraform documentation section Module Composition, and in particular the sub-section Conditional Creation of Objects. That guide is focused on interactions between modules in a single configuration, but the same underlying principles apply to dependencies between configurations (via data sources) too.