Efficient variable validation with Terraform - terraform

Is there an efficient way to apply validation logic to variables used in a terraform run?
Specifically I want to check the length and casing of some variables. The variables are a combination of ones declared in tfvars files, in variables.tf files, and collected during runtime by terraform.
Thanks.

Custom Validation Rules
Terraform document - Input Variables - Custom Validation Rules
Results
Failure case
provider aws {
profile="default"
}
terraform {
experiments = [variable_validation]
}
## Custom Validation Rules
variable "test" {
type = string
description = "Example to test the case and length of the variable"
default = "TEsT"
validation {
condition = length(var.test) > 4 && upper(var.test) == var.test
error_message = "Validation condition of the test variable did not meet."
}
}
Execution
$ terraform plan
Warning: Experimental feature "variable_validation" is active
on main.tf line 5, in terraform:
5: experiments = [variable_validation]
Experimental features are subject to breaking changes in future minor or patch
releases, based on feedback.
If you have feedback on the design of this feature, please open a GitHub issue
to discuss it.
Error: Invalid value for variable # <---------------------------
on main.tf line 9:
9: variable "test" {
Validation condition of the test variable did not meet.
This was checked by the validation rule at main.tf:14,3-13.
Pass case
terraform {
experiments = [variable_validation]
}
## Custom Validation Rules
variable "test" {
type = string
description = "Example to test the case and length of the variable"
default = "TESTED"
validation {
condition = length(var.test) > 4 && upper(var.test) == var.test
error_message = "Validation condition of the test variable did not meet."
}
}
Execution
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
No changes. Infrastructure is up-to-date.
Others
Alternatively, use null_resource local-exec to implement logic in shell script, or use external provider to send the variable to an external program to validate?

This isn't something you can currently do directly with Terraform but I find it easier to just mangle the input variables to the required format if necessary.
As an example the aws_lb_target_group resource takes a protocol parameter that currently requires it to be uppercased instead of automatically upper casing things and suppressing the diff like the aws_lb_listener resource does for the protocol (or even the protocol in the health_check block).
To solve this I just use the upper function when creating the resource:
variable "protocol" {
default = "http"
}
resource "aws_vpc" "main" {
cidr_block = "10.0.0.0/16"
}
resource "aws_lb_target_group" "test" {
name = "tf-example-lb-tg"
port = 80
protocol = "${upper(var.protocol)}"
vpc_id = "${aws_vpc.main.id}"
}
As for checking length I just substring things to make them the right length. I currently do this for ALBs as the name has a max length of 32 and I have Gitlab CI create review environments for some services that get a name based on the slug of the Git branch name so have little control over the length that is used.
variable "environment" {}
variable "service_name" {}
variable "internal" {
default = true
}
resource "aws_lb" "load_balancer" {
name = "${substr(var.environment, 0, min(length(var.environment), 27 - length(var.service_name)))}-${var.service_name}-${var.internal ? "int" : "ext"}"
internal = "${var.internal}"
security_groups = ["${aws_security_group.load_balancer.id}"]
subnets = ["${data.aws_subnet_ids.selected.ids}"]
}
With the above then any combination of length of environment or service name will lead to the environment/service name pair being trimmed to 27 characters at most which leaves room for the extra characters that I want to specify.

Got inspired by this conversation and found the following already existing provider:
https://github.com/craigmonson/terraform-provider-validate

Related

Module enabling to deploy based on a specific variable ( Terraform/Datadog )

I'm looking to automate on specific part based on a very complicated Terraform script that I have.
To make a bit clear I have created TF template with Deployment of entire infra into Azure with App Services, Storage account, Security groups, Windows based VM's, Linux based VM's split for MongoDB and RabbitMQ. Inside my script I was able to automate deployment to use the name of the application and create Synthetic Test and using a local variable plus based on to the environment to use a specific Datadog Key using the local variable ability
keyTouse = lower(var.environment) != "production" ? var.DatadogNPD : var.DatadogPRD
right now the point that bothers me is the following.
Since we are not in a need to use Synthetic tests based on NON Production environment i would like to use some sort of logic and not deploy Synthetic tests if the var.environment is not "PRODUCTION"
To make this part more interesting i also have the ability to deploy multiple Synthetic Test using the "count" and "length" shown below
inside main.tf
module "Datadog" {
source = "./Datadog"
webapp_name = [ azurerm_linux_web_app.service1.name, azurerm_linux_web_app.service2.name ]
}
and for Datadog Synthetic Test
resource "datadog_synthetics_test" "app_service_monitoring" {
count = length(var.webapp_name)
type = "api"
subtype = "http"
request_definition {
method = "GET"
url = "https://${element(var.webapp_name, count.index)}.azurewebsites.net/health"
}
Could you help me and suggest how can I enable or disable modules deployment using the variable based on environment?
Based on my understanding of the question, the change would have to be two-fold:
Add an environment variable to the module code
Use that variable for deciding if the synthetics test resource should be created or not
The above translates to creating another variable in the module and later on providing that variable a value when calling the module. The last part would be deciding based on that if the resource gets created.
# module level variable
variable "environment" {
type = string
description = "Environment in which to deploy resources."
}
Then, in the resource, you would add the following:
resource "datadog_synthetics_test" "app_service_monitoring" {
count = var.environment == "production" ? length(var.webapp_name) : 0
type = "api"
subtype = "http"
request_definition {
method = "GET"
url = "https://${element(var.webapp_name, count.index)}.azurewebsites.net/health"
}
}
And finally, in the root module:
module "Datadog" {
source = "./Datadog"
webapp_name = [ azurerm_linux_web_app.service1.name, azurerm_linux_web_app.service2.name ]
environment = var.environment
}
The environment = var.environment line will work if you have also defined the variable environment in the root module. If not you can always set it to a value you want:
module "Datadog" {
source = "./Datadog"
webapp_name = [ azurerm_linux_web_app.service1.name, azurerm_linux_web_app.service2.name ]
environment = "dev" # <--- or "production" or any other environment you have
}

how to make terraform error when data is empty

I have problem with terraform especially with google load balancer backend_service
here's my sample tf config
var backends {
default = [
{
'name':'red'
'service': 'abcd'
},
{
'name':'blue'
'service': 'efgh'
}
]
}
local {
backend = {for i in var.backends: "${i.name}-${i.service}" => i}
}
data "external" "mydata" {
for_each = local.backend
program = ["/bin/sh", "script.sh", each.value.name]
}
//that script will get some data like this //
{"id": "sometext"}
//
resource "google_compute_backend_service" "default" {
project = local.project_id
for_each = local.backend
name = each.key
dynamic "backend" {
for_each = {for zone in var.zone: zone => "projects/${local.project_id}/zones/${zone}/networkEndpointGroups/${data.external.mydata[each.key].result.id}"}
}
}
the problem is, if data.external.mydata[each.key].result.id is empty,the terraform plan is success/without error.
I want terraform plan to be error/failed in especially in google_compute_backend_service.
here's sample external.data if empty
{"id":""}
I want this to be error/failed in especially in google_compute_backend_service.
is it posible to reproduce error?
I know it can use count for checking the external.data value, but since I use for_each, so I cannot use count
Part of the expected protocol for programs you execute with the external data source is that they can indicate an error by printing an error message to stderr and then exiting with a non-zero exit status.
In a shell script you could do that via some commands like the following:
echo >&2 "No matching object is available."
exit 1
If your command exits unsuccessfully like this then the external provider will detect that and return a failure, which will then in turn prevent Terraform from processing any resources that depend on that result.

MySQL firewall rules from an app service's outbound IPs on Azure using Terraform

I'm using Terraform to deploy an app to Azure, including a MySQL server and an App Service, and want to restrict database access to only the app service. The app service has a list of outbound IPs, so I think I need to create firewall rules for these on the database. I've found that in Terraform, I can't use count or for_each to dynamically create these rules, as the value isn't known in advance.
We've also considered hard coding the count but the Azure docs don't confirm the number of IPs. With this, and after seeing different numbers in stackoverflow comments, I'm worried that the number could change at some point and break future deployments.
The output error suggests using -target as a workaround, but the Terraform docs explicitly advise against this due to potential risks.
Any suggestions for a solution? Is there a workaround, or is there another approach that would be better suited?
Non-functional code I'm using so far to give a better idea of what I'm trying to do:
...
locals {
appIps = split(",", azurerm_app_service.appService.outbound_ip_addresses)
}
resource "azurerm_mysql_firewall_rule" "appFirewallRule" {
count = length(appIps)
depends_on = [azurerm_app_service.appService]
name = "appService-${count.index}"
resource_group_name = "myResourceGroup"
server_name = azurerm_mysql_server.databaseServer.name
start_ip_address = local.appIps[count.index]
end_ip_address = local.appIps[count.index]
}
...
This returns the error:
Error: Invalid count argument
on main.tf line 331, in resource "azurerm_mysql_firewall_rule" "appFirewallRule":
331: count = length(local.appIps)
The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.
I dig deeper there and I think I have a solution which works at least for me :)
The core of the problem here is in the necessity to do the whole thing in 2 steps (we can't have not yet known values as arguments for count and for_each). It can be solved with explicit imperative logic or actions (like using -targetonce or commenting out and then uncommenting). Besides, it's not declarative it's also not suitable for automation via CI/CD (I am using Terraform Cloud and not the local environment).
So I am doing it just with Terraform resources, the only "imperative" pattern is to trigger the pipeline (or local run) twice.
Check my snippet:
data "azurerm_resources" "web_apps_filter" {
resource_group_name = var.rg_system_name
type = "Microsoft.Web/sites"
required_tags = {
ProvisionedWith = "Terraform"
}
}
data "azurerm_app_service" "web_apps" {
count = length(data.azurerm_resources.web_apps_filter.resources)
resource_group_name = var.rg_system_name
name = data.azurerm_resources.web_apps_filter.resources[count.index].name
}
data "azurerm_resources" "func_apps_filter" {
resource_group_name = var.rg_storage_name
type = "Microsoft.Web/sites"
required_tags = {
ProvisionedWith = "Terraform"
}
}
data "azurerm_app_service" "func_apps" {
count = length(data.azurerm_resources.func_apps_filter.resources)
resource_group_name = var.rg_storage_name
name = data.azurerm_resources.func_apps_filter.resources[count.index].name
}
locals {
# flatten ensures that this local value is a flat list of IPs, rather
# than a list of lists of IPs.
# distinct ensures that we have only uniq IPs
web_ips = distinct(flatten([
for app in data.azurerm_app_service.web_apps : [
split(",", app.possible_outbound_ip_addresses)
]
]))
func_ips = distinct(flatten([
for app in data.azurerm_app_service.func_apps : [
split(",", app.possible_outbound_ip_addresses)
]
]))
}
resource "azurerm_postgresql_firewall_rule" "pgfr_func" {
for_each = toset(local.web_ips)
name = "web_app_ip_${replace(each.value, ".", "_")}"
resource_group_name = var.rg_storage_name
server_name = "${var.project_abbrev}-pgdb-${local.region_abbrev}-${local.environment_abbrev}"
start_ip_address = each.value
end_ip_address = each.value
}
resource "azurerm_postgresql_firewall_rule" "pgfr_web" {
for_each = toset(local.func_ips)
name = "func_app_ip_${replace(each.value, ".", "_")}"
resource_group_name = var.rg_storage_name
server_name = "${var.project_abbrev}-pgdb-${local.region_abbrev}-${local.environment_abbrev}"
start_ip_address = each.value
end_ip_address = each.value
}
The most important piece there is azurerm_resources resource - I am using it to do filtering on what web apps are already existing in my resource group (and managed by automation). I am doing DB firewall rules on that list, next terraform run, when newly created web app is there, it will also whitelist the lastly created web app.
An interesting thing is also the filtering of IPs - a lot of them are duplicated.
At the moment, using -target as a workaround is a better choice. Because with how Terraform works at present, it considers this sort of configuration to be incorrect. Using resource computed outputs as arguments to count and for_each is not recommended. Instead, using variables or derived local values which are known at plan time is the preferred approach. If you choose to go ahead with using computed values for count/for_each, this will sometimes require you to work around this using -target as illustrated above. For more details, please refer to here
Besides, the bug will be fixed in the pre-release 0.14 code. For more details, please

Terraform module output not resolving for input variables in other module

Terraform Version
Terraform v0.11.11
+ provider.azurerm v1.21.0
Terraform Configuration Files
I have left many required fields for brevity (all other config worked before I added the connection strings).
# modules/function/main.tf
variable "conn-value" {}
locals {
conn = "${map("name", "mydb", "value", "${var.conn-value}", "type", "SQLAzure")}"
}
resource "azurerm_function_app" "functions" {
connection_string = "${list(local.conn)}"
# ...
}
# modules/db/main.tf
# ... other variables declared
resource "azurerm_sql_server" "server" {
# ...
}
output "connection-string" {
value = "Server=tcp:${azurerm_sql_server.server.fully_qualified_domain_name},1433;Initial Catalog=${var.catalog};Persist Security Info=False;User ID=${var.login};Password=${var.login-password};MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=200;"
}
# main.tf
module "my_db" {
source = "modules/db"
}
module "my_app" {
source = "modules/function"
conn-value = "${module.my_db.connection-string}"
# ...
}
Expected Behavior on terraform plan
The module.my_db.connection-string output resolves to a string when passed to the my_app conn-value variable and is able to be used in the map/list passed to the azurerm_function_app.functions.connection_string variable.
Actual Behavior on terraform plan
I get this error:
module.my_app.azurerm_function_app.functions: connection_string: should be a list
If I replace "${var.conn-value}" in the modules/function/main.tf locals with just a string, it works.
Update
In response to to this comment, I updated the script above with the connection string construction.
I finally found the GitHub issue that references the problem I am having (I found the issue through this gist comment). This describes the problem exactly:
Assigning values to nested blocks is not supported, but appears to work in certain cases due to a number of coincidences...
This limitation is in <= v0.11, but is apparently fixed in v0.12 with the dynamic block.

How to use dynamic resource names in Terraform?

I would like to use the same terraform template for several dev and production environments.
My approach:
As I understand it, the resource name needs to be unique, and terraform stores the state of the resource internally. I therefore tried to use variables for the resource names - but it seems to be not supported. I get an error message:
$ terraform plan
var.env1
Enter a value: abc
Error asking for user input: Error parsing address 'aws_sqs_queue.SqsIntegrationOrderIn${var.env1}': invalid resource address "aws_sqs_queue.SqsIntegrationOrderIn${var.env1}"
My terraform template:
variable "env1" {}
provider "aws" {
region = "ap-southeast-2"
}
resource "aws_sqs_queue" "SqsIntegrationOrderIn${var.env1}" {
name = "Integration_Order_In__${var.env1}"
message_retention_seconds = 86400
receive_wait_time_seconds = 5
}
I think, either my approach is wrong, or the syntax. Any ideas?
You can't interpolate inside the resource name. Instead what you should do is as #BMW have mentioned in the comments, you should make a terraform module that contains that SqsIntegrationOrderIn inside and takes env variable. Then you can use the module twice, and they simply won't clash. You can also have a look at a similar question I answered.
I recommend using a different workspace for each environment. This allows you to specify your configuration like this:
variable "env1" {}
provider "aws" {
region = "ap-southeast-2"
}
resource "aws_sqs_queue" "SqsIntegrationOrderIn" {
name = "Integration_Order_In__${var.env1}"
message_retention_seconds = 86400
receive_wait_time_seconds = 5
}
Make sure to make the name of the "aws_sqs_queue" resource depending on the environment (e.g. by including it in the name) to avoid name conflicts in AWS.

Resources