locals.tf file - parsing jsonencode body - terraform

Wondering if anyone has ran tackled it. So, I need to be able to generate list of egress CIDR blocks that is currently available for listing over an API. Sample output is the following:
[
{
"description": "blahnet-public-acl",
"metadata": {
"broadcast": "192.168.1.191",
"cidr": "192.168.1.128/26",
"ip": "192.168.1.128",
"ip_range": {
"start": "192.168.1.128",
"end": "192.168.1.191"
},
"netmask": "255.255.255.192",
"network": "192.168.1.128",
"prefix": "26",
"size": "64"
}
},
{
"description": "blahnet-public-acl",
"metadata": {
"broadcast": "192.168.160.127",
"cidr": "192.168.160.0/25",
"ip": "192.168.160.0",
"ip_range": {
"start": "192.168.160.0",
"end": "192.168.160.127"
},
"netmask": "255.255.255.128",
"network": "192.168.160.0",
"prefix": "25",
"size": "128"
}
}
]
So, I need convert it to Azure Firewall
###############################################################################
# Firewall Rules - Allow Access To TEST VMs
###############################################################################
resource "azurerm_firewall_network_rule_collection" "azure-firewall-azure-test-access" {
for_each = local.egress_ips
name = "azure-firewall-azure-test-rule"
azure_firewall_name = azurerm_firewall.public_to_test.name
resource_group_name = var.resource_group_name
priority = 105
action = "Allow"
rule {
name = "test-access"
source_addresses = local.egress_ips[each.key]
destination_ports = ["43043"]
destination_addresses = ["172.16.0.*"]
protocols = [ "TCP"]
}
}
So, bottom line is that allowed IP addresses have to be a list of strings for the "source_addresses" parameter, such as this:
["192.168.44.0/24","192.168.7.0/27","192.168.196.0/24","192.168.229.0/24","192.168.138.0/25",]
I configured data_sources.tf file:
data "http" "allowed_networks_v1" {
url = "https://testapiserver.com/api/allowed/networks/v1"
}
...and in locals.tf, I need to configure
locals {
allowed_networks_json = jsondecode(data.http.allowed_networks_v1.body)
egress_ips = ...
}
...and that's where I am stuck. How can parse that data in locals.tf file so I can reference it from within TF ?
Thanks a metric ton!!

I'm assuming that the list of string you are referring to are the objects under: metadata.cidr we can extract that with a for loop in a local, and also do a distinct just in case we get duplicates.
Here is a sample code
data "http" "allowed_networks_v1" {
url = "https://raw.githack.com/heldersepu/hs-scripts/master/json/networks.json"
}
locals {
allowed_networks_json = jsondecode(data.http.allowed_networks_v1.body)
distinct_cidrs = distinct(flatten([
for key, value in local.allowed_networks_json : [
value.metadata.cidr
]
]))
}
output "data" {
value = local.distinct_cidrs
}
and here is the output of a plan on that:
terraform plan
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
Terraform will perform the following actions:
Plan: 0 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ data = [
+ "192.168.1.128/26",
+ "192.168.160.0/25",
]
Here is the code for your second sample:
data "http" "allowed_networks_v1" {
url = "https://raw.githack.com/akamalov/testfile/master/networks.json"
}
locals {
allowed_networks_json = jsondecode(data.http.allowed_networks_v1.body)
distinct_cidrs = distinct(flatten([
for key, value in local.allowed_networks_json.egress_nat_ranges : [
value.metadata.cidr
]
]))
}
output "data" {
value = local.distinct_cidrs
}

Related

terraform templatefile Function Invalid template interpolation value(AKS Cluster Name)

I try to provision an azure dashboard using terraform module and template file but I get an error message:
Call to function "templatefile" failed:
src/main/terraform/environment/modules/someservice/resources/dashboards/infrastructure-dashboard.tpl:41,39-55:
Invalid template interpolation value; Cannot include the given value in a
string template: string required., and 25 other diagnostic(s).
The error is being caused by the code in the template. It does not accept the variable declared in the dashboard properties. The code in the template causing the error:
"clusterName": "${k8s_cluster_name}",
Terraform code:
resource "azurerm_dashboard" "infra-dashboard" {
name = "${upper(terraform.workspace)}-Infrastructure"
resource_group_name = azurerm_resource_group.rgp-dashboards.name
location = azurerm_resource_group.rgp-dashboards.location
tags = {
source = "terraform"
}
dashboard_properties = templatefile("${path.module}/resources/dashboards/infrastructure-dashboard.tpl",
{
md_content = var.infra_dashboard_md_content,
dashboard_title = var.infra_dashboard_title,
dashboard_subtitle = var.infra_dashboard_subtitle,
sub_id = data.azurerm_subscription.current.subscription_id,
rgp_k8s = azurerm_resource_group.seervice-k8s.name,
k8s_cluster_name = azurerm_kubernetes_cluster.service-k8s.*.name,
rgp_service_env_global_name = azurerm_resource_group.service-env-global.name,
log_analyt_wrkspc = local.log_analyt_wrkspc
})
}
Snippet from the tmeplate where the error occurs:
"1": {
"position": {
"x": 0,
"y": 4,
"colSpan": 9,
"rowSpan": 4
},
"metadata": {
"inputs": [
{
"name": "queryParams",
"value": {
"metricQueryId": "node-count",
"clusterName": "${k8s_cluster_name}",
"clusterResourceId": "/subscriptions/${sub_id}/resourceGroups/${rgp_k8s}/providers/Microsoft.ContainerService/managedClusters/${k8s_cluster_name}",
"workspaceResourceId": "/subscriptions/${sub_id}/resourcegroups/${rgp_service_env_global_name}/providers/microsoft.operationalinsights/workspaces/${log_analyt_wrkspc}",
"timeRange": {
"options": {},
"relative": {
"duration": 21600000
}
},
"cpuFilterSelection": "total",
"memoryFilterSelection": "total_memoryrss"
}
}
For example:
join(",", azurerm_kubernetes_cluster.service-k8s.*.name)
will join all entries with an ",". Replace it with whatever you need
As luk2302 said azurerm_kubernetes_cluster.service-k8s.*.name it is not a string. In fact it is a tuple

Terraform - How to convert lists into map (How to fetch AMI tags using terraform)

I am trying to fetch the tags of AMI using AWS CLI and want to reuse the values from the output.
I have a terraform code below which is returning outputs in string format(Maybe not sure of format) which I want to convert into a map object.
variable "ami" {
default = "ami-xxxx"
}
locals {
tags = {
"platform" = lookup(data.local_file.read_tags.content, "platform", "") #Expecting to get platform from Map of read_tags
}
}
data "template_file" "log_name" {
template = "${path.module}/output.log"
}
resource "null_resource" "ami_tags" {
provisioner "local-exec" {
command = "aws ec2 describe-tags --filters Name=resource-id,Values=${var.ami} --query Tags[*].[Key,Value] > ${data.template_file.log_name.rendered}"
}
}
data "local_file" "read_tags" {
filename = "${data.template_file.log_name.rendered}"
depends_on = ["null_resource.ami_tags"]
}
output "tags" {
value = local.tags
}
output "cli-output-tags" {
value = "${concat(data.local_file.read_tags.content)}"
}
output of cli-output-tags is below:
[
[
"ENV",
"DEV"
],
[
"Name",
"Base-AMI"
],
[
"platform",
"Linux"
]
]
How can I convert this output into Map as below using terraform/(jq command), or is there any other way to fetch required values directly from cli-output-tags output:
{
ENV = "DEV",
Name = "Base-AMI",
platform = "Linux"
}
I have also tried changing the CLI command a bit like below but still not able to fetch values as expected:
'Tags[].{Key:Key,Value:Value}'
Resulted below output:
[
{
"Key": "ENV",
"Value": "DEV"
},
{
"Key": "Name",
"Value": "Base-AMI"
},
{
"Key": "platform",
"Value": "Linux"
}
]
You could use zipmap:
output "cli-output-tags" {
value = zipmap(
jsondecode(data.local_file.read_tags.content)[*][0],
jsondecode(data.local_file.read_tags.content)[*][1]
)
}
The code first changes string data from your file to json, then
gets all first elements [*][0] (same for second elements [*][1]), and zips them into map.
How can I convert this output into Map as below
One way would be to use jq as follows (assuming cli-output-tags is the name of the file holding the JSON array of arrays):
jq -r -f '"{", (.[] | "\(.[0]) = \"\(.[1])\""), "}"' cli-output-tags

Failed to create AWSConfig rule: InvalidParameterValueException: Blank spaces are not acceptable for input parameter: threshold

I am trying to create an aws config rule for checking that cloudtrail alarms are enabled. I get the following error Error: Error creating AWSConfig rule: Failed to create AWSConfig rule: InvalidParameterValueException: Blank spaces are not acceptable for input parameter: threshold. when I run terraform apply. I'm not sure what the formatting issue is in the input parameters argument (see input_parameters). The apply works if I remove everything except for metricName i.e
input_parameters = "{\"metricName\":\"CloudTrailConfigChanges\"}"
Any help would be greatly appreciated.
resource aws_config_config_rule ensure-log-alarm-exists-for-cloudtrail {
name = "ensure-log-alarm-exists-for-cloudtrail"
description = "Checks whether cloudwatch alarm is on for cloudtrail configuration changes"
source {
owner = "AWS"
source_identifier = "CLOUDWATCH_ALARM_SETTINGS_CHECK"
}
input_parameters = "{\"metricName\":\"CloudTrailConfigChanges\",\"threshold\":1,\"evaluationPeriod\":1,\"period\":300,\"comparisionOperator\":\"GreaterThanOrEqualToThreshold\",\"statistic\":\"Sum\"}"
}
It seems like there is an issue parsing type ints from json strings: https://github.com/hashicorp/terraform-provider-aws/issues/773#issuecomment-385454229
I get the same error even with
input_parameters =<<EOF
{
"metricName":"CloudTrailConfigChanges",
"threshold":1
}
EOF
or
input_parameters = jsonencode({"metricName":"CloudTrailConfigChanges","threshold"=1})
Converting wrapping the int value in quotes does not work either.
resource "aws_config_config_rule" "ensure-log-alarm-exists-for-cloudtrail" {
name = "ensure-log-alarm-exists-for-cloudtrail"
description = "Checks whether cloudwatch alarm is on for cloudtrail configuration changes"
source {
owner = "AWS"
source_identifier = "CLOUDWATCH_ALARM_SETTINGS_CHECK"
}
input_parameters = jsonencode({
metricName = "CloudTrailConfigChanges"
threshold = "1"
})
}
The code above produces the following error:
Unknown parameters provided in the inputParameters:
With your examples you're still specifying the threshold as an integer. Try making it a string.
resource "aws_config_config_rule" "ensure-log-alarm-exists-for-cloudtrail" {
name = "ensure-log-alarm-exists-for-cloudtrail"
description = "Checks whether cloudwatch alarm is on for cloudtrail configuration changes"
source {
owner = "AWS"
source_identifier = "CLOUDWATCH_ALARM_SETTINGS_CHECK"
}
input_parameters = jsonencode({
metricName = "CloudTrailConfigChanges"
threshold = "1"
})
}
I ran into an error like this, and what resolved it for me was to add a condition. I don't fully understand why this worked and why it caused this error without the condition, but I saw the condition used in an AWS example.
For example, I first tried using something straightforward like this to reference a parameter:
"InputParameters": {
"appNames": {
"Ref": "ApplicationNames"
}
}
When my resource referenced the ApplicationNames parameter directly like this, it was giving that error. But using Conditions and referencing the parameter this way caused it to work, as in this full template example:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Just a stripped-down example",
"Parameters": {
"ApplicationNames": {
"Type": "String",
"Default": "This Has Spaces",
"MinLength": "1",
"ConstraintDescription": "This parameter is required."
}
},
"Conditions": {
"ApplicationNamesDefined": {
"Fn::Not": [
{
"Fn::Equals": [
"",
{
"Ref": "ApplicationNames"
}
]
}
]
}
},
"Resources": {
"SampleRule": {
"Type": "AWS::Config::ConfigRule",
"DependsOn": "SecurityHubCustomUpdaterFunction",
"Properties": {
"ConfigRuleName": "TheName",
"Description": "It was here that I was getting 'Blank spaces are not acceptable for input parameter: applicationNames' before I added the Conditions and Fn::If to reference it",
"InputParameters": {
"appNames": {
"Fn::If": [
"ApplicationNamesDefined",
{
"Ref": "ApplicationNames"
},
{
"Ref": "AWS::NoValue"
}
]
}
},
"Scope": {
"ComplianceResourceTypes": [
"AWS::SSM::ManagedInstanceInventory"
]
},
"Source": {
"Owner": "AWS",
"SourceIdentifier": "EC2_MANAGEDINSTANCE_APPLICATIONS_REQUIRED"
}
}
}
}
}
So you may want to try with Conditions usage.

How do I make my data template file recognise a number in terraform

THe problem I am facing now is this. I am trying to make my policy more flexible. So I shifted them into a file instead of using EOF.
How to make the template file recognise a number value?
"${max_untagged_images}" and "${max_tagged_images}" are suppose to be numbers.
Aws lifecycle policy:
resource "aws_ecr_lifecycle_policy" "lifecycle" {
count = length(aws_ecr_repository.repo)
repository = aws_ecr_repository.repo[count.index].name
depends_on = [aws_ecr_repository.repo]
policy = var.policy_type == "app" ? data.template_file.lifecycle_policy_app.rendered : data.template_file.lifecycle_policy_infra.rendered
}
Data template:
data "template_file" "lifecycle_policy_app" {
template = file("lifecyclePolicyApp.json")
vars = {
max_untagged_images = var.max_untagged_images
max_tagged_images = var.max_tagged_images
env = var.env
}
}
Policy:
{
"rules": [
{
"rulePriority": 1,
"description": "Expire untagged images older than ${max_untagged_images} days",
"selection": {
"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days",
"countNumber": "${max_untagged_images}"
},
"action": {
"type": "expire"
}
},
{
"rulePriority": 2,
"description": "Expire tagged images of ${env}, older than ${max_tagged_images} days",
"selection": {
"tagStatus": "tagged",
"countType": "imageCountMoreThan",
"countNumber": "${max_tagged_images}",
"tagPrefixList": [
"${env}"
]
},
"action": {
"type": "expire"
}
}
]
}
I would try the following 2 steps:
Remove the double quotes that around the "${max_tagged_images}"
Use terraform function called tonumber in order to convert it to a number:
tonumber("1")
(Follow the official documentation: https://www.terraform.io/docs/configuration/functions/tonumber.html)

Terraform - Passing type Object as a parameter to Azure Template Deployment

I am tying to to provision Azure AD Domain Service using Terraform by giving Terraform the Azure ARM template, this is because Terrafrom does not support provisioning Azure AD Domain Service natively.
I have exported the ARM Template and it's parameters, one of the parameters is called "notificationSettings" which is a type Object and looks like below :
"notificationSettings": {
"value": {
"notifyGlobalAdmins": "Enabled",
"notifyDcAdmins": "Enabled",
"additionalRecipients": []
}
}
Other parameters are all strings and I can pass them without any issue, for example:
"apiVersion" = "2017-06-01"
I have tried passing this object to parameters like below :
"notificationSettings" = [{
"notifyGlobalAdmins" = "Enabled"
"notifyDcAdmins" ="Enabled"
"additionalRecipients" = []
}]
However, when I execute terrafrom apply, terrafrom complains and say:
Inappropriate value for attribute "parameters": element
"notificationSettings": string required.
How do I pass parameters type of Object to template body?
I have also tried giving the entire ARM json parameter as a file to terrafrom by using parameters_body option like below :
parameters_body = "${file("${path.module}/temp/params.json")}"
however, I am getting the followig error when executing the terrafrom script:
The request content was invalid and could not be deserialized: 'Error
converting value
"https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#"
to type
'Microsoft.WindowsAzure.ResourceStack.Frontdoor.Data.Definitions.DeploymentParameterDefinition'.
Path 'properties.parameters.$schema', line 1, position 2952.'.
Below is the params.json file:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"apiVersion": {
"value": "2017-06-01"
},
"sku": {
"value": "Standard"
"location": {
"value": "westus"
},
"notificationSettings": {
"value": {
"notifyGlobalAdmins": "Enabled",
"notifyDcAdmins": "Enabled",
"additionalRecipients": []
}
},
"subnetName": {
"value": "xxxx"
},
"vnetName": {
"value": "xxxx"
},
"vnetAddressPrefixes": {
"value": [
"10.0.1.0/24"
]
},
"subnetAddressPrefix": {
"value": "10.0.1.0/24"
},
"nsgName": {
"value": "xxxxx"
}
}
}
There is a way to pass arbitrary data structures from Terraform to ARM.
There are two ways to pass data to the ARM template within the azure_template_deployment provider
use the parameters block, which is limited to string parameters only
use the parameters_body block, which is pretty much arbitrary JSON.
I find the easiest way to use the parameters block is to create a local variable with the structure I require, then call jsonencode on it. I also like to keep the ARM template in a separate file and pull it in via a file() call, reducing the complexity of the terraform.
locals {
location = "string"
members = [
"array",
"of",
"members"
]
enabled = true
tags = {
"key" = "value",
"simple" = "store"
}
# this is the format required by ARM templates
parameters_body = {
location = {
value = "${local.location}"
},
properties = {
value = {
users = {
members = "${local.members}"
}
boolparameter = "${local.enabled}"
}
}
tags = {
value = "${module.global.tags}"
}
}
}
resource "azurerm_template_deployment" "sample" {
name = "sample"
resource_group_name = "rg"
deployment_mode = "Incremental"
template_body = "${file("${path.module}/arm/sample_arm.json")}"
parameter_body = "${jsonencode(local.parameters_body)}"
}
The only caveat I've found is that the bool parameters pass as a string, so declare them as a string in the ARM parameters section, then use a ARM function to convert to bool
"parameters: {
"boolParameter": {
"type": "string"
}
},
"variables": {
"boolVariable": "[bool(parameters('boolParameter'))]"
},
"resources": [
...
"boolArm": "[variables('boolVariable')]",
...
]

Resources