Question
If there a way to get the assigned IP address of an aws_lb resource at the time aws_lb is created by Terraform?
As in AWS documentation - NLB - To find the private IP addresses to whitelist, we can find out the IP address associated to ELB.
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose Network Interfaces.
In the search field, type the name of your Network Load Balancer.
There is one network interface per load balancer subnet.
On the Details tab for each network interface, copy the address from
Primary private IPv4 IP.
Background
To be able to setup security group to white list the ELB IP address as Network Load Balancer cannot not have Security Group as in Network Load Balancers don't have Security Groups.
Considered aws_network_interface but it does not work with an error.
Error: no matching network interface found
Also I think datasource assumes the resource already exists and cannot be used for the resource to be created by Terraform.
More elegent solution using only HCL in Terraform :
data "aws_network_interface" "lb" {
for_each = var.subnets
filter {
name = "description"
values = ["ELB ${aws_lb.example_lb.arn_suffix}"]
}
filter {
name = "subnet-id"
values = [each.value]
}
}
resource "aws_security_group" "lb_sg" {
vpc_id = var.vpc_id
ingress {
from_port = 0
to_port = 0
protocol = "tcp"
cidr_blocks = formatlist("%s/32", [for eni in data.aws_network_interface.lb : eni.private_ip])
description = "Allow connection from NLB"
}
}
Source : https://github.com/terraform-providers/terraform-provider-aws/issues/3007
Hope this helps.
The solution from #user1297406 leads to an exeption. data.aws_network_interface.lb is tuple with 2 elements. Correct syntax is:
data "aws_network_interface" "lb" {
count = length(var.vpc_private_subnets)
filter {
name = "description"
values = ["ELB ${aws_alb.lb.arn_suffix}"]
}
filter {
name = "subnet-id"
values = [var.vpc_private_subnets[count.index]]
}
}
resource "aws_security_group_rule" "lb_sg" {
from_port = 0
protocol = "TCP"
to_port = 0
type = "ingress"
cidr_blocks = formatlist("%s/32", data.aws_network_interface.lb.*.private_ip)
}
Using external provider
Get the NLB IP using Python/boto3 invoking from external provider.
nlb_private_ips.tf
variable "nlb_name" {
}
variable "vpc_id" {
}
variable "region" {
}
data "external" "get_nlb_ips" {
program = ["python", "${path.module}/get_nlb_private_ips.py"]
query = {
aws_nlb_name = "${var.nlb_name}"
aws_vpc_id = "${var.vpc_id}"
aws_region = "${var.region}"
}
}
output "aws_nlb_ip_decoded" {
value = "${jsondecode(data.external.get_nlb_ips.result.private_ips)}"
}
output "aws_nlb_ip_encoded" {
value = "${data.external.get_nlb_ips.result.private_ips}"
}
get_nlb_private_ips.py
import boto3
import json
import sys
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code
Args:
obj: object to serialize into JSON
"""
_serialize = {
"int": lambda o: int(o),
"float": lambda o: float(o),
"decimal": lambda o: float(o) if o % 1 > 0 else int(o),
"date": lambda o: o.isoformat(),
"datetime": lambda o: o.isoformat(),
"str": lambda o: o,
}
return _serialize[type(obj).__name__.lower()](obj)
def pretty_json(dict):
"""
Pretty print Python dictionary
Args:
dict: Python dictionary
Returns:
Pretty JSON
"""
return json.dumps(dict, indent=2, default=json_serial, sort_keys=True, )
def get_nlb_private_ips(data):
ec2 = boto3.client('ec2', region_name=data['aws_region'])
response = ec2.describe_network_interfaces(
Filters=[
{
'Name': 'description',
'Values': [
"ELB net/{AWS_NLB_NAME}/*".format(
AWS_NLB_NAME=data['aws_nlb_name'])
]
},
{
'Name': 'vpc-id',
'Values': [
data['aws_vpc_id']
]
},
{
'Name': 'status',
'Values': [
"in-use"
]
},
{
'Name': 'attachment.status',
'Values': [
"attached"
]
}
]
)
# print(pretty_json(response))
interfaces = response['NetworkInterfaces']
# ifs = list(map(lamba index: interfaces[index]['PrivateIpAddresses'], xrange(len(interfaces))))
# --------------------------------------------------------------------------------
# Private IP addresses associated to an interface (ENI)
# Each association has the format:
# {
# "Association": {
# "IpOwnerId": "693054447076",
# "PublicDnsName": "ec2-52-88-47-177.us-west-2.compute.amazonaws.com",
# "PublicIp": "52.88.47.177"
# },
# "Primary": true,
# "PrivateDnsName": "ip-10-5-1-205.us-west-2.compute.internal",
# "PrivateIpAddress": "10.5.1.205"
# },
# --------------------------------------------------------------------------------
associations = [
association for interface in interfaces
for association in interface['PrivateIpAddresses']
]
# --------------------------------------------------------------------------------
# Get IP from each IP association
# --------------------------------------------------------------------------------
private_ips = [
association['PrivateIpAddress'] for association in associations
]
return private_ips
def load_json():
data = json.load(sys.stdin)
return data
def main():
data = load_json()
"""
print(data['aws_region'])
print(data['aws_vpc_id'])
print(data['aws_nlb_name'])
"""
ips = get_nlb_private_ips(data)
print(json.dumps({"private_ips": json.dumps(ips)}))
if __name__ == '__main__':
main()
Using aws_network_interfaces datasource
After aws_lb has been created.
data "aws_network_interfaces" "this" {
filter {
name = "description"
values = ["ELB net/${aws_lb.this.name}/*"]
}
filter {
name = "vpc-id"
values = ["${var.vpc_id}"]
}
filter {
name = "status"
values = ["in-use"]
}
filter {
name = "attachment.status"
values = ["attached"]
}
}
locals {
nlb_interface_ids = "${flatten(["${data.aws_network_interfaces.this.ids}"])}"
}
data "aws_network_interface" "ifs" {
count = "${length(local.nlb_interface_ids)}"
id = "${local.nlb_interface_ids[count.index]}"
}
output "aws_lb_network_interface_ips" {
value = "${flatten([data.aws_network_interface.ifs.*.private_ips])}"
}
i would suggest to use the data "dns_a_record_set" to get the IPs:
data "dns_a_record_set" "nlb_ips" {
host = aws_lb.<your_alb>.dns_name
}
You can find the documentation under https://registry.terraform.io/providers/hashicorp/dns/latest/docs/data-sources/dns_a_record_set
Related
So i have a terraform variable type list(string) that is called zones and contains
zones = [
"example.com",
"example2.com",
"example3.com",
...
]
and i m using data cloudflare_zones resource to fetch all zones info
data "cloudflare_zones" "zones" {
for_each = toset(var.zones)
filter {
name = each.value
}
}
Output for each of the zones
data.cloudflare_zones.zones["example.com"]
{
"filter" = tolist([
{
"account_id" = ""
"lookup_type" = "exact"
"match" = ""
"name" = "example.com"
"paused" = false
"status" = ""
},
])
"id" = "9f7xxx3xxxx"
"zones" = tolist([
{
"id" = "e13xxxx"
"name" = "example.com"
},
])
}
To fetch the zone id you need to parse data.cloudflare_zones as below:
data.cloudflare_zones.zones["example.com"].zones[0].id
What i want to create then is a variable that will be an object with all the zones names as keys and zone ids ad values, so i can use them in other resources.
For Example:
zones_ids =
{
"example.com" = "xxxzone_idxxx",
"example2.com" = "xxxzone_id2xxx",
"example3.com" = "xxxzone_id3xxx",
...
}
I would like to achieve this inside locals block
locals {
...
}
That should be easy:
locals {
zones_ids = { for k,v in data.cloudflare_zones.zones: k => v.zones[0].id }
}
Or alternatively:
locals {
zones_ids = { for k,v in data.cloudflare_zones.zones: v.zones[0].name => v.zones[0].id }
}
The above answers helped me here but did not give me the final answer. For anyone looking to update A records for cloudflare with a list of domain names that gets the zone_ids for you. Here is how I did it:
locals {
domains = ["example1.com", "example2.com"]
}
data "cloudflare_zones" "zones" {
count = "${length(local.domains)}"
filter {
name = "${element(local.domains, count.index)}"
}
}
locals {
zones_ids = { for k,v in data.cloudflare_zones.zones: k => v.zones[0].id }
}
resource "cloudflare_record" "redir-A-record" {
for_each = local.zones_ids
zone_id = each.value
name = "#"
value = "24.1.1.1"
type = "A"
proxied = false
}
resource "cloudflare_record" "redir-A-record-www" {
for_each = local.zones_ids
zone_id = each.value
name = "www"
value = "24.1.1.1"
type = "A"
proxied = false
}
Getting output for these values did not seem to work based on the above answer. This could of just been my confusion but I wanted to print out the zone_id for each domain. I found since it is a tuple it requires the use of a number instead of a name so I was required to do the following to get the proper output:
# Get information for Domain 1
output "Domain_Information" {
value = data.cloudflare_zones.zones[0].zones[0].id
}
# Get information for Domain 2
output "Domain_Information2" {
value = data.cloudflare_zones.zones[1].zones[0].id
}
There is a way to loop this in output with Terraform but in my case I only had 2 domains and did not need to spend additional time on this.
Now when I want to spin up a server in AWS and have multiple domains point to 1 IP address this code works.
This line here posted by #Marko E was the solution to my issues for looping and saving the data that could be used later.:
locals {
zones_ids = { for k,v in data.cloudflare_zones.zones: k => v.zones[0].id }
}
So, I have a list of items such as
public_subnet_ip = flatten(module.master_nodes.*.public_ip)
Which contain a list of IPs of my master nodes. The output of the variable is something like this
public_subnet_ip = ["10.0.0.1", "10.0.0.2", "10.0.0.3"]
Now, I have a resource with these arguments.
resource "some_resource" "this" {
name = var.name
master_nodes = [
{
address = "an-ip-address"
},
]
}
What I would like to do is to iterate through the public_subnet_id variable, pick each element, and fill in the address field in the resource.
I tried this and it created 3 separate master_nodes resources instead of a single master node resource which contains 3 nodes in it.
resource "some_resource" "this" {
name = var.name
count = length(var.public_subnet_ip)
master_nodes = [
{
address = element(var.public_subnet_ip, count.index)
},
]
}
This is what I'm having right now
resource "some_resource" "this" {
name = var.name
master_nodes = [
{
address = public_subnet_ip[0]
},
{
address = public_subnet_ip[1]
},
{
address = public_subnet_ip[2]
},
]
}
I have seeing some examples regarding how to pass annotations when deploying a helm chart via terraform but none of then are working as expected, in this case, im trying to create a service assining a private ip on a specific subnet, but instead, its creating a public IP.
My terraform files:
locals {
helm_general = {
# Reference values
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
elasticsearch = {
name = "elasticsearch"
chart = "elastic/elasticsearch"
tag = "7.14.0"
namespace = "elasticsearch"
set = [
{
name = "nodeSelector.agentpool"
value = "general"
},
{
name = "replicas"
value = "1"
},
{
name = "minimumMasterNodes"
value = "1"
},
{
name = "image"
value = "docker.elastic.co/elasticsearch/elasticsearch"
},
{
name = "imageTag"
value = "7.14.0"
},
{
name = "resources.requests.cpu"
value = "10m"
},
{
name = "resources.requests.memory"
value = "128Mi"
},
{
name = "volumeClaimTemplate.reosources.requests.storage"
value = "4Gi"
},
{
name = "persistence.enabled"
value = "false"
},
{
name = "service.type"
value = "LoadBalancer"
},
{
name = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
value = "true"
},
{
name = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal-subnet"
value = "somesubnet"
},
]
timeout = "900"
}
}
}
Helm deployment
resource "helm_release" "helm" {
provider = helm.general
for_each = local.helm_general
name = each.value.name
chart = each.value.chart
namespace = format(each.value.namespace)
dynamic "set" {
iterator = item
for_each = each.value.set == null ? [] : each.value.set
content {
name = item.value.name
value = item.value.value
}
}
depends_on = [kubernetes_namespace.general]
}
Plan / apply output
https://i.imgur.com/uAdmblM.png
And what is currently being deployed is a public ip instead of a private ip:
Namespace: elasticsearch
Labels: app=elasticsearch-master
app.kubernetes.io/managed-by=Helm
chart=elasticsearch
heritage=Helm
release=elasticsearch
Annotations: meta.helm.sh/release-name: elasticsearch
meta.helm.sh/release-namespace: elasticsearch
Selector: app=elasticsearch-master,chart=elasticsearch,release=elasticsearch
Type: LoadBalancer
IP Families: <none>
IP: xx
IPs: xxx
LoadBalancer Ingress: redacted public ip
Port: http 9200/TCP
TargetPort: 9200/TCP
NodePort: http 32083/TCP
Endpoints:
Port: transport 9300/TCP
TargetPort: 9300/TCP
NodePort: transport 32638/TCP
Endpoints:
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 1s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer <invalid> service-controller Ensured load balancer
References that i have been following:
https://github.com/hashicorp/terraform-provider-helm/issues/125
https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release
Edit:
error message:
Error: unable to decode "": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|nternal":true},"labe|..., bigger context
..|beta.kubernetes.io/azure-load-balancer-internal":true},"labels":{"app":"elasticsearch-master","chart|...
with helm_release.helm["elasticsearch"],
on aks-general-helm.tf line 1, in resource "helm_release" "helm":
1: resource "helm_release" "helm" {
I just faced a similar issue, and here is what worked for me:
{
name = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
value = "true"
},
I think the issue is how it is concatenated. The service in the chart manifest for elastic official is service.annotations:{} so you need to append .service then use \\.
if you have multiple you can try something like this, pass annotations as map and set them using
dynamic "set" {
for_each = var.ingress_annotations
content {
name = replace(set.key, ".", "\\.")
value = set.value
}
}
One way of doing this without the escape characters and keeping the original YAML format would be using values attribute of the helm_release resource. Would be curious to know if there was a specific used-case to not use it in the first place.
resource "helm_release" "helm" {
provider = helm.general
for_each = local.helm_general
name = each.value.name
chart = each.value.chart
namespace = format(each.value.namespace)
values = each.value.values ##CHANGE IS HERE ##
dynamic "set" {
iterator = item
for_each = each.value.set == null ? [] : each.value.set
content {
name = item.value.name
value = item.value.value
}
}
depends_on = [kubernetes_namespace.general]
}
The local in your case would be adjusted to below, you can still keep set for something which has a dependency on any terraform resources or with any other logical reasons.
locals {
helm_general = {
# Reference values
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
elasticsearch = {
[...]
values = [file("${path.module}/elasticsearch-values.yaml")]
[...]
}
}
}
There has to be a new file elasticsearch-values.yaml at the same path (which can be adjusted with any relative path as per the local.helm_general.elasticsearch.values) location, where this terraform configurations exist.
# Reference values can be adapted as per the upstream chart.
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
service:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "somesubnet"
I am attempting to loop over a module equal to the number of times a map appears inside a nested list of maps as follows:
vars.tf
variable "http_tcp_listeners" {
description = "A list of maps describing the HTTP listeners or TCP ports for this NLB"
type = any
default = [
{
"http_tcp_listener" = [
{
port = "80"
protocol = "TCP"
},
{
port = "7364"
protocol = "TCP"
}
]
},
{
"http_tcp_listener" = [
{
port = "8080"
protocol = "TCP"
},
{
port = "7365"
protocol = "TCP"
}
]
}
]
}
main.tf
module "create_network_lb" {
count = length(var."http_tcp_listeners")
source = "../../modules/lb"
subnets = tolist(data.aws_subnet_ids.private_compute[0].ids)
vpc_id = sort(data.aws_vpcs.platform_private_vpc.ids)[0]
target_groups = lookup(var.target_groups[count.index], "target_group", null)
http_tcp_listeners = lookup(var.http_tcp_listeners[count.index], "http_tcp_listener", null)
module
resource "aws_lb_listener" "frontend_http_tcp" {
count = var.create_lb ? length(var.http_tcp_listeners) : 0
load_balancer_arn = aws_lb.default[0].arn
port = var.http_tcp_listeners[count.index]["port"]
protocol = var.http_tcp_listeners[count.index]["protocol"]
dynamic "default_action" {
for_each = length(keys(var.http_tcp_listeners[count.index])) == 0 ? [] : [var.http_tcp_listeners[count.index]]
content {
type = lookup(default_action.value, "action_type", "forward")
target_group_arn = contains([null, "", "forward"], lookup(default_action.value, "action_type", "")) ? aws_lb_target_group.main[lookup(default_action.value, "target_group_index", count.index)].id : null
dynamic "redirect" {
for_each = length(keys(lookup(default_action.value, "redirect", {}))) == 0 ? [] : [lookup(default_action.value, "redirect", {})]
content {
path = lookup(redirect.value, "path", null)
host = lookup(redirect.value, "host", null)
port = lookup(redirect.value, "port", null)
protocol = lookup(redirect.value, "protocol", null)
query = lookup(redirect.value, "query", null)
status_code = redirect.value["status_code"]
}
}
dynamic "fixed_response" {
for_each = length(keys(lookup(default_action.value, "fixed_response", {}))) == 0 ? [] : [lookup(default_action.value, "fixed_response", {})]
content {
content_type = fixed_response.value["content_type"]
message_body = lookup(fixed_response.value, "message_body", null)
status_code = lookup(fixed_response.value, "status_code", null)
}
}
}
}
}
When performing a "terraform plan", it displays only the last "http_tcp_listener" value. The variable for the module must be in format "[{port=80, protocol="TCP"},{port=7364, protocol="TCP"}]" hence, everything after each iteration of "http_tcp_listener".
During troubleshooting, Terraform seems to think that the variable is a tuple with one element per the error:
Error: Invalid index
on main.tf line 86, in module "create_network_lb":
86: http_tcp_listeners = [lookup(var.http_tcp_listeners[1], "http_tcp_listener")]
|----------------
| var.http_tcp_listeners is tuple with 1 element
The given key does not identify an element in this collection value.
If I manually change one of the keys from "http_tcp_listener" to "http_tcp_listener1", and reflect this in the main.tf lookup value, it will display that value. i.e, if I rename the first key and reference it, terraform plan will display ports 80 and 7364 instead of 8080 and 7365.
Any help would be greatly appreciated.
Solved by re-creating the data structure, and using for_each to call the module. Details here.
I have a terraform template that creates multiple EC2 instances.
I then create a few Elastic Network interfaces in the AWS console and added them as locals in the terraform template.
Now, I want to map the appropriate ENI to the instance hence I added locals and variables as below.
locals {
instance_ami = {
A = "ami-11111"
B = "ami-22222"
C = "ami-33333"
D = "ami-4444"
}
}
variable "instance_eni" {
description = "Pre created Network Interfaces"
default = [
{
name = "A"
id = "eni-0a15890a6f567f487"
},
{
name = "B"
id = "eni-089a68a526af5775b"
},
{
name = "C"
id = "eni-09ec8ad891c8e9d91"
},
{
name = "D"
id = "eni-0fd5ca23d3af654a9"
}
]
}
resource "aws_instance" "instance" {
for_each = local.instance_ami
ami = each.value
instance_type = var.instance_type
key_name = var.keypair
root_block_device {
delete_on_termination = true
volume_size = 80
volume_type = "gp2"
}
dynamic "network_interface" {
for_each = [for eni in var.instance_eni : {
eni_id = eni.id
}]
content {
device_index = 0
network_interface_id = network_interface.value.eni_id
delete_on_termination = false
}
}
}
I am getting below error:
Error: Error launching source instance: InvalidParameterValue: Each network interface requires a
unique device index.
status code: 400, request id: 4a482753-bddc-4fc3-90f4-2f1c5e2472c7
I think terraform is tyring to attach all 4 ENI's to single instance only.
What should be done to attach ENI's to an individual instance?
The configuration you shared in your question is asking Terraform to manage four instances, each of which has four network interfaces associated with it. That's problematic in two different ways:
All for of the network interfaces on each instance are configured with the same device_index, which is invalid and is what the error message here is reporting.
Even if you were to fix that, it would then try to attach the same four network interfaces to four different EC2 instances, which is invalid: each network interface can be attached to only one instance at a time.
To address that and get the behavior you wanted, you only need one network_interface block, whose content is different for each of the instances:
locals {
instance_ami = {
A = "ami-11111"
B = "ami-22222"
C = "ami-33333"
D = "ami-4444"
}
}
variable "instance_eni" {
description = "Pre created Network Interfaces"
default = [
{
name = "A"
id = "eni-0a15890a6f567f487"
},
{
name = "B"
id = "eni-089a68a526af5775b"
},
{
name = "C"
id = "eni-09ec8ad891c8e9d91"
},
{
name = "D"
id = "eni-0fd5ca23d3af654a9"
}
]
}
locals {
# This expression is transforming the instance_eni
# value into a more convenient shape: a map from
# instance key to network interface id. You could
# also choose to just change directly the
# definition of variable "instance_eni" to already
# be such a map, but I did it this way to preserve
# your module interface as given.
instance_network_interfaces = {
for ni in var.instance_eni : ni.name => ni.id
}
}
resource "aws_instance" "instance" {
for_each = local.instance_ami
ami = each.value
instance_type = var.instance_type
key_name = var.keypair
root_block_device {
delete_on_termination = true
volume_size = 80
volume_type = "gp2"
}
network_interface {
device_index = 0
network_interface_id = local.instance_network_interfaces[each.key]
delete_on_termination = false
}
}
Now each instance has only one network interface, with each one attaching to the corresponding ENI ID given in your input variable. Referring to each.key and each.value is how we can create differences between each of the instances declared when using resource for_each; we don't need any other repetition constructs inside unless we want to create nested repetitions, like having a dynamic number of network interfaces for each instance.