I've Terraformed an AWS EKS cluster and EFS filesystem.
Everytime Terraform is applied, the EFS is replaced. This should not be happening.
Versions:
Terraform: v1.1.7
EKS v1.20
The resources are:
resource "aws_efs_file_system" "efs_fs" {
creation_token = var.eks_efs_name
encrypted = "true"
kms_key_id = data.aws_kms_alias.efs.arn
performance_mode = "generalPurpose"
throughput_mode = "bursting"
tags = merge(var.aws_tags, {
Name = "${var.eks_efs_name}"
})
}
resource "aws_efs_mount_target" "efs_mt" {
for_each = toset(var.eks_wrk_subnets)
file_system_id = aws_efs_file_system.efs_fs.id
security_groups = [aws_security_group.eks_efs.id]
subnet_id = each.value
}
resource "aws_efs_access_point" "efs_ap" {
file_system_id = aws_efs_file_system.efs_fs.id
tags = merge(var.aws_tags, {
Name = "${var.eks_efs_name}"
})
}
resource "aws_security_group" "eks_efs" {
name = "${var.eks_efs_name}-sg"
vpc_id = var.vpc_id
ingress {
from_port = 2049
to_port = 2049
protocol = "tcp"
cidr_blocks = [var.vpc_cidr_block]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
ipv6_cidr_blocks = ["::/0"]
}
}
Is there a lifecycle hook missing?
Any pointers or ideas would be much appreciated :-)
Plan output
There's too much sensitive info to present here, but most EFS resources are being recreated, EFS, Mounts, Access Point and an SG Rule . But what I did noticed is when kms_key_id uses a variable with the KMS key ARN, rather than a datasource, the issue stops. Why is this?
resource "aws_efs_file_system" "efs_fs" {
creation_token = var.eks_efs_name
encrypted = "true"
// kms_key_id = data.aws_kms_alias.efs.arn
kms_key_id = "arn:aws:kms:eu-west-2:012345678901:key/d7812345c-b388-421a-bf1b-0b11212345c"
performance_mode = "generalPurpose"
throughput_mode = "bursting"
tags = merge(var.aws_tags, {
Name = "${var.eks_efs_name}"
})
}
var.eks_efs_name
This value is not changing.
Datasource with depends_on triggers recreate
I came across this article whereby similar issues were seen, when using a datasource lookup inside a security group resource, the resource was recreated on each apply. The fix was to remove depends_on. Is this a known bug?
Related
I am struggling to get an ECS task to be able to see an EFS volume.
The terraform config is:
EFS DEFINITION
resource "aws_efs_file_system" "persistent" {
encrypted = true
}
resource "aws_efs_access_point" "access" {
file_system_id = aws_efs_file_system.persistent.id
}
resource "aws_efs_mount_target" "mount" {
for_each = {for net in aws_subnet.private : net.id => {id = net.id}}
file_system_id = aws_efs_file_system.persistent.id
subnet_id = each.value.id
security_groups = [aws_security_group.efs.id]
}
TASK DEFINITION
resource "aws_ecs_task_definition" "app" {
family = "backend-app-task"
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_role.arn
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = var.fargate_cpu
memory = var.fargate_memory
container_definitions = data.template_file.backendapp.rendered
volume {
name = "persistent"
efs_volume_configuration {
file_system_id = aws_efs_file_system.persistent.id
root_directory = "/opt/data"
transit_encryption = "ENABLED"
transit_encryption_port = 2999
authorization_config {
access_point_id = aws_efs_access_point.access.id
iam = "ENABLED"
}
}
}
}
SECURITY GROUP
resource "aws_security_group" "efs" {
name = "efs-security-group"
vpc_id = aws_vpc.main.id
ingress {
protocol = "tcp"
from_port = 2999
to_port = 2999
security_groups = [aws_security_group.ecs_tasks.id]
cidr_blocks = [for net in aws_subnet.private : net.cidr_block]
}
}
TASK ROLE
resource "aws_iam_role" "ecs_task_role" {
name = "ecsTaskRole"
assume_role_policy = data.aws_iam_policy_document.ecs_task_execution_role_base.json
managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonElasticFileSystemFullAccess","arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy", aws_iam_policy.ecs_exec_policy.arn]
}
As I understand the AWS docs, the IAM role should have access, and the security group should be passing traffic, but the error suggests that the task cannot resolve the EFS instance.
The error message is:
ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-0000000000000.efs.eu-west-2.amazonaws.com" - check that your file system ID is correct.
I've manually confirmed in the console that the EFS id is correct, so I can only conclude that it cannot resolve due to a network/permissions issue.
-- EDIT --
ECS SERVICE DEFINITION
resource "aws_ecs_service" "main" {
name = "backendservice"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.app.arn
desired_count = var.app_count
launch_type = "FARGATE"
enable_execute_command = true
network_configuration {
security_groups = [aws_security_group.ecs_tasks.id]
subnets = aws_subnet.private.*.id
assign_public_ip = true
}
load_balancer {
target_group_arn = aws_alb_target_group.app.id
container_name = "server"
container_port = var.app_port
}
depends_on = [aws_alb_listener.backend]
}
ECS TASK SECURITY GROUP
resource "aws_security_group" "ecs_tasks" {
name = "backend-ecs-tasks-security-group"
description = "allow inbound access from the ALB only"
vpc_id = aws_vpc.main.id
ingress {
protocol = "tcp"
from_port = var.app_port
to_port = var.app_port
security_groups = [aws_security_group.lb.id]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
}
}
VPC DEFINITION (minus internet gateway)
data "aws_availability_zones" "available" {
}
resource "aws_vpc" "main" {
cidr_block = "172.17.0.0/16"
}
# Create var.az_count private subnets, each in a different AZ
resource "aws_subnet" "private" {
count = var.az_count
cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
vpc_id = aws_vpc.main.id
}
resource "aws_subnet" "public" {
count = var.az_count
cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, var.az_count + count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
vpc_id = aws_vpc.main.id
map_public_ip_on_launch = true
}
EDIT
It turned out the mountPoints block was missing from the container template. I have now added it, but the outcome is the same.
Run into this problem, this is what I did to resolve:
platform_version = "1.4.0" on the aws_ecs_service, its possible its no longer relevant but was in this blog post I used as a starting point https://medium.com/#ilia.lazebnik/attaching-an-efs-file-system-to-an-ecs-task-7bd15b76a6ef
Made sure aws_efs_mount_target has subnet_id and security_groups set to same ones used by the service. Since I use multiple subnets for multiple availability zones I created mount target for each of them.
Added standard EFS port 2049 to ingress as without that mount operation was failing with timeout error.
I am trying to create a AWS VPC module using Terraform. I am making VPC secondary CIDR an optional feature of the module.
if secondary_cidr = true, then create subnets using the seocandary_cidr and network acls
The issue I am running into is with Network ACLs. Network ACLs creation using terraform uses a list subnet IDs to associate to the NACL. I want to create one network ACL to associate primary subnets and secondary subnets only when secondary_cidr=true
See the code below:
cidr1_subnets = {
CIDR1_SUBNETS = [aws_subnet.app1-az1.id, aws_subnet.app1-az2.id, aws_subnet.app1-az3.id]
}
cidr2_subnets = {
exists = {
CIDR2_SUBNETS = [aws_subnet.app2-az1.id, aws_subnet.app2-az2.id, aws_subnet.app2-az3.id]
}
not_exists = {}
}
}
resource "aws_network_acl" "app" {
vpc_id = aws_vpc.main.id
egress {
protocol = "-1"
rule_no = 100
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 0
to_port = 0
}
ingress {
protocol = "-1"
rule_no = 100
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 0
to_port = 0
}
subnet_ids = merge(
local.cidr1_subnets,
local.cidr2_subnets[var.secondary_cidr == true ? true : false]
)
}```
I think you are after the following:
subnet_ids = var.secondary_cidr == true ? merge(local.cidr1_subnets, local.cidr2_subnets) : local.cidr1_subnets
Btw, your locals and the use of merge will fail anyway, but this is a problem of other issue I guess.
➜ terraform -v
Terraform v0.12.24
+ provider.aws v2.60.0
My terraform example.tf:
locals {
standard_tags = {
team = var.team
project = var.project
component = var.component
environment = var.environment
}
}
provider "aws" {
profile = "profile"
region = var.region
}
resource "aws_key_pair" "security_key" {
key_name = "security_key"
public_key = file(".ssh/key.pub")
}
# New resource for the S3 bucket our application will use.
resource "aws_s3_bucket" "project_bucket" {
# NOTE: S3 bucket names must be unique across _all_ AWS accounts, so
# this name must be changed before applying this example to avoid naming
# conflicts.
bucket = "project-bucket"
acl = "private"
}
resource "aws_security_group" "ssh_allow" {
name = "allow-all-ssh"
ingress {
cidr_blocks = [
"0.0.0.0/0"
]
from_port = 22
to_port = 22
protocol = "tcp"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "http_allow" {
name = "allow-all-http"
ingress {
cidr_blocks = [
"0.0.0.0/0"
]
from_port = 80
to_port = 80
protocol = "tcp"
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_instance" "example" {
ami = "ami-08ee2516c7709ea48"
instance_type = "t2.micro"
security_groups = [aws_security_group.ssh_allow.name, aws_security_group.http_allow.name]
key_name = aws_key_pair.security_key.key_name
connection {
type = "ssh"
user = "centos"
private_key = file(".ssh/key")
host = self.public_ip
}
provisioner "local-exec" {
command = "echo ${aws_instance.example.public_ip} > ip_address.txt"
}
provisioner "remote-exec" {
inline = [
"sudo yum -y install nginx",
"sudo systemctl start nginx"
]
}
depends_on = [aws_s3_bucket.project_bucket, aws_key_pair.security_key]
dynamic "tag" {
for_each = local.standard_tags
content {
key = tag.key
value = tag.value
propagate_at_launch = true
}
}
}
And when I run terraform plan
I got the following error:
➜ terraform plan
Error: Unsupported block type
on example.tf line 94, in resource "aws_instance" "example":
94: dynamic "tag" {
Blocks of type "tag" are not expected here.
There isn't a block type called tag defined in the schema for the aws_instance resource type. There is an argument called tags, which is I think the way to get the result you were looking for here:
tags = local.standard_tags
I expect you are thinking of the tag block in aws_autoscaling_group, which deviates from the usual design of tags arguments in AWS provider resources because for this resource type in particular each tag has the additional attribute propagate_at_launch. That attribute only applies to autoscaling groups because it decides whether instances launched from the autoscaling group will inherit a particular tag from the group itself.
unfortunately since the aws_instance resource's tags attribute is a map, w/in the HCL constructs atm, it cannot exist as repeatable blocks like a tag attribute in the aws_autoscaling_group example seen here in the Dynamic Nested Blocks section: https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/
but from your comment, it seems you're trying to set the tags attribute with perhaps a map of key/value pairs? in this case, this is certainly doable 😄 you should be able to directly set the field with tags = local.standard_tags
OR if you intend to set the tags attribute with a list of key/value pairs, a for loop can work as well by doing something like:
locals {
standard_tags = [
{
name = "a"
number = 1
},
{
name = "b"
number = 2
},
{
name = "c"
number = 3
},
]
}
resource "aws_instance" "test" {
...
tags = {
for tag in local.standard_tags:
tag.name => tag.number
}
}
I am trying to provision RDS instance using terraform template and my template looks like this
template.tf
resource "aws_security_group" "web-server-security"{
name = "webserver-sg"
description = "webserver security group"
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
tags{
Name = "web-server-sg"
}
resource "aws_security_group" "db-server-sg" {
name = "db-server"
description = "dbserver security group"
ingress {
from_port = 3306
to_port = 3306
protocol = "tcp"
security_groups = ["${aws_security_group.web-server-security.id}"]
}
tags{
Name = "db-server-sg"
}
}
resource "aws_db_instance" "echomany_db" {
name = "echomanydb"
engine = "mysql"
engine_version = "5.7"
storage_type = "gp2"
allocated_storage = 20
instance_class = "db.t2.micro"
username = "${var.AWS_DB_USERNAME}"
password = "${var.AWS_DB_PASSWORD}"
parameter_group_name = "default.mysql5.7"
skip_final_snapshot = true
security_group_names = [
"${aws_security_group.db-server-sg.id}"
]
tags{
Name = "db-server"
}
}
However i get this following error:
1 error(s) occurred:
* aws_db_instance.echomany_db: 1 error(s) occurred:
aws_db_instance.echomany_db: Error creating DB Instance:
InvalidParameterCombination: DB Security Groups can only be
associated with VPC DB Instances using API versions 2012-01-15
through 2012-09-17.
status code: 400, request id: a19ea8ea-8ea0-46e4-97c6-b946419df9a3
i dont know whats the problem and how to fix this issue.
As mentioned by the documentation vpc_security_group_ids should be used instead of security_group_names which is a deprecated argument.
Parameter named
security_group_names = [
"${aws_security_group.db-server-sg.id}"
]
can only be used while using ec2-classic mode or outside VPC.
Use vpc_security_group_ids instead.
you can use vpc_security_groups_ids = [ ] instead of security_group_names because it can only be used while using ec2-classic mode
example:
vpc_security_group_ids=["${aws_security_group.rds.id}"]
I've written a simple module to provision a variable AZ numbered AWS VPC. It creates the route tables, gateways, routes, etc., but I'm having trouble keeping the security groups part DRY, i.e. keeping the module re-usable when specifying security groups.
This is as close as I can get:
varibles.tf:
variable "staging_security_groups" {
type = "list"
default = [ {
"name" = "staging_ssh"
"from port" = "22"
"to port" = "22"
"protocol" = "tcp"
"cidrs" = "10.0.0.5/32,10.0.0.50/32,10.0.0.200/32"
"description" = "Port 22"
} ]
}
main.tf:
resource "aws_security_group" "this_security_group" {
count = "${length(var.security_groups)}"
name = "${lookup(var.security_groups[count.index], "name")}"
description = "${lookup(var.security_groups[count.index], "description")}"
vpc_id = "${aws_vpc.this_vpc.id}"
ingress {
from_port = "${lookup(var.security_groups[count.index], "from port")}"
to_port = "${lookup(var.security_groups[count.index], "to port")}"
protocol = "${lookup(var.security_groups[count.index], "protocol")}"
cidr_blocks = ["${split(",", lookup(var.security_groups[count.index], "cidrs"))}"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags {
Name = "${lookup(var.security_groups[count.index], "name")}"
environment = "${var.name}"
terraform = "true"
}
}
Now this is fine, as long as what you want is to create a security group per port :) What I really need, is some way to call ingress the number of times that there are values in the variable staging_security_groups[THE SECURITY GROUP].from_port (please excuse the made-up notation).
You could look at using aws_security_group_rule instead of having your rules inline. You can then create a module like this:
module/sg/sg.tf
resource "aws_security_group" "default" {
name = "${var.security_group_name}"
description = "${var.security_group_name} group managed by Terraform"
vpc_id = "${var.vpc_id}"
}
resource "aws_security_group_rule" "egress" {
type = "egress"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
description = "All egress traffic"
security_group_id = "${aws_security_group.default.id}"
}
resource "aws_security_group_rule" "tcp" {
count = "${var.tcp_ports == "default_null" ? 0 : length(split(",", var.tcp_ports))}"
type = "ingress"
from_port = "${element(split(",", var.tcp_ports), count.index)}"
to_port = "${element(split(",", var.tcp_ports), count.index)}"
protocol = "tcp"
cidr_blocks = ["${var.cidrs}"]
description = ""
security_group_id = "${aws_security_group.default.id}"
}
resource "aws_security_group_rule" "udp" {
count = "${var.udp_ports == "default_null" ? 0 : length(split(",", var.udp_ports))}"
type = "ingress"
from_port = "${element(split(",", var.udp_ports), count.index)}"
to_port = "${element(split(",", var.udp_ports), count.index)}"
protocol = "udp"
cidr_blocks = ["${var.cidrs}"]
description = ""
security_group_id = "${aws_security_group.default.id}"
}
modules/sg/variables.tf
variable "tcp_ports" {
default = "default_null"
}
variable "udp_ports" {
default = "default_null"
}
variable "cidrs" {
type = "list"
}
variable "security_group_name" {}
variable "vpc_id" {}
Use the module in your main.tf
module "sg1" {
source = "modules/sg"
tcp_ports = "22,80,443"
cidrs = ["10.0.0.5/32", "10.0.0.50/32", "10.0.0.200/32"]
security_group_name = "SomeGroup"
vpc_id = "${aws_vpc.this_vpc.id}"
}
module "sg2" {
source = "modules/sg"
tcp_ports = "22,80,443"
cidrs = ["10.0.0.5/32", "10.0.0.50/32", "10.0.0.200/32"]
security_group_name = "SomeOtherGroup"
vpc_id = "${aws_vpc.this_vpc.id}"
}
References:
For why optionally excluding a resource with count looks like this (source):
count = "${var.udp_ports == "default_null" ? 0 : length(split(",", var.udp_ports))}"
And the variable is set to:
variable "udp_ports" {
default = "default_null"
}
I managed to create really simple yet dynamic security group module that you can use. Idea here is to have ability to add any port you desire, and add to that port any range of ips you like. You can even remove egress from module as it will be created by default, or follow idea i used in ingress so you have granular egress rules (if you wish so).
module/sg/sg.tf
data "aws_subnet_ids" "selected" {
vpc_id = "${var.data_vpc_id}"
}
resource "aws_security_group" "main" {
name = "${var.sg_name}-sg"
vpc_id = "${var.data_vpc_id}"
description = "Managed by Terraform"
ingress = ["${var.ingress}"]
lifecycle {
create_before_destroy = true
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
module/sg/vars.tf
variable "sg_name" {}
variable "data_vpc_id" {}
variable "ingress" {
type = "list"
default = []
}
ingress var needs to be type list. If you call vpc id manually you dont need data bit in module, im calling my vpc_id from terraform state that is stored in s3.
main.tf
module "aws_security_group" {
source = "module/sg/"
sg_name = "name_of_sg"
data_vpc_id = "${data.terraform_remote_state.vpc.vpc_id}"
ingress = [
{
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
description = "Managed by Terraform"
},
{
from_port = 0
to_port = 100
protocol = "tcp"
cidr_blocks = ["10.10.10.10/32"]
description = "Managed by Terraform"
},
{
from_port = 2222
to_port = 2222
protocol = "tcp"
cidr_blocks = ["100.100.100.0/24"]
description = "Managed by Terraform"
},
]
}
You can add as many ingress blocks you like, i have only 3 for test purposes. Hope this helps.
Note: You can follow this idea for many resources like RDS, where you need to specify parameters in parameter group or even tags. Cheers
Not sure if it was available at the time Brandon Miller's answer was written, but avoid count loops as they are ordered. So if you add or delete one port, it will cause all rules after it to be rebuilt as they rely on the count index, which changes. Far better to use a for_each loop. Make sure you use set not lists for this eg
variable "tcp_ports" {
default = [ ]
# or maybe default = [ "22", "443" ]
type = set(string)
}
resource "aws_security_group_rule" "tcp" {
for_each = var.tcp_ports
description = "Allow ${var.cdir} to connect to TCP port ${each.key}"
type = "ingress"
from_port = each.key
to_port = each.key
protocol = "tcp"
cidr_blocks = var.cdir
security_group_id = aws_security_group.default.id
}
Now you can add and delete ports without incurring unnecessary create and destroys
you you cant alter your data from lists to sets for any reason just wrap it eg
toset(var.tcp_ports)
or use a local to munge your data accordingly. You can also use maps as well