How to attach Two target group against single ECS services - terraform

I am looking for a way to attach two target group against single ECS services, in other my container exposes two port but I am only able to map one port against my service to LB.
So far I am able to create a new listener and target group but after target group creation I can see everything as per expectation but the target group show There are no targets registered to this target group
Here are my target group and listener configuration
target_group:
resource "aws_lb_target_group" "e_admin" {
name = "${var.env_prefix_name}-admin"
port = 5280
protocol = "HTTP"
vpc_id = "${aws_vpc.VPC.id}"
health_check {
path = "/admin"
healthy_threshold = 2
unhealthy_threshold = 10
port = 5280
timeout = 90
interval = 100
matcher = "401,200"
}
}
Listener:'
resource "aws_lb_listener" "admin" {
load_balancer_arn = "${aws_lb.admin_lb.arn}"
port = "5280"
protocol = "HTTP"
default_action {
target_group_arn = "${aws_lb_target_group.e_admin.id}"
type = "forward"
}
}
My question is how I can add ECS cluster Autoscaling group or how I can add all the instances running in the ECS cluster to this target group?

AWS recently announced support for multiple target groups for an ECS service.
The, currently unreleased, 2.22.0 version of the AWS provider contains support for this by adding more load_balancer blocks to the aws_ecs_service resource. Example from the acceptance tests:
resource "aws_ecs_service" "with_alb" {
name = "example"
cluster = "${aws_ecs_cluster.main.id}"
task_definition = "${aws_ecs_task_definition.with_lb_changes.arn}"
desired_count = 1
iam_role = "${aws_iam_role.ecs_service.name}"
load_balancer {
target_group_arn = "${aws_lb_target_group.test.id}"
container_name = "ghost"
container_port = "2368"
}
load_balancer {
target_group_arn = "${aws_lb_target_group.static.id}"
container_name = "ghost"
container_port = "4501"
}
depends_on = [
"aws_iam_role_policy.ecs_service",
]
}

Accodring to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html,
There is a limit of one load balancer or target group per service.
If you want to attach a autoscaling group to the target group, use aws_autoscaling_attachment,
https://www.terraform.io/docs/providers/aws/r/autoscaling_attachment.html
resource "aws_autoscaling_attachment" "asg_attachment_bar" {
autoscaling_group_name = "${aws_autoscaling_group.your_asg.id}"
alb_target_group_arn = "${aws_alb_target_group.e_admin.arn}"
}

You can add multiple define multiple target groups for the same ecs service using the load_balancer block.
resource "aws_ecs_service" "ecs_service_1" {
name = "service-1"
cluster = aws_ecs_cluster.ecs_cluster_prod.id
task_definition = aws_ecs_task_definition.ecs_task_definition_1.arn
desired_count = 1
launch_type = "FARGATE"
enable_execute_command = true
# Target group 1
load_balancer {
target_group_arn = aws_lb_target_group.lb_tg_1.arn
container_name = "app"
container_port = 8080
}
# Target group 2
load_balancer {
target_group_arn = aws_lb_target_group.lb_tg_2.arn
container_name = "app"
container_port = 8080
}
network_configuration {
subnets = [aws_subnet.subnet_a.id, aws_subnet.subnet_b.id]
security_groups = [aws_security_group.sg_internal.id]
assign_public_ip = true
}
tags = {
Name = "service-1"
ManagedBy = "terraform"
Environment = "prod"
}
}
You can map the same container and port to both target groups in case of having an external and an internal load balancer for example

Related

Problems in the connection process between moodle and rds in aws eks

I'm using terraform to distribute eks, rds, and kubernetes and try to run moodle. The image for moodle is using bitnami/moodle:latest.
Both eks and rds work on the same vpc, and the are generated very well. However, when I try to distribute moodle in kubernetes with rds, I experience problems with not accessing the database. What's the problem with connecting databases?
root#master:~/moodle_deploy# kubectl get pod
NAME READY STATUS RESTARTS AGE
moodle-54f477b7f8-nh98h 0/1 CrashLoopBackOff 15 70m
moodle-54f477b7f8-rhzzj 0/1 CrashLoopBackOff 15 70m
root#master:~/moodle_deploy# kubectl logs moodle-54f477b7f8-rhzzj
moodle 06:33:12.43
moodle 06:33:12.43 Welcome to the Bitnami moodle container
moodle 06:33:12.43 Subscribe to project updates by watching https://github.com/bitnami/containers
moodle 06:33:12.43 Submit issues and feature requests at https://github.com/bitnami/containers/issuesmoodle 06:33:12.43
moodle 06:33:12.44 INFO ==> ** Starting Moodle setup **
realpath: /bitnami/apache/conf: No such file or directory
moodle 06:33:12.46 INFO ==> Configuring Apache ServerTokens directive
moodle 06:33:12.49 INFO ==> Configuring PHP options
moodle 06:33:12.49 INFO ==> Setting PHP expose_php option
moodle 06:33:12.51 INFO ==> Validating settings in MYSQL_CLIENT_* env vars
moodle 06:33:12.53 INFO ==> Validating settings in POSTGRESQL_CLIENT_* env vars
moodle 06:33:12.69 INFO ==> Ensuring Moodle directories exist
moodle 06:33:12.78 INFO ==> Trying to connect to the database server
moodle 06:34:12.90 ERROR ==> Could not connect to the database
root#master:~/moodle_deploy# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
moodle-pv-claim Bound pvc-eefbb1d9-a6fd-4cf5-b044-159145798114 1Gi RWO gp2 59m
root#master:~/moodle_deploy# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-eefbb1d9-a6fd-4cf5-b044-159145798114 1Gi RWO Delete Bound default/moodle-pv-claim gp2 59m
These are the code that defined rds and kubernetes among the terraform files I distributed
//SECURITY GROUP
resource "aws_security_group" "secgrp-rds" {
name = "secgrp-rdis"
description = "Allow MySQL Port"
vpc_id = module.vpc.vpc_id
ingress {
description = "Allowing Connection for mysql"
from_port = 3306
to_port = 3306
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "Allowing Connection for SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_db_subnet_group" "sub_ids" {
name = "main"
subnet_ids = module.vpc.private_subnets
tags = {
Name = "DB subnet group"
}
}
//RDS INSTANCE
resource "aws_db_instance" "rds" {
depends_on=[aws_security_group.secgrp-rds]
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t2.micro"
allocated_storage = 20
storage_type = "gp2"
name = "moodledb"
identifier = "moodledb"
username = "root"
password = "test12345"
parameter_group_name = "default.mysql5.7"
#publicly_accessible = true
skip_final_snapshot = true
apply_immediately = true
iam_database_authentication_enabled = true
db_subnet_group_name = aws_db_subnet_group.sub_ids.id
vpc_security_group_ids = [aws_security_group.secgrp-rds.id]
}
//CREATE DEPLOYMENT - provide rds environment variables
resource "kubernetes_deployment" "moodle" {
depends_on = [aws_db_instance.rds]
metadata {
name = "moodle"
labels = {
App = "moodle"
}
}
spec {
replicas = 2
selector {
match_labels = {
App = "moodle"
}
}
template {
metadata {
labels = {
App = "moodle"
}
}
spec {
container {
image = "bitnami/moodle:latest"
name = "moodle"
env{
name = "MOODLE_DATABASE_TYPE"
value = "mysqli"
}
env{
name = "MOODLE_DATABASE_HOST"
value = aws_db_instance.rds.endpoint
}
env{
name = "MOODLE_DATABASE_USER"
value = aws_db_instance.rds.username
}
env{
name = "MOODLE_DATABASE_PASSWORD"
value = aws_db_instance.rds.password
}
env{
name = "MOODLE_DATABASE_NAME"
value = aws_db_instance.rds.db_name
}
port {
container_port = 8080
}
volume_mount {
name = "moodle-ps"
mount_path = "/opt/bitnami/apache2/htdocs/"
}
resources {
limits= {
cpu = "0.5"
memory = "512Mi"
}
requests= {
cpu = "250m"
memory = "50Mi"
}
}
}
volume {
name = "moodle-ps"
persistent_volume_claim {
claim_name = "moodle-pv-claim"
}
}
}
}
}
}
resource "kubernetes_service" "moodle" {
depends_on = [kubernetes_deployment.moodle]
metadata {
name = "moodle"
}
spec {
port {
port = 80
target_port = 8080
}
type = "LoadBalancer"
}
}
resource "kubernetes_storage_class" "kubeSC" {
metadata {
name = "kubese"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
}
}
resource "kubernetes_persistent_volume_claim" "pvc" {
depends_on=[kubernetes_storage_class.kubeSC]
metadata {
name = "moodle-pv-claim"
labels = {
"app" = "moodle"
}
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "1Gi"
}
}
}
}

Providing access to EFS from ECS task

I am struggling to get an ECS task to be able to see an EFS volume.
The terraform config is:
EFS DEFINITION
resource "aws_efs_file_system" "persistent" {
encrypted = true
}
resource "aws_efs_access_point" "access" {
file_system_id = aws_efs_file_system.persistent.id
}
resource "aws_efs_mount_target" "mount" {
for_each = {for net in aws_subnet.private : net.id => {id = net.id}}
file_system_id = aws_efs_file_system.persistent.id
subnet_id = each.value.id
security_groups = [aws_security_group.efs.id]
}
TASK DEFINITION
resource "aws_ecs_task_definition" "app" {
family = "backend-app-task"
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_role.arn
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
cpu = var.fargate_cpu
memory = var.fargate_memory
container_definitions = data.template_file.backendapp.rendered
volume {
name = "persistent"
efs_volume_configuration {
file_system_id = aws_efs_file_system.persistent.id
root_directory = "/opt/data"
transit_encryption = "ENABLED"
transit_encryption_port = 2999
authorization_config {
access_point_id = aws_efs_access_point.access.id
iam = "ENABLED"
}
}
}
}
SECURITY GROUP
resource "aws_security_group" "efs" {
name = "efs-security-group"
vpc_id = aws_vpc.main.id
ingress {
protocol = "tcp"
from_port = 2999
to_port = 2999
security_groups = [aws_security_group.ecs_tasks.id]
cidr_blocks = [for net in aws_subnet.private : net.cidr_block]
}
}
TASK ROLE
resource "aws_iam_role" "ecs_task_role" {
name = "ecsTaskRole"
assume_role_policy = data.aws_iam_policy_document.ecs_task_execution_role_base.json
managed_policy_arns = ["arn:aws:iam::aws:policy/AmazonElasticFileSystemFullAccess","arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy", aws_iam_policy.ecs_exec_policy.arn]
}
As I understand the AWS docs, the IAM role should have access, and the security group should be passing traffic, but the error suggests that the task cannot resolve the EFS instance.
The error message is:
ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-0000000000000.efs.eu-west-2.amazonaws.com" - check that your file system ID is correct.
I've manually confirmed in the console that the EFS id is correct, so I can only conclude that it cannot resolve due to a network/permissions issue.
-- EDIT --
ECS SERVICE DEFINITION
resource "aws_ecs_service" "main" {
name = "backendservice"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.app.arn
desired_count = var.app_count
launch_type = "FARGATE"
enable_execute_command = true
network_configuration {
security_groups = [aws_security_group.ecs_tasks.id]
subnets = aws_subnet.private.*.id
assign_public_ip = true
}
load_balancer {
target_group_arn = aws_alb_target_group.app.id
container_name = "server"
container_port = var.app_port
}
depends_on = [aws_alb_listener.backend]
}
ECS TASK SECURITY GROUP
resource "aws_security_group" "ecs_tasks" {
name = "backend-ecs-tasks-security-group"
description = "allow inbound access from the ALB only"
vpc_id = aws_vpc.main.id
ingress {
protocol = "tcp"
from_port = var.app_port
to_port = var.app_port
security_groups = [aws_security_group.lb.id]
}
egress {
protocol = "-1"
from_port = 0
to_port = 0
cidr_blocks = ["0.0.0.0/0"]
}
}
VPC DEFINITION (minus internet gateway)
data "aws_availability_zones" "available" {
}
resource "aws_vpc" "main" {
cidr_block = "172.17.0.0/16"
}
# Create var.az_count private subnets, each in a different AZ
resource "aws_subnet" "private" {
count = var.az_count
cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
vpc_id = aws_vpc.main.id
}
resource "aws_subnet" "public" {
count = var.az_count
cidr_block = cidrsubnet(aws_vpc.main.cidr_block, 8, var.az_count + count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
vpc_id = aws_vpc.main.id
map_public_ip_on_launch = true
}
EDIT
It turned out the mountPoints block was missing from the container template. I have now added it, but the outcome is the same.
Run into this problem, this is what I did to resolve:
platform_version = "1.4.0" on the aws_ecs_service, its possible its no longer relevant but was in this blog post I used as a starting point https://medium.com/#ilia.lazebnik/attaching-an-efs-file-system-to-an-ecs-task-7bd15b76a6ef
Made sure aws_efs_mount_target has subnet_id and security_groups set to same ones used by the service. Since I use multiple subnets for multiple availability zones I created mount target for each of them.
Added standard EFS port 2049 to ingress as without that mount operation was failing with timeout error.

ECS Fargate task fails health-check when created with Terraform

I created an ECS cluster, along with a Load Balancer, to expose a basc hello-world node app on Fargate using Terraform. Terraform manages to create my aws resources just fine, and deploys the correct image on ECS Fargate, but the task never passes the initial health-check and restarts indefinitely. I think this is a port-forwarding problem, but I believe my Dockerfile, Load Balancer and Task Definition all expose the correct ports.
Below is the error I see when looking at my service's "events" tab on the ECS dashboard:
service my-first-service (port 2021) is unhealthy in target-group target-group due to (reason Request timed out).
Below is my Application code, the Dockerfile, and the Terraform files I am using to deploy to Fargate:
index.js
const express = require('express')
const app = express()
const port = 2021
app.get('/', (req, res) => res.send('Hello World!'))
app.listen(port, () => console.log(`Example app listening on port ${port}!`))
Dockerfile
# Use an official Node runtime as a parent image
FROM node:12.7.0-alpine
# Set the working directory to /app
WORKDIR '/app'
# Copy package.json to the working directory
COPY package.json .
# Install any needed packages specified in package.json
RUN yarn
# Copying the rest of the code to the working directory
COPY . .
# Make port 2021 available to the world outside this container
EXPOSE 2021
# Run index.js when the container launches
CMD ["node", "index.js"]
application_load_balancer_target_group.tf
resource "aws_lb_target_group" "target_group" {
name = "target-group"
port = 80
protocol = "HTTP"
target_type = "ip"
vpc_id = "${aws_default_vpc.default_vpc.id}" # Referencing the default VPC
health_check {
matcher = "200,301,302"
path = "/"
}
}
resource "aws_lb_listener" "listener" {
load_balancer_arn = "${aws_alb.application_load_balancer.arn}" # Referencing our load balancer
port = "80"
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = "${aws_lb_target_group.target_group.arn}" # Referencing our tagrte group
}
}
application_load_balaner.tf
resource "aws_alb" "application_load_balancer" {
name = "test-lb-tf" # Naming our load balancer
load_balancer_type = "application"
subnets = [ # Referencing the default subnets
"${aws_default_subnet.default_subnet_a.id}",
"${aws_default_subnet.default_subnet_b.id}",
"${aws_default_subnet.default_subnet_c.id}"
]
# Referencing the security group
security_groups = ["${aws_security_group.load_balancer_security_group.id}"]
}
# Creating a security group for the load balancer:
resource "aws_security_group" "load_balancer_security_group" {
ingress {
from_port = 80 # Allowing traffic in from port 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic in from all sources
}
egress {
from_port = 0 # Allowing any incoming port
to_port = 0 # Allowing any outgoing port
protocol = "-1" # Allowing any outgoing protocol
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic out to all IP addresses
}
}
ecs_cluster.tf
resource "aws_ecs_cluster" "my_cluster" {
name = "my-cluster" # Naming the cluster
}
ecs_service.tf
# Providing a reference to our default VPC (these are needed by the aws_ecs_service at the bottom of this file)
resource "aws_default_vpc" "default_vpc" {
}
# Providing a reference to our default subnets (NOTE: Make sure the availability zones match your zone)
resource "aws_default_subnet" "default_subnet_a" {
availability_zone = "us-east-2a"
}
resource "aws_default_subnet" "default_subnet_b" {
availability_zone = "us-east-2b"
}
resource "aws_default_subnet" "default_subnet_c" {
availability_zone = "us-east-2c"
}
resource "aws_ecs_service" "my_first_service" {
name = "my-first-service" # Naming our first service
cluster = "${aws_ecs_cluster.my_cluster.id}" # Referencing our created Cluster
task_definition = "${aws_ecs_task_definition.my_first_task.arn}" # Referencing the task our service will spin up
launch_type = "FARGATE"
desired_count = 1 # Setting the number of containers we want deployed to 1
# NOTE: The following 'load_balancer' snippet was added here after the creation of the application_load_balancer files.
load_balancer {
target_group_arn = "${aws_lb_target_group.target_group.arn}" # Referencing our target group
container_name = "${aws_ecs_task_definition.my_first_task.family}"
container_port = 2021 # Specifying the container port
}
network_configuration {
subnets = ["${aws_default_subnet.default_subnet_a.id}", "${aws_default_subnet.default_subnet_b.id}", "${aws_default_subnet.default_subnet_c.id}"]
assign_public_ip = true # Providing our containers with public IPs
}
}
resource "aws_security_group" "service_security_group" {
ingress {
from_port = 0
to_port = 0
protocol = "-1"
# Only allowing traffic in from the load balancer security group
security_groups = ["${aws_security_group.load_balancer_security_group.id}"]
}
egress {
from_port = 0 # Allowing any incoming port
to_port = 0 # Allowing any outgoing port
protocol = "-1" # Allowing any outgoing protocol
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic out to all IP addresses
}
}
ecs_task_definition.tf
resource "aws_ecs_task_definition" "my_first_task" {
family = "my-first-task" # Naming our first task
container_definitions = <<DEFINITION
[
{
"name": "my-first-task",
"image": "${var.ECR_IMAGE_URL}",
"essential": true,
"portMappings": [
{
"containerPort": 2021,
"hostPort": 2021
}
],
"memory": 512,
"cpu": 256
}
]
DEFINITION
requires_compatibilities = ["FARGATE"] # Stating that we are using ECS Fargate
network_mode = "awsvpc" # Using awsvpc as our network mode as this is required for Fargate
memory = 512 # Specifying the memory our container requires
cpu = 256 # Specifying the CPU our container requires
execution_role_arn = "${aws_iam_role.ecsTaskExecutionRole.arn}"
}
resource "aws_iam_role" "ecsTaskExecutionRole" {
name = "ecsTaskExecutionRole"
assume_role_policy = "${data.aws_iam_policy_document.assume_role_policy.json}"
}
data "aws_iam_policy_document" "assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
role = "${aws_iam_role.ecsTaskExecutionRole.name}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
Where am I going wrong here?
I had same similar issue when I was migrating from k8s to ECS Fargate.
My task could not start, it was nightmare.
Same image in k8s was working great with same health checks.
I can see that you are missing healthCheck in task_definition, at least that was issue for me.
here is my containerDefinition :
container_definitions = jsonencode([{
name = "${var.app_name}-container-${var.environment}"
image = "${var.container_repository}:${var.container_image_version}"
essential = true
environment: concat(
var.custom_env_variables,
[
{
name = "JAVA_TOOL_OPTIONS"
value = "-Xmx${var.container_memory_max_ram}m -XX:MaxRAM=${var.container_memory_max_ram}m -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4"
},
{
name = "SPRING_PROFILES_ACTIVE"
value = var.spring_profile
},
{
name = "APP_NAME"
value = var.spring_app_name
}
]
)
portMappings = [
{
protocol = "tcp"
containerPort = var.container_port
},
{
protocol = "tcp"
containerPort = var.container_actuator_port
}
]
healthCheck = {
retries = 10
command = [ "CMD-SHELL", "curl -f http://localhost:8081/actuator/liveness || exit 1" ]
timeout: 5
interval: 10
startPeriod: var.health_start_period
}
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.main.name
awslogs-stream-prefix = "ecs"
awslogs-region = var.aws_region
}
}
mountPoints = [{
sourceVolume = "backend_efs",
containerPath = "/data",
readOnly = false
}]
}])
there is healthCheck aprt:
healthCheck = {
retries = 10
command = [ "CMD-SHELL", "curl -f http://localhost:8081/actuator/liveness || exit 1" ]
timeout: 5
interval: 10
startPeriod: var.health_start_period
}
container in order to start needs to have a way to check is that task running OK.
And I could only get that via curl . I have one endpoint that returns me is it live or not. You need to specify your, it is jut important that return 200.
Also there is no curl command by default, you need to add it in you DockerFile as that was next issue where I spent few hours, as there was not clear error on ECS.
I added this line:
RUN apt-get update && apt-get install -y --no-install-recommends curl
By the look of it, you are create new VPC with subnets, but there are no route tables defined, no internet gateway and attached to the VPC. So your VPC is simply private and not accessible from the internet, nor it can access ECR to get your docker image.
Maybe instead of creating a new VPC called default_vpc, you want to use an existing default vpc. If so you have to use data source:
data "aws_vpc" "default_vpc" {
default = true
}
to get subnets:
data "aws_subnet_ids" "default" {
vpc_id = data.aws_vpc.default_vpc.id
}
and modify the remaining of the code to reference these data sources.
Also for Fargate, it should remove:
"hostPort": 2021
And you forgot to setup security group for your ECS service. It should be:
network_configuration {
subnets = data.aws_subnet_ids.default.ids
assign_public_ip = true # Providing our containers with public IPs
security_groups = [aws_security_group.service_security_group.id]
}

Argument or block definition required Terraform - Error

I have autoscaling group and I need to create an application load balancer for accessing the application this is my two code for autoscaling and application load balancer but I get this issue
autoscaling group
resource "aws_launch_configuration" "OS-Type"{
name_prefix = "OS-Type"
image_id = "ami-0996d3051b72b5b2c"
instance_type = "t2.micro"
lifecycle {
create_before_destroy = true
}
}
resource "aws_autoscaling_group" "Dynamic-IN"{
name = "Dynamic-EC2-instance"
min_size = 1
max_size = 4
desired_capacity = 2
health_check_type = "ELB"
launch_configuration = aws_launch_configuration.OS-Type.name
vpc_zone_identifier = [aws_subnet.P-AV1.id, aws_subnet.P-AV2.id]
target_group_arns="aws_lb.App-lb.name"
lifecycle {
create_before_destroy = true
}
}
Application load balancer
resource "aws_lb_target_group" "Target-group"{
name = "Target-group"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.main.id
}
resource "aws_lb" "App-lb"{
name = "Application-load-balancer"
load_balancer_type = "application"
subnets = [aws_subnet.P-AV1.id , aws_subnet.P-AV2.id]
internal = false
}
resource "aws_autoscaling_attachment" "TG-attach" {
autoscaling_group_name = aws_autoscaling_group.Dynamic-IN.id
alb_target_group_arn = aws_lb_target_group.Target-group.arn
}
I get this error
Error: Argument or block definition required
on autoscalling-group.tf line 20, in resource "aws_autoscaling_group" "Dynamic-IN":
20: target_group.arns="aws_lb.App-lb.name"
An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.
I have tried
I have tried aws_lb.App-lb.arns for the target group alos but not working in the both ways
Yes like you suspect there should not quotes there:
target_group_arns="aws_lb.App-lb.name"
and that 'target_group_arns' is a set not a single item: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group#target_group_arns
target_group_arns (Optional) A set of aws_alb_target_group ARNs, for use with Application or Network Load Balancing.
Your code should probably be something like:
resource "aws_autoscaling_group" "Dynamic-IN" {
name = "Dynamic-EC2-instance"
min_size = 1
max_size = 4
desired_capacity = 2
health_check_type = "ELB"
launch_configuration = aws_launch_configuration.OS-Type.name
vpc_zone_identifier = [ aws_subnet.P-AV1.id, aws_subnet.P-AV2.id ]
target_group_arns = [ aws_lb_target_group.Target-group.arn ]
lifecycle {
create_before_destroy = true
}
}

Terraform resource recreation dynamic AWS RDS instance counts

I have a question relating to AWS RDS cluster and instance creation.
Environment
We recently experimented with:
Terraform v0.11.11
provider.aws v1.41.0
Background
Creating some AWS RDS databases. Our mission was that in some environment (e.g. staging) we may run fewer instances than in others (e.g. production.). With this in mind and not wanting to have totally different terraform files per environment we instead decided to specify the database resources just once and use a variable for the number of instances which is set in our staging.tf and production.tf files respectively for the number of instances.
Potentially one more "quirk" of our setup, is that the VPC in which the subnets exist is not defined in terraform, the VPC already existed via manual creation in the AWS console, so this is provided as a data provider and the subnets for the RDS are specific in terraform - but again this is dynamic in the sense that in some environments we might have 3 subnets (1 in each AZ), whereas in others perhaps we have only 2 subnets. Again to achieve this we used iteration as shown below:
Structure
|-/environments
-/staging
-staging.tf
-/production
-production.tf
|- /resources
- database.tf
Example Environment Variables File
dev.tf
terraform {
terraform {
backend "s3" {
bucket = "my-bucket-dev"
key = "terraform"
region = "eu-west-1"
encrypt = "true"
acl = "private"
dynamodb_table = "terraform-state-locking"
}
version = "~> 0.11.8"
}
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
version = "~> 1.33"
allowed_account_ids = ["XXX"]
}
module "main" {
source = "../../resources"
vpc_name = "test"
test_db_name = "terraform-test-db-dev"
test_db_instance_count = 1
test_db_backup_retention_period = 7
test_db_backup_window = "00:57-01:27"
test_db_maintenance_window = "tue:04:40-tue:05:10"
test_db_subnet_count = 2
test_db_subnet_cidr_blocks = ["10.2.4.0/24", "10.2.5.0/24"]
}
We came to this module based structure for environment isolation mainly due to these discussions:
https://github.com/hashicorp/terraform/issues/18632#issuecomment-412247266
https://github.com/hashicorp/terraform/issues/13700
https://www.terraform.io/docs/state/workspaces.html#when-to-use-multiple-workspaces
Our Issue
Initial resource creation works fine, our subnets are created, the database cluster starts up.
Our issues start the next time we subsequently run a terraform plan or terraform apply (with no changes to the files), at which point we see interesting things like:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
module.main.aws_rds_cluster.test_db (new resource required)
id: "terraform-test-db-dev" => (forces new resource)
availability_zones.#: "3" => "1" (forces new resource)
availability_zones.1924028850: "eu-west-1b" => "" (forces new resource)
availability_zones.3953592328: "eu-west-1a" => "eu-west-1a"
availability_zones.94988580: "eu-west-1c" => "" (forces new resource)
and
module.main.aws_rds_cluster_instance.test_db (new resource required)
id: "terraform-test-db-dev" => (forces new resource)
cluster_identifier: "terraform-test-db-dev" => "${aws_rds_cluster.test_db.id}" (forces new resource)
Something about the way we are approaching this appears to be causing terraform to believe that the resource has changed to such an extent that it must destroy the existing resource and create a brand new one.
Config
variable "aws_availability_zones" {
description = "Run the EC2 Instances in these Availability Zones"
type = "list"
default = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
}
variable "test_db_name" {
description = "Name of the RDS instance, must be unique per region and is provided by the module config"
}
variable "test_db_subnet_count" {
description = "Number of subnets to create, is provided by the module config"
}
resource "aws_security_group" "test_db_service" {
name = "${var.test_db_service_user_name}"
vpc_id = "${data.aws_vpc.vpc.id}"
}
resource "aws_security_group" "test_db" {
name = "${var.test_db_name}"
vpc_id = "${data.aws_vpc.vpc.id}"
}
resource "aws_security_group_rule" "test_db_ingress_app_server" {
security_group_id = "${aws_security_group.test_db.id}"
...
source_security_group_id = "${aws_security_group.test_db_service.id}"
}
variable "test_db_subnet_cidr_blocks" {
description = "Cidr block allocated to the subnets"
type = "list"
}
resource "aws_subnet" "test_db" {
count = "${var.test_db_subnet_count}"
vpc_id = "${data.aws_vpc.vpc.id}"
cidr_block = "${element(var.test_db_subnet_cidr_blocks, count.index)}"
availability_zone = "${element(var.aws_availability_zones, count.index)}"
}
resource "aws_db_subnet_group" "test_db" {
name = "${var.test_db_name}"
subnet_ids = ["${aws_subnet.test_db.*.id}"]
}
variable "test_db_backup_retention_period" {
description = "Number of days to keep the backup, is provided by the module config"
}
variable "test_db_backup_window" {
description = "Window during which the backup is done, is provided by the module config"
}
variable "test_db_maintenance_window" {
description = "Window during which the maintenance is done, is provided by the module config"
}
data "aws_secretsmanager_secret" "test_db_master_password" {
name = "terraform/db/test-db/root-password"
}
data "aws_secretsmanager_secret_version" "test_db_master_password" {
secret_id = "${data.aws_secretsmanager_secret.test_db_master_password.id}"
}
data "aws_iam_role" "rds-monitoring-role" {
name = "rds-monitoring-role"
}
resource "aws_rds_cluster" "test_db" {
cluster_identifier = "${var.test_db_name}"
engine = "aurora-mysql"
engine_version = "5.7.12"
# can only request to deploy in AZ's where there is a subnet in the subnet group.
availability_zones = "${slice(var.aws_availability_zones, 0, var.test_db_instance_count)}"
database_name = "${var.test_db_schema_name}"
master_username = "root"
master_password = "${data.aws_secretsmanager_secret_version.test_db_master_password.secret_string}"
preferred_backup_window = "${var.test_db_backup_window}"
preferred_maintenance_window = "${var.test_db_maintenance_window}"
backup_retention_period = "${var.test_db_backup_retention_period}"
db_subnet_group_name = "${aws_db_subnet_group.test_db.name}"
storage_encrypted = true
kms_key_id = "${data.aws_kms_key.kms_rds_key.arn}"
deletion_protection = true
enabled_cloudwatch_logs_exports = ["audit", "error", "general", "slowquery"]
vpc_security_group_ids = ["${aws_security_group.test_db.id}"]
final_snapshot_identifier = "test-db-final-snapshot"
}
variable "test_db_instance_count" {
description = "Number of instances to create, is provided by the module config"
}
resource "aws_rds_cluster_instance" "test_db" {
count = "${var.test_db_instance_count}"
identifier = "${var.test_db_name}"
cluster_identifier = "${aws_rds_cluster.test_db.id}"
availability_zone = "${element(var.aws_availability_zones, count.index)}"
instance_class = "db.t2.small"
db_subnet_group_name = "${aws_db_subnet_group.test_db.name}"
monitoring_interval = 60
engine = "aurora-mysql"
engine_version = "5.7.12"
monitoring_role_arn = "${data.aws_iam_role.rds-monitoring-role.arn}"
tags {
Name = "test_db-${count.index}"
}
}
My question is, is there a way to achieve this so that terraform would not try to recreate the resource (e.g. ensure that the availability zones of the cluster and ID of the instance do not change each time we run terraform.
Turns out that simply by just removing the explicit availability zones definitions from the aws_rds_cluster and aws_rds_cluster_instance then this issue goes away and everything so far appears to work as expected. See also https://github.com/terraform-providers/terraform-provider-aws/issues/7307#issuecomment-457441633

Resources