I'm using terraform to distribute eks, rds, and kubernetes and try to run moodle. The image for moodle is using bitnami/moodle:latest.
Both eks and rds work on the same vpc, and the are generated very well. However, when I try to distribute moodle in kubernetes with rds, I experience problems with not accessing the database. What's the problem with connecting databases?
root#master:~/moodle_deploy# kubectl get pod
NAME READY STATUS RESTARTS AGE
moodle-54f477b7f8-nh98h 0/1 CrashLoopBackOff 15 70m
moodle-54f477b7f8-rhzzj 0/1 CrashLoopBackOff 15 70m
root#master:~/moodle_deploy# kubectl logs moodle-54f477b7f8-rhzzj
moodle 06:33:12.43
moodle 06:33:12.43 Welcome to the Bitnami moodle container
moodle 06:33:12.43 Subscribe to project updates by watching https://github.com/bitnami/containers
moodle 06:33:12.43 Submit issues and feature requests at https://github.com/bitnami/containers/issuesmoodle 06:33:12.43
moodle 06:33:12.44 INFO ==> ** Starting Moodle setup **
realpath: /bitnami/apache/conf: No such file or directory
moodle 06:33:12.46 INFO ==> Configuring Apache ServerTokens directive
moodle 06:33:12.49 INFO ==> Configuring PHP options
moodle 06:33:12.49 INFO ==> Setting PHP expose_php option
moodle 06:33:12.51 INFO ==> Validating settings in MYSQL_CLIENT_* env vars
moodle 06:33:12.53 INFO ==> Validating settings in POSTGRESQL_CLIENT_* env vars
moodle 06:33:12.69 INFO ==> Ensuring Moodle directories exist
moodle 06:33:12.78 INFO ==> Trying to connect to the database server
moodle 06:34:12.90 ERROR ==> Could not connect to the database
root#master:~/moodle_deploy# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
moodle-pv-claim Bound pvc-eefbb1d9-a6fd-4cf5-b044-159145798114 1Gi RWO gp2 59m
root#master:~/moodle_deploy# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-eefbb1d9-a6fd-4cf5-b044-159145798114 1Gi RWO Delete Bound default/moodle-pv-claim gp2 59m
These are the code that defined rds and kubernetes among the terraform files I distributed
//SECURITY GROUP
resource "aws_security_group" "secgrp-rds" {
name = "secgrp-rdis"
description = "Allow MySQL Port"
vpc_id = module.vpc.vpc_id
ingress {
description = "Allowing Connection for mysql"
from_port = 3306
to_port = 3306
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "Allowing Connection for SSH"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_db_subnet_group" "sub_ids" {
name = "main"
subnet_ids = module.vpc.private_subnets
tags = {
Name = "DB subnet group"
}
}
//RDS INSTANCE
resource "aws_db_instance" "rds" {
depends_on=[aws_security_group.secgrp-rds]
engine = "mysql"
engine_version = "5.7"
instance_class = "db.t2.micro"
allocated_storage = 20
storage_type = "gp2"
name = "moodledb"
identifier = "moodledb"
username = "root"
password = "test12345"
parameter_group_name = "default.mysql5.7"
#publicly_accessible = true
skip_final_snapshot = true
apply_immediately = true
iam_database_authentication_enabled = true
db_subnet_group_name = aws_db_subnet_group.sub_ids.id
vpc_security_group_ids = [aws_security_group.secgrp-rds.id]
}
//CREATE DEPLOYMENT - provide rds environment variables
resource "kubernetes_deployment" "moodle" {
depends_on = [aws_db_instance.rds]
metadata {
name = "moodle"
labels = {
App = "moodle"
}
}
spec {
replicas = 2
selector {
match_labels = {
App = "moodle"
}
}
template {
metadata {
labels = {
App = "moodle"
}
}
spec {
container {
image = "bitnami/moodle:latest"
name = "moodle"
env{
name = "MOODLE_DATABASE_TYPE"
value = "mysqli"
}
env{
name = "MOODLE_DATABASE_HOST"
value = aws_db_instance.rds.endpoint
}
env{
name = "MOODLE_DATABASE_USER"
value = aws_db_instance.rds.username
}
env{
name = "MOODLE_DATABASE_PASSWORD"
value = aws_db_instance.rds.password
}
env{
name = "MOODLE_DATABASE_NAME"
value = aws_db_instance.rds.db_name
}
port {
container_port = 8080
}
volume_mount {
name = "moodle-ps"
mount_path = "/opt/bitnami/apache2/htdocs/"
}
resources {
limits= {
cpu = "0.5"
memory = "512Mi"
}
requests= {
cpu = "250m"
memory = "50Mi"
}
}
}
volume {
name = "moodle-ps"
persistent_volume_claim {
claim_name = "moodle-pv-claim"
}
}
}
}
}
}
resource "kubernetes_service" "moodle" {
depends_on = [kubernetes_deployment.moodle]
metadata {
name = "moodle"
}
spec {
port {
port = 80
target_port = 8080
}
type = "LoadBalancer"
}
}
resource "kubernetes_storage_class" "kubeSC" {
metadata {
name = "kubese"
}
storage_provisioner = "kubernetes.io/aws-ebs"
reclaim_policy = "Retain"
parameters = {
type = "gp2"
}
}
resource "kubernetes_persistent_volume_claim" "pvc" {
depends_on=[kubernetes_storage_class.kubeSC]
metadata {
name = "moodle-pv-claim"
labels = {
"app" = "moodle"
}
}
spec {
access_modes = ["ReadWriteOnce"]
resources {
requests = {
storage = "1Gi"
}
}
}
}
Related
I receive 2 errors when i deploy AWS EKS module via Terraform. How to solve it?
Error: unexpected EKS Add-On (my-cluster:coredns) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)
Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp [::1]:80: connect: connection refused
What role should i write in aws_auth_roles parameter? AWSServiceRoleForAmazonEKS or a custom role with policies: AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy?
What role should i add to instance-profile? AWSServiceRoleForAmazonEKS or a custom role with policies: AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, AmazonEKS_CNI_Policy?
Terraform deploys EC2 machines for worker node, but i dont see a nodegroup with worker nodes in eks, probably coredns issue is here.
My config:
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 18.20.2"
cluster_name = var.cluster_name
cluster_version = var.cluster_version
cluster_endpoint_private_access = true
cluster_endpoint_public_access = false
cluster_addons = {
coredns = {
resolve_conflicts = "OVERWRITE"
}
kube-proxy = {}
vpc-cni = {
resolve_conflicts = "OVERWRITE"
}
}
subnet_ids = ["...","..."]
self_managed_node_group_defaults = {
instance_type = "t2.micro"
update_launch_template_default_version = true
}
self_managed_node_groups = {
one = {
name = "test-1"
max_size = 2
desired_size = 1
use_mixed_instances_policy = true
mixed_instances_policy = {
instances_distribution = {
on_demand_base_capacity = 0
on_demand_percentage_above_base_capacity = 10
spot_allocation_strategy = "capacity-optimized"
}
}
}
}
create_aws_auth_configmap = true
manage_aws_auth_configmap = true
aws_auth_users = [
{
userarn = "arn:aws:iam::...:user/..."
username = "..."
groups = ["system:masters"]
}
]
aws_auth_roles = [
{
rolearn = "arn:aws:iam::...:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS"
username = "AWSServiceRoleForAmazonEKS"
groups = ["system:masters"]
}
]
aws_auth_accounts = [
"..."
]
}
I created an ECS cluster, along with a Load Balancer, to expose a basc hello-world node app on Fargate using Terraform. Terraform manages to create my aws resources just fine, and deploys the correct image on ECS Fargate, but the task never passes the initial health-check and restarts indefinitely. I think this is a port-forwarding problem, but I believe my Dockerfile, Load Balancer and Task Definition all expose the correct ports.
Below is the error I see when looking at my service's "events" tab on the ECS dashboard:
service my-first-service (port 2021) is unhealthy in target-group target-group due to (reason Request timed out).
Below is my Application code, the Dockerfile, and the Terraform files I am using to deploy to Fargate:
index.js
const express = require('express')
const app = express()
const port = 2021
app.get('/', (req, res) => res.send('Hello World!'))
app.listen(port, () => console.log(`Example app listening on port ${port}!`))
Dockerfile
# Use an official Node runtime as a parent image
FROM node:12.7.0-alpine
# Set the working directory to /app
WORKDIR '/app'
# Copy package.json to the working directory
COPY package.json .
# Install any needed packages specified in package.json
RUN yarn
# Copying the rest of the code to the working directory
COPY . .
# Make port 2021 available to the world outside this container
EXPOSE 2021
# Run index.js when the container launches
CMD ["node", "index.js"]
application_load_balancer_target_group.tf
resource "aws_lb_target_group" "target_group" {
name = "target-group"
port = 80
protocol = "HTTP"
target_type = "ip"
vpc_id = "${aws_default_vpc.default_vpc.id}" # Referencing the default VPC
health_check {
matcher = "200,301,302"
path = "/"
}
}
resource "aws_lb_listener" "listener" {
load_balancer_arn = "${aws_alb.application_load_balancer.arn}" # Referencing our load balancer
port = "80"
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = "${aws_lb_target_group.target_group.arn}" # Referencing our tagrte group
}
}
application_load_balaner.tf
resource "aws_alb" "application_load_balancer" {
name = "test-lb-tf" # Naming our load balancer
load_balancer_type = "application"
subnets = [ # Referencing the default subnets
"${aws_default_subnet.default_subnet_a.id}",
"${aws_default_subnet.default_subnet_b.id}",
"${aws_default_subnet.default_subnet_c.id}"
]
# Referencing the security group
security_groups = ["${aws_security_group.load_balancer_security_group.id}"]
}
# Creating a security group for the load balancer:
resource "aws_security_group" "load_balancer_security_group" {
ingress {
from_port = 80 # Allowing traffic in from port 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic in from all sources
}
egress {
from_port = 0 # Allowing any incoming port
to_port = 0 # Allowing any outgoing port
protocol = "-1" # Allowing any outgoing protocol
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic out to all IP addresses
}
}
ecs_cluster.tf
resource "aws_ecs_cluster" "my_cluster" {
name = "my-cluster" # Naming the cluster
}
ecs_service.tf
# Providing a reference to our default VPC (these are needed by the aws_ecs_service at the bottom of this file)
resource "aws_default_vpc" "default_vpc" {
}
# Providing a reference to our default subnets (NOTE: Make sure the availability zones match your zone)
resource "aws_default_subnet" "default_subnet_a" {
availability_zone = "us-east-2a"
}
resource "aws_default_subnet" "default_subnet_b" {
availability_zone = "us-east-2b"
}
resource "aws_default_subnet" "default_subnet_c" {
availability_zone = "us-east-2c"
}
resource "aws_ecs_service" "my_first_service" {
name = "my-first-service" # Naming our first service
cluster = "${aws_ecs_cluster.my_cluster.id}" # Referencing our created Cluster
task_definition = "${aws_ecs_task_definition.my_first_task.arn}" # Referencing the task our service will spin up
launch_type = "FARGATE"
desired_count = 1 # Setting the number of containers we want deployed to 1
# NOTE: The following 'load_balancer' snippet was added here after the creation of the application_load_balancer files.
load_balancer {
target_group_arn = "${aws_lb_target_group.target_group.arn}" # Referencing our target group
container_name = "${aws_ecs_task_definition.my_first_task.family}"
container_port = 2021 # Specifying the container port
}
network_configuration {
subnets = ["${aws_default_subnet.default_subnet_a.id}", "${aws_default_subnet.default_subnet_b.id}", "${aws_default_subnet.default_subnet_c.id}"]
assign_public_ip = true # Providing our containers with public IPs
}
}
resource "aws_security_group" "service_security_group" {
ingress {
from_port = 0
to_port = 0
protocol = "-1"
# Only allowing traffic in from the load balancer security group
security_groups = ["${aws_security_group.load_balancer_security_group.id}"]
}
egress {
from_port = 0 # Allowing any incoming port
to_port = 0 # Allowing any outgoing port
protocol = "-1" # Allowing any outgoing protocol
cidr_blocks = ["0.0.0.0/0"] # Allowing traffic out to all IP addresses
}
}
ecs_task_definition.tf
resource "aws_ecs_task_definition" "my_first_task" {
family = "my-first-task" # Naming our first task
container_definitions = <<DEFINITION
[
{
"name": "my-first-task",
"image": "${var.ECR_IMAGE_URL}",
"essential": true,
"portMappings": [
{
"containerPort": 2021,
"hostPort": 2021
}
],
"memory": 512,
"cpu": 256
}
]
DEFINITION
requires_compatibilities = ["FARGATE"] # Stating that we are using ECS Fargate
network_mode = "awsvpc" # Using awsvpc as our network mode as this is required for Fargate
memory = 512 # Specifying the memory our container requires
cpu = 256 # Specifying the CPU our container requires
execution_role_arn = "${aws_iam_role.ecsTaskExecutionRole.arn}"
}
resource "aws_iam_role" "ecsTaskExecutionRole" {
name = "ecsTaskExecutionRole"
assume_role_policy = "${data.aws_iam_policy_document.assume_role_policy.json}"
}
data "aws_iam_policy_document" "assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
role = "${aws_iam_role.ecsTaskExecutionRole.name}"
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
Where am I going wrong here?
I had same similar issue when I was migrating from k8s to ECS Fargate.
My task could not start, it was nightmare.
Same image in k8s was working great with same health checks.
I can see that you are missing healthCheck in task_definition, at least that was issue for me.
here is my containerDefinition :
container_definitions = jsonencode([{
name = "${var.app_name}-container-${var.environment}"
image = "${var.container_repository}:${var.container_image_version}"
essential = true
environment: concat(
var.custom_env_variables,
[
{
name = "JAVA_TOOL_OPTIONS"
value = "-Xmx${var.container_memory_max_ram}m -XX:MaxRAM=${var.container_memory_max_ram}m -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4"
},
{
name = "SPRING_PROFILES_ACTIVE"
value = var.spring_profile
},
{
name = "APP_NAME"
value = var.spring_app_name
}
]
)
portMappings = [
{
protocol = "tcp"
containerPort = var.container_port
},
{
protocol = "tcp"
containerPort = var.container_actuator_port
}
]
healthCheck = {
retries = 10
command = [ "CMD-SHELL", "curl -f http://localhost:8081/actuator/liveness || exit 1" ]
timeout: 5
interval: 10
startPeriod: var.health_start_period
}
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.main.name
awslogs-stream-prefix = "ecs"
awslogs-region = var.aws_region
}
}
mountPoints = [{
sourceVolume = "backend_efs",
containerPath = "/data",
readOnly = false
}]
}])
there is healthCheck aprt:
healthCheck = {
retries = 10
command = [ "CMD-SHELL", "curl -f http://localhost:8081/actuator/liveness || exit 1" ]
timeout: 5
interval: 10
startPeriod: var.health_start_period
}
container in order to start needs to have a way to check is that task running OK.
And I could only get that via curl . I have one endpoint that returns me is it live or not. You need to specify your, it is jut important that return 200.
Also there is no curl command by default, you need to add it in you DockerFile as that was next issue where I spent few hours, as there was not clear error on ECS.
I added this line:
RUN apt-get update && apt-get install -y --no-install-recommends curl
By the look of it, you are create new VPC with subnets, but there are no route tables defined, no internet gateway and attached to the VPC. So your VPC is simply private and not accessible from the internet, nor it can access ECR to get your docker image.
Maybe instead of creating a new VPC called default_vpc, you want to use an existing default vpc. If so you have to use data source:
data "aws_vpc" "default_vpc" {
default = true
}
to get subnets:
data "aws_subnet_ids" "default" {
vpc_id = data.aws_vpc.default_vpc.id
}
and modify the remaining of the code to reference these data sources.
Also for Fargate, it should remove:
"hostPort": 2021
And you forgot to setup security group for your ECS service. It should be:
network_configuration {
subnets = data.aws_subnet_ids.default.ids
assign_public_ip = true # Providing our containers with public IPs
security_groups = [aws_security_group.service_security_group.id]
}
I am trying to create an AWS EKS cluster with an ALB ingress using Terraform resources.
This document indicates that the ingress will automatically create a load balancer with associated listeners and target groups.
The Kubernetes Ingress creates an ALB load balancer, security group and rules but doesn't create target groups or listeners. I have tried using either the gateway or the application subnets but it makes no difference. I tried setting the security group but the ALB setup and used its own self managed security group.
I have relied on this guide
A curl to the ALB gets me
Failed to connect to
de59ecbf-default-mainingre-8687-1051686593.ap-southeast-1.elb.amazonaws.com
port 80: Connection refused
I created IAM roles and ACM certs separately as AWS has a quota limit on these. My roles for EKS cluster and nodes are standard and the nodes role has the latest policy attached.
I used kubectl to apply the kubernetes ingress separately but it had the same result. It creates the ALB and a security group with rules for the ports but no target group or listeners.
When I paste the cluster endpoint from aws eks describe-cluster --name my-tf-eks-cluster --query "cluster.endpoint" into the browser I get this:
{ "kind": "Status", "apiVersion": "v1", "metadata": {
}, "status": "Failure", "message": "forbidden: User "system:anonymous" cannot get path "/"", "reason": "Forbidden",
"details": {
}, "code": 403 }
Additionally, the ingress has no ip address.
kubectl describe ingresses
Name: main-ingress
Namespace: default
Address:
Default backend: go-hello-world:8080 (<none>)
Rules:
Host Path Backends
---- ---- --------
* * go-hello-world:8080 (<none>)
aws eks describe-cluster --name my-tf-eks-cluster --query cluster.endpoint"
"https://88888888B.gr7.ap-southeast-1.eks.amazonaws.com"
curl https://88888888B.gr7.ap-southeast-1.eks.amazonaws.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
edit: The IAM cluster policy is lacking these permissions. I have decided it may be better to use an ELB instead since they can terminate ssl certificates and then use traefik as a back end proxy so I can't really test this now. Can anyone confirm if these permissions are needed for ALB?
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates"
Here is my EKS master resource:
data "aws_iam_role" "tf-eks-master" {
name = "terraform-eks-cluster"
}
resource "aws_eks_cluster" "tf_eks" {
name = var.cluster_name
role_arn = data.aws_iam_role.tf-eks-master.arn
vpc_config {
security_group_ids = [aws_security_group.master.id]
subnet_ids = var.application_subnet_ids
endpoint_private_access = true
endpoint_public_access = true
}
}
ALB Ingress controller:
output "vpc_id" {
value = data.aws_vpc.selected
}
data "aws_subnet_ids" "selected" {
vpc_id = data.aws_vpc.selected.id
tags = map(
"Name", "application",
)
}
resource "kubernetes_deployment" "alb-ingress" {
metadata {
name = "alb-ingress-controller"
labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
namespace = "kube-system"
}
spec {
selector {
match_labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
}
template {
metadata {
labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
}
spec {
volume {
name = kubernetes_service_account.alb-ingress.default_secret_name
secret {
secret_name = kubernetes_service_account.alb-ingress.default_secret_name
}
}
container {
# This is where you change the version when Amazon comes out with a new version of the ingress controller
image = "docker.io/amazon/aws-alb-ingress-controller:v1.1.8"
name = "alb-ingress-controller"
args = [
"--ingress-class=alb",
"--cluster-name=${var.cluster_name}",
"--aws-vpc-id=${data.aws_vpc.selected.id}",
"--aws-region=${var.aws_region}"
]
volume_mount {
name = kubernetes_service_account.alb-ingress.default_secret_name
mount_path = "/var/run/secrets/kubernetes.io/serviceaccount"
read_only = true
}
}
service_account_name = "alb-ingress-controller"
}
}
}
}
resource "kubernetes_service_account" "alb-ingress" {
metadata {
name = "alb-ingress-controller"
namespace = "kube-system"
labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
}
automount_service_account_token = true
}
kubernetes_ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: main-ingress
annotations:
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: "internet-facing"
alb.ingress.kubernetes.io/target-type: "ip"
alb.ingress.kubernetes.io/subnets: 'subnet-0ab65d9cec9451287, subnet-034bf8856ab9157b7, subnet-0c16b1d382fadd0b4'
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
spec:
backend:
serviceName: go-hello-world
servicePort: 8080
roles
resource "kubernetes_cluster_role" "alb-ingress" {
metadata {
name = "alb-ingress-controller"
labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
}
rule {
api_groups = ["", "extensions"]
resources = ["configmaps", "endpoints", "events", "ingresses", "ingresses/status", "services"]
verbs = ["create", "get", "list", "update", "watch", "patch"]
}
rule {
api_groups = ["", "extensions"]
resources = ["nodes", "pods", "secrets", "services", "namespaces"]
verbs = ["get", "list", "watch"]
}
}
resource "kubernetes_cluster_role_binding" "alb-ingress" {
metadata {
name = "alb-ingress-controller"
labels = {
"app.kubernetes.io/name" = "alb-ingress-controller"
}
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = "alb-ingress-controller"
}
subject {
kind = "ServiceAccount"
name = "alb-ingress-controller"
namespace = "kube-system"
}
}
Some code from the VPC
data "aws_availability_zones" "available" {}
resource "aws_subnet" "gateway" {
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.1${count.index}.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "gateway",
)
}
resource "aws_subnet" "application" {
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.2${count.index}.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "application",
"kubernetes.io/cluster/${var.cluster_name}", "shared",
"kubernetes.io/role/elb", "1",
)
}
resource "aws_subnet" "database" {
count = var.subnet_count
availability_zone = data.aws_availability_zones.available.names[count.index]
cidr_block = "10.0.3${count.index}.0/24"
vpc_id = aws_vpc.tf_eks.id
tags = map(
"Name", "database"
)
}
resource "aws_route_table" "application" {
count = var.subnet_count
vpc_id = aws_vpc.tf_eks.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.tf_eks.*.id[count.index]
}
tags = {
Name = "application"
}
}
resource "aws_route_table" "database" {
vpc_id = aws_vpc.tf_eks.id
tags = {
Name = "database"
}
}
resource "aws_route_table" "gateway" {
vpc_id = aws_vpc.tf_eks.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.tf_eks.id
}
tags = {
Name = "gateway"
}
}
resource "aws_route_table_association" "application" {
count = var.subnet_count
subnet_id = aws_subnet.application.*.id[count.index]
route_table_id = aws_route_table.application.*.id[count.index]
}
resource "aws_route_table_association" "database" {
count = var.subnet_count
subnet_id = aws_subnet.database.*.id[count.index]
route_table_id = aws_route_table.database.id
}
resource "aws_route_table_association" "gateway" {
count = var.subnet_count
subnet_id = aws_subnet.gateway.*.id[count.index]
route_table_id = aws_route_table.gateway.id
}
resource "aws_internet_gateway" "tf_eks" {
vpc_id = aws_vpc.tf_eks.id
tags = {
Name = "internet_gateway"
}
}
resource "aws_eip" "nat_gateway" {
count = var.subnet_count
vpc = true
}
resource "aws_nat_gateway" "tf_eks" {
count = var.subnet_count
allocation_id = aws_eip.nat_gateway.*.id[count.index]
subnet_id = aws_subnet.gateway.*.id[count.index]
tags = {
Name = "nat_gateway"
}
depends_on = [aws_internet_gateway.tf_eks]
}
Security groups
resource "aws_security_group" "eks" {
name = "tf-eks-master"
description = "Cluster communication with worker nodes"
vpc_id = var.vpc_id
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "node" {
name = "tf-eks-node"
description = "Security group for all nodes in the cluster"
vpc_id = var.vpc_id
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group_rule" "main-node-ingress-self" {
type = "ingress"
description = "Allow node to communicate with each other"
from_port = 0
protocol = "-1"
security_group_id = aws_security_group.node.id
to_port = 65535
cidr_blocks = var.subnet_cidrs
}
resource "aws_security_group_rule" "main-node-ingress-cluster" {
type = "ingress"
description = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
from_port = 1025
protocol = "tcp"
security_group_id = aws_security_group.node.id
source_security_group_id = aws_security_group.eks.id
to_port = 65535
}
kubectl get all --all-namespaces
kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/go-hello-world-68545f84bc-5st4s 1/1 Running 0 35s
default pod/go-hello-world-68545f84bc-bkwpb 1/1 Running 0 35s
default pod/go-hello-world-68545f84bc-kmfbq 1/1 Running 0 35s
kube-system pod/alb-ingress-controller-5f9cb4b7c4-w858g 1/1 Running 0 2m7s
kube-system pod/aws-node-8jfkf 1/1 Running 0 67m
kube-system pod/aws-node-d7s7w 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-g5fmj 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-q5tz5 1/1 Running 0 67m
kube-system pod/aws-node-termination-handler-tmzmr 1/1 Running 0 67m
kube-system pod/aws-node-vswpf 1/1 Running 0 67m
kube-system pod/coredns-5c4dd4cc7-sk474 1/1 Running 0 71m
kube-system pod/coredns-5c4dd4cc7-zplwg 1/1 Running 0 71m
kube-system pod/kube-proxy-5m9dn 1/1 Running 0 67m
kube-system pod/kube-proxy-8tn9l 1/1 Running 0 67m
kube-system pod/kube-proxy-qs652 1/1 Running 0 67m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 71m
kube-system service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 71m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-node 3 3 3 3 3 <none> 71m
kube-system daemonset.apps/aws-node-termination-handler 3 3 3 3 3 <none> 68m
kube-system daemonset.apps/kube-proxy 3 3 3 3 3 <none> 71m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/go-hello-world 3/3 3 3 37s
kube-system deployment.apps/alb-ingress-controller 1/1 1 1 2m9s
kube-system deployment.apps/coredns 2/2 2 2 71m
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/go-hello-world-68545f84bc 3 3 3 37s
kube-system replicaset.apps/alb-ingress-controller-5f9cb4b7c4 1 1 1 2m9s
kube-system replicaset.apps/coredns-5c4dd4cc7 2 2
Can you try adding these lines and try kubectl commands
# ALB's Target Group Configurations
alb.ingress.kubernetes.io/backend-protocol: HTTPS
alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
Check this
still the Target group is not created, check the logs of controller
kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller
Did not test it but I am going to point out the issue that I saw going through your resources.
In the alb resource args you have the below:
"--aws-vpc-id=${data.aws_vpc.selected.id}",
Yet, you do not have any data resource to pull this VPC.
In addition, data resource runs without any dependency and will not pull any information about a VPC that was not yet created.
If you have your terraform modulized, output the VPC ID from the VPC module and use it instead. If all the resources are in the same file/folder just directly reference it in this line: aws_vpc.tf_eks.id
I am looking for a way to attach two target group against single ECS services, in other my container exposes two port but I am only able to map one port against my service to LB.
So far I am able to create a new listener and target group but after target group creation I can see everything as per expectation but the target group show There are no targets registered to this target group
Here are my target group and listener configuration
target_group:
resource "aws_lb_target_group" "e_admin" {
name = "${var.env_prefix_name}-admin"
port = 5280
protocol = "HTTP"
vpc_id = "${aws_vpc.VPC.id}"
health_check {
path = "/admin"
healthy_threshold = 2
unhealthy_threshold = 10
port = 5280
timeout = 90
interval = 100
matcher = "401,200"
}
}
Listener:'
resource "aws_lb_listener" "admin" {
load_balancer_arn = "${aws_lb.admin_lb.arn}"
port = "5280"
protocol = "HTTP"
default_action {
target_group_arn = "${aws_lb_target_group.e_admin.id}"
type = "forward"
}
}
My question is how I can add ECS cluster Autoscaling group or how I can add all the instances running in the ECS cluster to this target group?
AWS recently announced support for multiple target groups for an ECS service.
The, currently unreleased, 2.22.0 version of the AWS provider contains support for this by adding more load_balancer blocks to the aws_ecs_service resource. Example from the acceptance tests:
resource "aws_ecs_service" "with_alb" {
name = "example"
cluster = "${aws_ecs_cluster.main.id}"
task_definition = "${aws_ecs_task_definition.with_lb_changes.arn}"
desired_count = 1
iam_role = "${aws_iam_role.ecs_service.name}"
load_balancer {
target_group_arn = "${aws_lb_target_group.test.id}"
container_name = "ghost"
container_port = "2368"
}
load_balancer {
target_group_arn = "${aws_lb_target_group.static.id}"
container_name = "ghost"
container_port = "4501"
}
depends_on = [
"aws_iam_role_policy.ecs_service",
]
}
Accodring to https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html,
There is a limit of one load balancer or target group per service.
If you want to attach a autoscaling group to the target group, use aws_autoscaling_attachment,
https://www.terraform.io/docs/providers/aws/r/autoscaling_attachment.html
resource "aws_autoscaling_attachment" "asg_attachment_bar" {
autoscaling_group_name = "${aws_autoscaling_group.your_asg.id}"
alb_target_group_arn = "${aws_alb_target_group.e_admin.arn}"
}
You can add multiple define multiple target groups for the same ecs service using the load_balancer block.
resource "aws_ecs_service" "ecs_service_1" {
name = "service-1"
cluster = aws_ecs_cluster.ecs_cluster_prod.id
task_definition = aws_ecs_task_definition.ecs_task_definition_1.arn
desired_count = 1
launch_type = "FARGATE"
enable_execute_command = true
# Target group 1
load_balancer {
target_group_arn = aws_lb_target_group.lb_tg_1.arn
container_name = "app"
container_port = 8080
}
# Target group 2
load_balancer {
target_group_arn = aws_lb_target_group.lb_tg_2.arn
container_name = "app"
container_port = 8080
}
network_configuration {
subnets = [aws_subnet.subnet_a.id, aws_subnet.subnet_b.id]
security_groups = [aws_security_group.sg_internal.id]
assign_public_ip = true
}
tags = {
Name = "service-1"
ManagedBy = "terraform"
Environment = "prod"
}
}
You can map the same container and port to both target groups in case of having an external and an internal load balancer for example
I am trying to provision RDS instance using terraform template and my template looks like this
template.tf
resource "aws_security_group" "web-server-security"{
name = "webserver-sg"
description = "webserver security group"
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
tags{
Name = "web-server-sg"
}
resource "aws_security_group" "db-server-sg" {
name = "db-server"
description = "dbserver security group"
ingress {
from_port = 3306
to_port = 3306
protocol = "tcp"
security_groups = ["${aws_security_group.web-server-security.id}"]
}
tags{
Name = "db-server-sg"
}
}
resource "aws_db_instance" "echomany_db" {
name = "echomanydb"
engine = "mysql"
engine_version = "5.7"
storage_type = "gp2"
allocated_storage = 20
instance_class = "db.t2.micro"
username = "${var.AWS_DB_USERNAME}"
password = "${var.AWS_DB_PASSWORD}"
parameter_group_name = "default.mysql5.7"
skip_final_snapshot = true
security_group_names = [
"${aws_security_group.db-server-sg.id}"
]
tags{
Name = "db-server"
}
}
However i get this following error:
1 error(s) occurred:
* aws_db_instance.echomany_db: 1 error(s) occurred:
aws_db_instance.echomany_db: Error creating DB Instance:
InvalidParameterCombination: DB Security Groups can only be
associated with VPC DB Instances using API versions 2012-01-15
through 2012-09-17.
status code: 400, request id: a19ea8ea-8ea0-46e4-97c6-b946419df9a3
i dont know whats the problem and how to fix this issue.
As mentioned by the documentation vpc_security_group_ids should be used instead of security_group_names which is a deprecated argument.
Parameter named
security_group_names = [
"${aws_security_group.db-server-sg.id}"
]
can only be used while using ec2-classic mode or outside VPC.
Use vpc_security_group_ids instead.
you can use vpc_security_groups_ids = [ ] instead of security_group_names because it can only be used while using ec2-classic mode
example:
vpc_security_group_ids=["${aws_security_group.rds.id}"]