Terraform destroys the instance inside RDS cluster when upgrading - terraform

Question

I have created a RDS cluster with 2 instances using terraform. When I am upgrading the RDS from front-end, it modifies the cluster. But when I do the same using terraform, it destroys the instance.
We tried create_before_destroy, and it gives error.
We tried with ignore_changes=engine but that didn't make any changes.
Is there any way to prevent it?
resource "aws_rds_cluster" "rds_mysql" {
cluster_identifier = var.cluster_identifier
engine = var.engine
engine_version = var.engine_version
engine_mode = var.engine_mode
availability_zones = var.availability_zones
database_name = var.database_name
port = var.db_port
master_username = var.master_username
master_password = var.master_password
backup_retention_period = var.backup_retention_period
preferred_backup_window = var.engine_mode == "serverless" ? null : var.preferred_backup_window
db_subnet_group_name = var.create_db_subnet_group == "true" ? aws_db_subnet_group.rds_subnet_group[0].id : var.db_subnet_group_name
vpc_security_group_ids = var.vpc_security_group_ids
db_cluster_parameter_group_name = var.create_cluster_parameter_group == "true" ? aws_rds_cluster_parameter_group.rds_cluster_parameter_group[0].id : var.cluster_parameter_group
skip_final_snapshot = var.skip_final_snapshot
deletion_protection = var.deletion_protection
allow_major_version_upgrade = var.allow_major_version_upgrade
lifecycle {
create_before_destroy = false
ignore_changes = [availability_zones]
}
}
resource "aws_rds_cluster_instance" "cluster_instances" {
count = var.engine_mode == "serverless" ? 0 : var.cluster_instance_count
identifier = "${var.cluster_identifier}-${count.index}"
cluster_identifier = aws_rds_cluster.rds_mysql.id
instance_class = var.instance_class
engine = var.engine
engine_version = aws_rds_cluster.rds_mysql.engine_version
db_subnet_group_name = var.create_db_subnet_group == "true" ? aws_db_subnet_group.rds_subnet_group[0].id : var.db_subnet_group_name
db_parameter_group_name = var.create_db_parameter_group == "true" ? aws_db_parameter_group.rds_instance_parameter_group[0].id : var.db_parameter_group
apply_immediately = var.apply_immediately
auto_minor_version_upgrade = var.auto_minor_version_upgrade
lifecycle {
create_before_destroy = false
ignore_changes = [engine_version]
}
}
Error:
resource \"aws_rds_cluster_instance\" \"cluster_instances\" {\n\n\n\nError: error creating RDS Cluster (aurora-cluster-mysql) Instance: DBInstanceAlreadyExists: DB instance already exists\n\tstatus code: 400, request id: c6a063cc-4ffd-4710-aff2-eb0667b0774f\n\n on
Plan output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
+/- create replacement and then destroy
Terraform will perform the following actions:
# module.rds_aurora_create[0].aws_rds_cluster.rds_mysql will be updated in-place
~ resource "aws_rds_cluster" "rds_mysql" {
~ allow_major_version_upgrade = false -> true
~ engine_version = "5.7.mysql_aurora.2.07.1" -> "5.7.mysql_aurora.2.08.1"
id = "aurora-cluster-mysql"
tags = {}
# (33 unchanged attributes hidden)
}
# module.rds_aurora_create[0].aws_rds_cluster_instance.cluster_instances[0] must be replaced
+/- resource "aws_rds_cluster_instance" "cluster_instances" {
~ arn = "arn:aws:rds:us-east-1:account:db:aurora-cluster-mysql-0" -> (known after apply)
~ availability_zone = "us-east-1a" -> (known after apply)
~ ca_cert_identifier = "rds-ca-" -> (known after apply)
~ dbi_resource_id = "db-32432432SDF" -> (known after apply)
~ endpoint = "aurora-cluster-mysql-0.jkjk.us-east-1.rds.amazonaws.com" -> (known after apply)
~ engine_version = "5.7.mysql_aurora.2.07.1" -> "5.7.mysql_aurora.2.08.1" # forces replacement
~ id = "aurora-cluster-mysql-0" -> (known after apply)
+ identifier_prefix = (known after apply)
+ kms_key_id = (known after apply)
+ monitoring_role_arn = (known after apply)
~ performance_insights_enabled = false -> (known after apply)
+ performance_insights_kms_key_id = (known after apply)
~ port = 3306 -> (known after apply)
~ preferred_backup_window = "07:00-09:00" -> (known after apply)
~ preferred_maintenance_window = "thu:06:12-thu:06:42" -> (known after apply)
~ storage_encrypted = false -> (known after apply)
- tags = {} -> null
~ tags_all = {} -> (known after apply)
~ writer = true -> (known after apply)
# (12 unchanged attributes hidden)
}
Plan: 1 to add, 1 to change, 1 to destroy.

Answer 1

I see apply_immediately argument not there in aws_rds_cluster resource , can you add that and try.

Answer 2

Terraform is seeing the engine version change on the instances and is detecting this as an action that forces replacement.
Remove (or ignore changes to) the engine_version input for the aws_rds_cluster_instance resources.
AWS RDS upgrades the engine version for cluster instances itself when you upgrade the engine version of the cluster (this is why you can do an in-place upgrade via the AWS console).
By excluding the engine_version input, Terraform will see no changes made to the aws_rds_cluster_instances and will do nothing.
AWS will handle the engine upgrades for the instances internally.
If you decide to ignore changes, use the ignore_changes argument within a lifecycle block:
resource "aws_rds_cluster_instance" "cluster_instance" {
engine_version = aws_rds_cluster.main.engine_version
...
lifecycle {
ignore_changes = [engine_version]
}
}

Answer 3

I didn't know that, but after some Googling I found this:
https://github.com/hashicorp/terraform-provider-aws/issues/10714
i.e. a bug report to AWS Terraform provider:
resource/aws_rds_cluster_instance is being destroyed and re-created when updating engine_version while apply_immediately is set to false
which seems to be the very same issue you are facing.
One comment there seems to point to a solution:
As of v3.63.0 (EDITED) of the provider, updates to the engine_version parameter of aws_rds_cluster_instance resources no longer forces replacement of the resource.
The original comment seems to have a typo - 3.36 vs. 3.63.
Can you try upgrading your aws Terraform provider?