Unable to pass service annotations when deploying helm chart via terraform - terraform

Question

I have seeing some examples regarding how to pass annotations when deploying a helm chart via terraform but none of then are working as expected, in this case, im trying to create a service assining a private ip on a specific subnet, but instead, its creating a public IP.
My terraform files:
locals {
helm_general = {
# Reference values
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
elasticsearch = {
name = "elasticsearch"
chart = "elastic/elasticsearch"
tag = "7.14.0"
namespace = "elasticsearch"
set = [
{
name = "nodeSelector.agentpool"
value = "general"
},
{
name = "replicas"
value = "1"
},
{
name = "minimumMasterNodes"
value = "1"
},
{
name = "image"
value = "docker.elastic.co/elasticsearch/elasticsearch"
},
{
name = "imageTag"
value = "7.14.0"
},
{
name = "resources.requests.cpu"
value = "10m"
},
{
name = "resources.requests.memory"
value = "128Mi"
},
{
name = "volumeClaimTemplate.reosources.requests.storage"
value = "4Gi"
},
{
name = "persistence.enabled"
value = "false"
},
{
name = "service.type"
value = "LoadBalancer"
},
{
name = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
value = "true"
},
{
name = "service.annotations\\.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal-subnet"
value = "somesubnet"
},
]
timeout = "900"
}
}
}
Helm deployment
resource "helm_release" "helm" {
provider = helm.general
for_each = local.helm_general
name = each.value.name
chart = each.value.chart
namespace = format(each.value.namespace)
dynamic "set" {
iterator = item
for_each = each.value.set == null ? [] : each.value.set
content {
name = item.value.name
value = item.value.value
}
}
depends_on = [kubernetes_namespace.general]
}
Plan / apply output
https://i.imgur.com/uAdmblM.png
And what is currently being deployed is a public ip instead of a private ip:
Namespace: elasticsearch
Labels: app=elasticsearch-master
app.kubernetes.io/managed-by=Helm
chart=elasticsearch
heritage=Helm
release=elasticsearch
Annotations: meta.helm.sh/release-name: elasticsearch
meta.helm.sh/release-namespace: elasticsearch
Selector: app=elasticsearch-master,chart=elasticsearch,release=elasticsearch
Type: LoadBalancer
IP Families: <none>
IP: xx
IPs: xxx
LoadBalancer Ingress: redacted public ip
Port: http 9200/TCP
TargetPort: 9200/TCP
NodePort: http 32083/TCP
Endpoints:
Port: transport 9300/TCP
TargetPort: 9300/TCP
NodePort: transport 32638/TCP
Endpoints:
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 1s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer <invalid> service-controller Ensured load balancer
References that i have been following:
https://github.com/hashicorp/terraform-provider-helm/issues/125
https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release
Edit:
error message:
Error: unable to decode "": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|nternal":true},"labe|..., bigger context
..|beta.kubernetes.io/azure-load-balancer-internal":true},"labels":{"app":"elasticsearch-master","chart|...
with helm_release.helm["elasticsearch"],
on aks-general-helm.tf line 1, in resource "helm_release" "helm":
1: resource "helm_release" "helm" {

Answer 1

I just faced a similar issue, and here is what worked for me:
{
name = "service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-internal"
value = "true"
},
I think the issue is how it is concatenated. The service in the chart manifest for elastic official is service.annotations:{} so you need to append .service then use \\.

Answer 2

if you have multiple you can try something like this, pass annotations as map and set them using
dynamic "set" {
for_each = var.ingress_annotations
content {
name = replace(set.key, ".", "\\.")
value = set.value
}
}

Answer 3

One way of doing this without the escape characters and keeping the original YAML format would be using values attribute of the helm_release resource. Would be curious to know if there was a specific used-case to not use it in the first place.
resource "helm_release" "helm" {
provider = helm.general
for_each = local.helm_general
name = each.value.name
chart = each.value.chart
namespace = format(each.value.namespace)
values = each.value.values ##CHANGE IS HERE ##
dynamic "set" {
iterator = item
for_each = each.value.set == null ? [] : each.value.set
content {
name = item.value.name
value = item.value.value
}
}
depends_on = [kubernetes_namespace.general]
}
The local in your case would be adjusted to below, you can still keep set for something which has a dependency on any terraform resources or with any other logical reasons.
locals {
helm_general = {
# Reference values
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
elasticsearch = {
[...]
values = [file("${path.module}/elasticsearch-values.yaml")]
[...]
}
}
}
There has to be a new file elasticsearch-values.yaml at the same path (which can be adjusted with any relative path as per the local.helm_general.elasticsearch.values) location, where this terraform configurations exist.
# Reference values can be adapted as per the upstream chart.
# https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
service:
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "somesubnet"