How to use/reference Terraform output values in BigQuery Schema file - terraform

Question

Using Terraform BigQuery Module to deploy BQ schema. Trying to define policy tags but not sure how to reference newly created Taxonomy and Policy Tag Ids inside my JSON Schema. Below is a dummy extract of how my schema.json is looking with policy tags linked to fields
Problem:
Schema below referencing the ids of Taxonomy and Policy Tag as
${google_data_catalog_taxonomy.my_taxonomy.id}
but when I apply TF it's not replacing values and throwing exception
Error 400: Invalid value for policyTags:
projects/my_project/locations/europe-
west2/taxonomies/${google_data_catalog_taxonomy.my_taxonomy.id}/policyTags/${google_data_catalog_policy_tag.PII.id} is not a valid value. Expected value should follow the format "projects/<projectId>/locations/<locationId>/taxonomies/<taxonomyId>/policyTags/<policyTagId>".
Table_1.json looks like following
{
"fields": [
{
"mode": "NULLABLE",
"name": "Email",
"type": "STRING",
"policyTags":{
"names": [
"projects/my_project/locations/europe-west2/taxonomies/${google_data_catalog_taxonomy.my_taxonomy.id}/policyTags/${google_data_catalog_policy_tag.PII.id}"
]
}
},
{
"mode": "NULLABLE",
"name": "Mobile",
"type": "STRING",
"policyTags":{
"names": [
"projects/my_project/locations/europe-west2/taxonomies/${google_data_catalog_taxonomy.my_taxonomy.id}/policyTags/${google_data_catalog_policy_tag.PII.id}"
]
}
},
}
I am outputting Taxonomy and Policy tags as following. Can Anyone please suggest how this can be referenced in schema.json file.
outputs.tf
output "my_taxonomy" {
value = google_data_catalog_taxonomy.my_taxonomy.id
}
output "PII" {
value = google_data_catalog_policy_tag.PII.id
}
Edit:
I am using TF BigQuery module where my table schema exists in a separate file.
main.tf
module "bigquery" {
source = "terraform-google-modules/bigquery/google"
dataset_id = "my_Dataset"
dataset_name = "my_Dataset"
description = "my_Dataset"
project_id = "my_project_id"
location = "europe-west2"
default_table_expiration_ms = 3600000
tables = [
{
table_id = "table_!",
**schema = "table_1.json",**
time_partitioning = null,
range_partitioning = null,
expiration_time = null,
clustering = null,
labels = {
env = "dev"
}
}
},
]
}

Answer 1

You can import the json template using templatefile( path ,vars ).
Edit your json to include the vars using ${ ... } syntax.
{
"fields": [
{
"mode": "NULLABLE",
"name": "Email",
"type": "STRING",
"policyTags":{
"names": [
"projects/my_project/locations/europe-west2/taxonomies/${my_taxonomy}/policyTags/${PII}"
]
}
},
{
"mode": "NULLABLE",
"name": "Mobile",
"type": "STRING",
"policyTags":{
"names": [
"projects/my_project/locations/europe-west2/taxonomies/${my_taxonomy}/policyTags/${PII}"
]
}
},
}
In your terraform config edit the schema to use the templatefile function
module "bigquery" {
source = "terraform-google-modules/bigquery/google"
dataset_id = "my_Dataset"
dataset_name = "my_Dataset"
description = "my_Dataset"
project_id = "my_project_id"
location = "europe-west2"
default_table_expiration_ms = 3600000
tables = [
{
table_id = "table_!",
schema = templatefile(
"${path.module}/table_1.json",
{
my_taxonomy = "${google_data_catalog_taxonomy.my_taxonomy.id}",
PII = "${google_data_catalog_policy_tag.PII.id}"
}),
time_partitioning = null,
range_partitioning = null,
expiration_time = null,
clustering = null,
labels = {
env = "dev"
}
}
},
]
}