I've created a JSON string via template/interpolation.
I need to pass that to local-exec, which in turn uses a Powershell template to make a CLI call.
Originally I tried just referencing the json template in the Powershell command itself
--cli-input-json file://lfsetup.tpl
.. however, the template does not get interpolated.
Next, I tried setting the json to a local. However, this is multi-line and the CLI does not like that. Maybe if I could convert to single line ?
Any sugestions or guidance welcome !!
Thanks
JSON (.tpl or variable)
{
"CatalogId": "${account_id}",
"DataLakeSettings": {
"DataLakeAdmins": [
{
"DataLakePrincipalIdentifier": "arn:aws:iam::${account_id}:role/Role1"
},
{
"DataLakePrincipalIdentifier": "arn:aws:iam::${account_id}:role/Role2"
}
],
"CreateDatabaseDefaultPermissions": [],
"CreateTableDefaultPermissions": []
}
}
.tf
locals {
assume_role_arn = "arn:aws:iam::${local.account_id}:role/role_to_assume"
lf_json_settings = templatefile("${path.module}/lfsetup.tpl", { account_id = local.account_id})
cli_region = "region"
}
resource "null_resource" "settings" {
provisioner "local-exec" {
command = templatefile("${path.module}/scripts/settings.ps1", { role_arn = local.assume_role_arn, json_settings = local.lf_json_settings, region = local.cli_region})
interpreter = ["pwsh", "-Command"]
}
}
.ps
$ErrorActionPreference = "Stop"
$json = aws sts assume-role --role-arn ${role_arn} --role-session-name sessionname
$accessTokens = ConvertFrom-Json (-join $json)
$env:AWS_ACCESS_KEY_ID = $accessTokens.Credentials.AccessKeyId
$env:AWS_SECRET_ACCESS_KEY = $accessTokens.Credentials.SecretAccessKey
$env:AWS_SESSION_TOKEN = $accessTokens.Credentials.SessionToken
aws lakeformation put-data-lake-settings --cli-input-json file://lfsetup.tpl --region ${region}
$env:AWS_ACCESS_KEY_ID = ""
$env:AWS_SECRET_ACCESS_KEY = ""
$env:AWS_SESSION_TOKEN = ""
Output:
For these I put the template output into a local and passed the local to powershell. Then did variations with/out jsonencde and trying to replace '\n'. Strange results in some cases:
Use file provisioner to create .json file from rendered .tpl file:
locals {
...
settings_json_file = "/tmp/lfsetup.json"
}
resource "null_resource" "settings" {
provisioner "file" {
content = templatefile("${path.module}/lfsetup.tpl", { account_id = local.account_id})
destination = local.settings_json_file
}
provisioner "local-exec" {
command = templatefile("${path.module}/scripts/settings.ps1", role_arn = local.assume_role_arn, json_settings = local.settings_json_file, region = local.cli_region})
interpreter = ["pwsh", "-Command"]
}
}
Update your .ps file
replace file://lfsetup.tpl by file://${json_settings}
aws lakeformation put-data-lake-settings --cli-input-json file://${json_settings} --region ${region}
You may also use jsonencode function
Related
I am attempting to grab a value within a Terraform/Python null_resource and make it available to other Terraform processes like so:
print_env.py
import os
# Create TF_VAR key value
param_tfvar = "TF_VAR_helloworld"
# Publish value through TF_VAR value
os.environ[param_tfvar] = "Hey Bob!"
print(param_tfvar, os.getenv(param_tfvar))
The Terraform *.tf files look like this:
variables.tf
variable "helloworld" {
description = "Display greeting"
type = string
default = ""
}
main.tf
resource "null_resource" "helloworld" {
provisioner "local-exec" {
command = <<EOC
python3 print_env.py
EOC
}
}
resource "null_resource" "echo_helloworld" {
provisioner "local-exec" {
command = "echo ${var.helloworld}"
}
depends_on = [
null_resource.helloworld
]
}
My issue is that the echo command does not display Hey Bob! but just a blank (the default value of helloworld defined in the variables.tf file.
I was looking at: https://www.terraform.io/language/values/variables#environment-variables on how to craft this solution.
Am I running into some type of scoping issue where the TF_VAR value published in the null_resource.helloworld block is not visible to the null_resource.echo_helloworld block?
Environment variables should be exported in the shell before you invoke terraform. Setting those at run-time like you are doing is not going to work.
Here's how I would solve your problem:
main.tf:
data "external" "get_greeting" {
program = ["python3", "get_env.py", var.greeting]
}
variable "greeting" {
description = "Display greeting"
type = string
default = ""
}
output "show_greeting" {
value = data.external.get_greeting.result.greeting
}
get_env.py:
import json
import sys
if len(sys.argv) > 1:
greeting = f"Hello {sys.argv[1]}"
else:
greeting = "Hello"
tf_dict = ({"greeting": greeting})
print(json.dumps(tf_dict))
And this is what you would see if you invoked it with no variable override:
$ terraform apply
outputs:
show_greeting = "Hello"
and with override:
$ terraform apply -var greeting="world"
outputs:
show_greeting = "Hello world"
I have a following example of Terraform resources where I fetch values from secrets manager and pass them to the Lambda function. The question is how can I add extra values to an object before passing it to environment variable without replicating the values?
resource "aws_secretsmanager_secret" "example" {
name = "example"
}
resource "aws_secretsmanager_secret_version" "example" {
secret_id = aws_secretsmanager_secret.example.id
secret_string = <<EOF
{
"FOO": "bar"
}
EOF
}
data "aws_secretsmanager_secret_version" "example" {
secret_id = aws_secretsmanager_secret.example.id
depends_on = [aws_secretsmanager_secret_version.example]
}
locals {
original_secrets = jsondecode(
data.aws_secretsmanager_secret_version.example.secret_string
)
}
resource "aws_lambda_function" "example" {
...
environment {
variables = local.original_secrets
}
}
As a pseudo code I'd like to do something like this:
local.original_secrets["LOG_LEVEL"] = "debug"
The current approach I have is just to replicate the original values and add a new but of course this is not DRY.
locals {
...
updated_secrets = {
FOO = try(local.original_secrets.FOO, "")
DEBUG = "false"
}
}
You can use Terraform merge function to produce new combined map of environment variables.
lambda_environment_variables = merge(local.lambda_secrets, local.environment_variables)
How to iterate resources for different values of a parameter
For example, in my below terraform file I have one data block and one resource. If I pass value DB_NAME=test then its working fine. But what if I have multiple values of DB_NAME and I want it to run multiple time DB_NAME=test, app. How will I iterate over data and resource block? :
data "template_file" "search-index" {
template = "${file("search-index/search-index.sh")}"
vars {
DB_NAME = "${var.DB_NAME}"
}
}
resource "null_resource" "script" {
triggers = {
DB_NAME = "${var.DB_NAME}"
script_sha = "${sha256(file("search-index/search-index.sh"))}"
}
provisioner "local-exec" {
command = "${data.template_file.search-index.rendered}"
interpreter = ["/bin/bash", "-c"]
}
}
I'm not sure if I fully understood what you are trying to achieve but you can create multiple DB by defining a variables.tf like this:
variable "DB_NAME" {
description = "A list of databases"
type = list(string)
default = ["db1", "db2", "db3"]
}
And then using the for_each functionality in your terraform file:
data "template_file" "search-index" {
for_each = toset(var.DB_NAME)
template = "${file("search-index/search-index.sh")}"
vars = {
DB_NAME = each.value
}
}
resource "null_resource" "script" {
for_each = toset(var.DB_NAME)
triggers = {
DB_NAME = each.value
script_sha = "${sha256(file("search-index/search-index.sh"))}"
}
provisioner "local-exec" {
command = "${data.template_file.search-index[each.value]}"
interpreter = ["/bin/bash", "-c"]
}
}
I tried to create a custom module to encrypt our access and secret access keys when outputs are used. Thus when a build runs it tries to print the out the access key after encrypting it using kms.
But currently when we use this module to create multiple users, it's just printing the 1st user's access key and secret key for other users as well.
Someone please suggest me how i should fix this. Using terraform 0.12.18
variable "iam_username" {
description = "IAM username"
}
variable "path" {
description = "path for IAM user"
default = "/"
}
resource "aws_iam_user" "iam_user" {
name = var.iam_username
path = var.path
}
resource "aws_iam_access_key" "iam_keys" {
user = aws_iam_user.iam_user.name
}
data "external" "stdout" {
program = [
"bash",
"${path.module}/encrypt_credentials.sh"]
query = {
access_key = aws_iam_access_key.iam_keys.id
secret_key = aws_iam_access_key.iam_keys.secret
}
}
encrypt_credentials.sh
function encrypt() {
aws kms encrypt --key-id alias/xxxx --plaintext $ACCESS_KEY --output text --query CiphertextBlob --region us-east-1 > encrypted_access_key
aws kms encrypt --key-id alias/xxxx --plaintext $SECRET_KEY --output text --query CiphertextBlob --region us-east-1 > encrypted_secret_key
}
function output() {
access_key=$(cat encrypted_access_key )
secret_key=$(cat encrypted_secret_key)
jq -n \
--arg access_value "$access_key" \
--arg secret_value "$secret_key"\
'{"access_value":$access_value,"secret_value":$secret_value}'
}
encrypt
output
outputs.tf
output "aws_iam_access_key" {
value = chomp(data.external.stdout.result["access_value"])
}
output "aws_iam_secret_access_key" {
value = chomp(data.external.stdout.result["secret_value"])
}
I tested this module, I'm trying to create two users, test1, test2 ..here is the output, it as the same access key and secret key for both users
Terraform
module "test1user" {
source = "../../"
iam_username = "test1"
path = "/"
}
module "test2user" {
source = "../../"
iam_username = "test2"
path = "/"
}
outputs.tf
output "user1_access_key" {
value = module.test1user.aws_iam_access_key
}
output "user1_secret_key" {
value = module.test1user.aws_iam_secret_access_key
}
output "user2_access_key" {
value = module.test2user.aws_iam_access_key
}
output "user2_secret_key" {
value = module.test2user.aws_iam_secret_access_key
}
14:47:47 TestTerraformAwsNetworkExample 2020-07-22T18:47:47Z logger.go:66: user1_access_key = AQECAHj0ior/LD5LXMzmwFwEYlbqXWdHuCRWGQNeqhU6VNir+gAAAHIwcAYJKoZIhvcNAQcGoGMwYQIBADBcBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDULiS2JecmxLYdv9QIBEIAvjB60Maw5IuryzukItn8awWXnqfUzUcnPJNq7mFHQ2MYRBtOqBJJo0zbPo1i+pgw=
14:47:47 TestTerraformAwsNetworkExample 2020-07-22T18:47:47Z logger.go:66: user1_secret_key = AQECAHj0ior/LD5LXMzmwFwEYlbqXWdHuCRWGQNeqhU6VNir+gAAAIcwgYQGCSqGSIb3DQEHBqB3MHUCAQAwcAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAxyo66cMnxkOCrHjhoCARCAQzbpGYCzH6Ed+XvDFinBSbrK0LDk0YMXh39JCcztYwoJDFMbAtnWlS4cUyrmncf5paxE2oB7w2ujtpds/dBxUtsw6Lg=
14:47:47 TestTerraformAwsNetworkExample 2020-07-22T18:47:47Z logger.go:66: user2_access_key = AQECAHj0ior/LD5LXMzmwFwEYlbqXWdHuCRWGQNeqhU6VNir+gAAAHIwcAYJKoZIhvcNAQcGoGMwYQIBADBcBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDULiS2JecmxLYdv9QIBEIAvjB60Maw5IuryzukItn8awWXnqfUzUcnPJNq7mFHQ2MYRBtOqBJJo0zbPo1i+pgw=
14:47:47 TestTerraformAwsNetworkExample 2020-07-22T18:47:47Z logger.go:66: user2_secret_key = AQECAHj0ior/LD5LXMzmwFwEYlbqXWdHuCRWGQNeqhU6VNir+gAAAIcwgYQGCSqGSIb3DQEHBqB3MHUCAQAwcAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAxyo66cMnxkOCrHjhoCARCAQzbpGYCzH6Ed+XvDFinBSbrK0LDk0YMXh39JCcztYwoJDFMbAtnWlS4cUyrmncf5paxE2oB7w2ujtpds/dBxUtsw6Lg=
I refactored a lot of the code as I tried to reproduce it...
Finally got it working.
What I found suspicious was your > encrypted_access_key to latter read it back, we can just load that into a var and consume it without the intermediary file, and that is what I did.
module
variable "name" {
type = string
}
resource "aws_iam_user" "iam_user" {
name = var.name
}
resource "aws_iam_access_key" "iam_keys" {
user = aws_iam_user.iam_user.name
}
data "external" "stdout" {
program = [ "bash", "${path.module}/encrypt.sh"]
query = {
id = aws_iam_access_key.iam_keys.id
se = aws_iam_access_key.iam_keys.secret
}
}
output "out" {
value = data.external.stdout.result
}
#!/bin/bash
eval "$(jq -r '#sh "ID=\(.id) SE=\(.se)"')"
access=$(aws kms encrypt --key-id alias/xxxx --plaintext $ID --output text --query CiphertextBlob --region us-east-1)
secret=$(aws kms encrypt --key-id alias/xxxx --plaintext $SE --output text --query CiphertextBlob --region us-east-1)
jq -n --arg a "$access" --arg s "$secret" '{"access_value":$a,"secret_value":$s}'
main
provider "aws" {
region = "us-east-1"
}
module "test1user" {
source = "./aws_user"
name = "test1"
}
output "user1_out" {
value = module.test1user.out
}
module "test2user" {
source = "./aws_user"
name = "test2"
}
output "user2_out" {
value = module.test2user.out
}
output
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Outputs:
user1_out = {
"access_value" = "AQICAHgxynd50R/zNmpbsZ8biySxfHUL9kNuyyylE5GSqkiK7wHYbkBH3jxR3zvkFLogYVAsAAAAcjBwBgkqhkiG9w0BBwagYzBhAgEAMFwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMWhETOYT+qhL/IibfAgEQgC+kdJy7fJLZBW/AUk7YdjqDeAyymt6xBxeS1kBJIOWdVnwOujAkLG0wI+JAUqin8w=="
"secret_value" = "AQICAHgxynd50R/zNmpbsZ8biySxfHUL9kNuyyylE5GSqkiK7wFvozPjgGKbxj61aKEbxYUwAAAAhzCBhAYJKoZIhvcNAQcGoHcwdQIBADBwBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJOwAiWgVWPtIzwURAIBEIBD7Q78YneG+/FMlkDTUnCkczf8TQBezQyMCI5cUx4qVX7iECvzx/5qAfKdy3tI4ViUGR5XV12WBvWIXj8iRN55D0jK4A=="
}
user2_out = {
"access_value" = "AQICAHgxynd50R/zNmpbsZ8biySxfHUL9kNuyyylE5GSqkiK7wFIms+isXNTAl6xWDiXcz1gAAAAcjBwBgkqhkiG9w0BBwagYzBhAgEAMFwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMxiChWwPDGCdImUtXAgEQgC9vJfi6GaHXbqal/2nSc9FSkXEOPOsn7J+a5u8JiI2x6flBoeia9QMjVv9tOxpzYA=="
"secret_value" = "AQICAHgxynd50R/zNmpbsZ8biySxfHUL9kNuyyylE5GSqkiK7wFBLdzTFeCSk2Zv16sSHZ8bAAAAhzCBhAYJKoZIhvcNAQcGoHcwdQIBADBwBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDNGAphqIxZPthA+IkgIBEIBDAufp2xtAsfNctmnEa4grTb15MatDKJuqIB8qWCBaht563qp+RbL1aoZ8oxPYYtiU2LuHUnvbhHtWklvn2SkdSDN90w=="
}
I tested locally on Ubuntu 18.04.4 with:
Terraform v0.12.24
+ provider.aws v2.54.0
+ provider.external v1.2.0
Here is the entire code:
https://github.com/heldersepu/hs-scripts/tree/master/TerraForm/encrypt_output
I am trying to have a common user_data file for common tasks such as folder creation and certain package install and a separate user_data file for application specific configuration
I am trying the below -
user_data = "${data.template_file.userdata_common.rendered}", "${data.template_file.userdata_master.rendered}"
With these configs -
Common User Data Template
data "template_file" "userdata_common" {
template = "${file("${path.module}/userdata_common.sh")}"
vars {
"ALBTarget" = "${var.ALBTarget}"
"s3bucket" = "${var.s3bucket}"
"centrifydomain" = "${lookup(var.centrifydomain, format("%s-%s", lower(var.env),var.region))}"
"centrifyadgroup" = "${lookup(var.centrifyadgroup, format("%s-%s", lower(var.env),var.region))}"
}
}
Application Specific Config
data "template_file" "userdata_master" {
template = "${file("${path.module}/userdata_master.sh")}"
vars {
"ALBTarget" = "${var.ALBTarget}"
"s3bucket" = "${var.s3bucket}"
"centrifydomain" = "${lookup(var.centrifydomain, format("%s-%s", lower(var.env),var.region))}"
"centrifyadgroup" = "${lookup(var.centrifyadgroup, format("%s-%s", lower(var.env),var.region))}"
}
}
I get the below Error when i do Plan -
Failed to load root config module: Error parsing /terraform/main.tf: key ${data.template_file.userdata_common.rendered}"' expected start of object ('{') or assignment ('=')
Is this possible using Terraform (0.9.3)?
If not, what's the best way to do this with Terraform?
Did you try template_cloudinit_config?
Add below codes.
data "template_cloudinit_config" "master" {
gzip = true
base64_encode = true
# get common user_data
part {
filename = "common.cfg"
content_type = "text/part-handler"
content = "${data.template_file.userdata_common.rendered}"
}
# get master user_data
part {
filename = "master.cfg"
content_type = "text/part-handler"
content = "${data.template_file.userdata_master.rendered}"
}
}
# sample code to use it.
resource "aws_instance" "web" {
ami = "ami-d05e75b8"
instance_type = "t2.micro"
user_data = "${data.template_cloudinit_config.master.rendered}"
}
Let me know if it works.
You can use "provisioner" to modify the infrastructure you are creating using the Terraform, Here is the example from them https://www.terraform.io/intro/getting-started/provision.html