I'm building out my infrastructure in terraform and one of the outputs that I would like is the ssh config formatted for the newly created server.
I created a template file:
Host ${serverLabel}
HostName ${hostname}
User ${user}
IdentityFile ${identityFile}
And used it in an output:
output "ssh-config" {
description = "The new server's ssh config"
value = templatefile("${path.module}/ssh-config.tpl", {
serverLabel=aws_instance.minecraft_ec2.tags.Name
hostname= aws_instance.minecraft_ec2.public_ip
user= "ec2-user",
identityFile = "~/.ssh/minecraft_key"
})
}
Which works fine except for one little hiccup. I end up with some heredoc tags in the output:
It's not the end of the world because I can just copy formatted text between the heredocs, but is there a way of using a template file as an output value without getting the tags??
Update
In reply to Mark B's question:
What happens if you run terraform output --raw?
Apparently when I output it with the --raw flag the heredoc tags go away
Which feels weird, but reading the description for the output command I guess makes sense:
-raw - If specified, Terraform will convert the specified output value to a string and print that string directly to the output, without any special formatting. This can be convenient when working with shell scripts, but it only supports string, number, and boolean values. Use -json instead for processing complex data types.
So I guess the heredoc tags are considered "special formatting". Makes sense, thought it was not what I expected.
It appears this is how Terraform now outputs multiline values, similar to how regular values had quotes added to them a few versions ago. To get the values without quotes or heredoc tags wrapping them, use terraform output --raw.
Related
I wanted to ask if there is a way to ignore whitespace changes when creating a terraform plan.
This question is related to this one, I created a new one because I wanted to give a new example of the issue.
Terraform shows unnecessary changes due to whitespace
For example, when running
terraform plan
I get the following change for a helm provider resource
# helm_release.cert-manager will be updated in-place
~ resource "helm_release" "cert-manager" {
id = "cert-manager"
name = "cert-manager"
~ values = [
- <<-EOT
installCRDs: true
EOT,
+ <<-EOT
installCRDs: true
EOT,
]
# (27 unchanged attributes hidden)
}
I found out that the change was due to line endings. Deployed was CRLF and my local source file had LF as line ending.
Is there an option to ignore whitespaces and/or line ending characters?
It's typically the responsibility of the provider itself to determine whether the prior value and the new value are equivalent despite not being exactly equal, and so making this work automatically would require a change to the provider itself to notice that this argument is defined as being YAML and YAML doesn't ascribe any meaning to the decision between CRLF and just LF. The provider would ideally perform this check itself and thus avoid you needing to worry about it, and I would suggest opening a feature request with the provider developer to see if they would be interested in handling that.
However, if a provider isn't performing that job correctly itself then you can potentially work around it by doing your own normalization of the value using Terraform language features, so that the value passed to the provider is always the same when the meaning is the same.
One straightforward way to achieve that in this case would be to round-trip the value through both yamldecode and yamlencode, thereby normalizing the input to be in the style that yamlencode produces:
values = [yamlencode(yamldecode(var.something))]
If you want to be more surgical about it and only normalize the line endings, you could use replace to remove the CR character from any CRLF pair:
values = [replace(var.something, "\r\n", "\n")]
The above solution assumes that the difference in whitespace is being caused by something in your module, such as if you're storing your Terraform configuration in a misconfigured Git repository that's rewriting LF to CRLF when you clone it on a Windows system. This config-based normalization can undo that sort of transformation so that the provider will always see the value in the same way.
This solution cannot address problems that are caused by the provider itself misbehaving. Unfortunately some providers have bugs where they will silently rewrite the stored values for some arguments during the "refresh" step, regardless of how you wrote it in the configuration. In that case the only recourse is to fix the provider, because that incorrect value is originating inside the provider itself and isn't under the control of the module author.
I have a Python program that is being called by an "External Data Source" explained here:
https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/data_source
The code looks like something like this:
data "external" "example" {
program = ["python", "${path.module}/example-data-source.py"]
query = {
# arbitrary map from strings to strings, passed
# to the external program as the data query.
id = "abc123"
}
}
Is there technique or workaround for my external Python program to know which terraform operation (plan, apply or destroy) is calling the external data?
Is there any way for a hcl code to find what operation is it being called under?
Update #1
I understand that we can pass parameters as query argument.
The question is that is there any way a Terraform script know which Terraform operation type is it being called under? so that I have the operation (Apply/Destroy) in a terraform variable then pass it as query argument to the Python program?
The external data source isn't able to infer what the Terraform operation type is as it is only sent the data in the query argument, as mentioned and shown in the data source's documentation.
As an example, if we look at the example in the documentation:
#!/bin/bash
# Exit if any of the intermediate steps fail
set -e
# Extract "foo" and "baz" arguments from the input into
# FOO and BAZ shell variables.
# jq will ensure that the values are properly quoted
# and escaped for consumption by the shell.
eval "$(jq -r '#sh "FOO=\(.foo) BAZ=\(.baz)"')"
# Placeholder for whatever data-fetching logic your script implements
FOOBAZ="$FOO $BAZ"
# Safely produce a JSON object containing the result value.
# jq will ensure that the value is properly quoted
# and escaped to produce a valid JSON string.
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
We can see that it's reading the foo and baz keys from whatever is sent to the program via stdin.
If we ran this in the most simple way we can see that it does as we expect:
data "external" "example" {
program = ["sh", "${path.module}/example-data-source.sh"]
query = {
foo = "abc1234"
baz = "123abc"
}
}
output "example" {
value = data.external.example
}
Gives the following output:
Outputs:
example = {
"id" = "-"
"program" = tolist([
"sh",
"./example-data-source.sh",
])
"query" = tomap({
"baz" = "123abc"
"foo" = "abc1234"
})
"result" = tomap({
"foobaz" = "abc1234 123abc"
})
"working_dir" = tostring(null)
}
Just to check that there's nothing sneaky there that Terraform communicates over stdin we can modify the script to log out to a file what is passed on stdin (note that this will break the example script so that jq no longer sees the input from Terraform):
#!/bin/bash
# Exit if any of the intermediate steps fail
set -e
# Log stdin inputs to an output file so we only output the formatted
# JSON on stdout for the external data source to work correctly.
cat - >> log.out
# Extract "foo" and "baz" arguments from the input into
# FOO and BAZ shell variables.
# jq will ensure that the values are properly quoted
# and escaped for consumption by the shell.
eval "$(jq -r '#sh "FOO=\(.foo) BAZ=\(.baz)"')"
# Placeholder for whatever data-fetching logic your script implements
FOOBAZ="$FOO $BAZ"
# Safely produce a JSON object containing the result value.
# jq will ensure that the value is properly quoted
# and escaped to produce a valid JSON string.
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
And then if we run a plan we can see just the expected JSON in the log.out file:
{"baz":"123abc","foo":"abc1234"}
If you could access the operation type in your Terraform code instead then you could pass that in as a parameter to your external data source but that's also not exposed anywhere that you can access unfortunately.
It's also worth mentioning that as a data source, it's expected to be side effect free to run the script. This means it shouldn't really matter what operation Terraform is running when it invokes your data source. If you need to do something else that isn't side effect free then you might need to consider another option such as a provisioner, potentially attached to a null_resource but all of these options are meant to be last resorts where you need an escape hatch. Most of the time it's better to either raise a feature request in the appropriate provider or to call these things separately and orchestrate them together via a wrapper script or some orchestration software that calls them in succession.
Haven't tested this, but you should be able to get terraform's pid with os.getppid(). Then you can get more information with psutil (available in pip), /proc (linux only) or subprocessing a ps command. Finding the verb on the command line may be brittle, but depending on context, it can probably work.
That said, if you need to do this, you should reconsider how your code is organized. A pure data source shouldn't care.
I'm trying to obtain a secret out of my KeyVault.
The variable name is secretVar.
Obtaining the secret like this: $(secretVar) works fine however I would like to retrieve it from a variable like this:
I keep getting command not found and I've no idea why this shouldn't be working.
So the name of the secret I want to extract is inside a bash variable. For this question I've simplified the problem but in my real use case I have a bash for loop which loops through secret names and inside the for loop I want to extract the appropriate value from the KeyVault with the corresponding secret name like this:
for secretname in secrets; do
echo $($secretname) # This should contain the value of the secret but gives command not found
done
If anyone has an idea what could be happening, any help is very appreciated.
Thanks in Advance!
Look at the syntax you're using.
variable=secretVar
You are creating an environment variable with the literal value secretVar
Then you try to execute the value of the variable $variable with $($variable). So it tries to run the command secretVar, which obviously doesn't exist, and you get an error message.
The syntax you're looking for is
variable=$(secretVar)
just like you used in the first echo command in the script.
If you don't want to run the variable value as a command, the syntax would be $variable, not $($variable)
$variable is the syntax for a Bash environment variable.
$(variable) is the syntax for referencing Azure DevOps variables.
First of all, the script keyword is a shortcut for the command-line task. The task runs a script using cmd.exe on Windows and Bash on other platforms. You need to pay attention to the agent you are using.
If you want to set variables in scripts, you can use task.setvariable logging command. For example:
- script: |
echo $(secretvar)
echo "##vso[task.setvariable variable=variable]$(secretvar)"
- script: |
echo $(variable)
You can find more detailed information in this document.
How can I retrieve a bash variable value if I have the variable name as string? var1="this is the real value"
a="var1"
Do something to get value of var1 just using variable a.
Context:
I have some AMI's (Amazon Machine Image) and I want to fire up a few instances of each AMI. As soon as they finish booting, I want to setup each instance according to its AMI type. I don't want to bake lots of scripts or secret keys inside any AMI so I prepared a generalized startup script and I put it on S3 with a publicly accessible link. In rc.local I put small piece of code which fetches the startup script and executes it. This is all I have in the AMIs. Then each AMI accesses a common configuration script which is applicable to all AMIs and special setup scripts for each. These scripts are private and require a signed URL to access them.
So now, when I fire an instance of an AMI (my_private_ami_1), I pass a signed URL for one more file presented on S3 which contains signed URL for all private scripts in terms of key/value pair.config_url="http://s3.amazo.../config?signature"
my_private_ami_1="http://s3.amazo.../ami_1?signature"
...
When the startup script runs, it downloads the above file and source's it. Then it checks for its AMI type and picks the correct setup script for itself.
ami\_type=GET AMI TYPE #ex: sets ami\_type to my\_private\_ami\_1
setup\_url=GET THE SETUP FILE URL BASED ON AMI\_TYPE # this is where this problem arises
So now I can have a generic code which can fire instances irrespective of their AMI types and instances can take care of themselves.
You can use ${!a}:
var1="this is the real value"
a="var1"
echo "${!a}" # outputs 'this is the real value'
This is an example of indirect parameter expansion:
The basic form of parameter expansion is ${parameter}. The value of
parameter is substituted.
If the first character of parameter is an exclamation point (!), it
introduces a level of variable indirection. Bash uses the value of the
variable formed from the rest of parameter as the name of the
variable; this variable is then expanded and that value is used in the
rest of the substitution, rather than the value of parameter itself.
X=foo
Y=X
eval "Z=\$$Y"
sets Z to foo.
For my fellow zsh users, the way to accomplish the same thing as the accepted answer is to use:
echo ${(P)a} # outputs 'this is the real value'
It is appropriately called Parameter name replacement
This forces the value of the parameter name to be interpreted as a
further parameter name, whose value will be used where appropriate.
Note that flags set with one of the typeset family of commands (in
particular case transformations) are not applied to the value of name
used in this fashion.
If used with a nested parameter or command substitution, the result of
that will be taken as a parameter name in the same way. For example,
if you have ‘foo=bar’ and ‘bar=baz’, the strings ${(P)foo},
${(P)${foo}}, and ${(P)$(echo bar)} will be expanded to ‘baz’.
Likewise, if the reference is itself nested, the expression with the
flag is treated as if it were directly replaced by the parameter name.
It is an error if this nested substitution produces an array with more
than one word. For example, if ‘name=assoc’ where the parameter assoc
is an associative array, then ‘${${(P)name}[elt]}’ refers to the
element of the associative subscripted ‘elt’.
Modified my search keywords and Got it :). eval a=\$$a Thanks for your time.
In bash 4.3+, you can use declare -n:
#!/usr/bin/env bash
var="this is the real value"
var_name="var"
declare -n var_ref=$var_name
echo "${var_ref}"
Had the same issue with arrays, here is how to do it if you're manipulating arrays too :
array_name="ARRAY_NAME"
ARRAY_NAME=("Val0" "Val1" "Val2")
ARRAY=$array_name[#]
echo "ARRAY=${ARRAY}"
ARRAY=("${!ARRAY}")
echo "ARRAY=${ARRAY[#]}"
echo "ARRAY[0]=${ARRAY[0]}"
echo "ARRAY[1]=${ARRAY[1]}"
echo "ARRAY[2]=${ARRAY[2]}"
This will output :
ARRAY=ARRAY_NAME[#]
ARRAY=Val0 Val1 Val2
ARRAY[0]=Val0
ARRAY[1]=Val1
ARRAY[2]=Val2
In bash 4.3, the '-v' test for set variables was introduced. At the same time, 'nameref' declaration was added. These two features together with the indirection operator (!) enable a simplified version of the previous example:
get_value()
{
declare -n var_name=$1
if [[ -v var_name ]]
then
echo "${var_name}"
else
echo "variable with name <${!var_name}> is not set"
fi
}
test=123
get_value test
123
test="\$(echo \"something nasty\")"
get_value test
$(echo "something nasty")
unset test
get_value test
variable with name <test> is not set
As this approach eliminates the need for 'eval', it is safer.
This code checked under bash 5.0.3(1).
modern shells already support arrays( and even associative arrays). So please do use them, and use less of eval.
var1="this is the real value"
array=("$var1")
# or array[0]="$var1"
then when you want to call it , echo ${array[0]}
Based on the answer: https://unix.stackexchange.com/a/111627
###############################################################################
# Summary: Returns the value of a variable given it's name as a string.
# Required Positional Argument:
# variable_name - The name of the variable to return the value of
# Returns: The value if variable exists; otherwise, empty string ("").
###############################################################################
get_value_of()
{
variable_name=$1
variable_value=""
if set | grep -q "^$variable_name="; then
eval variable_value="\$$variable_name"
fi
echo "$variable_value"
}
test=123
get_value_of test
# 123
test="\$(echo \"something nasty\")"
get_value_of test
# $(echo "something nasty")
VALUE=$(eval "echo \$$SOME_VAR_NAME")
Example
SSH_KEY_FILE_PATH_FOO="/tmp/key"
SSH_KEY_FILE_PATH_VAR_NAME_PREFIX="SSH_KEY_FILE_PATH"
SSH_KEY_FILE_PATH_VAR_NAME_SUFFIX="FOO"
SSH_KEY_FILE_PATH=$(eval "echo \$${SSH_KEY_FILE_PATH_VAR_NAME_PREFIX}_${SSH_KEY_FILE_PATH_VAR_NAME_SUFFIX}")
echo "$SSH_KEY_FILE_PATH"
/tmp/key
How can I retrieve a bash variable value if I have the variable name as string? var1="this is the real value"
a="var1"
Do something to get value of var1 just using variable a.
Context:
I have some AMI's (Amazon Machine Image) and I want to fire up a few instances of each AMI. As soon as they finish booting, I want to setup each instance according to its AMI type. I don't want to bake lots of scripts or secret keys inside any AMI so I prepared a generalized startup script and I put it on S3 with a publicly accessible link. In rc.local I put small piece of code which fetches the startup script and executes it. This is all I have in the AMIs. Then each AMI accesses a common configuration script which is applicable to all AMIs and special setup scripts for each. These scripts are private and require a signed URL to access them.
So now, when I fire an instance of an AMI (my_private_ami_1), I pass a signed URL for one more file presented on S3 which contains signed URL for all private scripts in terms of key/value pair.config_url="http://s3.amazo.../config?signature"
my_private_ami_1="http://s3.amazo.../ami_1?signature"
...
When the startup script runs, it downloads the above file and source's it. Then it checks for its AMI type and picks the correct setup script for itself.
ami\_type=GET AMI TYPE #ex: sets ami\_type to my\_private\_ami\_1
setup\_url=GET THE SETUP FILE URL BASED ON AMI\_TYPE # this is where this problem arises
So now I can have a generic code which can fire instances irrespective of their AMI types and instances can take care of themselves.
You can use ${!a}:
var1="this is the real value"
a="var1"
echo "${!a}" # outputs 'this is the real value'
This is an example of indirect parameter expansion:
The basic form of parameter expansion is ${parameter}. The value of
parameter is substituted.
If the first character of parameter is an exclamation point (!), it
introduces a level of variable indirection. Bash uses the value of the
variable formed from the rest of parameter as the name of the
variable; this variable is then expanded and that value is used in the
rest of the substitution, rather than the value of parameter itself.
X=foo
Y=X
eval "Z=\$$Y"
sets Z to foo.
For my fellow zsh users, the way to accomplish the same thing as the accepted answer is to use:
echo ${(P)a} # outputs 'this is the real value'
It is appropriately called Parameter name replacement
This forces the value of the parameter name to be interpreted as a
further parameter name, whose value will be used where appropriate.
Note that flags set with one of the typeset family of commands (in
particular case transformations) are not applied to the value of name
used in this fashion.
If used with a nested parameter or command substitution, the result of
that will be taken as a parameter name in the same way. For example,
if you have ‘foo=bar’ and ‘bar=baz’, the strings ${(P)foo},
${(P)${foo}}, and ${(P)$(echo bar)} will be expanded to ‘baz’.
Likewise, if the reference is itself nested, the expression with the
flag is treated as if it were directly replaced by the parameter name.
It is an error if this nested substitution produces an array with more
than one word. For example, if ‘name=assoc’ where the parameter assoc
is an associative array, then ‘${${(P)name}[elt]}’ refers to the
element of the associative subscripted ‘elt’.
Modified my search keywords and Got it :). eval a=\$$a Thanks for your time.
In bash 4.3+, you can use declare -n:
#!/usr/bin/env bash
var="this is the real value"
var_name="var"
declare -n var_ref=$var_name
echo "${var_ref}"
Had the same issue with arrays, here is how to do it if you're manipulating arrays too :
array_name="ARRAY_NAME"
ARRAY_NAME=("Val0" "Val1" "Val2")
ARRAY=$array_name[#]
echo "ARRAY=${ARRAY}"
ARRAY=("${!ARRAY}")
echo "ARRAY=${ARRAY[#]}"
echo "ARRAY[0]=${ARRAY[0]}"
echo "ARRAY[1]=${ARRAY[1]}"
echo "ARRAY[2]=${ARRAY[2]}"
This will output :
ARRAY=ARRAY_NAME[#]
ARRAY=Val0 Val1 Val2
ARRAY[0]=Val0
ARRAY[1]=Val1
ARRAY[2]=Val2
In bash 4.3, the '-v' test for set variables was introduced. At the same time, 'nameref' declaration was added. These two features together with the indirection operator (!) enable a simplified version of the previous example:
get_value()
{
declare -n var_name=$1
if [[ -v var_name ]]
then
echo "${var_name}"
else
echo "variable with name <${!var_name}> is not set"
fi
}
test=123
get_value test
123
test="\$(echo \"something nasty\")"
get_value test
$(echo "something nasty")
unset test
get_value test
variable with name <test> is not set
As this approach eliminates the need for 'eval', it is safer.
This code checked under bash 5.0.3(1).
modern shells already support arrays( and even associative arrays). So please do use them, and use less of eval.
var1="this is the real value"
array=("$var1")
# or array[0]="$var1"
then when you want to call it , echo ${array[0]}
Based on the answer: https://unix.stackexchange.com/a/111627
###############################################################################
# Summary: Returns the value of a variable given it's name as a string.
# Required Positional Argument:
# variable_name - The name of the variable to return the value of
# Returns: The value if variable exists; otherwise, empty string ("").
###############################################################################
get_value_of()
{
variable_name=$1
variable_value=""
if set | grep -q "^$variable_name="; then
eval variable_value="\$$variable_name"
fi
echo "$variable_value"
}
test=123
get_value_of test
# 123
test="\$(echo \"something nasty\")"
get_value_of test
# $(echo "something nasty")
VALUE=$(eval "echo \$$SOME_VAR_NAME")
Example
SSH_KEY_FILE_PATH_FOO="/tmp/key"
SSH_KEY_FILE_PATH_VAR_NAME_PREFIX="SSH_KEY_FILE_PATH"
SSH_KEY_FILE_PATH_VAR_NAME_SUFFIX="FOO"
SSH_KEY_FILE_PATH=$(eval "echo \$${SSH_KEY_FILE_PATH_VAR_NAME_PREFIX}_${SSH_KEY_FILE_PATH_VAR_NAME_SUFFIX}")
echo "$SSH_KEY_FILE_PATH"
/tmp/key