Unable to fetch data using powershell as external data source - terraform

# main.tf
data "external" "extDateTime" {
program = ["pwsh", "${path.module}/getDateTime.ps1"]
}
output "value" {
value = "${data.external.extDateTime.result.dateTime}"
}
This is the Powershell file getDateTime.ps1 code section
# getDateTime.ps1
$DateTime = Get-Date -Format "yyyyMMddHHmmss"
Write-Output "{""dateTime"": $DateTime}"
Then I run the following command: terraform plan
Error: Unexpected External Program Results with data.external.extDateTime, on main.tf line 26, in data "external" "extDateTime":
26: program = ["Powershell.exe", "${path.module}/getDateTime.ps1"]
The data source received unexpected results after executing the program. Program output must be a JSON encoded map of string keys and string values.
Program: C:\WINDOWS\System32\WindowsPowerShell\v1.0\Powershell.exe
Result Error: invalid character '{' after top-level value
My understanding is the PS script has to return in JSON format but I keep getting the Result Error. Any ideas would be appreciated.

Ensure that the JSON values are correctly formatted as strings (using back ticks to escape double quotes):
$DateTime = Get-Date -Format "yyyyMMddHHmmss"
Write-Output "{`"dateTime`": `"$DateTime`"}"
As per the Terraform docs for the external data source:
The JSON object contains the contents of the query argument and its values will always be strings.
Another way that this can be done is to create a PowerShell custom object and convert it to JSON:
$DateTime = Get-Date -Format "yyyyMMddHHmmss"
$myObject = [PSCustomObject]#{
dateTime = $DateTime
}
ConvertTo-Json $myObject
If you need the output to be a number, you can use the tonumber() function:
output "value" {
value = tonumber(data.external.extDateTime.result.dateTime)
}

I figured out what the issue was. I was running terraform commands in PS 7 (pwsh.exe) but my data.external program was calling PS 5 (powershell.exe). Once I updated to the following: program = ["pwsh.exe", "${path.module}/getDateTime.ps1"] it worked fine. The error message referring to the { was misleading me. Thanks again Cody.

Related

Cloudwatch alarm creation fails due to heredoc

I am trying to create a composite cloudwatch alarm using terraform. But unfortunately my terraform code breaks with the following error:
Error: error creating CloudWatch Composite Alarm
(node-count-office-time-composite-alarm-DP-1474-desert):
ValidationError: AlarmRule must not contain leading or trailing
whitespace or be null
status code: 400, request id: 272b14ae-e6bd-4e65-8bb8-25372d9a5f7c
Following is my terraform code:
resource "aws_cloudwatch_composite_alarm" "node_count_office_time_alarm" {
depends_on = [aws_cloudwatch_metric_alarm.node_count, aws_cloudwatch_metric_alarm.office_time]
alarm_description = "Composite alarm for node count & office time"
alarm_name = "node-count-office-time-composite-alarm-${local.postfix}"
alarm_actions = [var.sns_topic_arn]
ok_actions = [var.sns_topic_arn]
alarm_rule =<<-EOF
ALARM(${aws_cloudwatch_metric_alarm.node_count.alarm_name}) AND
ALARM(${aws_cloudwatch_metric_alarm.office_time.alarm_name})
EOF
}
I checked many times and there are no leading or trailing spaces in my alarm_rule. Only new line after AND operator. I am using terraform 0.15.3 version. Anyone faces similar issues and how can I resolve this issue? thanks
I did not find the solution to how to make the heredoc working. But I fixed it for the time being using direct string expression instead of heredoc block. Following is the string expression:
alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.node_count.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.office_time.alarm_name})"
I hope it is useful for others if they face the same issue. thanks
Terraform instructions https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_composite_alarm are not accurate as of this writing in 2021.
alarm_rule accepts a single string argument, EOF/heredoc has to be processed to create a literal string:
locals {
alarm_rule_with_newlines = <<-EOF
ALARM(${aws_cloudwatch_metric_alarm.alpha.alarm_name}) OR
ALARM(${aws_cloudwatch_metric_alarm.bravo.alarm_name})
EOF
}
[...]
alarm_rule = trimspace(replace(local.alarm_rule_with_newlines, "/\n+/", " "))
I was not satisfied with neither of proposed answers so I have another solution.
Move your composite alert rules to separate file and just read it:
alarm_rule = file("./composite-alert-rule")
or
alarm_rule = templatefile("./composite-alert-rule", { arg = ... })
if you need to pass some dynamic args.
Check terraform docs for reference:
https://www.terraform.io/language/functions/templatefile
https://www.terraform.io/language/functions/file

Terraform syntax for putting json as value in a map

I'm new to terraform. I have a json object that I need to set as the value in a terraform map so that the resource gets created with the json as the value.
The .tf file looks like this in that section:
...
config_overrides = {
override_1 = "True"
override_2 = '{"key1":"val1","key2":"val2"}' #this is the json object
}
...
However, the terraform lint command terraform lint -check is failing on the json object.
$ terraform fmt -check
Error: Invalid character
on myterraform.tf line 28, in resource <<resource name>> :
28: override_2 = '{"key1":"val1","key2":"val2"}'
Single quotes are not valid. Use double quotes (") to enclose strings.
Error: Invalid expression
on myterraform.tf line 28, in resource <<resource name>>:
28: override_2 = '{"key1":"val1","key2":"val2"}'
Expected the start of an expression, but found an invalid expression token.
I have tried many different variations and cant get the linter to accept it. Please advise.
You can use Terraform's jsonencode function so that Terraform itself is responsible for generating the JSON and you only need to worry about the data structure:
override_2 = jsonencode({
"key1": "val1",
"key2": "val2",
})
Terraform's object expression syntax happens to be similar to JSON's and so the argument to jsonencode here looks a lot like the JSON string it'll convert to, but that is really just a normal Terraform expression and so you can include any Terraform expression constructs in there. For example:
override_2 = jsonencode({
"key1": "val1",
"key2": var.any_variable,
})
You will need to use the \ in the value ' isn't going to work.
config_overrides = {
override_1 = "True"
override_2 = "{\"key1\":\"val1\",\"key2\":\"val2\"}"
}

Terraform windows vs linux issue

So this issue is a bit convoluted but I need this for a very specific case in azure. I'm trying to create an APIM subnet inside an azure k8s vnet, but I haven't been able to find a return value from the k8s terraform call that gives me the ID/name for the vnet. Instead I used a powershell command to query Azure and get the name of the new vnet. I was working on this code locally on my windows box and it works fine:
data "external" "cluster_vnet_name" {
program = [var.runPSVer6 == true ? "pwsh" : "powershell","Select-AzSubscription '${var.subscriptionName}' |out-null; Get-AzVirtualNetwork -ResourceGroupName '${module.kubernetes-service.k8ResourceGroup}' | Select Name | convertto-json}"]
depends_on = [module.kubernetes-service]
}
I have a toogle in my variables for runpsver6 so when I run on a linux machine it will change powershell to pwsh. Now, this is were is starts getting a little weird. When I run this on my windows machine, its not an issue, however when I run this from a linux box I get the following error:
can't find external program "pwsh"
I have tried a number of different work arounds (such as using the full powershell snapin path /snap/bin/powershell and putting the commands in a .ps1 file) to no avail. Every single time it throws the error that it can't find pwsh as an external program.
I use this same runPSVer6 toggle for local-exec terraform commands with no issue, but I need the name of the Vnet as a response.
Anyone have any ideas what I'm missing here?
ADDED AFTER SEPT 30th
So I tried the alternative way of firing commands:
variable "runPSVer6" {
type = bool
default = true
}
variable "subscriptionName" {
type = string
}
variable "ResourceGroup" {
type = string
}
data "external" "runpwsh" {
program = [var.runPSVer6 == true ? "pwsh" : "powershell", "test.ps1"]
query = {
subscriptionName = var.subscriptionName
ResourceGroup = var.ResourceGroup
}
}
output "vnet" {
value = data.external.runpwsh.result.name
}
and this appears to allow the command to execute, however its not pulling back the result of the json response (even when I confirmed that I do get a response).
This is what I'm using for my .ps1:
Param($subscriptionName,$ResourceGroup)
$subscription = Select-AzSubscription $subscriptionName
$name = (Get-AzVirtualNetwork -ResourceGroupName $ResourceGroup | Select Name).Name
Write-Output "{`n`t""name"":""$name""`n}"
When i don't use the .name in the out, this is what I get:
data.external.runpwsh: Refreshing state...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
vnet = { "name" = "" }
And this is the output from the .ps1:
{
"name":"vnettest"
}
Can you check if pwsh is working in the terminal. It should bring up the powershell prompt...
The path of pwsh must be added to the PATH.. /usr/bin is in my PATH as you can see below.
ubuntu#myhost:~$ whereis pwsh
pwsh: /usr/bin/pwsh
ubuntu#myhost:~$
ubuntu#myhost:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ubuntu#myhost:~$ pwsh
PowerShell 7.0.3
Copyright (c) Microsoft Corporation. All rights reserved.
https://aka.ms/powershell
Type 'help' to get help.
PS /home/ubuntu>
PS /home/ubuntu> exit
ubuntu#myhost:~$
============
Added later after 29 Sep 2020.
I tried again in Ubuntu 20.
Terraform 13 was downloaded as Zip from main site
PowerShell 7.0.3 installed with snap install powershell --classic
I tried the below test code which worked.
varriable runPSVer6 {}
default = true
}
data "external" "testps" {
program = [var.runPSVer6 == true ? "pwsh" : "powershell","/tmp/testScript.ps1"]
}
output "ps_out" {
value = data.external.testps.result
}
The output was like...
Outputs:
ps_out = {
"name" = "My Resource Name"
"region" = "West Europe"
}
/tmp/testScript.ps1 code was simple output statement...
Write-Output '{ "name" : "My Resource Name", "region" : "West Europe" }'
I tried to null out the path variable just to see if i get the error message you mentioned. I did, as expected..
ubuntu#ip-172-31-53-128:~$ ./terraform apply
data.external.test_ps: Refreshing state...
Error: can't find external program "pwsh"
on main.tf line 5, in data "external" "test_ps":
5: data "external" "test_ps" {
but when i used the full path, it worked again. (even /snap/bin/powershell works)
program = [var.runPSVer6 == true ? "/snap/bin/pwsh" : "powershell","/tmp/testScript.ps1"]
I ealier wrognly blamed snap with my issue, but snap did work now.
This does not give any clue here or pin-point the issue you are having. But may be you try a couple of things just to be sure,
1.) issue "pwsh" in the current directory and see that Powershell prompt does come up.. not sure if you already checked this, but sometime some other characters in path could cause an issue.
2.) can you run tf once after exporting PATH=/snap/bin ... (do it inside a shell and exit later so that you back to old path. or. export the correct path later after test)
3.) If you used full path, the error message must have been different other "Error: can't find external program "pwsh" ... can you cross check if there was diff error msg
this is how the pwsh bin and the sym link looks like in my machine...
ubuntu#ip-172-31-53-128:~$ /usr/bin/ls -lt /snap/bin/pwsh
lrwxrwxrwx 1 root root 10 Sep 29 15:40 /snap/bin/pwsh -> powershell
ubuntu#ip-172-31-53-128:~$
ubuntu#ip-172-31-53-128:~$ /usr/bin/ls -lt /snap/bin/powershell
lrwxrwxrwx 1 root root 13 Sep 29 15:40 /snap/bin/powershell -> /usr/bin/snap
ubuntu#ip-172-31-53-128:~$

Terraform Module - Output error when count = 0

I'm relatively new to Terraform - I have a module setup as below, the issue I'm having is with the outputs if the module count is '0' when running a terraform plan. Output PW works fine now that I've used the element(concat workaround but the Output I'm having issues with is DCPWUn, I get the following error:
Error: Error refreshing state: 1 error(s) occurred:
* module.PrimaryDC.output.DCPWUn: At column 21, line 1: rsadecrypt: argument 1 should be type string, got type list in:
${element(concat("${rsadecrypt(aws_spot_instance_request.PrimaryDC.*.password_data,file("${var.PATH_TO_PRIVATE_KEY}"))}", list("")), 0)}
Code:
resource "aws_spot_instance_request" "PrimaryDC" {
wait_for_fulfillment = true
provisioner "local-exec" {
command = "aws ec2 create-tags --resources ${self.spot_instance_id} --tags Key=Name,Value=${var.ServerName}0${count.index +01}"
}
ami = "ami-629a7405"
spot_price = "0.01"
instance_type = "t2.micro"
count = "${var.count}"
key_name = "${var.KeyPair}"
subnet_id = "${var.Subnet}"
vpc_security_group_ids = ["${var.SecurityGroup}"]
get_password_data = "true"
user_data = <<EOF
<powershell>
Rename-computer -NewName "${var.ServerName}0${count.index +01}"
</powershell>
EOF
tags {
Name = "${var.ServerName}0${count.index +01}"
}
}
output "PW" {
value = "${element(concat("${aws_spot_instance_request.PrimaryDC.*.password_data}", list("")), 0)}"
}
output "DCPWUn" {
value = "${element(concat("${rsadecrypt(aws_spot_instance_request.PrimaryDC.*.password_data,file("${var.PATH_TO_PRIVATE_KEY}"))}", list("")), 0)}"
}
As the error says, rsadecrypt has an argument that is of type list, not string as it should be. If you want to ensure that the argument is a string, you need to invert your function call nesting to make sure that rsadecrypt gets a string:
output "DCPWUn" {
value = "${rsadecrypt(element(concat(aws_spot_instance_request.PrimaryDC.*.password_data, list("")), 0),file("${var.PATH_TO_PRIVATE_KEY}"))}"
}
The problem lies within this line
${element(concat("${rsadecrypt(aws_spot_instance_request.PrimaryDC.*.password_data,file("${var.PATH_TO_PRIVATE_KEY}"))}", list("")), 0)}
What are you trying to achieve? Let's break it down a little
element(…, 0): Get the first element of the following list.
concat(…,list("")): Concatenate the following list of strings and then append the concatenation of a list containing the empty string (Note that the second part is not useful, since you are appending an empty string).
rsadecrypt(…,file("${var.PATH_TO_PRIVATE_KEY}")): decrypt the following expression with the private key (Error: The following thing needs to be a string, you will be supplying a list)
aws_spot_instance_request.PrimaryDC.*.password_data This is a list of all password data (and not a string).
I don't know what your desired output should look like, but with the above list, you may be able to mix-and-match the functions to suit your needs.
edit: Fixed a mistake thanks to the comment by rahuljain1311.

Terraform External Provider resource does not have attribute for variable during plan phase

When running terraform plan with the below scripts I Gert the following error message:
Error: Error running plan: 1 error(s) occurred:
* output.foobaz: Resource 'data.external.example' does not have attribute 'result.foobaz' for variable 'data.external.example.result.foobaz'
It doesn't appear from testing that the external script is actually executed during the plan phase, however, it does appear that the plan phase is trying to interpolate the expected response, which seem s incorrect to me. Is there something I'm missing?
provider "scaleway" {
region = "ams1"
}
resource "scaleway_ip" "swarm_manager_ip" {
count = 1
}
data "external" "example" {
program = ["./scripts/test.sh"]
query = {
# arbitrary map from strings to strings, passed
# to the external program as the data query.
foo = "${scaleway_ip.swarm_manager_ip.0.ip}"
baz = "i-am-baz"
}
}
output "foobaz" {
value = "${data.external.example.result.foobaz}"
}
output "scaleway_ip_address" {
value = "${scaleway_ip.swarm_manager_ip.0.ip}"
}
Here is the external script:
#!/bin/bash
# Exit if any of the intermediate steps fail
set -e
# Extract "foo" and "baz" arguments from the input into
# FOO and BAZ shell variables.
# jq will ensure that the values are properly quoted
# and escaped for consumption by the shell.
eval "$(jq -r '#sh "FOO=\(.foo) BAZ=\(.baz)"')"
# Placeholder for whatever data-fetching logic your script implements
FOOBAZ="$FOO BAZ"
# Safely produce a JSON object containing the result value.
# jq will ensure that the value is properly quoted
# and escaped to produce a valid JSON string.
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
Your Terraform syntax is incorrect. data.external.example.result is a map. To access its entry foobaz you need to code
"${data.external.example.result["foobaz"]}"
See https://www.terraform.io/docs/configuration/interpolation.html

Resources