I'm going to parsing a token value from other .tf file into other .tf file
I have tried to understand this link and also from this article
data.tf
data "external" "get_token" {
program = ["/bin/sh", "${path.module}/get-token.sh"]
}
get-token.sh
#!/bin/bash
token=$(kubectl -n kube-system exec [POD_NAME] cat /var/lib/kube-proxy/kubeconfig 2>/dev/null | grep token | awk '{print $2}'
proxy.tf
...
metadata_startup_script = <<-EOT
- name: kube-proxy
user:
token: ${lookup(data.external.get_token.result, "token")}
certificate-authority-data: ${google_container_cluster.new_container_cluster.master_auth.0.cluster_ca_certificate}
...
EOT
My expectation is
token has the value as same as with certificate-authority-data.
certificate-authority-data has a exact value like i expect but the token is nil or blank.
I have run my get-token.sh manually and it's good. But when terraform want to parse it, the value is not parsed successfully. I have added ' before and after the variable ${lookup(data.external.get_token.result, "token")}. Seems not to work.
https://www.terraform.io/docs/providers/external/data_source.html
The program must then produce a valid JSON object on stdout, which
will be used to populate the result attribute exported to the rest of
the Terraform configuration. This JSON object must again have all of
its values as strings. On successful completion it must exit with
status zero.
So the script should return a json object.
#!/bin/bash
...
# add below line for make a json result
jq -n --arg token "$token" '{"token":$token}'
or if there is no jq,
#!/bin/bash
...
#add below
echo -n "{\"token\":\"${token}\"}"
Related
Trying to add sql dbs created from azure portal to azure failover group.
Terraform block will call bash script :
data "external" "database_names" {
program = ["sh", "${path.module}/scripts/fetch_db_id.sh"]
query = {
db_rg = azurerm_resource_group.mssql.name
server_name = azurerm_mssql_server.mssqlserver.name
}
}
fetch_db_id.sh bash script:
#!/usr/bin/env bash
# This script will get the database names at runtime.
eval "$(jq -r '#sh "export DB_RG=\(.db_rg) SERVER_NAME=\(.server_name)"')"
if [[ -z $DB_RG || -z $SERVER_NAME ]]; then
echo "Required variables DB_RG & SERVER_NAME not set" 1>&2
exit 1
fi
db_id=$(az sql db list --resource-group $DB_RG --server $SERVER_NAME --query [*].id | grep -v master 2>/dev/null)
jq -n --arg db_id "$db_id" '{"db_id":$db_id}'
unset DB_RG SERVER_NAME NODE_RG db_id
exit 0
Bash script output : Ran it locally on a linux VM without terraform :
"db_id": "[\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db1",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db2",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db3"\n]"
}
Terraform resource block to add database inside failover group:
resource "azurerm_sql_failover_group" "mssql_failover" {
count = (var.enable_read_replica && var.environment == "prd") ? 0 : 1
name = var.mssql_failover_group
resource_group_name = azurerm_resource_group.mssql.name
server_name = azurerm_mssql_server.mssqlserver.name
databases = toset(jsondecode(data.external.database_names.result["db_id"]))
partner_servers {
id = azurerm_mssql_server.replica[0].id
}
read_write_endpoint_failover_policy {
mode = "Automatic"
grace_minutes = 60
}
depends_on = [
azurerm_mssql_server.replica
]
}
terraform error code: when executed via terraform jenkins pipeline
[1mdata.external.database_names.result["db_id"][0m is "[\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db1",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db2",\n "/subscriptions/my_subscription_id/resourceGroups/sql_rg/providers/Microsoft.Sql/servers/sql_server_name/databases/databaseprd-db3",\n]"
Call to function "jsondecode" failed: invalid character ']' looking for
beginning of value
Note: it is introducing an extra "," when we run it with terraform jenkins pipeline, which could lead to json error.
I don't think there's enough information here to be certain, but I can give a partial, speculative answer.
az sql db list --resource-group $DB_RG --server $SERVER_NAME --query [*].id | grep -v master 2>/dev/null
This looks suspicious: the az command outputs JSON, but you're filtering it with grep. What does the output from az look like here? Do you expect the result to be valid JSON?
You say that our output has a comma that you don't expect. This is what you'd expect to see if the az command spat out something like:
[
"/blah/blah/databaseprd-db1",
"/blah/blah/databaseprd-db2",
"/blah/blah/databaseprd-db3",
"/blah/blah/master"
]
The grep -v master would remove the line containing the term "master", leaving you with invalid JSON:
[
"/blah/blah/databaseprd-db1",
"/blah/blah/databaseprd-db2",
"/blah/blah/databaseprd-db3",
]
If you want to use jq, you could replace the grep with something like
jq 'map(select(index("master")|not))'
ip="192.168.1.1"
if [ -n "$(ip = 192.168.1.1)" ];
then
IPADDR=$(ip addr show |grep 'inet '|grep -v 127.0.0.1 |awk '{print $2}'| cut -d/ -f1)
else
"${ip}"="${IPADDR}"
fi
echo "${IPADDR}"
Im trying to assigning ip="192.168.1.1" to variable IPADDR
the error im geting atm is
Object "=" is unknown, try "ip help".
./test: line 7: 192.168.1.1=: command not found
Here ...
if [ -n "$(ip = 192.168.1.1)" ];
... you are executing the command ip, passing arguments = and 192.168.1.1, and capturing its standard output. These are not valid arguments for that command, so ip emits an error message to its standard error, but nothing to its standard output.
The test [ -n "$(ip = 192.168.1.1)" ] evaluates whether the captured output is non-empty, so this test fails, and consequently, the else block is executed. Here ...
"${ip}"="${IPADDR}"
... is not a variable assignment but rather a command, because the text to the left of the = is not (just) an identifier. The IPADDR variable is unset or null, so that expands to 192.168.1.1=. The shell reports that there is no command of that name.
It's hard to be sure what you were really trying to accomplish, but going with the question title, if you want to assign a value to a variable then the syntax is
variable_name=value
The variable_name must be an identifier. The value is subject to expansion and quote removal, but not word splitting, so either of these would work for what you appear to be trying to do
ip="${IPADDR}"
or
ip=${IPADDR}
. With that said, the code does not make sense, because it does not assign a value to variable IPADDR before attempting that assignment.
Earlier, it is possible that you meant
if [ "$ip" != 192.168.1.1 ]
... which tests whether the value of variable $ip, considered as a string, is equal to the string 192.168.1.1. Even then, however, the code does not make sense overall.
Below is my bash script.
#!/bin/bash
message=$1
hostname=$2
severity=$3
eventname=$4
tagpath=$5
appname=$6
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
echo "Total number of args : $#"
echo "message = $message"
echo "hostname = $hostname"
echo "appname = $appname"
echo "data = $data"
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d $data
and when i try to run with sh tcp.sh value value value value value value
host:
'host,component:host,severity:host,evclasskey:nxlog,evclass:/nxlog/perf,monitor:localhost}],type:rpc,tid:1}'
is not a legal name (unexpected end of input) Total number of args : 6
message = message hostname = test appname = host data = curl: option
-d: requires parameter
I see that data has no value included.
This json has to be sent in this order for it to be accepted in the endpoint. Help me correct this.
Using jq to safely generate the desired JSON:
#!/bin/bash
parameters=(
--arg message "$1"
--arg hostname "$2"
--arg severity "$3"
--arg eventname "$4"
--arg tagpath "$5"
--arg appname "$6"
)
data=$(jq -n "${parameters[#]}" '
{action: "EventsRouter",
method: "add_event",
data: [ {summary: $message,
device: $hostname,
message: "\($message) \($eventname\) \($tagpath)",
component: $appname,
severity: $severity,
evclasskey: "nxlog",
evclass: "/nxlog/perf",
monitor: "localhost"
}
],
type: "rpc",
tid: 1
}'
curl -u uname:password -k https://myurlcom/zport/dmd/evconsole_router -d "$data"
Assuming:
message="my_message"
hostname="my_host"
severity="my_severity"
eventname="my_event"
tagpath="my_path"
appname="my_app"
If you run:
data="{"action":"EventsRouter","method":"add_event","data":[{"summary":"$message"},"device":"$hostname","message": "$message $eventname $tagpath","component":"$appname","severity":"$severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}"
you will get an error, because there is a not escaped white space before the string "my_event"
my_event: command not found
What happened? Since your json input has a lot of words between double quotes, you will have to enclose the whole string into single quotes, in order to preserve the double quotes inside of the string. But between single quotes, the bash variables will not be replaced by their value. So you will need to close the single quotes before each variable and reopen these again immediately after.
So that line of your script must become:
data='{"action":"EventsRouter","method":"add_event","data":[{"summary":"'$message'"},"device":"'$hostname'","message": "'$message $eventname $tagpath'","component":"'$appname'","severity":"'$severity'","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}'
If you execute:
echo "$data"
you will get:
{"action":"EventsRouter","method":"add_event","data":[{"summary":"my_message"},"device":"my_host","message": "my_message my_event my_path","component":"my_app","severity":"my_severity","evclasskey":"nxlog","evclass":"/nxlog/perf","monitor":"localhost"}],"type":"rpc","tid":1}
which is correct, I assume: the double quotes didn't disappear from your json data structure.
I am working on a bash script and I got a list of IP's that I wanted to add one by one in a CURL command.
For example given list on a file named list.txt
8.8.8.8
10.10.10.10
136.34.24.22
192.168.10.32
I wanted to add each value on curl command
curl -k -u $user:$password "https://logservice/jobs" --data-urlencode 'search=search index=test $ARRAYVALUE | head 1' > output.txt
Where $ARRAYVALUE is the IP address to be used on the command.
I will appreciate any hint.
Thanks
If I understood correctly, you want to:
map each line of a "list.txt" to an item of an array
loop over the newly created array inserting items one by one into your command invocation
Consider this, heavily commented, snippet. Look especially at mapfile and how variable is used in curl invocation, surrounded by double quotes.
#!/bin/bash
# declare a (non-associative) array
# each item is indexed numerically, starting from 0
declare -a ips
#put proper values here
user="userName"
password="password"
# put file into array, one line per array item
mapfile -t ips < list.txt
# counter used to access items with given index in an array
ii=0
# ${#ips[#]} returns array length
# -lt makes "less than" check
# while loops as long as condition is true
while [ ${ii} -lt ${#ips[#]} ] ; do
# ${ips[$ii]} accesses array item with the given (${ii}) index
# be sure to use __double__ quotes around variable, otherwise it will not be expanded (value will not be inserted) but treated as a string
curl -k -u $user:$password "https://logservice/jobs" --data-urlencode "search=search index=test ${ips[$ii]} | head -1" > output.txt
# increase counter to avoid infinite loop
# and access the next item in an array
((ii++))
done
You may read about mapfile in GNU Bash reference: Built-ins.
You may read about creating and accessing arrays in GNU Bash reference: Arrays
Check this great post about quotes in bash.
I hope you found this answer helpful.
I believe you need something like this :
#!/bin/bash
function FN()
{
filename=$1
declare -a IPs_ARRAY
i=0
user=$2
password=$3
while read ip
do
IPs_ARRAY[$i]=$ip
echo ${IPs_ARRAY[$i]}
# Uncomment for your actions ::
#curl -k -u $user:$password "https://logservice/jobs" --data-urlencode 'search=search index=test ${IPs_ARRAY[$i]} | head 1' > output.txt
(( i++ ))
done < $filename
}
#############
### MAIN ###
###########
read -p "Enter username: " username
read -p "Enter password: " password
# Call your function
filename="list.txt"
FN $filename $username $password
I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).