When i am passing value of a variable declared in jenkins Groovy script its value is not retained in for loop which is running on a remote server. Strange thing is i am able to access the same value outside the for loop.
Here is the sample code i am trying to use
#!/usr/bin/env groovy
def config
def COMMANDS_TO_CHECK='curl grep hello awk tr mkdir bc'
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """
#!/bin/bash
/usr/bin/sshpass -p passwrd ssh user#host << EOF
hostname
echo $COMMANDS_TO_CHECK ---> This is printed
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo ${CURRENT_COMMAND} ---> Why This is not printed?
echo \${CURRENT_COMMAND} ----> Why This is not printed?
done
hostname
EOF
exit
"""
}
}
}
}
}
Output
[workspace#3] Running shell script
+ /usr/bin/sshpass -p passwrd ssh user#host
Pseudo-terminal will not be allocated because stdin is not a terminal.
illinsduck01
curl grep hello awk tr mkdir bc
illinsduck01
+ exit
You can wrap sh in """ ... """ as below
#!/usr/bin/env groovy
def config
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """#!/bin/sh
/usr/bin/sshpass -p password ssh username#hostname << EOF
COMMANDS_TO_CHECK="curl grep hello awk tr mkdir bc"
hostname
echo \$COMMANDS_TO_CHECK
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo \$CURRENT_COMMAND
which \$CURRENT_COMMAND
status=\$?
if [ \${status} -eq 0 ]
then
echo \${CURRENT_COMMAND} command is OK
else
echo "Failed to find the \${CURRENT_COMMAND} command"
fi
done
hostname
EOF
exit
"""
}
}
}
}
}
This the code I,m trying to execute:
#!/bin/bash
set -e -a
err_report() {
echo "Error on line $1"
}
trap 'err_report $LINENO' ERR
IFS=$'\n'
value=($(cat builds.txt))
prevBuild=${value[0]}
prevBuildDir=$prevBuild
currentBuild=${value[1]}
currentBuildDir=$currentBuild
splitPrevBuild=(${prevBuild//./$'\n'})
splitcurrentBuild=(${currentBuild//./$'\n'})
prevBuildSR=${splitPrevBuild[2]}
prevBuildHF=${splitPrevBuild[3]}
prevBuildNum=${splitPrevBuild[4]}
currentBuildSR=${splitcurrentBuild[2]}
currentBuildHF=${splitcurrentBuild[3]}
currentBuildNum=${splitcurrentBuild[4]}
function change {
cd ~/InstallationFiles/${currentBuildSR}/${currentBuildDir}/${currentBuild}extracted
}
change
pwd
ssh -T atao52#kvs-in-hsglap17.in.kronos.com <<-EOF
$(typeset -f)
change
pwd
echo hi
exit
EOF
How to access the variables as currently, it says
cd: /home/atao52/InstallationFiles///extracted: No such file or directory
I have this Exec declaration:
exec { 'Normalize MP3 filename':
environment => ["t=0"],
command => 'for i in *mp3; do mv -v $i track_`seq -f "%03g" $t $t`.mp3 ; t=`expr $t + 1`; done',
cwd => "$resource_path/res/raw",
} ->
When my manifest runs it get this error:
Error: Could not find command 'for'
Error: /Stage[main]/Make_it::App_mp3files/Exec[Normalize MP3 filename]/returns: change from notrun to 0 failed: Could not find command 'for'
The default exec provider on *nix OSes is posix, which does not support shell built-ins such as for. You should change the provider to shell to use shell built-ins.
exec { 'Normalize MP3 filename':
environment => ["t=0"],
command => 'for i in *mp3; do mv -v $i track_`seq -f "%03g" $t $t`.mp3 ; t=`expr $t + 1`; done',
cwd => "$resource_path/res/raw",
provider => 'shell',
} ->
I am trying to reference custom modules shortcuts (ie use ts paths mapping feature) for my typescript app, with the following config.
Project structure
dist/
src/
lyrics/
... ts files
app/
... ts files
Full project structure is here: github.com/adadgio/npm-lyrics-ts, dist folder is not commited of course)
tsconfig.json
{
"compilerOptions": {
"outDir": "dist",
"module": "commonjs",
"target": "es6",
"sourceMap": true,
"moduleResolution": "node",
"experimentalDecorators": true,
"emitDecoratorMetadata": true,
"removeComments": true,
"noImplicitAny": false,
"baseUrl": ".",
"paths": {
"*": ["src/lyrics/*"], // to much here !! but none work
"zutils/*": ["./src/lyrics/*", "src/lyrics/utils/*", "dist/lyrics/utils/*"]
},
"rootDir": "."
},
"exclude": [
"dist",
"node_modules"
],
"include": [
"src/**/*.ts"
]
}
When i run my npm start/compile or watch script, i get no Typescript errors. The following works (Atom is my IDE)
// string-utils.ts does exist, no IDE error, typescript DOES compile
`import { StringUtils } from 'zutils/string-utils';`
But i then get the NodeJS following error:
Error: Cannot find module 'zutils/string-utils'
at Function.Module._resolveFilename (module.js:470:15)
at Function.Module._load (module.js:418:25)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/home/adadgio/WebServer/projects/adadgio/npm-lyrics-ts/src/index.ts:7:1)
at Module._compile (module.js:571:32)
at Module.m._compile (/home/adadgio/WebServer/projects/adadgio/npm-lyrics-ts/node_modules/ts-node/src/index.ts:413:23)
at Module._extensions..js (module.js:580:10)
at Object.require.extensions.(anonymous function) [as .ts] (/home/adadgio/WebServer/projects/adadgio/npm-lyrics-ts/node_modules/ts-node/src/index.ts:416:12)
at Module.load (module.js:488:32)
It looks like the module is trying to be resolved from the node_modules folder. I've read docs about Typescript paths mapping but i cannot get it to work.
I was doing lot of research on this.
I am using atom, typescript and nodejs.
The thing is, when you compile typescript, it does search for paths (the path to a .ts file to include). However, the final compiled .js file does not get path substituted.
The solution:
Compile ts files, use paths in tsconfig
Use script to substitute path token in final .js files
Run node application
So essentially, part of tsconfig will look like this
"baseUrl": "./app",
"paths" : {
"#GameInstance" : ["model/game/GameInstance"],
"#Map" : ["model/game/map/Map"],
"#MapCreator" : ["model/game/map/creator/MapCreator"],
"#GameLoop" : ["model/game/GameLoop"],
"#Point" : ["model/other/math/geometry/Point"],
"#Rectangle" : ["model/other/math/geometry/Rectangle"],
"#Functions" : ["model/other/Functions"]
}
And consider Rectangle.ts file
import { Point } from '#Point';
import { Vector } from '#Vector';
/**
* Represents a rectangle.
* You can query it for collisions or whether rectangles are touching
*/
export class Rectangle {
//more code
Where Rectangle.ts is in
./src/app/model/other/math/geometry/Rectangle/Rectangle.ts
We run
tsc
which will compile all .ts files we set up. There, the paths will be substituted in runtime, if you get error, run
tsc --traceResolution > tmp && gedit tmp
and search for the fiel wehere is undefined path include. You will be able to see substitution log
Then we are left with Rectangle.js (builded)
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
var _Point_1 = require("#Point");
//more code
As you see, the #Point will not exist and we can not run node application like this.
For that, however, I created a script, that recursivelly searches js files and replaces the tokens by going up to root and then to target path.
There is requirePaths.data file where you define token and the path.
'#GameLoop' : "./app/model/game/GameLoop"
'#GameInstance' : "./app/model/game/GameInstance"
"#Map" : "./app/model/game/map/Map"
"#MapCreator" : "./app/model/game/map/creator/MapCreator"
"#Point" : "./app/model/other/math/geometry/Point"
"#Rectangle" : "./app/model/other/math/geometry/Point"
Now, this is NOT universal, it is just my hot script. Please, take a note, that structure is
src
|-app
| |-model
-build
|-src
|-app
|-model
|-test
Technically, the app/model... structure in src, is just copied to the src/build
The tsc takes source from /src/app and compiles it. Compiled result is in /src/build
Then, there is the substitutePathsInJS.sh script.
This scand for build path, and whenever it finds token #Rectangle, it replaces it (more explanation below...) Code:
#!/bin/bash
function testreqLevel()
{
local srcPath="$1"
local replacingIn="$2"
local expectedLevel=$3
getPathLevel "$replacingIn"
local res=$?
if [ ! $res -eq $expectedLevel ]; then
echo "[-] test $srcPath , $replacingIn FAILED. ($res != $expectedLevel)"
fi
}
function assertreqPath()
{
local input="$1"
local expected="$2"
if [ ! "$input" = "$expected" ]; then
echo "[-] test $expected FAILED"
echo "computed: $input"
echo "expected: $expected"
fi
}
function testGetPathLevel()
{
testreqLevel "./build/src" "./build/src/app/model/game/GameObject.js" 4
testreqLevel "./build/src" "./build/src/file.js" 1
testreqLevel "./build/src" "./build/src/app/model/game/GameObject.js" 4
}
function testGetPathToRoot()
{
local path=$(getPathToRoot "./build/src" "./build/src/app/model/game/GameObject.js")
assertreqPath "$path" "../../../../../"
path=$(getPathToRoot "./" "./server.js")
assertreqPath "$path" "./"
path=$(getPathToRoot "./" "./app/model/game/GameInstance.js")
assertreqPath "$path" "../../../"
}
function err()
{
echo "[-] $1"
}
function getPathLevel()
{
#get rid of starting ./
local input=$(echo "$1" | sed "s/^\.\///")
local contains=$(echo "$input" | grep '/')
if [ -z "$contains" ]; then
return 0
fi
#echo "$input"
local slashInput=$(echo "$input" | awk -F/ '{print NF-1}')
return $(($slashInput - 1))
}
#given ROOT, and PATH, returns a path P such as being in directory PATH, from there using P we get to root
#example:
#ROOT=./src
#PATH=./src/model/game/objects/a.js
#returns ../../
function getPathToRoot()
{
local root="$1"
local input="$2"
getPathLevel "$input"
local level=$?
if [ $level -eq 0 ]; then
echo "./"
return 0
fi
for ((i=1; i <= level + 1; i++)); do
echo -n '../'
done
#echo "$root" | sed 's/^\.\///'
}
function parseData()
{
echo "**************"
echo "**************"
local data="$1"
let lineNum=1
while read -r line; do
parseLine "$line" $lineNum
if [ $? -eq 1 ]; then
return 1
fi
let lineNum++
done <<< "$data"
echo 'Parsing ok'
echo "**************"
echo "**************"
return 0
}
function parseLine()
{
if [[ "$1" =~ ^\#.*$ ]] || [[ "$1" =~ ^\ *$ ]]; then
#comment line
return 0
fi
local line=$(echo "$1" | sed "s/\"/'/g")
let lineNum=$2
local QUOTE=\'
local WORD_IN_QUOTES=$QUOTE[^:$QUOTE]*$QUOTE
if [[ "$line" =~ ^\ *$WORD_IN_QUOTES\ *:\ *$WORD_IN_QUOTES\ *$ ]]; then
# valid key : value pair
local key=$(echo "$line" | awk -F: '{print $1}' | sed 's/^ *//g' \
| sed 's/ *$//g' | sed 's/\//\\\//g' | sed "s/'//g" | sed "s/\./\\\./g")
local val=$(echo "$line" | awk -F: '{print $2}' | sed 's/^ *//g' \
| sed 's/ *$//g' | sed "s/'//g")
echo "[+] Found substitution from '$key' : '$val'"
if [ -z "$REPLACEMENT_KEY_VAL" ]; then
REPLACEMENT_KEY_VAL="$key|$val"
else
REPLACEMENT_KEY_VAL="$REPLACEMENT_KEY_VAL;$key|$val"
fi
else
err "Parse error on line $lineNum"
echo "Expecting lines 'token' : 'value'"
return 1
fi
return 0
}
function replaceInFiles()
{
cd "$WHERE_SUBSTITUTE"
echo "substitution root $WHERE_SUBSTITUTE"
local fileList=`find . -type f -name "*.js" | grep -v "$EXCLUDE"`
echo "$fileList"| while read fname; do
export IFS=";"
echo "Replacing in file '$WHERE_SUBSTITUTE$fname'"
for line in $REPLACEMENT_KEY_VAL; do
local key=`echo "$line" | awk -F\| '{print $1}'`
local val=`echo "$line" | awk -F\| '{print $2}'`
local finalPath=$(getPathToRoot "./" "$fname")"$val"
if [ $VERBOSE -eq 1 ]; then
echo -e "\tsubstitute '$key' => '$val'"
#echo -e "\t$finalPath"
echo -e "\treplacing $key -> $finalPath"
fi
#escape final path for sed
#slashes, dots
finalPath=$(echo "$finalPath" | sed 's/\//\\\//g'| sed 's/\./\\\./g')
if [ $VERBOSE -eq 1 ]; then
echo -e '\t\tsed -i.bak '"s/require(\(.\)$key/require(\1$finalPath/g"\ "$fname"
fi
sed -i.bak "s/require(\(.\)$key\(.\))/require(\1$finalPath\2)/g" "$fname"
done
done
return 0
}
function quit()
{
echo "*************************************"
echo "*****SUBSTITUTING PATHS EXITING******"
echo "*************************************"
echo
exit $1
}
#######################################
CURRENTDIR=`dirname "$(realpath $0)"`
WHERE_SUBSTITUTE='./build/src'
REPLACEMENT_KEY_VAL="";
VERBOSE=0
FILE="$CURRENTDIR/requirePaths.data"
EXCLUDE='./app/view'
if [ "$1" = "-t" ]; then
testGetPathLevel
testGetPathToRoot
echo "[+] tests done"
exit 0
fi
if [ "$1" = "-v" ]; then
VERBOSE=1
fi
echo "*************************************"
echo "********SUBSTITUTING PATHS***********"
echo "*************************************"
if [ ! -f "$FILE" ]; then
err "File $FILE does not exist"
quit 1
fi
DATA=`cat "$FILE"`
parseData "$DATA"
if [ $? -eq 1 ]; then
quit 1
fi
replaceInFiles
quit $?
This seems confusing, but consider excample.
We have Rectangle.js file.
The script loads bunch of input tokens from requirePaths.data file, in this case, lets focus on line
"#Point" : "./app/model/other/math/geometry/Point"
Script runs from ./src, and is given root directory ./src/build/src
Script does cd ./src/build/src
Executes find. There, it will receive
./model/other/math/geometry/Rectangle/Rectangle.ts
The absolute path of that is
./src/build/src/app/model/other/math/geometry/Rectangle/Rectangle.ts
But we do not care about absolute path now.
Calculates path such as he gets from the directory up
Whis will result is something like
./../../../../
Where he will like that get from directory
/src/build/app/model/other/math/geometry/Rectangle
to directory
/src/build/app
Then, behind that string, we add the path provided from the data file
./../../../.././app/model/other/math/geometry/Point
So the final substitution for file Rectangle.js (in BUILD folder somewhere) is
before
require("#Point")
after
require("./../../../.././app/model/other/math/geometry/Point")
Which is terrible, but we do not care about what is in js anyway. Main thing is that it works.
Drawbacks
You can NOT combine it with code monitor. Monitoring tsc and then, when change in code is done, do automatic tsc compile, then automatically run the shell path substitution and then tun nodeJS upon final js files is possible, BUT for some reason, then the sh script substitutes paths, the monitoring software considers is as change in code (no idea why, it has build excluded from monitor) and compiles again. Therefore, you gen an infinite loop
You must compile it manually, step by step, or JUST use monitor on tsc compilation. When you write some code, go and run substitution and test the nodeJS functionality.
When adding new Class Food, you must define a token for it (#Food) and path to file in 2 places (tsconfig) and the input for shell script
You make entire compilation process longer. To be honest, tsc takes most of time anyway, and the bash script is not so time consuming surprisingly....
When implementing tests with mocha, you must again do step by step compilation and when finished, run mocha above final js files. But for that you can write scripts....
Some people usually substitute only like #app or some directories. The problem with that is, whenever you move source file around, you have to do lot of changes...
The good sides
When moving file around, you change one string (in two places....)
No more relative paths that make big project impossible to maintain
It is interesting, but it really works, and I did not encounter major problems (if used properly)
I have perl script
perl script (install.pl):
use strict; use warnings;
use Term::ANSIColor;
my $logfilename = "install.log";
open LOG,">$logfilename" or die "failed to open log file reason:$!";
sub logMsg
{
my ($msg) = #_;
my ($error) = $_[1];
$msg .= "\n";
print LOG $msg;
if( $error ) { print color 'red' ;}
{print $msg}
if( $error ) { print color 'reset' ;}
}
sub lampInstall
{
logMsg("Installing apache2, php, and mysql db");
# Install apache2 and mysql:
my $cmdOutput = `./ubuntu-lamp-install.sh`;
print LOG "$cmdOutput\n";
}
lampInstall();
close LOG;
bash script (ubuntu-lamp-install.sh)
#!/bin/bash
sudo apt-get update
sudo apt-get install ntp
sudo /etc/init.d/ntp start
export DEBIAN_FRONTEND=noninteractive
echo "mysql-server mysql-server/root_password password test" | sudo debconf-set-selections
echo "mysql-server mysql-server/root_password_again password test" | sudo debconf-set-selections
sudo apt-get install -y apache2 php5 libapache2-mod-php5
sudo service apache2 restart
sudo a2enmod ssl
The issues is that , when install.pl is invoked, it waits for long time and does not give any output information of what is being installed.I need it to change the perl script to display the output information from bash script(better if instantaneous) as well as log the output to log file. The script is written by someone else , I have very little knowledge of perl scripting.
You could do something like
open my $cmdOutut, "-|", "./ubuntu-lamp-install.sh";
while (<$cmdOutput>) {
print LOG "$_\n";
print "$_\n";
}
in your lampInstall().
This is the new lampInstall():
sub lampInstall
{
logMsg("Installing apache2, php, and mysql db");
# Install apache2 and mysql:
open my $cmdOutut, "-|", "./ubuntu-lamp-install.sh";
while (<$cmdOutput>) {
print LOG "$_\n";
print "$_\n";
}
}