Using bash I can get the file extension by simply:
FILE_BASENAME="abcd.001.xyz.txt"
FILE_EXTENSION="${FILE_BASENAME##*.}"
echo $FILE_EXTENSION
I can also get the same result using a bit different syntax:
FILE_BASENAME="abcd.001.xyz.txt"
RESULT=$(echo "${FILE_BASENAME##*.}")
echo $FILE_EXTENSION
Regardless what way is used the both approaches produce a string txt.
Unfortunately the same syntax in Jenkins pipeline results to an empty string:
FILE_EXTENSION = sh(script: '$(echo "${FILE_BASENAME##*.}")', returnStdout: true).trim()
I have also tried a variation of this command with
FILE_EXTENSION = sh(script: 'echo $(echo "${FILE_BASENAME##*.}")', returnStdout: true).trim()
which doesn't work as well.
How to get the file extension in Jenkins?
Designed to support the three characters file extension:
FILE_BASENAME="abcd.001.xyz.txt"
FILE_NAME = FILE_BASENAME.replaceAll(".[a-zA-Z]{3}\$", "")
FILE_EXT = FILE_BASENAME.replaceAll(FILE_NAME, "")
echo ("FILE_NAME: ${FILE_NAME} FILE_EXT: ${FILE_EXT}")
Output:
FILE_NAME: abcd.001.xyz FILE_EXT: .txt
Related
I will post my script here
#!/bin/tcsh
echo 'Running'
set fileN = '2021-02-07-0448-04S.JKH_RR.SAC'
set fileE = '2021-02-07-0448-04S.JKH_RR_BHE.SAC'
set compR=BHR
set compT=BHT
set compR_name=BHR.SAC
set compT_name=BHT.SAC
set fileN_rot = `echo $fileN | awk '{split($0,a,".SAC"); print a[1]}'`
set fileE_rot = `echo $fileE | awk '{split($0,a,".SAC"); print a[1]}'`
echo 'output1'
echo $fileN
echo $fileE
echo 'output2'
echo $fileN_rot
echo $fileE_rot
echo 'output3'
echo $fileE_rot-$compR_name
echo $fileN_rot-$compT_name
The output is:
Running
output1
2021-02-07-0448-04S.JKH_RR_BHN.SAC 2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR_BHN
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHN
-BHR.SAC
2021-02-07-0448-04S.JKH_RR_BHE-BHT.SAC
echo $fileE_rot-$compR_name giving wrong output.
Here the out is copy-pasted from the output file,so -BHR.SAC showing in new line.
But in shell terminal it is showing -BHR.SAC07-0448-04S.JKH_RR_BHN.
I find it strange.
Looks like you have some control chars in your strings. Run cat -Ev script to see them and if you see ^Ms in the output then read Why does my tool output overwrite itself and how do I fix it? for how to deal with them.
Don't write scripts in [t]csh, though, as it wasn't designed for that. Writing a script in csh is like digging a hole with a toothbrush - sure you CAN kinda get there in the end but there are better alternatives. See https://www.google.com/search?q=google+csh+why+not.
Having said that, it's not obvious why you're trying to manipulate text in any shell. Shells exist to manipulate (create/destroy) files and processes and sequence calls to tools. The people who invented shell also invented tools such as awk for shell to call when appropriate to manipulate text. So, here is how to really write a shell script to do what you want (the shell part is to call awk to manipulate the text):
$ cat tst.sh
#!/usr/bin/env bash
awk '
BEGIN {
print "Running"
fileN = "2021-02-07-0448-04S.JKH_RR.SAC"
fileE = "2021-02-07-0448-04S.JKH_RR_BHE.SAC"
compR = "BHR"
compT = "BHT"
compR_name = "BHR.SAC"
compT_name = "BHT.SAC"
fileN_rot = fileN
sub(/\.SAC$/,"",fileN_rot)
fileE_rot = fileE
sub(/\.SAC$/,"",fileE_rot)
print "output1"
print fileN
print fileE
print "output2"
print fileN_rot
print fileE_rot
print "output3"
print fileE_rot "-" compR_name
print fileN_rot "-" compT_name
}
'
$ ./tst.sh
Running
output1
2021-02-07-0448-04S.JKH_RR.SAC
2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHE-BHR.SAC
2021-02-07-0448-04S.JKH_RR-BHT.SAC
or if there really was some reason to want to do it directly in a shell (e.g. this code is in some loop manipulating files named based on these variables) then:
$ cat tst.sh
#!/usr/bin/env bash
fileN='2021-02-07-0448-04S.JKH_RR.SAC'
fileE='2021-02-07-0448-04S.JKH_RR_BHE.SAC'
compR='BHR'
compT='BHT'
compR_name='BHR.SAC'
compT_name='BHT.SAC'
fileN_rot="${fileN%*.SAC}"
fileE_rot="${fileE%*.SAC}"
echo 'output1'
echo "$fileN"
echo "$fileE"
echo 'output2'
echo "$fileN_rot"
echo "$fileE_rot"
echo 'output3'
echo "${fileE_rot}-${compR_name}"
echo "${fileN_rot}-${compT_name}"
$ ./tst.sh
output1
2021-02-07-0448-04S.JKH_RR.SAC
2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHE-BHR.SAC
2021-02-07-0448-04S.JKH_RR-BHT.SAC
I need to output the result of this shell script "git diff --name-only commit2 commit1" to an array in groovy. How can I do it?
Already tried to create a variable
def diff = sh(script: "git diff --name-only commit2 commit1", returnStdout: true)
and then work it with Pattern and Matcher, but probably due to its formatting it is always returning an empty array.
The output of the shell script is something like:
directory/file1.java
directory/file2.java
Found a way to do it with bash but can't figure out how to adapt it to groovy (mapfile -t my_array < <( my_command ))
If you want to get the output as a list, where each line is a separate entry in that list, you could call split('\n') on the output of the sh step:
def diff = sh(script: "git diff --name-only commit2 commit1", returnStdout: true).trim().split('\n')
I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later.
sh "ls -l"
How can I do this, especially as it seems that you cannot really run any kind of groovy code inside the Jenkinsfile?
The latest version of the pipeline sh step allows you to do the following;
// Git committer email
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
Another feature is the returnStatus option.
// Test commit message for flags
BUILD_FULL = sh (
script: "git log -1 --pretty=%B | grep '\\[jenkins-full]'",
returnStatus: true
) == 0
echo "Build full flag: ${BUILD_FULL}"
These options where added based on this issue.
See official documentation for the sh command.
For declarative pipelines (see comments), you need to wrap code into script step:
script {
GIT_COMMIT_EMAIL = sh (
script: 'git --no-pager show -s --format=\'%ae\'',
returnStdout: true
).trim()
echo "Git committer email: ${GIT_COMMIT_EMAIL}"
}
Current Pipeline version natively supports returnStdout and returnStatus, which make it possible to get output or status from sh/bat steps.
An example:
def ret = sh(script: 'uname', returnStdout: true)
println ret
An official documentation.
quick answer is this:
sh "ls -l > commandResult"
result = readFile('commandResult').trim()
I think there exist a feature request to be able to get the result of sh step, but as far as I know, currently there is no other option.
EDIT: JENKINS-26133
EDIT2: Not quite sure since what version, but sh/bat steps now can return the std output, simply:
def output = sh returnStdout: true, script: 'ls -l'
If you want to get the stdout AND know whether the command succeeded or not, just use returnStdout and wrap it in an exception handler:
scripted pipeline
try {
// Fails with non-zero exit if dir1 does not exist
def dir1 = sh(script:'ls -la dir1', returnStdout:true).trim()
} catch (Exception ex) {
println("Unable to read dir1: ${ex}")
}
output:
[Pipeline] sh
[Test-Pipeline] Running shell script
+ ls -la dir1
ls: cannot access dir1: No such file or directory
[Pipeline] echo
unable to read dir1: hudson.AbortException: script returned exit code 2
Unfortunately hudson.AbortException is missing any useful method to obtain that exit status, so if the actual value is required you'd need to parse it out of the message (ugh!)
Contrary to the Javadoc https://javadoc.jenkins-ci.org/hudson/AbortException.html the build is not failed when this exception is caught. It fails when it's not caught!
Update:
If you also want the STDERR output from the shell command, Jenkins unfortunately fails to properly support that common use-case. A 2017 ticket JENKINS-44930 is stuck in a state of opinionated ping-pong whilst making no progress towards a solution - please consider adding your upvote to it.
As to a solution now, there could be a couple of possible approaches:
a) Redirect STDERR to STDOUT 2>&1
- but it's then up to you to parse that out of the main output though, and you won't get the output if the command failed - because you're in the exception handler.
b) redirect STDERR to a temporary file (the name of which you prepare earlier) 2>filename (but remember to clean up the file afterwards) - ie. main code becomes:
def stderrfile = 'stderr.out'
try {
def dir1 = sh(script:"ls -la dir1 2>${stderrfile}", returnStdout:true).trim()
} catch (Exception ex) {
def errmsg = readFile(stderrfile)
println("Unable to read dir1: ${ex} - ${errmsg}")
}
c) Go the other way, set returnStatus=true instead, dispense with the exception handler and always capture output to a file, ie:
def outfile = 'stdout.out'
def status = sh(script:"ls -la dir1 >${outfile} 2>&1", returnStatus:true)
def output = readFile(outfile).trim()
if (status == 0) {
// output is directory listing from stdout
} else {
// output is error message from stderr
}
Caveat: the above code is Unix/Linux-specific - Windows requires completely different shell commands.
this is a sample case, which will make sense I believe!
node('master'){
stage('stage1'){
def commit = sh (returnStdout: true, script: '''echo hi
echo bye | grep -o "e"
date
echo lol''').split()
echo "${commit[-1]} "
}
}
For those who need to use the output in subsequent shell commands, rather than groovy, something like this example could be done:
stage('Show Files') {
environment {
MY_FILES = sh(script: 'cd mydir && ls -l', returnStdout: true)
}
steps {
sh '''
echo "$MY_FILES"
'''
}
}
I found the examples on code maven to be quite useful.
All the above method will work. but to use the var as env variable inside your code you need to export the var first.
script{
sh " 'shell command here' > command"
command_var = readFile('command').trim()
sh "export command_var=$command_var"
}
replace the shell command with the command of your choice. Now if you are using python code you can just specify os.getenv("command_var") that will return the output of the shell command executed previously.
How to read the shell variable in groovy / how to assign shell return value to groovy variable.
Requirement : Open a text file read the lines using shell and store the value in groovy and get the parameter for each line .
Here , is delimiter
Ex: releaseModule.txt
./APP_TSBASE/app/team/i-home/deployments/ip-cc.war/cs_workflowReport.jar,configurable-wf-report,94,23crb1,artifact
./APP_TSBASE/app/team/i-home/deployments/ip.war/cs_workflowReport.jar,configurable-temppweb-report,394,rvu3crb1,artifact
========================
Here want to get module name 2nd Parameter (configurable-wf-report) , build no 3rd Parameter (94), commit id 4th (23crb1)
def module = sh(script: """awk -F',' '{ print \$2 "," \$3 "," \$4 }' releaseModules.txt | sort -u """, returnStdout: true).trim()
echo module
List lines = module.split( '\n' ).findAll { !it.startsWith( ',' ) }
def buildid
def Modname
lines.each {
List det1 = it.split(',')
buildid=det1[1].trim()
Modname = det1[0].trim()
tag= det1[2].trim()
echo Modname
echo buildid
echo tag
}
If you don't have a single sh command but a block of sh commands, returnstdout wont work then.
I had a similar issue where I applied something which is not a clean way of doing this but eventually it worked and served the purpose.
Solution -
In the shell block , echo the value and add it into some file.
Outside the shell block and inside the script block , read this file ,trim it and assign it to any local/params/environment variable.
example -
steps {
script {
sh '''
echo $PATH>path.txt
// I am using '>' because I want to create a new file every time to get the newest value of PATH
'''
path = readFile(file: 'path.txt')
path = path.trim() //local groovy variable assignment
//One can assign these values to env and params as below -
env.PATH = path //if you want to assign it to env var
params.PATH = path //if you want to assign it to params var
}
}
Easiest way is use this way
my_var=`echo 2`
echo $my_var
output
: 2
note that is not simple single quote is back quote ( ` ).
I wrote simple script as follow
#!/bin/bash
auth_type=""
SM_Read-only="Yes"
SM_write-only="No"
echo -e ${SM_Read-only}
echo -e ${SM_Write-only}
if [ "${SM_Read-only}" == "Yes" ] && [ "${SM_Write-only}" == "Yes" ]
then
auth_type="Read Write"
else
auth_type="Read"
fi
echo -e $auth_type
And when i execute it i got following output with errors.
./script.bash: line 5: SM_Read-only=Yes: command not found
./script.bash: line 6: SM_write-only=No: command not found
only
only
Read
Any one know correct way to declare the variable with "-" (dash)?
EDIT:
have getting response from c code and evaluate the variables for example
RESP=`getValue SM_ Read-only ,Write-only 2>${ERR_DEV}`
RC=$?
eval "$RESP"
from above scripts code my c binary getValue know that script want Read-only and Write-only and return value to script.So during eval $RESP in cause error and in my script i access variable by
echo -e ${SM_Read-only}
echo -e ${SM_Write-only}
which also cause error.
Rename the variable name as follows:
SM_Read_only="Yes"
SM_write_only="No"
Please, don't use - minus sign in variable names in bash, please refer to the answer, on how to set the proper variable name in bash.
However if you generate the code, based on others output, you can simply process their output with sed:
RESP=$(getValue SM_ Read-rule,Write-rule 2>${ERR_DEV}|sed "s/-/_/g")
RC=$?
eval "$RESP"
- is not allowed in shell variable names. Only letters, numbers, and underscore, and the first character must be a letter or underscore.
I think you cant have a dash in your variables names, only letters, digits and "_"
Try:
SM_Read_only
Or
SM_ReadOnly
I have one file with -| as delimiter after each section...need to create separate files for each section using unix.
example of input file
wertretr
ewretrtret
1212132323
000232
-|
ereteertetet
232434234
erewesdfsfsfs
0234342343
-|
jdhg3875jdfsgfd
sjdhfdbfjds
347674657435
-|
Expected result in File 1
wertretr
ewretrtret
1212132323
000232
-|
Expected result in File 2
ereteertetet
232434234
erewesdfsfsfs
0234342343
-|
Expected result in File 3
jdhg3875jdfsgfd
sjdhfdbfjds
347674657435
-|
A one liner, no programming. (except the regexp etc.)
csplit --digits=2 --quiet --prefix=outfile infile "/-|/+1" "{*}"
tested on:
csplit (GNU coreutils) 8.30
Notes about usage on Apple Mac
"For OS X users, note that the version of csplit that comes with the OS doesn't work. You'll want the version in coreutils (installable via Homebrew), which is called gcsplit." — #Danial
"Just to add, you can get the version for OS X to work (at least with High Sierra). You just need to tweak the args a bit csplit -k -f=outfile infile "/-\|/+1" "{3}". Features that don't seem to work are the "{*}", I had to be specific on the number of separators, and needed to add -k to avoid it deleting all outfiles if it can't find a final separator. Also if you want --digits, you need to use -n instead." — #Pebbl
awk '{f="file" NR; print $0 " -|"> f}' RS='-\\|' input-file
Explanation (edited):
RS is the record separator, and this solution uses a gnu awk extension which allows it to be more than one character. NR is the record number.
The print statement prints a record followed by " -|" into a file that contains the record number in its name.
Debian has csplit, but I don't know if that's common to all/most/other distributions. If not, though, it shouldn't be too hard to track down the source and compile it...
I solved a slightly different problem, where the file contains a line with the name where the text that follows should go. This perl code does the trick for me:
#!/path/to/perl -w
#comment the line below for UNIX systems
use Win32::Clipboard;
# Get command line flags
#print ($#ARGV, "\n");
if($#ARGV == 0) {
print STDERR "usage: ncsplit.pl --mff -- filename.txt [...] \n\nNote that no space is allowed between the '--' and the related parameter.\n\nThe mff is found on a line followed by a filename. All of the contents of filename.txt are written to that file until another mff is found.\n";
exit;
}
# this package sets the ARGV count variable to -1;
use Getopt::Long;
my $mff = "";
GetOptions('mff' => \$mff);
# set a default $mff variable
if ($mff eq "") {$mff = "-#-"};
print ("using file switch=", $mff, "\n\n");
while($_ = shift #ARGV) {
if(-f "$_") {
push #filelist, $_;
}
}
# Could be more than one file name on the command line,
# but this version throws away the subsequent ones.
$readfile = $filelist[0];
open SOURCEFILE, "<$readfile" or die "File not found...\n\n";
#print SOURCEFILE;
while (<SOURCEFILE>) {
/^$mff (.*$)/o;
$outname = $1;
# print $outname;
# print "right is: $1 \n";
if (/^$mff /) {
open OUTFILE, ">$outname" ;
print "opened $outname\n";
}
else {print OUTFILE "$_"};
}
The following command works for me. Hope it helps.
awk 'BEGIN{file = 0; filename = "output_" file ".txt"}
/-|/ {getline; file ++; filename = "output_" file ".txt"}
{print $0 > filename}' input
You can also use awk. I'm not very familiar with awk, but the following did seem to work for me. It generated part1.txt, part2.txt, part3.txt, and part4.txt. Do note, that the last partn.txt file that this generates is empty. I'm not sure how fix that, but I'm sure it could be done with a little tweaking. Any suggestions anyone?
awk_pattern file:
BEGIN{ fn = "part1.txt"; n = 1 }
{
print > fn
if (substr($0,1,2) == "-|") {
close (fn)
n++
fn = "part" n ".txt"
}
}
bash command:
awk -f awk_pattern input.file
Here's a Python 3 script that splits a file into multiple files based on a filename provided by the delimiters. Example input file:
# Ignored
######## FILTER BEGIN foo.conf
This goes in foo.conf.
######## FILTER END
# Ignored
######## FILTER BEGIN bar.conf
This goes in bar.conf.
######## FILTER END
Here's the script:
#!/usr/bin/env python3
import os
import argparse
# global settings
start_delimiter = '######## FILTER BEGIN'
end_delimiter = '######## FILTER END'
# parse command line arguments
parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input-file", required=True, help="input filename")
parser.add_argument("-o", "--output-dir", required=True, help="output directory")
args = parser.parse_args()
# read the input file
with open(args.input_file, 'r') as input_file:
input_data = input_file.read()
# iterate through the input data by line
input_lines = input_data.splitlines()
while input_lines:
# discard lines until the next start delimiter
while input_lines and not input_lines[0].startswith(start_delimiter):
input_lines.pop(0)
# corner case: no delimiter found and no more lines left
if not input_lines:
break
# extract the output filename from the start delimiter
output_filename = input_lines.pop(0).replace(start_delimiter, "").strip()
output_path = os.path.join(args.output_dir, output_filename)
# open the output file
print("extracting file: {0}".format(output_path))
with open(output_path, 'w') as output_file:
# while we have lines left and they don't match the end delimiter
while input_lines and not input_lines[0].startswith(end_delimiter):
output_file.write("{0}\n".format(input_lines.pop(0)))
# remove end delimiter if present
if not input_lines:
input_lines.pop(0)
Finally here's how you run it:
$ python3 script.py -i input-file.txt -o ./output-folder/
Use csplit if you have it.
If you don't, but you have Python... don't use Perl.
Lazy reading of the file
Your file may be too large to hold in memory all at once - reading line by line may be preferable. Assume the input file is named "samplein":
$ python3 -c "from itertools import count
with open('samplein') as file:
for i in count():
firstline = next(file, None)
if firstline is None:
break
with open(f'out{i}', 'w') as out:
out.write(firstline)
for line in file:
out.write(line)
if line == '-|\n':
break"
cat file| ( I=0; echo -n "">file0; while read line; do echo $line >> file$I; if [ "$line" == '-|' ]; then I=$[I+1]; echo -n "" > file$I; fi; done )
and the formated version:
#!/bin/bash
cat FILE | (
I=0;
echo -n"">file0;
while read line;
do
echo $line >> file$I;
if [ "$line" == '-|' ];
then I=$[I+1];
echo -n "" > file$I;
fi;
done;
)
This is the sort of problem I wrote context-split for:
http://stromberg.dnsalias.org/~strombrg/context-split.html
$ ./context-split -h
usage:
./context-split [-s separator] [-n name] [-z length]
-s specifies what regex should separate output files
-n specifies how output files are named (default: numeric
-z specifies how long numbered filenames (if any) should be
-i include line containing separator in output files
operations are always performed on stdin
Here is a perl code that will do the thing
#!/usr/bin/perl
open(FI,"file.txt") or die "Input file not found";
$cur=0;
open(FO,">res.$cur.txt") or die "Cannot open output file $cur";
while(<FI>)
{
print FO $_;
if(/^-\|/)
{
close(FO);
$cur++;
open(FO,">res.$cur.txt") or die "Cannot open output file $cur"
}
}
close(FO);
Try this python script:
import os
import argparse
delimiter = '-|'
parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input-file", required=True, help="input txt")
parser.add_argument("-o", "--output-dir", required=True, help="output directory")
args = parser.parse_args()
counter = 1;
output_filename = 'part-'+str(counter)
with open(args.input_file, 'r') as input_file:
for line in input_file.read().split('\n'):
if delimiter in line:
counter = counter+1
output_filename = 'part-'+str(counter)
print('Section '+str(counter)+' Started')
else:
#skips empty lines (change the condition if you want empty lines too)
if line.strip() :
output_path = os.path.join(args.output_dir, output_filename+'.txt')
with open(output_path, 'a') as output_file:
output_file.write("{0}\n".format(line))
ex:
python split.py -i ./to-split.txt -o ./output-dir