How can I use the version from my build.gradle file in my bitbucket-pipelines.yml file - bitbucket-pipelines

Currently when I change the version number in my project I have to define it in my build.gradle file and several places in my bitbucket-pipelines.yml. This is because I have scripts to upload files to my project's Downloads folder. Is there a way to get the version from the build.gradle file? I would like to just update one file when I increment the version.
build.gradle
plugins {
id 'java'
}
def buildNumber = project.properties['buildNumber'] ?:'0'
group 'edu.csusb.iset'
version "2020.1-BETA"
jar {
manifest {
attributes('Main-Class': 'edu.csusb.iset.projectname.Main',
'Implementation-Title': 'ProjectName',
'Implementation-Version': "$version, build $buildNumber")
}
from {
configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
}
}
task packSrc(type: Tar) {
archiveAppendix.set("src")
archiveVersion.set("$project.version")
archiveExtension.set("tar.gz")
destinationDirectory = file("$buildDir/dist")
compression = Compression.GZIP
from 'build.gradle'
from 'settings.gradle'
from('src') {
include '**/*'
into "src"
}
}
repositories {
mavenCentral()
}
dependencies {
...
}
bitbucket-pipelines.yml
image: openjdk:11
pipelines:
default:
- step:
name: Gradle build
caches:
- gradle
script:
- bash ./gradlew build
- step:
name: Deploy downloads
trigger: manual
caches:
- gradle
script:
- bash ./gradlew build
- pipe: atlassian/bitbucket-upload-file:0.1.6
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: build/libs/ProjectName-2020.1-BETA.jar
- bash ./gradlew packSrc
- pipe: atlassian/bitbucket-upload-file:0.1.6
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: build/dist/ProjectName-src-2020.1-BETA.tar.gz

Bitbucket Pipelines is just running bash/shell commands on a Unix docker machine. So you can use the normal unix shell commands, like awk or perl, to extract text from a text file:
Use awk
build.gradle contains a line:
version "2020.1-BETA"
awk '/^version/{gsub(/\"/,"",$2);print $2}' build.gradle
Output: 2020.1-BETA
Explanation:
awk = extract text on a line, using the default space character " " as a delimiter
^ = Regular expression to start at the beginning of a line only
version = Look for the word "version", case-sensitive
{gsub(/\"/,"",$2);print $2} = strip out all speech mark characters ", and print column 2
More info and source: how to extract text which matches particular fields in text file using linux commands
Another way with perl
build.gradle contains a line:
version "2020.1-BETA"
perl -nle 'print $& while m{(?<=version ").*?(?=")}g' build.gradle
Output: 2020.1-BETA
Explanation:
Extract the text between two words of: version " and ", using a regular expression
perl -nle = extract text from a line in a file, using regular expressions
(?<=version ") = Look for the start tag of version ", using negative lookahead ?<= to make the capture start at the end of the tag
.*? = Capture any string, after the first tag
(?=") = Look for the end tag of a speech mark ", using positive lookahead ?= to make the capture stop before the tag
g = global search on all lines of a multi-line file
More info and sources: grep -P no longer works. How can I rewrite my searches? and How to use sed/grep to extract text between two words?
Use the shell commands in the yml script
You can therefore use either awk or perl in your yml script. Capture the output as an environment variable called $APP_VERSION, and reuse this environment variable:
script:
- bash ./gradlew build
# Extract the version string from build.gradle into an environment variable
- export APP_VERSION=$(awk '/^version/{gsub(/\"/,"",$2);print $2}' build.gradle)
- echo $APP_VERSION # debug to print the extracted text of "2020.1-BETA"
- pipe: atlassian/bitbucket-upload-file:0.1.6
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: "build/libs/ProjectName-$APP_VERSION.jar" # Use the extracted text here
Alternative way:
script:
- bash ./gradlew build
# Extract the version string from build.gradle into an environment variable
- export APP_VERSION=$( perl -nle 'print $& while m{(?<=version ").*?(?=")}g' build.gradle )
- echo $APP_VERSION # debug to print the extracted text of "2020.1-BETA"
- pipe: atlassian/bitbucket-upload-file:0.1.6
variables:
BITBUCKET_USERNAME: $BITBUCKET_USERNAME
BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
FILENAME: "build/libs/ProjectName-${APP_VERSION}.jar" # Use the extracted text here

Related

Find matching pattern and replace variable value in next line

I am trying to replace the value of a variable "Tag" next to the searched pattern, like:
If name in images is "repo.url/app1"
Replace Tag: NEW_VALUE
Below is the yaml sample file where I need to replace the values.
namespace: dev
resources:
- file.yaml
- ../../..
images:
- name: repo.url/app1
Tag: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- name: repo.url/app2
Tag: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
So far I tried
sed -i "/\-\ name\:\ repo.url\/app1/{n;n;s/\(Tag\).*/\1: $NEW_VALUE/}" file
which doesn't do anything and doesn't throw any error. I am not sure, if there's a better way to handle such substitutions.
Simplest way with sed is just to match the line and then perform the substitution on the next (\w used to match all word-characters at the end of the line for replacement), e.g.
sed -E '/repo\.url\/app1$/{n;s/\w+$/yyyyyyyyyyyyyyyyyyyyyy/}' file.yaml
Example Use/Output
sed -E '/repo\.url\/app1$/{n;s/\w+$/yyyyyyyyyyyyyyyyyyyyyy/}' file.yaml
namespace: dev
resources:
- file.yaml
- ../../..
images:
- name: repo.url/app1
Tag: yyyyyyyyyyyyyyyyyyyyyy
- name: repo.url/app2
Tag: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Using awk
Another way is to use awk and a flag to indicate replace in next line (rep below). Then use sub() to replace the 'x's with whatever you need, e.g.
awk '
rep { sub(/\w+$/,"yyyyyyyyyyyyyyyyyyyyyyyy"); rep = 0 }
$NF == "repo.url/app1" { rep=1 }
1
' file.yaml
Where 1 is just shorthand for print current record.
Example Use/Output
awk 'rep { sub(/\w+$/,"yyyyyyyyyyyyyyyyyyyyyyyy"); rep = 0 } $NF == "repo.url/app1" { rep=1 }1' file.yaml
namespace: dev
resources:
- file.yaml
- ../../..
images:
- name: repo.url/app1
Tag: yyyyyyyyyyyyyyyyyyyyyyyy
- name: repo.url/app2
Tag: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Just redirect the output to a new file to save, e.g. awk '{...}' file.yaml > newfile.yaml.

Update YAML parameter file from CLI

I have a question regarding mkdocs YAML file update from the CLI. YAML file has following string:
...
site_name: Material for MkDocs
...
What I need is to manipulate site_name parameter:
add src/ before
remove all spaces and special characters from parameter.
Output should be:
...
site_name: src/MaterialforMkDocs
#OR
site_name: src/Material_for_MkDocs
...
I did following and looks that it's working:
newSiteName=$(grep "site_name:" mkdocs.yml | sed 's/ //g' | sed 's/[^a-z A-Z 0-9 _]//g' | sed 's/site_name/site_name: src\//')
sed -i "s|site_name:.*|$newSiteName|" mkdocs.yml
I strongly believe that this could be solved in much easy way.
One call to sed:
sed -E '
/^[[:blank:]]*site_name:[[:blank:]]*/ {
h ;# copy the line
s/// ;# remove the previous regex
s/[[:blank:]]+/_/g ;# convert whitespace to underscores
s,^,src/, ;# add the prefix
x ;# swap hold and pattern spaces
s/:.*/: / ;# remove the value
G ;# append the hold space
s/\n// ;# remove the newline
}
' mkdocs.yml
If it looks right; add the -i option.
You can also use yq - a tool specifically for yaml docs, including editing front-matter:
yq e --front-matter=process '.site_name |= "src/" + sub("[^a-zA-Z]", "")' file.yaml
See https://mikefarah.gitbook.io/yq/usage/front-matter#process-front-matter
Disclosure: I wrote yq

Using sed to add text after a pattern, but the added text comes from a list file

How can I use sed to locate a string, and add text from another file after the string?
File 1:
stage ('Clone Repo31') {
steps {
git credentialsId: '', url: '/stash/scm/'
}
}
stage ('Zip Repo31') {
steps {
sh"""
tar --exclude='*.tar' -cvf .tar *
"""
}
}
steps {
git credentialsId: '', url: '/stash/scm/'
}
}
stage ('Zip Repo32') {
steps {
sh"""
tar --exclude='*.tar' -cvf .tar *
"""
}
}
File 2:
randomRepo.git
differentRandomRepo.git
I want to be able to use sed to read the second file, and add the contents of each line from the second file after each occurance of stash/scm/
Desired output:
stage ('Clone Repo31') {
steps {
git credentialsId: '', url: '/stash/scm/randomRepo.git'
}
}
stage ('Zip Repo31') {
steps {
sh"""
tar --exclude='*.tar' -cvf .tar *
"""
}
}
steps {
git credentialsId: '', url: '/stash/scm/differentRandomRepo.git'
}
}
stage ('Zip Repo32') {
steps {
sh"""
tar --exclude='*.tar' -cvf .tar *
"""
}
}
Can this be done with sed? I'm having issues reading it from a list file and it's confusing since it has a lot of slashes in it. I've been able to use normal sed substitution but I don't know how to do substitution by reading another file.
In the following I present an almost pure sed solution.
sed has an r command to read files, so you could in principle use that to read the file2. However, no subsequent command will affect the lines read from the file, so I cannot think of any way of using the r command effectively to do what you ask.
However, a solution is possible if file1 and file2 are both given in input to sed.
In the following, in order to distinguish the two files, I put a marker line (-----) that I give for granted is not in file2; it could be anywhere in file1 without creating any problems, however.
cat file2 <(echo '-----') file1 | sed -f script.sed
where script.sed is the following:
1{ # only on line 1
:a # begin while
/-----/!{ # while the line does not contain the marker
N # append the following line
ba # end while
} # here the pattern space is a multiline containing the list
s/\n-----// # remove the last newline and the marker
h # put the multiline in the hold space
d # delete, as we don't want to print anything so far
} # that's it, lines from 1 to the marker are processed
/stash\/scm\//{ # for lines matching this pattern
G # we append the full hold space
s/'\n\([^\n]*\)/\1'/ # and position the first entry in the list appropriately
x # then we swap pattern and hold space
s/[^\n]*\n// # remove the first element of the list
x # and swap again
} # now the hold space has one item less
This is a bash script that uses sed and reads File_2 (The file containing the replacements) line by line, thus reading one replacement at a time. I then replaced the lines in File_1 with a sed script.
while IFS= read -r line; do
sed -i "0,/\/stash\/scm\/'/{s|/stash/scm/'|/stash/scm/${line}'|}" File_1.txt
done < File_2.txt
Some tricks used to do this:
sed '0,/Apple/{s/Apple/Banana/}' input_filename Replace only the first occurrence in filename of the string Apple with the string Banana
Using double quotes for the sed script to allow for variable expansion ${line}
Making sure the search string to replace was being changed each iteration. This was done by including the ending single quote char ' for the search argument in the sed script s|/stash/scm/'|
Reading a file line by line in a bash script
while IFS= read -r line; do
echo $line
done < File_2.txt
Read File line by line in bash
You want to have lines like
sed 's#'/stash/scm/'#&something_from_file2#' file1
You can make these lines with
# Note:
# / is not a delimiter, but part of the path
# % is the delimiter in the current sed-command
# # is the delimiter in the generated command.
sed 's%.*%s#/stash/scm/#\&&#%' file2
You can generate these commands on the fly and execute them on file1.
sed -f <(sed 's%.*%s#/stash/scm/#\&&#%' file2) file1
One problem left. Both commands will substitute all matches.
I will use the single quote given after the match.
When something is put before the single quote in /stash/scm/' this is different than the match files when you look for the /stash/scm/' string including the quote.
You want to generate lines like
s#(/stash/scm/)(')#\1randomRepo.git\2#
s#(/stash/scm/)(')#\1differentRandomRepo.git\2#
Each substition should be done only once, so we consider file2 as one long line using the option -z:
sed -rzf <(sed 's%.*%s#(/stash/scm/)('\'')#\\1&\\2#%' file2) file1

How can I distinguish between file names with and without suffix?

I am writing a bash shell script to output the suffixes of filenames.
In this case I use:
sed 's|.*\.||'
So the output is e.g.:
png
exe
c
But what do I do if the file name has no suffix and therefore no dot? My output should be "no suffix", but I don't know how to do this with sed.
EDIT
What I've already tried:
Directory:
abc.x
abc.y
abc
Input:
find . -type f | sed -E 's/^[^.]+$/no suffix/; s/.*\.//'
Output:
x
y
/abc
Use 2 consecutive substitutions:
sed -E 's/^[^.]+$/no suffix/; s/.+\.//'
One in awk. First some test material that was not provided:
$ cat foo
this.foo
that.bar
nothing
The awk:
$ awk '{n=split($0,a,".");print (n>1?a[n]:"no suffix")}' foo
foo
bar
no suffix
$ cat file
abc.x
abc.y
abc
$ awk -F'.' '{print (NF>1 ? $NF : "no suffix")}' file
x
y
no suffix
How about
sed '/.*\./s///;t;s/.*/no suffix/'
The regex matches lines with a dot. On those lines, we perform a substitution. If a substitution occurred, we are done. Otherwise, perform the other substitution.
The use of an empty regex in the substitution pattern uses the previous pattern. The t command branches if a substitution occurred; without an argument, we branch to the end of the script. (Otherwise, you can set a label with :label and branch to that with tlabel.)
You can accomplish the same with the POSIX shell parameter expansions without invoking separate utilities. For example, to test whether a file contains a '.' you can simply use test, e.g.:
[ "$i" = "${i%.*}" ]
See Posix Programmer's Manual - Shell Command Language - Parameter Expansion
If it tests TRUE, then no extension is present, otherwise, you can use an additional parameter expansion to obtain the extension itself, e.g.
[ "$i" = "${i%.*}" ] && echo "$i - no suffix" || echo "$i - ${i##*.}"
(note: you would need an additional test to exclude .foo (e.g. dotfiles), but that is left to you)
Wrap that in a loop and exclude directory files and you can test every file within a directory or use read within a loop and pipe a list of names to it. For example, looping over the files in a directory would results in:
...
ftlcdfil.c - c
geany-plugin_build_w_gtk+2.patch - patch
geany-plugins.spec - spec
geany.spec - spec
geany_build_w_gtk+2.patch - patch
getfl - no suffix
gtkw_save_test.txt - txt
gtkwrite-master.zip - zip
helloleap - no suffix
helloleap.c - c
jnl - no suffix
messages - no suffix
opensuse_15.0_1s_delay.svg - svg
...
Using Perl
/tmp> ls ab*
abc abc.x abc.y
/tmp> perl -e ' print /\./? "$_\n" : "no suffix\n" for(glob("ab*")) '
no suffix
abc.x
abc.y
/tmp>

Get string from file

In my Packages file i have multiple packages. I'm able to check the file if a string is inside, and if so, i would like to get the version of the file.
Package: depictiontest
Version: 1.0
Filename: ./debs/com.icr8zy.depictiontest.deb
Size: 810
Description: Do not install. Testing Depiction.
Name: Depiction Test
so the above is part of the many similar looking info of a package. Each time i detected if the package exists i would like to get the Version. is there any possible way?
btw, this is what i use to get check if the file exists.
if grep -q "$filename" /location/Packages; then
#file exists
#get file version <-- stuck here
else
#file does not exists
fi
EDIT:
Sorry but maybe i wasn't clear in explaining myself, I would already have the Name of the package and would like to extract the Version of that package only. I do not need a loop to get all the Names and Versions. Hope this clears it... :)
How do you extract the file name in the first place? Why not parse the whole file, then filter out nonexistent file names.
awk '/^Package:/{p=$2}
/^Version:/{v=$2}
/^Filename:/{f=$2}
/^$/{print p, v, f}' Packages |
while read p v f; do
test -e "$f" || continue
echo "$p $v"
done
This is not robust with e.g. file names with spaces, but Packages files don't have file names with spaces anyway. (Your example filename is nonstandard, though; let's assume it's no worse than this.)
You want to make sure there's an empty line at the end of Packages, or force it with { sed '$/^$/d' Packages; echo; } | awk ...
Edit: This assumes a fairly well-formed Packages file, with an empty line between records. If a record lacks one of these fields, the output will repeat the value from the previous record - that's nasty. If there are multiple adjacent empty lines, it will output the same package twice. Etc. If you want robust parsing, I'd switch to Perl or Python, or use a standard Debian tool (I'm sure there must be one).
With grep, you can pick a certain amount of lines before or after a keyword.
egrep -A1 "^Package: depictiontest" /path/to/file
would yield 1 additional line after the match.
egrep -B1 "^Filename: .*depictiontest.*" /path/to/file
would yield 1 additional line before the match.
egrep "^(Package|Version): " "^Package: depictiontest" /path/to/file
would lead to package and Version lines only, so would rely on them being in the correct order, to find out easily, which version belongs to which package.
If the order is same then you can parse the whole file and feed values in to an array -
awk -F": " '
/^Package/{p=$2;getline;v=$2;getline;f=$2;ary[p"\n"v"\n"f"\n"]}
END{for (x in ary) print x}' file
Test:
[jaypal:~/Temp] cat file
Package: depictiontest
Version: 1.0
Filename: ./debs/com.icr8zy.depictiontest.deb
Size: 810
Description: Do not install. Testing Depiction.
Name: Depiction2fdf Test
Package: depi444ctiontest
Version: 1.05
Filename: ./debs/coffm.icr8zy.depictiontest.deb
Size: 810
Description: Do not install. Testing Depiction.
Name: Depiction Test
Package: depic33tiontest
Version: 1.01
Filename: ./d3ebs/com.icr8zy.depictiontest.deb
Size: 810
Description: Do not install. Testing Depiction.
Name: Depiction Test
[jaypal:~/Temp] awk -F": " '/^Package/{p=$2;getline;v=$2;getline;f=$2;ary[p"\n"v"\n"f"\n"]}END{for (x in ary) print x}' file
depi444ctiontest
1.05
./debs/coffm.icr8zy.depictiontest.deb
depic33tiontest
1.01
./d3ebs/com.icr8zy.depictiontest.deb
depictiontest
1.0
./debs/com.icr8zy.depictiontest.deb
If "Version: ..." line is always exactly 1 line before the "Filename: ..." line, then you may try something like this:
line_number=$(grep -n "$filename" /location/Packages | head -1 | cut -d: -f1)
if (( $line_number > 0 )); then
#file exists
version=$(head -n $(( $line_number - 1 )) /location/Packages | tail -1 | cut -d' ' -f2)
else
#file doesn't exist
fi
The simplest awk implementation that I can think of:
$ awk -F':' -v package='depictiontest' '
$1 == "Package" {
trimmed_package_name = gensub(/^ */, "", "", $2)
found_package = (trimmed_package_name == package)
}
found_package && $1 == "Version" {
trimmed_version_number = gensub(/^ */, "", "", $2)
print trimmed_version_number
}
' Packages
1.0
This processes the file (Packages) line-by-line and sets the found_package flag if the line starts with 'Package' and the value after the field separator (-F), which is : (and any whitespace) is the value passed into the package variable (-v). Then, if the flag is set an we find a line beginning with a 'Version' field, we print the value after the field separator (trimming leading whitespace). If another 'Package' field is found and the name is not the one that we are looking for, the flag is reset and the subsequent version number will not be printed.

Resources