linux - simplify semantic versioning script - linux

I have semantic versioning for an app component and this is how I update the major version number and then store it back in version.txt. This seems like a lot of lines for a simple operation. Could someone please help me trim this down? No use of bc as the docker python image I'm on doesn't seem to have that command.
This is extracted from a yml file and version.txt only contains a major and minor number. 1.3 for example. The code below updates only the major number (1) and resets the minor number to 0. So if I ran the code on 1.3, I would get 2.
- echo $(<version.txt) 1 | awk '{print $1 + $2}' > version.txt
VERSION=$(<version.txt)
VERSION=${VERSION%.*}
echo $VERSION > version.txt
echo "New version = $(<version.txt)"

About Simplicity
"Simple" and "short" are not the same thing. echo $foo is shorter than echo "$foo", but it actually does far more things: It splits the value of foo apart on characters in IFS, evaluates each result of that split as a glob expression, and then recombines them.
Similarly, making your code simpler -- as in, limiting the number of steps in the process it goes through -- is not at all the same thing as making it shorter.
Incrementing One Piece, Leaving Others Unmodified
if IFS=. read -r major rest <version.txt || [ -n "$major" ]; then
echo "$((major + 1)).$rest" >"version.txt.$$" && mv "version.txt.$$" version.txt
else
echo "ERROR: Unable to read version number from version.txt" >&2
exit 1
fi
Incrementing Major Version, Discarding Others
if IFS=. read -r major rest <version.txt || [ -n "$major" ]; then
echo "$((major + 1))" >"version.txt.$$" && mv "version.txt.$$" "version.txt"
else
echo "ERROR: Unable to read version number from version.txt" >&2
exit 1
fi
Rationale
Both of the above are POSIX-compliant, and avoid relying on any capabilities not built into the shell.
IFS=. read -r first second third <input reads the first line of input, and splits it on .s into the shell variables first, second and third; notably, the third column in this example includes everything after the first two, so if you had a.b.c.d.e.f, you would get first=a; second=b; third=d.e.f -- hence the name rest to make this clear. See BashFAQ #1 for a detailed explanation.
$(( ... )) creates an arithmetic context in all POSIX-compliant shells. It's only useful for integer math, but since we split the pieces out with the read, we only need integer math. See http://wiki.bash-hackers.org/syntax/arith_expr
Writing to version.txt.$$ and renaming if that write is successful prevents version.txt from being left empty or corrupt if a failure takes place between the open and the write. (A version that was worried about symlink attacks would use mktemp, instead of relying on $$ to generate a unique tempfile name).
Proceeding through to the write only if the read succeeds or [ -n "$major" ] is true prevents the code from resetting the version to 1 (by adding 1 to an empty string, which evaluates in an arithmetic context as 0) if the read fails.

Related

Check file's whole raw text content exists as part of another file

big1.txt:
a
b
c
d
e
big2.txt:
f
c
g
h
i
b
small.txt:
b
c
Within a bash script, how can I tell that the whole ordered content of small.txt exists in another file?
Example:
??? small.txt big1.txt should return true
??? small.txt big2.txt should return false
If big1.txt and big2.txt not too big (can be loaded in memory). Following test may be sufficient.
# to store file content into variables
big1=$(< big1.txt)
big2=$(< big2.txt)
small=$(< small.txt)
# to run from test case
big1=$'a\nb\nc\nd\ne\n'
big2=$'f\nc\ng\nh\ni\nb\n'
small=$'b\nc\n'
if [[ ${big1} = *${small}* ]]; then echo "big1"; fi
if [[ ${big2} = *${small}* ]]; then echo "big2"; fi
Sometimes the way to discover that two complicated things are 'equal' is to do some cheap test that is true if they are equal and rarely true if they are not. Those that pass this hueristic test are then checked more carefully ... but only rarely so full equality test can be expensive and yet not be triggered on every compare.
What I would do in this circumstance is take all the files, and sort their lines. (You might want to suppress blank lines if you are looking for matching text, and strip lines with trailing blanks, but that's your choice). Probably useful to delete duplicate lines.
Now compare each file, to all longer files to see if it is a prefix. (Can't be a prefix if the other file is shorter, thus we get rid of 1/2 the compares just based on sizes). If sorted file A is a prefix of sorted file B, then you can run a more complicated test to see if the real file A is embeded in file B (which I suspect will be true with high probability if the sorted files pass the prefix test).
Having had this idea, we can now optimize it. Instead of storing lines of text, we take each file, and hash each line, giving a file of hash codes. Sort these. Follow the rest of the procedure.
Next trick: decide our hash codes are 8 bits or 16 bits in size. This makes them fit into a character of your favorite programming language. Now your prefix-compare test can consist of collecting the character-sized hash codes per file, and doing a string compare of shorter ones against longer ones. At this point we've moved the problem from reading the disk to comparing efficiently in memory; we probably can't speed it up much because disk reads are very expensive compared to in memory computations.
$ diff small big1.txt | grep -q '^<'
$ echo $?
1
$ diff small big2.txt | grep -q '^<'
$ echo $?
0
$ ! (diff small big1.txt | grep -q '^<')
$ echo $?
0
$ ! (diff small big2.txt | grep -q '^<')
$ echo $?
1
$ if diff small big1.txt | grep -q '^<'; then echo "does not exit"; else echo "does exist"; fi
does exist
$ if diff small big2.txt | grep -q '^<'; then echo "does not exit"; else echo "does exist"; fi
does not exit
check this please
if perl -0777 -e '$n = <>; $h = <>; exit(index($h,$n)<0)' small.txt big.txt
then echo small.txt is found in big.txt
fi

What would cause the BASH error "[: too many arguments" after taking measures for special characters in strings?

I'm writing a simple script to check some repositories updates and, if needed, I'm making new packages from these updates to install new versions of those programs it refers to (in Arch Linux). So I made some testing before executing the real script.
The problem is that I'm getting the error [: excessive number of arguments (but I think the proper translation would be [: too many arguments) from this piece of code:
# Won't work despite the double quoted $r
if [ "$r" == *"irt"* ]; then
echo "TEST"
fi
The code is fixed by adding double square brackets which I did thanks to this SO answer made by #user568458:
# Makes the code works
if [[ "$r" == *"irt"* ]]; then
echo "TEST"
fi
Note that $r is defined by:
# Double quotes should fix it, right? Those special characters/multi-lines
r="$(ls)"
Also note that everything is inside a loop and the loop progress with success. The problems occurs every time the if comparison matches, not printing the "TEST" issued, jumping straight to the next iteration of the loop (no problem: no code exists after this if).
My question is: why would the error happens every time the string matches? By my understanding, the double quotes would suffice to fix it. Also, If I count on double square brackets to fix it, some shells won't recognize it (refers to the answer mentioned above). What's the alternative?
Shell scripting seems a whole new programming paradigm.. I never quite grasp the details and fail to secure a great source for that.
The single bracket is a shell builtin, as opposed to the double bracket which is a shell keyword. The difference is that a builtin behaves like a command: word splitting, file pattern matching, etc. occur when the shell parses the command. If you have files that match the pattern *irt*, say file1irt.txt and file2irt.txt, then when the shell parses the command
[ "$r" = *irt* ]
it expands $r, matches all files matching the pattern *irt*, and eventually sees the command:
[ expansion_of_r = file1irt.txt file2irt.txt ]
which yields an error. No quotes can fix that. In fact, the single bracket form can't handle pattern matching at all.
On the other hand, the double brackets are not handled like commands; Bash will not perform any word splitting nor file pattern matching, so it really sees
[[ "expansion_of_r" = *irt* ]]
In this case, the right hand side is a pattern, so Bash tests whether the left hand side matches that pattern.
For a portable alternative, you can use:
case "$r" in
(*irt*) echo "TEST" ;;
esac
But now you have a horrible anti-pattern here. You're doing:
r=$(ls)
if [[ "$r" = *irt* ]]; then
echo "TEST"
fi
What I understand is that you want to know whether there are files matching the pattern *irt* in the current directory. A portable possibility is:
for f in *irt*; do
if [ -e "$f" ]; then
echo "TEST"
break
fi
done
Since you're checking for files with a certain file name, I'd suggest to use find explicitly. Something like
r="$(find . -name '*irt*' 2> /dev/null)"
if [ ! -z "$r" ]; then
echo "found: $r"
fi

nested functions and while loops returned `while: command not found` in bash [duplicate]

This question already has answers here:
Why does /bin/sh behave differently to /bin/bash even if one points to the other?
(4 answers)
Closed 7 years ago.
The main function execSteps executes emerge --pretend $package one by one, and these package's names(only names, no version information) are stored in a text file stepFile. Some of packages may have extra need for configuring package.use, package.license, this kind of extra information will be shown up after executed emerge --pretend $package. The second while loop in main function and function acceptPKGTypeItems are intended to deal with this kind of extra information.
When emerging one particular package, it may depend on a couple of more packages. For example, emerge --pretend ceph, I need to emerge more than 10 packages before ceph gets emerged. Along with Gentoo/Linux updated, new version of package may be applied. So text file stepFile only contained with package names which I need, and parsing the result of emerge --pretend $package, I'm able to get updated package emerged.
At case 0), this while loop is intended to parse the result of emerge --pretend $line(which is from stepFile), for example emerge --pretend ceph and get its dependant packages with current version, for example dev-libs/boost-1.57.0, pass it as an argument to function emgRecursion because dependant package dev-libs/boost-1.57.0 of package ceph may have its own dependant packages which are dev-libs/boost-build-1.57.0 and dev-libs/boost-1.57.0.
My problem is I get an error while : command not found in function emgRecursion when I enter 0 at case 0). Is it another different shell thing? I've added a pair parenthesis between the second while loop in main function which helped get readin answer from user for choosing package.use, package.license, or package.keywords. And I've tried to add another pair of parenthesis between the third while loop, the same problem. I've tested emgRecursion and acceptPKGTypeItems separately, both of them work fine and correctly.
Any ideas? Thank you very much.
function acceptPKGTypeItems() {
...
}
function emgRecursion() {
local output="$(emerge --pretend "="$1 | grep "\[ebuild")"
       while read -r line;
       do
done <<<"$output"
}
function execSteps() {
local running=0
while read -r line;
do
if (( running )); then
if [[ $line = "#"* ]] && [[ "${line/"step"}" = "$line" ]]; then
continue
else
if [[ ! "${line/"step"}" = "$line" ]]; then
echo "====== approaching to the next step which is not available at this time."
break
else
( output="$(emerge --pretend $line | grep "\[ebuild")"
echo "**************** $line is ready for emerging ****************"
while read -p "Which type of package would you like to add new item to (1-packageuse 2-packagelicense 3-packagekeywords 0-exit and continue)? " choice; do
case "$choice" in
1) echo "**************** $line is ready for emerging"
acceptPKGTypeItems $PACKAGEUSE
echo "**************** package.use has been updated."
;;
2) echo "**************** $line is ready for emerging"
acceptPKGTypeItems $PACKAGELICENSE
echo "**************** package.license has been updated."
;;
3) echo "**************** $line is ready for emerging"
acceptPKGTypeItems $PACKAGEKEYWORDS
echo "**************** package.keywords has been updated."
;;
0) echo "**************** $line starts emerging"
while read -r element;
do
local str="${element#*"] "}"
str="${str%%" "*}"
echo " $str is an element that need to be emerged. "
emgRecursion "$str"
done <<<"$output"
echo "**************** $line has been emerged. ****************"
break
;;
*) echo "~~~~~~~~~~~~~~~~ Invalid input, try again. ~~~~~~~~~~~~~~~~"
;;
esac
done) </dev/tty
fi
fi
else
[[ $line = "#"$1 ]] && running=1
done <$STEPS
}
execSteps step2
Nothing will stop while loop in main function,
output:
livecd / # ./step1
* Last emerge --sync was 32d 23h 4m 58s ago.
**************** sys-cluster/ceph is ready for emerging ****************
Which type of package would you like to add new item to (1-packageuse 2-packagelicense 3-package.keywords 0-exit and continue)?0
**************** sys-cluster/ceph starts emerging ****************
dev-libs/libaio-0.3.110 is an element that need to be emerged.
* Last emerge --sync was 32d 23h 5m 3s ago.
./step1: line 48:        while: command not found
./step1: line 49:        : command not found
./step1: line 50:                str=dev-libs/libaio-0.3.110: No such file or directory
Take a look at what dev-libs/libaio-0.3.110 looks like.
./step1: line 77:        done: command not found
sys-libs/libunwind-1.1 is an element that need to be emerged.
* Last emerge --sync was 32d 23h 5m 5s ago.
^C
Exiting on signal 2
Problem solved by copying functions other than emgRecursion to another file while I created for testing emgRecursion.
I realized that the difference between these two files(recursion for testing emgRecursion, step for test whole function) is recursion was originally created with #!/bin/bash, and step was originally a plain shell text file without any first line symbol and then I added #!/bin/bash to it. I thought there was no big difference between bash text file and shell text file in terms of syntax. In fact, THEY ARE TOTALLY DIFFERENT. If you mixed them up like my case, it's a WASTE of time.
This would be a template for your while loop. If you want to read a whole line then don't bother with putting a variable in the read line and stop useless Word splitting occurring.
while read -r; do
line=$REPLY
...
done <<<"$OUTPUT"
See Bash-Hackers

Scripts in sed - linux

I got the concept of bash, now, I found a site full of riddles for practising bash. I solve a couple of scripts (you should mention what do they do, what they are missing or so, depends on the question) and I bumped this script:
random_var="$(echo $1 | sed -e 's/[^[:alnum:]]//g')"
Correct me if I'm wrong about my basic assumptions on the following code:
$1 is the second argument that the script got (when the first is the script name)
There is a pipeline between the second argument and the sed script that removes all alpha numerics and... according to what I understand, this script can be "broken" by using a delimiter such as [/\]^$ and so ?
Now, there comes the difficulty (well, for me), the program gets an input from the user and, when the following script I just mention is found at a function returning true if the input is different than the result. I have no idea what is happening here, can someone enlighten me?
#!/bin/sh
func()
{
somevar="$(echo $1 | sed -e 's/[^[:alnum:]]//g')"
if [ "$somevar" != "$input" ] ; then
return 1
else
return 0
fi
}
# Sample usage of this function in a script
echo -n "Enter input: "
read input
if ! func "$input" ; then
echo "HELL NO"
exit 1
else
echo "YES!"
fi
exit 0
The script tests a string to see whether it contains any non-alphanumeric characters.
As Avinash has mentioned in the comments, the sed command removes all non-alphanumeric characters. Within the function, $input has the same value as it does in the calling scope, which is also the same as the first argument, $1. This is perhaps a little bit confusing...
If $somevar is different to $input (=$1), then this means that sed has changed the string in some way. Therefore, the string must contain at least one non-alphanumeric character.
If the function returns 1 (there were some non-alphanumeric characters in the input), then ! func is false, so the else branch will be executed and the script will return with an exit code of 0 (success). Otherwise, the script will return a non-zero exit code, indicating a failure.

BASH scripting: taking in multiple flags

I have this issue where I am doing a menu sort of program where users are supposed to type in different numbers depending on what operation(s) the want to do.
Right now I need to take in multiple flags to be placed in a command, but I can't seem to get it to work.
Catch: I cannot type in the flags directly into one variable and I will probably have to have this case list to be able to quit at will)
function fileCpy {
printf "Choices:\n1. Interactive copy, answer yes/no before doing the copy: \n2. Make backups of existing destination files\n3. Preserve file attributes\n4. Do a recursive copy\n0. Return to main menu\n"
#read choices
read $a
read $b
read $c
for choices in $choice ; do
case "$choices" in
1)
a=-i ;;
2)
b=--backup ;;
3)
c=-p ;;
#4)
#4= -R -v;;
0)
main;;
esac
echo "$a"
echo "$b"
echo "$c"
printf "\nType the name of the file you wish to copy/backup: "
read $fl1
printf "\nType the destination file: "
read $des1
cp $a $b $c $fl1 $des1
done
}
One problem is that read $a etc is wrong: you should write read a if you need the read at all. As it stands, the value read is stored in the variable with the name stored in $a.
Another problem is that it is far from clear to the innocent user that they're supposed to enter 3 lines of information before the script will continue, but the three read lines force that.
Another problem is that you don't read into $choice (via read choice) so the for loop has nothing to do.
Another problem is that your script will inherit the values of any environment variables that happen to be the same as the names of the variables you're using.
Another problem is that you don't quote the file names. It mostly won't matter unless you have a name that contains spaces or other similarly awkward characters.
A cosmetic issue is that the printf statement is ridiculously long. Use one printf per line. Or use echo. Stuff that scrolls off the RHS of the page is bad (though I don't regard 80 characters as a fixed length for lines, there's a quadratic penalty for lines that are longer than 80 — as (length-80)2 increases, the pain of the longer line goes up.
At another level altogether, the interface is modestly grotesque. As an exercise in shell scripting, it makes sense. As an exercise in how to design good shell scripts, it is a very bad design.
A design that might make sense is:
Set variables to empty: a=""; b=""; c=""; etc.
Offer a range of choices similar to those given now, but add an option to execute the command, and another to abandon ship.
Have a loop that reads choices, and sets flags.
When the user chooses execute, exit the loop and prompt for the file names.
If all's well, execute the command.
Note that you should check that the read commands work; if they don't, fail safe (don't damage anything).
Putting all those together (with some slight differences, but the same overall effect — witness the use of local for the variables):
fileCpy()
{
local a b c file dest
echo "Choices:"
echo "0. Return to main menu"
echo "1. Interactive copy, answer yes/no before doing the copy"
echo "2. Make backups of existing destination files"
echo "3. Preserve file attributes"
echo "4. Do a recursive copy"
echo "5. Execute the copy"
while printf "Your choice: " && read choice
do
[ -z "$choice" ] && return 1 # Empty input - failure
case "$choice" in
(0) return 0;;
(1) a="-i";;
(2) b="--backup";;
(3) c="-p";;
(4) d="-R";;
(5) break;;
(*) echo "Unrecognized response ($choice); please enter 0..5";;
esac
done
[ "$choice" != 5 ] && return 1 # EOF - failure
printf "Type the name of the file you wish to copy/backup: "
read file
[ -z "$file" ] && return 1 # Empty file name - failure
printf "Type the name of the destination file/directory: "
read dest
[ -z "$dest" ] && return 1 # Empty file name - failure
cp $a $b $c "$file" "$dest"
}
Test code:
echo "a=$a b=$b c=$c file=$file dest=$dest"
if fileCpy
then : OK
else echo "Failed"
fi
echo "a=$a b=$b c=$c file=$file dest=$dest"
The last block is a simple test harness. It reports on the values of the variables used inside the function, runs the function, reports if the function failed, and re-echoes the variables to demonstrate that they've not been set.
I would not use that interface unless you paid me to do so, but it more or less meets the goal of a training exercise.
if you want to do menu, you can use select construct (or case/esac). See here for info

Resources