I have stumbled upon this small difference between the set implementation of the 2 operating systems.
When running:
#!/bin/sh
set -eu
echo "${#}"
Running this on MacOS gives the following error:
#: unbound variable
Whereas running this in a Linux environment, results no errors, but echos an empty string.
Can this be resolved somehow, except changing ${#} to ${#:-} as this may result different results, if the amount of arguments is checked within the code.
The shell you have on Linux conforms to the latest POSIX standard while the one on MacOS does not (see spec., scroll downwards for -u; related: #0000155), hence the difference in behavior. A workaround would be:
#!/bin/sh
set -eu
echo "${1+"$#"}"
As you have an old version of bash on macos, you have to use "$#"
You can check the version with :
/bin/sh --version
Related
I have been working on a shell script to automate some tasks. What is the best way to make sure the shell script would run without any issues in most of the platforms. For ex., I have been using echo -n command to print some messages to the screen without a trailing new line and the -n switch doesn't work in some ksh shells. I was told the script must be POSIX compliant. How do I make sure that the script is POSIX compliant. Is there a tool? Or is there a shell that supports only bare minimum POSIX requirements?
POSIX
One first step, which gives you indications of what works or not and why, is to set the shebang to /bin/sh and use shellcheck site to analyze your script.
For example, paste this script in the shellcheck editor window:
#!/bin/sh
read -r a b <<<"$1"
echo $((a+b))
to get an indication that: "In POSIX sh, here-strings are undefined".
As a second step, you can use a shell that is as compatible with POSIX as possible.
One shell that is compatible with most other simple shells, is dash, Debian default system shell, which is a derivative of the older BSD ash.
Another shell compatible with posix is posh.
However, dash and/or posh may not be available for some systems.
There is lksh (with a ksh flavor), with the goal to be compatible with legacy (old) shell scripts. From its manual:
lksh is a command interpreter intended exclusively for running legacy shell scripts.
But there is the need to use options when calling lksh, like -o posix and -o sh:
Note that it's strongly recommended to invoke lksh with at least the -o posix option, if not both that and -o sh, to fully enjoy better compatibility to the POSIX standard (which is probably why you use lksh over mksh in the first place) or legacy scripts, respectively.
You would call lksh -o posix -o sh instead of the simple lksh.
Using options is a way to make other shells become POSIX compatible. Like lksh, using the option -o posix, like bash -o posix.
In bash, it is even possible to turn the POSIX option inside an script, with:
shopt -o posix # also with: set -o posix
It is also possible to make a local link to bash or zsh that makes both act like an old sh shell. Like this:
$ ln -s /bin/bash ./sh
$ ./sh
There are plenty of alternatives (dash, posh, lksh, bash, zsh, etc.) to get a shell that will work as a POSIX shell.
Portable
However, even so, all the above does not ensure "portability".
Unfortunately, making a shell script 'POSIX-compliant' is usually easier than making it run on any real-world shell.
The only real-world sensible recommendation is test your script in several shells.
Like the list above: dash, posh, lksh, and bash --posix.
Solaris is a world on its own, probably you will need to test against /bin/sh and xpg4/sh.
Followup:
How can I test for POSIX compliance for shell scripts?
Starting Bash with the --posix command-line option or executing ‘set -o posix’ while Bash is running will cause Bash to conform more closely to the POSIX standard by changing the behavior to match that specified by POSIX in areas where the Bash default differs.
Reference
Note:
This answer complements user8017719's great answer.
As requested in the question, a tool is discussed below: while it does not directly check for POSIX compliance, it runs a given script in multiple shells, notably including /bin/sh.
/bin/sh, the system default shell, should not be assumed to support any features other than POSIX-prescribed ones, though in practice it does, to varying degrees, depending on the specific implementation. Therefore, successfully running via /bin/sh on one platform does not guarantee that the script will work on another. Among widely used shells, dash comes closest to being a POSIX-features-only shell.
Running successfully in multiple shells is important:
if you're authoring a script that needs to be sourced in various shells.
if you know that your script will encounter only a limited set of known-in-advance shells.
For a proof-of-the-pudding-is-in-the-eating approach, consider using shall (a utility I wrote), which allows you to invoke a given script or command with multiple shells at once, with feedback about which of the targeted shells the script/command executed successfully with.
If you have Node.js installed, you can easily install it with npm install -g shall (if not, follow the above link to the GitHub repo for manual installation instructions) and then use it as follows:
shall scriptFile
or, with an ad-hoc command:
shall -c '<shell-commands>'
By default, it invokes sh, and, if installed, dash, bash, zsh, and ksh, but you can target any set of shells that you have installed by using the SHELLS environment variable.
Using the example of the echo -n command on macOS to only target shells sh and bash:
$ SHELLS=sh,bash shall -c 'echo -n hi'
✓ sh (bash variant) [0.00s]
-n hi
✓ bash [0.00s]
hi
OK - All 2 shells (sh, bash) report success.
On macOS, bash (effectively) acts as sh, and while echo -n didn't fail when used with sh, you can also see that -n wasn't recognized as an option when bash ran as sh.
Another macOS example that shows that bash permits certain Bash-specific extensions even when running as sh, such as using nonstandard [[ ... ]] conditionals (assumes that dash - which acts as sh on Ubuntu systems - was installed via Homebrew):
$ SHELLS=sh,bash,dash shall -c '[[ -n nonempty ]] && echo nonempty'
✓ sh (bash variant) [0.00s]
nonempty
✓ bash [0.00s]
nonempty
✗ dash [0.01s]
dash: 1: [[: not found
FAILED - 1 shell (dash) reports failure, 2 (sh, bash) report success.
As you can see, Bash running as sh still accepted [[ ... ]], whereas dash, which is a (mostly) POSIX-features-only shell, failed, because POSIX only mandates [ ... ] conditionals (as an alias of test ... commands).
I recently updated to Ubuntu 13.04 and one of my shell scripts broke with this:
#!/bin/sh
...
if [ "$SHOW_USAGE" == "true" ]; then
./install.sh: 248: [: false: unexpected operator
Now, I understand that I can fix this by replacing the double-equal ('==') with a single equal ('='), but what caused this difference in Ubuntu behavior between 13.04 and previous Ubuntu releases? And is there a way to restore the more flexible interpretation that the previous releases had?
/bin/sh (not /bin.sh as written in your post) used to be linked to /bin/bash, but is now linked to /bin/dash on Ubuntu. bash tolerates some bashisms even when called as sh.
If you don't mind requiring bash to run your script, you can use bash's [[ instead of trying to use [. There is some different behaviour, but [[ does support == operator, while [ never has (except maybe as an extension on particular systems).
Edit: Since you're using sh, the == was never correct; you were just getting lucky with some particular characteristic of your system setup.
I want to generate fixed a sequence of numbers that change with "seed" in linux. I am doing it with the following command.
export USR_SEED=91; export SEQ_START=53; export SEQ_LENGTH=15; bash -c 'RANDOM=$USR_SEED; for((i=1;i<=$SEQ_START;i++)); do echo -n $RANDOM >/dev/null; done; for((i=1;i<=$SEQ_LENGTH;i++)); do echo -n "$RANDOM "; done; echo'
This give a fixed sequence every-time I ran it, but the sequence changes when I do it on a different machine. Is there a way to generate a fixed sequence of numbers every-time I ran it, irrespective of machine (64bit, 32-bit linux platform).
-Mohan
I tried it on a few linux machines running:
3.8.0-23-generic, ubuntu, x86_64, bash version 4.2.45(1)-release
2.6.32-5-686, debian, i686, bash version 4.1.5(1)-release
2.6.32-5-amd64, debian, x86_64, bash version 4.2.37(1)-release
in each case if I used the same seed I had the same results.
Check to see if you're running different bash versions or if anything else strange might be interfering with your results. It should work exactly as you said.
I have uncovered another problem in the effort that we are making to port several hundreds of ksh scripts from AIX, Solaris and HPUX to Linux. See here for the previous problem.
This code:
#!/bin/ksh
if [ -a k* ]; then
echo "Oh yeah!"
else
echo "No way!"
fi
exit 0
(when run in a directory with several files whose name starts with k) produces "Oh yeah!" when called with the AT&T ksh variants (ksh88 and ksh93). On the other hand it produces and error message followed by "No way!" on the other ksh variants (pdksh, MKS ksh and bash).
Again, my question are:
Is there an environment variable that will cause pdksh to behave like ksh93? Failing that:
Is there an option on pdksh to get the required behavior?
I wouldn't use pdksh on Linux anymore.
Since AT&T ksh has become OpenSource there are packages available from the various Linux distributions. E.g. RedHat Enterprise Linux and CentOS include ksh93 as the "ksh" RPM package.
pdksh is still mentioned in many installation requirement documentations from software vendors. We replaced pdksh on all our Linux systems with ksh93 with no problems so far.
Well after one year there seems to be no solution to my problem.
I am adding this answer to say that I will have to live with it......
in Bash the test -a operation is for a single file.
I'm guessing that in Ksh88 the test -a operation is for a single file, but doesn't complain because the other test words are an unspecified condition to the -a.
you want something like
for K in /etc/rc2.d/K* ; do test -a $K && echo heck-yea ; done
I can say that ksh93 works just like bash in this regard.
Regrettably I think the code was written poorly, my opinion, and likely a bad opinion since the root cause of the problem is the ksh88 built-in test allowing for sloppy code.
You do realize that [ is an alias (often a link, symbolic or hard) for /usr/bin/test, right? So perhaps the actual problem is different versions of /usr/bin/test ?
OTOH, ksh overrides it with a builtin. Maybe there's a way to get it to not do that? or maybe you can explicitly alias [ to /usr/bin/test, if /usr/bin/test on all platforms is compatible?
I am involved in the process of porting a system containing several hundreds of ksh scripts from AIX, Solaris and HPUX to Linux. I have come across the following difference in the way ksh behaves on the two systems:
#!/bin/ksh
flag=false
echo "a\nb" | while read x
do
flag=true
done
echo "flag = ${flag}"
exit 0
On AIX, Solaris and HPUX the output is "flag = true" on Linux the output is "flag = false".
My questions are:
Is there an environment variable that I can set to get Linux's ksh to behave like the
other Os's'? Failing that:
Is there an option on Linux's ksh to get the required behavior? Failing that:
Is there a ksh implementation available for Linux with the desired behavior?
Other notes:
On AIX, Solaris and HPUX ksh is a variant of ksh88.
On Linux, ksh is the public domain ksh (pdksh)
On AIX, Solaris and HPUX dtksh and ksh93 (where I have them installed) are consistent with ksh
The Windows NT systems I have access to: Cygwin and MKS NT, are consistent with Linux.
On AIX, Solaris and Linux, bash is consistent, giving the incorrect (from my perspective) result of "flag = false".
The following table summarizes the systems the problem:
uname -s uname -r which ksh ksh version flag =
======== ======== ========= =========== ======
Linux 2.6.9-55.0.0.0.2.ELsmp /bin/ksh PD KSH v5.2.14 99/07/13.2 false
AIX 3 /bin/ksh Version M-11/16/88f true // AIX 5.3
/bin/ksh93 Version M-12/28/93e true
SunOS 5.8, 5.9 and 5.10 /bin/ksh Version M-11/16/88i true
/usr/dt/bin/dtksh Version M-12/28/93d true
HP-UX B.11.11 and B.11.23 /bin/ksh Version 11/16/88 true
/usr/dt/bin/dtksh Version M-12/28/93d true
CYGWIN_NT-5.1 1.5.25(0.156/4/2) /bin/ksh PD KSH v5.2.14 99/07/13.2 false
Windows_NT 5 .../mksnt/ksh.exe Version 8.7.0 build 1859... false // MKS
Update
After some advice from people in my company we decided to make the following modification to the code. This gives us the same result whether using the "real" ksh's (ksh88, ksh93) or any of the ksh clones (pdksh, MSK ksh). This also works correctly with bash.
#!/bin/ksh
echo "a\nb" > junk
flag=false
while read x
do
flag=true
done < junk
echo "flag = ${flag}"
exit 0
Thanks to jj33 for the previously accepted answer.
Instead of using pdksh on linux, use the "real" ksh from kornshell.org. pdksh is a blind re-implementation of ksh. kornshell.org is the original korn shell dating back 25 years or so (the one written by David Korn). AIX and Solaris use versions of the original ksh, so the kornshell.org version is usually feature- and bug- complete. Having cut my teeth with SunOS/Solaris, installing kornshell.org ksh is usually one of the first things I do on a new Linux box...
After some advice from people in my company we decided to make the following modification to the code. This gives us the same result whether using the "real" ksh's (ksh88, ksh93) or any of the ksh clones (pdksh, MSK ksh). This also works correctly with bash.
#!/bin/ksh
echo "a\nb" > junk
flag=false
while read x
do
flag=true
done < junk
echo "flag = ${flag}"
exit 0
Thanks to jj33 for the previous accepted answer.
I installed 'ksh' and 'pdksh' on my local Ubuntu Hardy system.
ii ksh 93s+20071105-1 The real, AT&T version of the Korn shell
ii pdksh 5.2.14-21ubunt A public domain version of the Korn shell
ksh has the "correct" behavior that you're expecting while pdksh does not. You might check your local Linux distribution's software repository for a "real" ksh, instead of using pdksh. The "Real Unix" OS's are going to install the AT&T version of Korn shell, rather than pdksh, by default, what with them being based off AT&T Unix (System V) :-).
Do you have to stay within ksh?
Even if you use the same ksh you'll still call all kinds of external commands (grep, ps, cat, etc...) part of them will have different parameters and different output from system to system. Either you'll have to take in account those differences or use the GNU version of each one of them to make things the same.
The Perl programming language originally was designed exactly to overcome this problem.
It includes all the features a unix shell programmer would want from he shell program but
it is the same on every Unix system. You might not have the latest version on all those
systems, but if you need to install something, maybe it is better to install perl.
The reason for the differences is whether the inside block is executed in the original shell context or in a subshell. You may be able to control this with the () and {} grouping commands. Using a temporary file, as you do in your update, will work most of the time but will run into problems if the script is run twice rapidly, or if it executes without clearing the file, etc.
#!/bin/ksh
flag=false
echo "a\nb" | { while read x
do
flag=true
done }
echo "flag = ${flag}"
exit 0
That may help with the problem you were getting on the Linux ksh. If you use parentheses instead of braces, you'll get the Linux behavior on the other ksh implementations.
Here is the another solution for echo "\n" issue
Steps:
Find ksh package name
$ rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE}(%{ARCH})\n" | grep "ksh"
ksh-20100621-19.el6_4.3(x86_64)
uninstall ksh
$ sudo yum remove ksh-20100621-19.el6_4.3.x86_64
down load pdksh-5.2.14-37.el5_8.1.x86_64.rpm (Please check OS for 32-bit or 64-bit and choose correct pkg)
Install pdksh-5.2.14-37.el5_8.1.x86_64.rpm
$ sudo yum -y install /SCRIPT_PATH/pdksh-5.2.14-37.el5_8.1.x86_64.rpm
Output before PDKSH install
$ ora_db_start_stop.sh
\n==============
Usage: START
==============\n\n
./ora_db_start_stop.sh START ALL \n
OR \n
./ora_db_start_stop.sh START ONE_OR_MORE \n
\n==============
Usage: STOP
==============\n\n
./ora_db_start_stop.sh STOP ALL \n
OR \n
./ora_db_start_stop.sh STOP ONE_OR_MORE \n\n
After PDKSH install
==============
Usage: START
./ora_db_start_stop.sh START ALL
OR
./ora_db_start_stop.sh START ONE_OR_MORE
==============
Usage: STOP
./ora_db_start_stop.sh STOP ALL
OR
./ora_db_start_stop.sh STOP ONE_OR_MORE
I don't know of any particular option to force ksh to be compatible with a particular older version. That said, perhaps you could install a very old version of ksh on your linux box, and have it behave in a compatible manner?
It might be easier to install a more modern version of amy shell on the AIX/HP-UX boxes, and just migrate your scripts to use sh. I know there are versions of bash available for all platforms.
Your script gives the correct (true) output when zsh is used with the emulate -L ksh option. If all else fails you may wish to try using zsh on Linux.