I have been working on a shell script to automate some tasks. What is the best way to make sure the shell script would run without any issues in most of the platforms. For ex., I have been using echo -n command to print some messages to the screen without a trailing new line and the -n switch doesn't work in some ksh shells. I was told the script must be POSIX compliant. How do I make sure that the script is POSIX compliant. Is there a tool? Or is there a shell that supports only bare minimum POSIX requirements?
POSIX
One first step, which gives you indications of what works or not and why, is to set the shebang to /bin/sh and use shellcheck site to analyze your script.
For example, paste this script in the shellcheck editor window:
#!/bin/sh
read -r a b <<<"$1"
echo $((a+b))
to get an indication that: "In POSIX sh, here-strings are undefined".
As a second step, you can use a shell that is as compatible with POSIX as possible.
One shell that is compatible with most other simple shells, is dash, Debian default system shell, which is a derivative of the older BSD ash.
Another shell compatible with posix is posh.
However, dash and/or posh may not be available for some systems.
There is lksh (with a ksh flavor), with the goal to be compatible with legacy (old) shell scripts. From its manual:
lksh is a command interpreter intended exclusively for running legacy shell scripts.
But there is the need to use options when calling lksh, like -o posix and -o sh:
Note that it's strongly recommended to invoke lksh with at least the -o posix option, if not both that and -o sh, to fully enjoy better compatibility to the POSIX standard (which is probably why you use lksh over mksh in the first place) or legacy scripts, respectively.
You would call lksh -o posix -o sh instead of the simple lksh.
Using options is a way to make other shells become POSIX compatible. Like lksh, using the option -o posix, like bash -o posix.
In bash, it is even possible to turn the POSIX option inside an script, with:
shopt -o posix # also with: set -o posix
It is also possible to make a local link to bash or zsh that makes both act like an old sh shell. Like this:
$ ln -s /bin/bash ./sh
$ ./sh
There are plenty of alternatives (dash, posh, lksh, bash, zsh, etc.) to get a shell that will work as a POSIX shell.
Portable
However, even so, all the above does not ensure "portability".
Unfortunately, making a shell script 'POSIX-compliant' is usually easier than making it run on any real-world shell.
The only real-world sensible recommendation is test your script in several shells.
Like the list above: dash, posh, lksh, and bash --posix.
Solaris is a world on its own, probably you will need to test against /bin/sh and xpg4/sh.
Followup:
How can I test for POSIX compliance for shell scripts?
Starting Bash with the --posix command-line option or executing ‘set -o posix’ while Bash is running will cause Bash to conform more closely to the POSIX standard by changing the behavior to match that specified by POSIX in areas where the Bash default differs.
Reference
Note:
This answer complements user8017719's great answer.
As requested in the question, a tool is discussed below: while it does not directly check for POSIX compliance, it runs a given script in multiple shells, notably including /bin/sh.
/bin/sh, the system default shell, should not be assumed to support any features other than POSIX-prescribed ones, though in practice it does, to varying degrees, depending on the specific implementation. Therefore, successfully running via /bin/sh on one platform does not guarantee that the script will work on another. Among widely used shells, dash comes closest to being a POSIX-features-only shell.
Running successfully in multiple shells is important:
if you're authoring a script that needs to be sourced in various shells.
if you know that your script will encounter only a limited set of known-in-advance shells.
For a proof-of-the-pudding-is-in-the-eating approach, consider using shall (a utility I wrote), which allows you to invoke a given script or command with multiple shells at once, with feedback about which of the targeted shells the script/command executed successfully with.
If you have Node.js installed, you can easily install it with npm install -g shall (if not, follow the above link to the GitHub repo for manual installation instructions) and then use it as follows:
shall scriptFile
or, with an ad-hoc command:
shall -c '<shell-commands>'
By default, it invokes sh, and, if installed, dash, bash, zsh, and ksh, but you can target any set of shells that you have installed by using the SHELLS environment variable.
Using the example of the echo -n command on macOS to only target shells sh and bash:
$ SHELLS=sh,bash shall -c 'echo -n hi'
✓ sh (bash variant) [0.00s]
-n hi
✓ bash [0.00s]
hi
OK - All 2 shells (sh, bash) report success.
On macOS, bash (effectively) acts as sh, and while echo -n didn't fail when used with sh, you can also see that -n wasn't recognized as an option when bash ran as sh.
Another macOS example that shows that bash permits certain Bash-specific extensions even when running as sh, such as using nonstandard [[ ... ]] conditionals (assumes that dash - which acts as sh on Ubuntu systems - was installed via Homebrew):
$ SHELLS=sh,bash,dash shall -c '[[ -n nonempty ]] && echo nonempty'
✓ sh (bash variant) [0.00s]
nonempty
✓ bash [0.00s]
nonempty
✗ dash [0.01s]
dash: 1: [[: not found
FAILED - 1 shell (dash) reports failure, 2 (sh, bash) report success.
As you can see, Bash running as sh still accepted [[ ... ]], whereas dash, which is a (mostly) POSIX-features-only shell, failed, because POSIX only mandates [ ... ] conditionals (as an alias of test ... commands).
Related
So I was trying to create a script on bash shell, I came to know that the script doesn't run on ksh or dash shells. So my question is how you make a script to run on all 3 (bash, dash & ksh) shells.
In order to write a script that is guaranteed to be portable between the various shells, the script must be POSIX Shell compliant. POSIX is a minimum set of builtins and commands that all conforming shells must support. Ash, Dash, Zsh, Bash, Ksh, etc.. are all shells capable of running scripts that are POSIX compliant.
What shells like Bash do is add nice features which make the shell more capable, like additional parameter expansions for conversion to upper/lower case, substring replacement, etc.. and new builtins like [[ ... ]] that provide regex matching capabilities, etc.. While this makes Bash more capable, it also means scripts written using "Bashisms" are no longer able to run under all other shells. Ash, Dash and other minimal shells have no idea how to handle the features added by Bash, Ksh or Zsh and therefore fail.
To write truly portable scripts, you must limit the content to that provided by the POSIX command language.
You need something file like this:
#!/bin/bash #isn't a simple comment
echo "hello bash"
#!/bin/sh #isn't a simple comment
echo "hello sh"
#!/bin/ksh #isn't a simple comment
echo "hello ksh"
( #!) it's called shebang tells the shell what program to interpret the script
called this file as you better prefert (file.bsk), but don't forget give it execute permission it with :
chmod +x file.bsk
then run ./file.bsk
Some commands or utilities are not available in all shells or they might have different behavior in different shell. If you know which command run on which shell or gives you desired output you can write shell specific commends as below
bash -c 'echo bash'
ksh -c 'echo ksh'
All other commands that are common to all shell can be written in normal way.
By running the following script in sh and ./ mode get different result on Ubuntu
#!/bin/bash
x=80
z=90
let "a=$x+$z"
echo $a
Result:
sh mode gives me "blank" output, while ./ mode yields 170.
Any explanation?
Shell Selection via Invocation Method
./mode honors the shebang, which specifies that your script be invoked with bash. By contrast, sh mode explicitly uses sh instead, ignoring the shebang. (. mode doesn't run a new shell at all, but executes the commands in the script inside the same shell you're already using interactively).
Why Shell Selection Matters
let is a bash extension for compatibility with pre-POSIX shells.
On a platform (such as Ubuntu) where /bin/sh is implemented by an ash derivative, very little functionality not specified in the POSIX sh standard is available.
To perform arithmetic in a manner compatible with all POSIX-compliant shells (from the initial 1992 publication of the POSIX sh standard), use the following syntax:
a=$(( x + z ))
I am trying to make my following script compatible with other platforms (unix) and i am not sure if it will be. Especially [[test]] and %%# symbols are compatible. At lease this script works fine on linux.
It would be great if someone (who is familiar with unix) can make some suggestions or fixes to make the following script portable across the platforms (except windows).
#!/bin/sh
INSTALL_HOME=/opt/prod/install0308
export INSTALL_HOME
export CONF_INSTALL_ARGS="-Dinstall.ext.dir=/opt/prod/installExt -Dinstall.alternateExtDir=/opt/dev/installExt/lib -Dinstall.type=OSD"
INSTALL_ALTERNATIVE_TYPES_DIR=''
if [[ ${CONF_INSTALL_ARGS} == *'-Dinstall.alternateExtDir'* ]]; then
INSTALL_ALT_TYPE_DIR_TEMP=${CONF_INSTALL_ARGS#*-Dinstall.alternateExtDir=}
INSTALL_TYPE_DIR=${INSTALL_ALT_TYPE_DIR_TEMP%%-D*}
FINAL_INST_TYPE_DIR="$(echo -e "${INSTALL_TYPE_DIR}" | sed 's/ *$//g')"
INSTALL_ALTERNATIVE_TYPES_DIR=','$FINAL_INST_TYPE_DIR
fi
TOTAL_CONF_ARGS="-Dinstall.ext.dir=${INSTALL_HOME}/lib/provider,${INSTALL_HOME}/lib/security${INSTALL_ALTERNATIVE_TYPES_DIR}"
echo $TOTAL_CONF_ARGS
This is not a compatibility problem between Operating Systems, this is a compatibility problem between Shells.
Your script has been written for bash-like shells, so you just need to replace the first line #!/bin/sh by #!/bin/bash (or any path where bash is located) for it to work on other systems (do not forget to install bash on them).
NB: This script works on your Linux with the shebang #!/bin/sh probably because your Linux distribution has chosen to replace legacy sh by a link to bash or you are explicitly running the script with bash like this: bash ./script.sh.
While I was playing around in my shell investigating the answer to this question, I noticed that, even though /bin/sh was pointing to /bin/bash on my system, the two commands behave differently. First of all, the output of
ls -lh /bin/sh
is:
lrwxrwxrwx 1 root root 4 Apr 22 2013 /bin/sh -> bash*
However, invoking the following command through /bin/sh:
/bin/sh -c "script.sh 2> >( grep -v FILTER 2>&1 )"
returns this error:
/bin/sh: -c: line 0: syntax error near unexpected token '>'
/bin/sh: -c: line 0: 'script.sh 2> >( grep -v FILTER 2>&1 )'
While running the same command through /bin/bash:
/bin/bash -c "script.sh 2> >( grep -v FILTER 2>&1 )"
executes successfully, here is the output:
This should be on stderr
For reference, here is the contents of script.sh:
#!/bin/sh
echo "FILTER: This should be filtered out" 1>&2
echo "This should be on stderr" 1>&2
echo "FILTER: This should be filtered out" 1>&2
Why do the two invocations behave differently?
bash looks at the value of $argv[0] (bash is implemented in C) to determine how it was invoked.
Its behavior when invoked as sh is documented in the manual:
If Bash is invoked with the name sh, it tries to mimic the startup
behavior of historical versions of sh as closely as possible, while
conforming to the POSIX standard as well.
When invoked as an interactive login shell, or as a non-interactive
shell with the -login option, it first attempts to read and execute
commands from /etc/profile and ~/.profile, in that order. The
--noprofile option may be used to inhibit this behavior. When invoked as an interactive shell with the name sh, Bash looks for the variable
ENV, expands its value if it is defined, and uses the expanded value
as the name of a file to read and execute. Since a shell invoked as sh
does not attempt to read and execute commands from any other startup
files, the --rcfile option has no effect. A non-interactive shell
invoked with the name sh does not attempt to read any other startup
files.
When invoked as sh, Bash enters POSIX mode after the startup files are
read
There's a long list (currently 46 items) of things that change when bash is in POSIX mode, documented here.
(POSIX mode is probably useful mostly as a way to test scripts for portability to non-bash shells.)
Incidentally, programs that change their behavior depending on the name under which they were invoked are fairly common. Some versions of grep, fgrep, and egrep are implemented as a single executable (though GNU grep doesn't do this). view is typically a symbolic link to vi or vim; invoking it as view causes to open in read-only mode. The Busybox system includes a number of individual commands that are all symlinks to the master busybox executable.
Invoking bash as sh causes it to enter posix mode after reading the startup files it would normally read (as opposed to the startup files a POSIX sh would read.) Bash has many different invocation modes. You can find out about these modes from the INVOCATION section of the manual. Here is some detail about the POSIX mode.
POSIX mode
This mode means bash will try, in various degrees, to conform to POSIX expectations. As explained here, bash has a few different invocations for this mode, with slightly different implications:
sh: Bash enters POSIX mode after reading startup files.
bash --posix: Bash enters POSIX mode before reading startup files.
set -o posix: Bash switches to POSIX mode.
POSIXLY_CORRECT: If this variable is in the environment when bash starts, the shell enters posix mode before reading the startup files, like bash --posix. If it is set while bash is running, like set -o posix.
From the Bash Reference Manual:
If Bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
Because the bash binary checks how it was invoked (via argv[0]) and enters a compatibility mode if it's being run as sh.
I have just started using Linux and I am curious how shell built-in commands such as cd are defined.
Also, I'd appreciate if someone could explain how they are implemented and executed.
If you want to see how bash builtins are defined then you just need to look at Section 4 of The Bash Man Page.
If, however, you want to know how bash bultins are implemented, you'll need to look at the Bash source code because these commands are compiled into the bash executable.
One fast and easy way to see whether or not a command is a bash builtin is to use the help command. Example, help cd will show you how the bash builtin of 'cd' is defined. Similarly for help echo.
The actual set of built-ins varies from shell to shell. There are:
Special built-in utilities, which must be built-in, because they have some special properties
Regular built-in utilities, which are almost always built-in, because of the performance or other considerations
Any standard utility can be also built-in if a shell implementer wishes.
You can find out whether the utility is built in using the type command, which is supported by most shells (although its output is not standardized). An example from dash:
$ type ls
ls is /bin/ls
$ type cd
cd is a shell builtin
$ type exit
exit is a special shell builtin
Re cd utility, theoretically there's nothing preventing a shell implementer to implement it as external command. cd cannot change the shell's current directory directly, but, for instance, cd could communicate new directory to the shell process via a socket. But nobody does so because there's no point. Except very old shells (where there was not a notion of built-ins), where cd used some dirty system hack to do its job.
How is cd implemented inside the shell? The basic algorithm is described here. It can also do some work to support shell's extra features.
Manjari,
Check the source code of bash shell from ftp://ftp.gnu.org/gnu/bash/bash-2.05b.tar.gz
You will find that the definition of shell built-in commands in not in a separate binary executable but its within the shell binary itself (the name shell built-in clearly suggests this).
Every Unix shell has at least some builtin commands. These builtin commands are part of the shell, and are implemented as part of the shell's source code. The shell recognizes that the command that it was asked to execute was one of its builtins, and it performs that action on its own, without calling out to a separate executable. Different shells have different builtins, though there will be a whole lot of overlap in the basic set.
Sometimes, builtins are builtin for performance reasons. In this case, there's often also a version of that command in $PATH (possibly with a different feature set, different set of recognized command line arguments, etc), but the shell decided to implement the command as a builtin as well so that it could save the work of spawning off a short-lived process to do some work that it could do itself. That's the case for bash and printf, for example:
$ type printf
printf is a shell builtin
$ which printf
/usr/bin/printf
$ printf
printf: usage: printf [-v var] format [arguments]
$ /usr/bin/printf
/usr/bin/printf: missing operand
Try `/usr/bin/printf --help' for more information.
Note that in the above example, printf is both a shell builtin (implemented as part of bash itself), as well as an external command (located at /usr/bin/printf). Note that they behave differently as well - when called with no arguments, the builtin version and the command version print different error messages. Note also the -v var option (store the results of this printf into a shell variable named var) can only be done as part of the shell - subprocesses like /usr/bin/printf have no access to the variables of the shell that executed them.
And that brings us to the 2nd part of the story: some commands are builtin because they need to be. Some commands, like chmod, are thin wrappers around system calls. When you run /bin/chmod 777 foo, the shell forks, execs /bin/chmod (passing "777" and "foo") as arguments, and the new chmod process runs the C code chmod("foo", 777); and then returns control to the shell. This wouldn't work for the cd command, though. Even though cd looks like the same case as chmod, it has to behave differently: if the shell spawned another process to execute the chdir system call, it would change the directory only for that newly spawned process, not the shell. Then, when the process returned, the shell would be left sitting in the same directory as it had been in all along - therefore cd needs to be implemented as a shell builtin.
A Shell builtin -- http://linux.about.com/library/cmd/blcmdl1_builtin.htm
for eg. -
which cd
/usr/bin/which: no cd in (/usr/bin:/usr/local/bin......
Not a shell builtin but a binary.
which ls
/bin/ls
http://ss64.com/bash/ this will help you.
and here is shell scripting guide
http://www.freeos.com/guides/lsst/