I infrequently have to write bash scripts for various unrelated purposes and while I usually have a good idea what commands I want in the script, I often have no idea what header to use or why I'm using one when I do find it. For example(s):
Standard shell script:
#!/bin/bash
Python:
#!/usr/bin/env python
Scripts seem to work fine without headers but if headers are the standard, there's a reason for them and they shouldn't be ignored. If it has an effect, then it's a valuable tool that could be used to accomplish more.
Minimally, I'd like to know what headers to use with MySQL scripts and what the headers do on Standard, Python, and MySQL scripts. Ideally, I'd like a generic list of headers or an understanding of how to create a header based on what program is being used.
How the Kernel Executes Things
Simplified (a bit), there are two ways the kernel in a POSIX system knows how to execute a program. One, if the program is in a binary format the kernel understands (such as ELF), the kernel can execute it "directly" (more detail out of scope). If the program is a text file starting with a shebang, such as
#!/usr/bin/somebinary -arg
or what-have-you, the kernel actually executes the command as if it had been directed to execute:
/usr/bin/somebinary -arg "$0"
where $0 here is the name of the script file you just tried to execute. (So you can immediately tell why so many scripting languages use # as a comment-starter – it means they don't have to treat the shebang as special.)
PATH and the env command
The kernel does not look at the PATH environment variable to determine which executable you're talking about, so if you are distributing a python script to systems that may have multiple versions of python installed, you can't guarantee that there will be a
#!/usr/bin/python
env, however, is POSIX, so you can count on it existing, and it will look up python in PATH. Thus,
#!/usr/bin/env python
will execute the script with the first python found in your PATH.
BASH, SH and Special Meanings for Invocation
Some programs have special semantics for how they're invoked. In particular, on many systems /bin/sh is a symlink to another shell, such as /bin/bash. While bash does not contain a perfectly POSIXLY_STRICT implementation of sh, when it is invoked as /bin/sh it is stricter than it would be if invoked as plain-old-bash.
MySQL and arg limitations
The shebang line can be length limited and technically, it can only support one argument, so mysql is a bit tricky – you can't expect to pass a username and database name to a mysql script.
#!/usr/bin/env mysql
use mydb;
select * from mytbl;
Will fail because the kernel will try mysql "$0". Even if you have your credentials in a .my.cnf file, mysql itself will try to treat "$0" as a database name. Likewise:
#!/usr/bin/mysql -e
use mydb;
select * from mytbl;
will fail because again, "$0" is not a table name (you hope).
There does not seem to be an appropriate syntax for directly executing a mysql script this way. Your best bet is to pipe the sql commands to mysql directly:
mysql < my_sql_commands
http://mywiki.wooledge.org/BashGuide/Practices#Choose_Your_Shell
When the first line of a script starts with #!, that's what's called a "shebang". When that script is run as an executable, the operating system uses that line to determine how to run the script -- that is to say, to find the program with which the script should be executed.
It's incorrect that "scripts work fine without headers" -- if you don't have a shebang line, you can't be invoked using the execve() call, which means that many (most?) programs won't be able to execute your script. Sometimes invocation from a shell will try to use that shell itself in the absence of a shebang, but you can't trust that to be the case.
(There's an exception to that -- if someone starts your script by running sh yourscript or bash yourscript, the shebang line isn't read at all, and the script they chose is used; however, running scripts this way is a bad practice, as the author typically knows better than the user what the correct interpreter is).
In short:
If you want to use modern features, and you want the user to be able to override the shell version in use by putting a different release of bash earlier in their path, use #!/usr/bin/env bash
If you want to use modern features and ensure that you always run with the system shell, use #!/bin/bash
If you're going to write your script to strictly conform with POSIX sh, use #!/bin/sh
There's not a limited list of shebang lines we can give you, since any native executable (non-script program) can be used as a script interpreter, and thus be placed in a shebang. If you created a file called myscript with #!/usr/bin/env yourprogram, gave it executable permissions, and ran ./myscript foo bar, this would result in /usr/bin/env yourprogram myscript foo bar being invoked; yourprogram would be run by /usr/bin/env (after a PATH lookup), and would be responsible for knowing what to do with myscript and its arguments.
For an extremely detailed history of shebang lines and how they work across systems both modern and ancient, see http://www.in-ulm.de/~mascheck/various/shebang/
Related
I have been working on a node script that does the installation of external dependencies and was wondering if its possible to set the PATH variable using Node.
For example, If I want to install Go using script, I can execute command brew install Go and this works but when I do export GOPATH=$PATH, this is never set.
I also tried using process.env but this also does not work. Also my assumption with process is that it must run in a sandbox. So it will not update/ set anything outside.
So the question is, can I do this? If yes, how?
It is not possible in any programming language. At least on Unix (BSD, MacOS, Illumos etc.) and Linux, I'm not sure about Windows.
As a security feature, Unix processes are designed such that they cannot touch the environment of their parent's process. Only the parent process can influence the environment of their children.
When you are on a command line you are running a shell program (either Bash or Korn Shell or TCSH or Fish etc.). When you execute something on the command line such as a node.js script or Google Chrome or Minecraft these programs will be the children to your shell program. On Unix and Linux these child programs inherit their parent's environment.
That is why you can export environment (global) variables and your programs can access them. However, this relationship is one-way and it was designed that way on purpose.
You may notice that in shell scripts you can "save" commands to set environment variables in a file. For example in Bash you sometimes see people do this:
# variables.sh
export HELLO "World"
However, this will not work if you execute the script as a script:
$ bash variables.sh
$ echo $HELLO
<--------------------- nothing is displayed here
the variable is not exported
You sometimes see people do this instead:
$ . variables.sh
$ echo $HELLO
World
And you may wonder how that works. Well.. in some shell languages the dot command imports the script instead of executing the script as a separate process. Think of it like import or require() in javascript.
Of course, Bash does not understand javascript syntax. It also does not understand C++ or Java or Ruby or PHP. It only understands Bash syntax.
So the longer answer is it is not possible in any programming language except for the language of the shell you are using (eg. if you are using Fish you need to write a script in Fish syntax or if you are using Ksh you need to write a script in Korn syntax).
I have a script; it needs to use bash's associative arrays (trust me on that one).
It needs to run on normal machines, as well as a certain additional machine that has /bin/bash 3.2.
It works fine if I declare the interpreter to be /opt/userwriteablefolder/bin/bash4, the location of bash 4.2 that I put there.. but it then only works on that machine.
I would like to have a test at the beginning of my script that checks what the interpreting shell is, and if it's bash3.2, calls bash4 $0 $#. The problem is that I can't figure out any way to determine what the interpreting shell is. I would really rather not do a $HOSTNAME based decision, but that will work if necessary (It's also awkward, because it needs to pass a "we've done this already" flag).
For a couple reasons, "Just have two scripts" is not a good solution.
You can check which interpreter is used by looking at $SHELL, which contains the full path to the shell executable (ex. /bin/bash)
Then, if it is Bash, you can check the Bash version in various ways:
${BASH_VERSINFO[*]} -- an array of version components, e.g. (4 1 5 1 release x86_64-pc-linux-gnu)
${BASH_VERSION} -- a string version, e.g. 4.1.5(1)-release
And of course, "$0" --version
This could be an option, depending on how you launch the script:
Install bash 4.2 as /opt/userwriteablefolder/bin/bash.
Use '#!/usr/bin/env bash' as the shebang in your script.
Add '/opt/userwriteablefolder/bin' to the front of PATH in the environment from which
your script is called, so that the bash there will be used if present, otherwise
the regular bash will be used.
The benefit would be to avoid having to detect the version of bash at runtime, but I realize your setup may not make step 3 desirable.
I have made Bash scripts before and they all ran fine without #!/bin/bash at the beginning.
What's the point of putting it in? Would things be any different?
Also, how do you pronounce #? I know that ! is pronounced as "bang."
How is #! pronounced?
It's a convention so the *nix shell knows what kind of interpreter to run.
For example, older flavors of ATT defaulted to sh (the Bourne shell), while older versions of BSD defaulted to csh (the C shell).
Even today (where most systems run bash, the "Bourne Again Shell"), scripts can be in bash, python, perl, ruby, PHP, etc, etc. For example, you might see #!/bin/perl or #!/bin/perl5.
PS:
The exclamation mark (!) is affectionately called "bang". The shell comment symbol (#) is sometimes called "hash".
PPS:
Remember - under *nix, associating a suffix with a file type is merely a convention, not a "rule". An executable can be a binary program, any one of a million script types and other things as well. Hence the need for #!/bin/bash.
To be more precise the shebang #!, when it is the first two bytes of an executable (x mode) file, is interpreted by the execve(2) system call (which execute programs). But POSIX specification for execve don't mention the shebang.
It must be followed by a file path of an interpreter executable (which BTW could even be relative, but most often is absolute).
A nice trick (or perhaps not so nice one) to find an interpreter (e.g. python) in the user's $PATH is to use the env program (always at /usr/bin/env on all Linux) like e.g.
#!/usr/bin/env python
Any ELF executable can be an interpreter. You could even use #!/bin/cat or #!/bin/true if you wanted to! (but that would be often useless)
It's called a shebang. In unix-speak, # is called sharp (like in music) or hash (like hashtags on twitter), and ! is called bang. (You can actually reference your previous shell command with !!, called bang-bang). So when put together, you get haSH-BANG, or shebang.
The part after the #! tells Unix what program to use to run it. If it isn't specified, it will try with bash (or sh, or zsh, or whatever your $SHELL variable is) but if it's there it will use that program. Plus, # is a comment in most languages, so the line gets ignored in the subsequent execution.
Every distribution has a default shell. Bash is the default on the majority of the systems. If you happen to work on a system that has a different default shell, then the scripts might not work as intended if they are written specific for Bash.
Bash has evolved over the years taking code from ksh and sh.
Adding #!/bin/bash as the first line of your script, tells the OS to invoke the specified shell to execute the commands that follow in the script.
#! is often referred to as a "hash-bang", "she-bang" or "sha-bang".
The shebang is a directive to the loader to use the program which is specified after the #! as the interpreter for the file in question when you try to execute it. So, if you try to run a file called foo.sh which has #!/bin/bash at the top, the actual command that runs is /bin/bash foo.sh. This is a flexible way of using different interpreters for different programs. This is something implemented at the system level and the user level API is the shebang convention.
It's also worth knowing that the shebang is a magic number - a human readable one that identifies the file as a script for the given interpreter.
Your point about it "working" even without the shebang is only because the program in question is a shell script written for the same shell as the one you are using. For example, you could very well write a javascript file and then put a #! /usr/bin/js (or something similar) to have a javascript "Shell script".
The operating system takes default shell to run your shell script. so mentioning shell path at the beginning of script, you are asking the OS to use that particular shell. It is also useful for portability.
It is called a shebang. It consists of a number sign and an exclamation point character (#!), followed by the full path to the interpreter such as /bin/bash. All scripts under UNIX and Linux execute using the interpreter specified on a first line.
Bash standards for “Bourne-Again shell” is just one type of many available
shells in Linux.
A shell is a command line interpreter that accepts and runs commands.
Bash is often the default shell in most Linux distributions. This is why bash is
synonymous to shell.
The shell scripts often have almost the same syntaxes, but they also differ sometimes. For example, array index starts at 1 in Zsh instead of 0 in bash. A script
written for Zsh shell won’t work the same in bash if it has arrays.
To avoid unpleasant surprises, you should tell the interpreter that your shell script
is written for bash shell. How do you do that?
simply begin your bash script into #!/bin/bash
Also you will see some other parameters after #!/bin/bash,
for example
#!/bin/bash -v -x
read this to get more idea.
https://unix.stackexchange.com/questions/124272/what-do-the-arguments-v-and-x-mean-to-bash .
It can be useful to someone that uses a different system that does not have that library readily available. If that is not declared and you have some functions in your script that are not supported by that system, you should declare #/bin/bash. I've ran into this problem before at work and now I just include it as a practice.
I read that the executables for the commands issued using exec() calls are supposed to be stored in directories that are part of the PATH variable.
Accordingly, I found the executables for ls, chmod, grep, cat in /bin.
However, I could not find the executable for cd.
Where is it located?
A process can only affect its own working directory. When an executable is executed by the shell it executes as a child process, so a cd executable (if one existed) would change that child process's working directory without affecting the parent process (the shell), hence the cd command must be implemented as a shell built-in that actually executes in the shell's own process.
cd is a shell built-in, unfortunately.
$ type cd
cd is a shell builtin
...from http://www.linuxquestions.org/questions/linux-newbie-8/whereis-cd-sudo-doesnt-find-cd-464767/
But you should be able to get it working with:
sh -c "cd /somedir; do something"
Not all utilities that you can execute at a shell prompt need actually exist as actual executables in the filesystem. They can also be so-called shell built-ins, which means – you guessed it – that they are built into the shell.
The Single Unix Specification does, in general, not specify whether a utility has to be provided as an executable or as a built-in, that is left as a private internal implementation detail to the OS vendor.
The only exceptions are the so-called special built-ins, which must be provided as built-ins, because they affect the behavior of the shell itself in a manner that regular executables (or even regular built-ins) can't (for example set, which sets variables that persist even after set exits). Those special built-ins are:
break
:
continue
.
eval
exec
exit
export
readonly
return
set
shift
times
trap
unset
Note that cd is not on that list, which means that cd is not a special built-in. In fact, according to the specification, it would be perfectly legal to implement cd as a regular executable. It's just not possible, for the reasons given by the other answers.
And if you scroll down to the non-normative section of the specification, i.e. to the part that is not officially part of the specification but only purely informational, you will find that fact explicitly mentioned:
Since cd affects the current shell execution environment, it is always provided as a shell regular built-in.
So, the specification doesn't require cd to be a built-in, but it's simply impossible to do otherwise.
Note that sometimes utilities are provided both as a built-in and as an executable. A good example is the time utility, which on a typical GNU system is provided both as an executable by the Coreutils package and as a shell regular built-in by Bash. This can lead to confusion, because when you do man time, you get the manpage of the time executable (the time builtin is documented in man builtins), but when you execute time you get the time built-in, which does not support the same features as the time executable whose manpage you just read. You have to explicitly run /usr/bin/time (or whatever path you installed Coreutils into) to get the executable.
According to this, cd is always a built-in command and never an executable:
Since cd affects the current shell execution environment, it is always provided as a shell regular built-in.
cd is part of the shell; an internal command. There is no binary for it.
The command cd is built-in in your command line shell. It could not affect the working directory of your shell otherwise.
I also searched the executable of "cd" and there is no such.
You can work with chdir (pathname) in C, it has the same effect.
I have just started using Linux and I am curious how shell built-in commands such as cd are defined.
Also, I'd appreciate if someone could explain how they are implemented and executed.
If you want to see how bash builtins are defined then you just need to look at Section 4 of The Bash Man Page.
If, however, you want to know how bash bultins are implemented, you'll need to look at the Bash source code because these commands are compiled into the bash executable.
One fast and easy way to see whether or not a command is a bash builtin is to use the help command. Example, help cd will show you how the bash builtin of 'cd' is defined. Similarly for help echo.
The actual set of built-ins varies from shell to shell. There are:
Special built-in utilities, which must be built-in, because they have some special properties
Regular built-in utilities, which are almost always built-in, because of the performance or other considerations
Any standard utility can be also built-in if a shell implementer wishes.
You can find out whether the utility is built in using the type command, which is supported by most shells (although its output is not standardized). An example from dash:
$ type ls
ls is /bin/ls
$ type cd
cd is a shell builtin
$ type exit
exit is a special shell builtin
Re cd utility, theoretically there's nothing preventing a shell implementer to implement it as external command. cd cannot change the shell's current directory directly, but, for instance, cd could communicate new directory to the shell process via a socket. But nobody does so because there's no point. Except very old shells (where there was not a notion of built-ins), where cd used some dirty system hack to do its job.
How is cd implemented inside the shell? The basic algorithm is described here. It can also do some work to support shell's extra features.
Manjari,
Check the source code of bash shell from ftp://ftp.gnu.org/gnu/bash/bash-2.05b.tar.gz
You will find that the definition of shell built-in commands in not in a separate binary executable but its within the shell binary itself (the name shell built-in clearly suggests this).
Every Unix shell has at least some builtin commands. These builtin commands are part of the shell, and are implemented as part of the shell's source code. The shell recognizes that the command that it was asked to execute was one of its builtins, and it performs that action on its own, without calling out to a separate executable. Different shells have different builtins, though there will be a whole lot of overlap in the basic set.
Sometimes, builtins are builtin for performance reasons. In this case, there's often also a version of that command in $PATH (possibly with a different feature set, different set of recognized command line arguments, etc), but the shell decided to implement the command as a builtin as well so that it could save the work of spawning off a short-lived process to do some work that it could do itself. That's the case for bash and printf, for example:
$ type printf
printf is a shell builtin
$ which printf
/usr/bin/printf
$ printf
printf: usage: printf [-v var] format [arguments]
$ /usr/bin/printf
/usr/bin/printf: missing operand
Try `/usr/bin/printf --help' for more information.
Note that in the above example, printf is both a shell builtin (implemented as part of bash itself), as well as an external command (located at /usr/bin/printf). Note that they behave differently as well - when called with no arguments, the builtin version and the command version print different error messages. Note also the -v var option (store the results of this printf into a shell variable named var) can only be done as part of the shell - subprocesses like /usr/bin/printf have no access to the variables of the shell that executed them.
And that brings us to the 2nd part of the story: some commands are builtin because they need to be. Some commands, like chmod, are thin wrappers around system calls. When you run /bin/chmod 777 foo, the shell forks, execs /bin/chmod (passing "777" and "foo") as arguments, and the new chmod process runs the C code chmod("foo", 777); and then returns control to the shell. This wouldn't work for the cd command, though. Even though cd looks like the same case as chmod, it has to behave differently: if the shell spawned another process to execute the chdir system call, it would change the directory only for that newly spawned process, not the shell. Then, when the process returned, the shell would be left sitting in the same directory as it had been in all along - therefore cd needs to be implemented as a shell builtin.
A Shell builtin -- http://linux.about.com/library/cmd/blcmdl1_builtin.htm
for eg. -
which cd
/usr/bin/which: no cd in (/usr/bin:/usr/local/bin......
Not a shell builtin but a binary.
which ls
/bin/ls
http://ss64.com/bash/ this will help you.
and here is shell scripting guide
http://www.freeos.com/guides/lsst/