Related
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
What Linux shell should I use?
I am starting to get proficient in a Linux environment and i'm trying to pick a weapon of choice in terms of command shell scripting (as i'm still a big n00b at this) that will help me (and others) manage, test and administer a set of server side applications running on a *NIX environment.
My question is: What is(are) the preferred command shell(s) out there when the following criteria are considered:
How easy is it to learn/understand for a junior dev who has never had an exposure to shell scripting?
Is there a big pool of developers out there that know this shell script?
Is it safe and easy to use - will script errors be silent or give intelligent error output, will it let the uninitiated shoot them selves in the foot?
How portable is it? - Can i expect the same script to run in OpenSolaris as well as Redhat, FreeBSD? (granted command syntax and options for specific OS will change accordingly)
How standard is it? Is it expected to be included on most distro's of *NIX or does it have to be installed additionally?
I understand that there are camps out there who hold strong feelings for/against specific command shells, i am just looking for an informed opinion.
These days, just about any non-embedded (or large embedded) operating system has a POSIX:2001 a.k.a. Single Unix v3 compatibility layer. This is native on unix platforms (Linux, Mac OS X, Solaris, *BSD, etc.) and installable on other platforms such as Windows and Android. POSIX specifies a shell language, usually known as POSIX sh. This language is derived from the Bourne shell.
Most unix systems have one of two implementations of POSIX sh: ksh or bash, which have additional useful features compared to POSIX. However some less mainstream systems (especially embedded ones) may have only POSIX-mandated features.
Given your objectives, I see three choices:
Restrict yourself to POSIX sh. Pro: you don't have to worry about differing variants, since there's a standard and compliant implementations are readily available. Con: you don't benefit from bash and ksh's extensions.
Use the intersection of ksh and bash. This is attractive in appearance, but it does mean you have to use two reference documents instead of just one — and even the features that bash and ksh have in common don't always use the same syntax. Figuring out which one you want to use on a given system is also a pain.
Choose one of ksh or bash. Both bash and ksh are available on all unix-like platforms and on Windows. Both have an open source implementation (the only one for bash, ATT ksh93 for ksh) that can be installed on most platforms. I'd go for bash over ksh for two reasons. First, it's the default on Linux, so you'll find more people who're used to it. Second, there are systems that come with an older, less-featured implementation of ksh; even if you can install ksh93, it's another thing you have to think about when deploying.
Forget about csh for scripting, and forget about zsh if you want common default availability.
See also What are the fundamental differences between the mainstream *NIX shells?, particularly the “for scripting” part of my answer.
Note that shell programming involves other utilities beyond the shell. POSIX specifies those other utilities. “Bash plus other POSIX utilities” is a reasonable choice, distinct from “POSIX utilities (including sh)”.
csh is almost always wrong.
Z shell (zsh)
It's said zsh is the most powerful for now so I would recommend trying it.
No matter which shell you learn - their syntax is very similar. Only built-in commands may slightly differ. But don't choose those old and unmaintained.
Bash is the most popular. But almost every command in bash works in zsh the same way. There are some exceptions of course.
AFAIK, every shell handles it the same way. But be warned - shells are stupid, they are not as smart as programming languages.
I saw zsh working on all Linuxes, FreeBSD and OpenSolaris.
See 4. Distros have zsh in their repos.
Why I prefer zsh (Z shell) to bash:
files matching like this: for file in ./**/*.java; do ... (I mean ./**/*.ext)
wants me to confirm when I do rm * :)
tab-autocompletion is a lot better, I can write dmdomi[tab] and it suggests dnddomainname. java wants class name as the first parameter, zsh will suggest all classes available in the package and all subpackages.
But you are not limited to zsh only. If something does not work for you, you just write it in bash or sh. This is what is "#!/bin/bash" on top of the script for. :-)
To start quickly, use my .zshrc config: http://www.rozne.geozone.pl/.zshrc The only thing you should change there is export LANG="pl_PL.UTF-8". You probably don't want Polish locale.
Shell scripts for any *nix shell are generally deceptively simple. Easy things are usually easy, sometimes hard things are easy, sometimes easy-seeming things are hard. No shell is particularly better than the others in this area but some are worse (I can't seriously recommend csh). Some will say that bash is the worst 'modern' shell, which may be true but you can't completely escape it anyway.
There's an argument to be made that using the most 'popular' shell is best for maintainability for the same reason Windows is best (and I'm not saying that it is): It's easy to find people you can hire who know how to use it. There are simply more people who have at least a passing familiarity with bash-specific features, say, than ksh or zsh. Finding people who actually understand what they're doing is another matter.
All shells have various gotchas, corner-cases and weird behaviors. Mostly it comes down to what you're used to. Shooting yourself in the foot is what I'd call a grand Unix tradition and no *nix shell can truly keep you safe.
Nearly every shell you'll see is highly portable to almost every platform. Even though this is true you won't necessarily be able to run the same (say) bash script on three different boxes unless you were careful about what utilities you used and which options you passed them. Writing portable shell scripts is hard for reasons having nothing to do with which shell they're written for.
Nearly every Linux uses bash by default and has most shells available. FreeBSD includes sh, csh and tcsh by default with bash and others in ports. Once upon a long time ago, Mac OS X used tcsh by default, but it now uses bash by default, and includes zsh along with most common shells. Beyond that I cannot comment.
Personally I use bash out of (mostly) inertia. If I weren't so familiar with it already I would use zsh instead.
bash is the standard and is very good at interactive use (good completion supporting many programs, history, readline support, many kinds of string expansion). It is also good at scripting, for a shell (arrays and hashes, quoting, string manipulation); though writing reliable scripts requires you to learn a lot more.
If you want your programs to be able to grow, work with elaborate data structures, and use some useful libraries, you should learn a language like python, ruby or perl. Most of those have interactive interpreters as well, not as convenient as a shell but useful for quick testing. IPython, for Python, is particularly useful; it lets you explore documentation very easily, can load and reload source, includes a debugger. It also includes some standard shell commands and can pass the rest to a standard shell by prefixing them with a !.
Thanks to being interactive most shells are easy enough to learn once you start using them exclusively
I believe bash, and the posix subset, is better known by a wide margin. But the languages I mentioned are as well known as many shells.
You can easily shoot yourself in the foot, convenience often makes undesirable things easy.
and 5. Portability of the shell itself shouldn't be a problem; you may need to recompile to get more modern features on some of the OSes you mention. Using a full-blown language with its own libraries will help smoothe the variation of your multiplicity of platforms.
Ok this is a 2 parter.
First of all, this may sound stupid, but are all Unix Commands on Linux Distros? Does any Linux Distro add any.
And commands like....pwd or......ssh, like what are these "written" in programming language wise. C?
Second part, What is a good book that lists and explains all the Unix commands (or at least most of them)
thanks
The biggest difference you will find
between the Unix (and its
derivatives) and Linux commands are
the options that exist for those
commands. Specifically, the GNU
versions tend to add more
functionality to their commands but
their use is not portable to non-GNU
systems like BSD or Unix.
A common example of this would be
the GNU sed -i option which allows
you to do an in-place edit on the
file. Non-GNU versions of sed
require you to first write to a temp
file then copy the temp file over
the original.
In order to help define what is
portable code and what is not, there
are the
POSIX
and
SUS
specifications. Any shell which
claims to be POSIX compliant
guarantees that any code written
against the POSIX standard will
perform identically. The bash
shell is a POSIX compliant shell but
that doesn't mean all bash
commands are POSIX compliant.
For your second question, yes, most
commands in binary form were
written in the C language.
Sometimes you'll find certain
commands that are written in
scripting languages such as perl,
python or bash. The rename
command is one example that exists
in both perl and binary form
written from C.
For your 3rd question on Book
Recommendations, see this SO
question
Not all linux distributions come with one and the same set of commands. There is, however, The Single UNIX® Specification, which lists canonical commands and utilities.
In short: The majority of programs is written in C.
Again, take a look at The Single UNIX® Specification to get an overview. Once you have opened a terminal with a shell, just press Tab - this will yield a list of commands on your machine.
$ # press
Display all 3358 possibilities? (y or n)
Most commands have man pages, too, if you want to know more about them.
$ man pwd
First off, I think it would be better to refer to shells here, since each *nix distro can come with numerous shells. To answer your questions:
Many commands are available on all shells, but of course each shell can add their own commands. Additionally, many programs can add their own shell accessible programs.
I found the O'Reilly book on learning unix shells to be extremely useful.
I learned much of the basics from http://linuxcommand.org/. Maybe that will help you and the price is right
And visiting it just now I see that the author has compiled it into book for free under a CC licence. Check out the description and a download link here.
1. Yes / C. 2. O'Reilly, and Apress
I. Common Commands
There is a core set of commands that true Unix and all of the Linux distros have.
For some time, the distributions have been evolving to install a set of packages, and since there are thousands and thousands to choose from, the idea of a "base system" has given way to that of a "default install". For example, some can be installed without X or graphics.
Some of the original Unix commands or rewrites of them are available with BSD licenses, and you will run into many of those on true Unix, *BSD, and on the Mac. There are also a set of GPL-licensed rewrites of the Unix commands. These tend to be quite elaborate, leading to confusing manpages, but sometimes they are considerably more useful than the original Unix program. Groff is a good example of a case where the original code has been completely superseded. In any case, Linux distros stick as much as possible to the GPL/GNU versions.
Language: The core commands are almost entirely written in C. As the C++ and C compilers have merged, a few packages are written in C++. A few are written in Posix shell code. As the distributions have grown in scope and size, more packages that are included are written in languages like Perl and Python. I presume you may see packages written in Ruby as well. (Ruby comes with the Mac now.)
II. Books
O'Reilly and Apress are two good sources for Unix and Linux publications.
III. Well, there wasn't a III but Mac OS X is built on a version of Unix. You can learn Unix perfectly well in the Terminal app on your Mac.
Is there a programming language which can be programmed entirely in interactive mode, without needing to write files which are interpreted or compiled. Think maybe something like IRB for Ruby, but a system which is designed to let you write the whole program from the command line.
I assume you are looking for something similar to how BASIC used to work (boot up to a BASIC prompt and start coding).
IPython allows you to do this quite intuitively. Unix shells such as Bash use the same concept, but you cannot re-use and save your work nearly as intuitively as with IPython. Python is also a far better general-purpose language.
Edit: I was going to type up some examples and provide some links, but the IPython interactive tutorial seems to do this a lot better than I could. Good starting points for what you are looking for are the sections on source code handling tips and lightweight version control. Note this tutorial doesn't spell out how to do everything you are looking for precisely, but it does provide a jumping off point to understand the interactive features on the IPython shell.
Also take a look at the IPython "magic" reference, as it provides a lot of utilities that do things specific to what you want to do, and allows you to easily define your own. This is very "meta", but the example that shows how to create an IPython magic function is probably the most concise example of a "complete application" built in IPython.
Smalltalk can be programmed entirely interactively, but I wouldn't call the smalltalk prompt a "command line". Most lisp environments are like this as well. Also postscript (as in printers) if memory serves.
Are you saying that you want to write a program while never seeing more code than what fits in the scrollback buffer of your command window?
There's always lisp, the original alternative to Smalltalk with this characteristic.
The only way to avoid writing any files is to move completely to a running interactive environment. When you program this way (that is, interactively such as in IRB or F# interactive), how do you distribute your programs? When you exit IRB or F# interactive console, you lose all code you interactively wrote.
Smalltalk (see modern implementation such as Squeak) solves this and I'm not aware of any other environment where you could fully avoid files. The solution is that you distribute an image of running environment (which includes your interactively created program). In Smalltalk, these are called images.
Any unix shell conforms to your question. This goes from bash, sh, csh, ksh to tclsh for TCL or wish for TK GUI writing.
As already mentioned, Python has a few good interactive shells, I would recommend bpython for starters instead of ipython, the advantage of bpython here is the support for autocompletion and help dialogs to help you know what arguments the function accepts or what it does (if it has docstrings).
Screenshots: http://bpython-interpreter.org/screenshots/
This is really a question about implementations, not languages, but
Smalltalk (try out the Squeak version) keeps all your work in an "interactive workspace", but it is graphical and not oriented toward the command line.
APL, which was first deployed on IBM 360 and 370 systems, was entirely interactive, using a command line on a modified IBM Selectric typewriter! Your APL functions were kept in a "workspace" which did not at all resemble an ordinary file.
Many, many language implementations come with pure command-line interactive interpreters, like say Standard ML of New Jersey, but because they don't offer any sort of persistent namespace (i.e., when you exit the program, all your work is lost), I don't think they should really count.
Interestingly, the prime movers behind Smalltalk and APL (Kay and Iverson respectively) both won Turing Awards. (Iverson got his Turing award after being denied tenure at Harvard.)
TCL can be programmed entirely interactivly, and you can cetainly define new tcl procs (or redefine existing ones) without saving to a file.
Of course if you are developing and entire application at some point you do want to save to a file, else you lose everything. Using TCLs introspective abilities its relatively easy to dump some or all of the current interpreter state into a tcl file (I've written a proc to make this easier before, however mostly I would just develop in the file in the first place, and have a function in the application to resources itself if its source changes).
Not sure about that, but this system is impressively interactive: http://rigsomelight.com/2014/05/01/interactive-programming-flappy-bird-clojurescript.html
Most variations of Lisp make it easy to save your interactive work product as program files, since code is just data.
Charles Simonyi's Intentional Programming concept might be part way there, too, but it's not like you can go and buy that yet. The Intentional Workbench project may be worth exploring.
Many Forths can be used like this.
Someone already mentioned Forth but I would like to elaborate a bit on the history of Forth. Traditionally, Forth is a programming language which is it's own operating system. The traditional Forth saves the program directly onto disk sectors without using a "real" filesystem. It could afford to do that because it didn't ran directly on the CPU without an operating system so it didn't need to play nice.
Indeed, some implementations have Forth as not only the operating system but also the CPU (a lot of more modern stack based CPUs are in fact designed as Forth machines).
In the original implementation of Forth, code is always compiled each time a line is entered and saved on disk. This is feasible because Forth is very easy to compile. You just start the interpreter, play around with Forth defining functions as necessary then simply quit the interpreter. The next time you start the interpreter again all your previous functions are still there. Of course, not all modern implementations of Forth works this way.
Clojure
It's a functional Lisp on the JVM. You can connect to a REPL server called nREPL, and from there you can start writing code in a text file and loading it up interactively as you go.
Clojure gives you something akin to interactive unit testing.
I think Clojure is more interactive then other Lisps because of it's strong emphasis of the functional paradigm. It's easier to hot-swap functions when they are pure.
The best way to try it out is here: http://web.clojurerepl.com/
ELM
ELM is probably the most interactive you can get that I know of. It's a very pure functional language with syntax close to Haskell. What makes it special is that it's designed around a reactive model that allows hot-swapping(modifying running code(functions or values)) of code. The reactive bit makes it that whenever you change one thing, everything is re-evaluated.
Now ELM is compiled to HTML-CSS-JavaScript. So you won't be able to use it for everything.
ELM gives you something akin to interactive integration testing.
The best way to try it out is here: http://elm-lang.org/try
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
What language do you prefer for writing scripts for common tasks (backup, sync, etc.) and why? I'm not talking about programming web pages or applications.
I've came up with this question when thinking about why bash is still popular. For example Python looks more comfortable for me. Do you use just because you know it or for some special reasons?
If it's "create this directory, run this command, if that worked do run this"-level, I just use bash shell-scripts..
Anything more complicated, say something that parses the output of a command and acts upon it, becomes a Python script - I find it just as quick to write, mainly because shell scripts are difficult to debug (bash script error messages aren't exactly useful compared to Python's tracebacks..), and the end code becomes much more readable
...why bash is still popular?
Well, because Bourne Shell ( sh , and not necessarily bash ) is pretty much available in any +*n*x installation.
A good command of sh and vi its extremely helpful when connecting to remote servers via telnet/ssh
For local admin ( when you own the server ) you can use python/perl/ruby and customize them at your will. But most certainly, any day you could be asked to "quickly fix" other server where the two defaults are installed: sh+vi
That's why.
Unix has a philosophy of small tools which do one particular job and do it well. Often the easiest way to solve problems is to use a combination of such tools. Shell scripting is the king for this, no questions about that.
Of course, there's also the "when all you have is a hammer" syndrome :)
This really depends on the type of script. I am starting to use ruby for many sysadmin type tasks however bash is still my first choice for quick and dirty scripts. The advantage of bash, in my eyes, is the interactive nature of it.
To give an example. The other day I was searching for some particular values in approx 200 compressed log files, re-formatting the output and mailing the results.
It was very easy to use bash to do this iteratively, so, zcat one file piping the output to grep, retrying a few times to get the regex correct. Then take that output and reformat the result using awk, again retrying several times to get the format correct.
This process took a couple of minutes after which I wrote the bash commands into a script file, paramaterized it, wrapped a for loop around it, mailed the result and the job was done.
I find this process much simpler in bash just using command editing and retrying the regexes etc than I would in a separate script file where I have to keep editing the script and retrying etc.
G'day,
Different tasks call for different languages. I tend to use either shell, usually bash, or Perl depending on the task.
Now I'm getting more comfortable with Ruby, for those tasks that might suit an OO approach, I'll use that.
HTH
cheers,
Perl would be best in handling system administration tasks. I have never come across a *nix system that does not have Perl installed.
Python for me at the moment, I like using python because it has an interactive terminal that I can use to build up and execute the script as I go along - but I used perl in the past.
Bash, or various sh dialects in the broader sense can be assumed to be present on pretty much any unix system. Often, production Unix systems (Solaris, HP/UX, AIX etc.) have a very plain vanilla install; quite often they will not have perl or python installed. There may be company policies restricting this, so getting it installed may not be an option either. If you want something that will work on this type of platform, you will probably be limited to sh/sed/awk.
Bash is quite good for tasks that primarily involve running other commands, so you shouldn't underrate it. However, it rapidly becomes a write-only language at fairly trivial levels of complexity, so Perl or Python might be a better choice if you are programming something with a lot of internal processing.
For scheduling a backup, a bash script run from cron is quite possibly the best way to do the job. For something that involves parsing multiple log files, formatting the output to a summary status file and sending you an email notification if it notices certain types of events you might be better off with perl or python.
Bash is the preferred scripting language for these kinds of tasks. It's pretty ubiquitous, and it's intended to be a glue language, in the sense that you can glue together a bunch of commands that you would normally do in the terminal pretty much unchanged.
I use Ruby for most of my shell scripting tasks. I can never remember some of the nuanaces/gotchas of Bash scripting.
I use Ruby because I am most comfortable in it. It's one of the few languages in which I find myself struggling with the logic of my problem, rather than the syntax or restrictions of the language. Compare this to C++ or Perl, in which I get frustrated over pointers and sigils. I find recursive directory traversal and running system commands very easy to do in Ruby, e.g. using Ruby to rename files and edit their content.
I use perl, typically. The module library at CPAN makes many tasks simple. Net::SSH is a great tool for automating system administration tasks.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I use bash, and have done so for over a decade - but occasionally I wonder whether there has been any significant new developments in the world of Linux shells.
A few years back Microsoft released PowerShell, which seemed very interesting. Is there any comparable innovation going on in Linux shells?
You do realize bash 4 has very recently been released with a load of new features and language additions?
Shell options globstar (**/foo) does a recursive search, dirspell fixes typos during pathname expansion.
Associative arrays, map strings to strings, instead of just numbers to strings.
The autocd shell option allows changing directories by just typing the directory path instead of having to put cd in front.
Coprocesses
&>> and |& redirection operators that redirect both stdout and stderr
Loads of additions to existing builtins for improved scripting convenience.
Check out:
The "official" changelog: http://tiswww.case.edu/php/chet/bash/CHANGES
A short guide to some of the new features: http://bash-hackers.org/wiki/doku.php/bash4
I'd take a look at zsh or fishshell.
One of the least touted features of Bash (and several other shells) is the ability to write your own loadables, and have the shell run them as builtins.
Lets say you write the loadable 'on' .. and you want it to work like this:
on node 123 run some command
on class nodes run some command
on all nodes run some command
... etc ..
You can follow simple examples on how to write a loadable, then enable it as a bash built in via enable -f /path/to/loadable loadable_name
So in our case, enable -f /opt/bash/loadables/on on
... in your bashrc , and you've got it.
So, if you want to have bash interpret your spiffy new language natively, you would write a loadable named 'use' or 'switch_to', then modify the parser to load a different grammar / runtime if a certain environment variable was set.
I.e.:
#/bin/bash
switch_to my-way-cool-language
funkyfunc Zippy(int p) [[
jive.wassup(p) ]]
Most people are not going to want to hack their shell, however. I did want to point out that facilities exist to take Bash and make it the way you want it, without fiddling too much with core code.
See /path-to-bash-source/examples/loadables, you might be able to get that to fly where you work, since you're still using Bash.
You can run PowerShell on Linux via Pash. It uses Mono the way PowerShell uses .NET.
I think the "original improved shell" is ksh93. bash came into existence at a time when the ksh source code was proprietary; if ksh had been open-source then, it might not have been deemed necessary to have a new shell (though with the FSF you never know). ksh is worth studying, especially for its ability to be extended through C modules, but it's not a clear win over bash. bash's autocompletion is clearly superior, which may be enough to make bash a win overall. In any case bash and ksh have made substantial effort to converge, so differences are minor.
The other interesting shell is zsh, which attempts to be everything that ksh is while also including csh. Since I never saw any point or use to csh, I am not the right person to advocate for zsh. I will point out one unusual incompatibility: by default, in zsh a variable $var always expands to a single token, even if it contains spaces. This behavior is incompatible with all other sh-derived shells, and it is occasionally inconvenient, but really it makes a lot more sense than the original, and it saves a hell of a lot of quoting.
csh was the first shell to have job control, but in my mind it (and its successors) has been superseded by bash and ksh. It was never mucn fun to write scripts in.
Finally, there are many tiny shells designed for rescue floppies (!) and other Spartan environments, but it sounds like you have little interest in those.
(In the matter of innovation, I should add that more than half the scripts I used to write as shell scripts are now Lua scripts. Others could say the same for Python or Ruby, or back in the day, Perl or Tcl. So I think the real innovation is migration away from the shell for programmable interaction at the command line.)
IIRC, Powershell is Object Oriented, whereas most unix shells and utilities operate on text. On that regard, Squirrel Shell might interest you. I've never used it, though.
If you’re willing to lose sh compatibility, you could look at using a scripting language like Python or Tcl as your shell. rlwrap can be very handy if the interpreter doesn't provide line editing, command history, completion, etc.
One philosophy regarding shells is that they should primarily only be used to connect processes with files (here is one page that espouses that approach). That said, people have written some remarkably complex software using them.
Shells don't come much more inovative than the Scheme Schell. All the power of Scheme combined with the ability to run Unix commands and an embedded awk interpreter (written in Scheme, of course). The only drawback is that it needs a tiny bit of patching to build on 64 bit Linux.
It's not exactly Bourne-shell, but it's different. Of course, you have to learn Scheme - bonus!
if you like ruby, you can use rush (ruby-unix shell, not irb)
see the presentation here
http://www.slideshare.net/adamwiggins/rush-the-ruby-shell-and-unix-integration-library
or official website to see more examples
http://rush.heroku.com/