I am in the process of answering a request for tender on a contract that requires a decent amount of text processing. The main trouble is that the customer wants to be able to run this on any UNIX (HPUX, Solaris, AIX, FreeBSD) or Linux (SLES, RHEL) platform, which may constrain what I use to do it. They don't want to make the installation of extra tools a pre-requisite.
I'm torn between Perl and awk. I know that Perl is an ideal tool for processing text (and I'm reasonably adept at it) but, before I put in the RFT response that Perl will be required, I'd like to find out if anyone is running on a platform where Perl is not installed by default.
It'd be handy to list those platforms in the RFT and give the client the option of which way they want to go. I have a vague recollection that it's not on FreeBSD in the default install and it may also be that the non-Linux platforms also don't have it.
Any other tools can be suggested but, given my familiarity with Perl and awk, they'll probably be the only ones on the shortlist.
You can get a version of Perl compiled for just about any unix variant. Perl doesn't have to be 'installed', but can run inside of your application's directory. I would bundle Perl with my distribution, so you can ensure you're running the same version.
It is very difficult to write a completely cross platform shell script, without testing on the target OS. If you develop an awk script, you are probably going to develop using the GNU variant on Linux, which is a superset of the POSIX awk. I'm often configuring opening source packages on Solaris, and I'm constantly finding issues where people assume your running a modern version of a tool. For example, on Solaris bash is not the standard bourne shell (/bin/sh), and echo does not take any parameters. If you do try to code with POSIX awk, you may find yourself limited by the regex library, or out dated conventions.
Perl's artistic license allows you to bundle it with your program, as long as you follow a couple of simple things like keeping the copyright in tact.
Almost every *nix (except some for very limited disk space) has Perl installed. AFAIK, even FreeBSD. Just in case it isn't, you can transform Perl program into an executable that will not need perl with PAR::Packer.
If the client does not have Perl on their machines you can always use Par::Packer to create an executable for that platform. This also means that you don't have to worry about using modules, since they will be included in the executable as well.
even if Perl might be installed on almost every *nix platforms, they might not be the same version , so be aware of this. With the requirement that it needs to works on most *nix, you can just code with shell+ utilities. For parsing files, awk+shell can do the job as well. you just have to write it in a "portable" format. check this out for more info
Although I would guess that all current versions of the operating systems you mention install Perl, there will of course be older versions around that didn't and haven't. You should also be aware that even tools like awk were not routinely installed on very old UN*X versions, as it and other programming tools were optional extras (at extra cost). I remember Altos systems where even the TCP/IP stack was an extra cost item, but presumably you won't be going that far back :-)
Bottom line: If your app really needs Perl you should check it is installed (via a Bourne shell script - if theat doesn't work you are really screwed) and if not provide some way of installing it.
not all has perl :)
mmd-sparc% cat /etc/motd
Sun Microsystems Inc. SunOS 5.6 Generic August 1997
mmd-sparc% which perl
no perl in /usr/bin .
mmd-sparc%
mmd-sparc% uname -a
SunOS mmd-sparc 5.6 Generic_105181-05 sun4m sparc SUNW,SPARCstation-5
mmd-sparc% date
Mon Jun 3 10:11:22 EDT 2019
Related
I have been looking for a linux distribution that is not for embedded systems and does not use many of the GNU utilities found in many popular distributions. I want to develop a (pet project) linux distribution that uses musl-libc, bsd userland, and Plan 9 from user space. Before I start and possibly waste time doing the impossible, is it feasible/practical to use the BSD userland as a replacement for GNU coreutils? If not, what is an alternative?
Your goal appears to be much close to stali project (the only difference is the BSD userland requirement).
http://sta.li/
I don't know much about the stage of this project, but you can get some help in the project mailing-list.
As far as I know, the BSD tools uses a lot of direct syscalls and little usage of the POSIX API. I don't believe that bsd guys had written code using a lot of #ifdefs to get fully compliant programs (but I can be wrong)...
The suckless site ported the plan9 userland to unix (based on plan9port too), it's called 9base (and is available on archlinux repo to install).
I think you'll have the same problem I had in the past trying to assembly a similar distro: Too much effort to get rid of GNU... The base system is easy, but for something useful you'll need a C compiler and then you're out of good alternatives. GCC is gnu and have dozens of gnu dependencies and the sane freebsd gcc port never will work on linux for obvious reasons.
My current try is help finish the ken-c (or 9-cc) port for linux.
I'm considering a switch to BSD. I'm interested in what tool(s) are used in BSD for scripting of the kind done with applescript in OSX.
Although shell scripts are at the core of scripting in the different BSDs, much is done with Perl, Python, Ruby and Lua. I'm sure there are plenty more options, it really depends on what you like and possibly what comes with your distribution.
OpenBSD comes with Perl in the base, NetBSD has Lua now partly integrated into the kernel but I don't believe FreeBSD has anything but shell scripting available in a default install.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
What Linux shell should I use?
I am starting to get proficient in a Linux environment and i'm trying to pick a weapon of choice in terms of command shell scripting (as i'm still a big n00b at this) that will help me (and others) manage, test and administer a set of server side applications running on a *NIX environment.
My question is: What is(are) the preferred command shell(s) out there when the following criteria are considered:
How easy is it to learn/understand for a junior dev who has never had an exposure to shell scripting?
Is there a big pool of developers out there that know this shell script?
Is it safe and easy to use - will script errors be silent or give intelligent error output, will it let the uninitiated shoot them selves in the foot?
How portable is it? - Can i expect the same script to run in OpenSolaris as well as Redhat, FreeBSD? (granted command syntax and options for specific OS will change accordingly)
How standard is it? Is it expected to be included on most distro's of *NIX or does it have to be installed additionally?
I understand that there are camps out there who hold strong feelings for/against specific command shells, i am just looking for an informed opinion.
These days, just about any non-embedded (or large embedded) operating system has a POSIX:2001 a.k.a. Single Unix v3 compatibility layer. This is native on unix platforms (Linux, Mac OS X, Solaris, *BSD, etc.) and installable on other platforms such as Windows and Android. POSIX specifies a shell language, usually known as POSIX sh. This language is derived from the Bourne shell.
Most unix systems have one of two implementations of POSIX sh: ksh or bash, which have additional useful features compared to POSIX. However some less mainstream systems (especially embedded ones) may have only POSIX-mandated features.
Given your objectives, I see three choices:
Restrict yourself to POSIX sh. Pro: you don't have to worry about differing variants, since there's a standard and compliant implementations are readily available. Con: you don't benefit from bash and ksh's extensions.
Use the intersection of ksh and bash. This is attractive in appearance, but it does mean you have to use two reference documents instead of just one — and even the features that bash and ksh have in common don't always use the same syntax. Figuring out which one you want to use on a given system is also a pain.
Choose one of ksh or bash. Both bash and ksh are available on all unix-like platforms and on Windows. Both have an open source implementation (the only one for bash, ATT ksh93 for ksh) that can be installed on most platforms. I'd go for bash over ksh for two reasons. First, it's the default on Linux, so you'll find more people who're used to it. Second, there are systems that come with an older, less-featured implementation of ksh; even if you can install ksh93, it's another thing you have to think about when deploying.
Forget about csh for scripting, and forget about zsh if you want common default availability.
See also What are the fundamental differences between the mainstream *NIX shells?, particularly the “for scripting” part of my answer.
Note that shell programming involves other utilities beyond the shell. POSIX specifies those other utilities. “Bash plus other POSIX utilities” is a reasonable choice, distinct from “POSIX utilities (including sh)”.
csh is almost always wrong.
Z shell (zsh)
It's said zsh is the most powerful for now so I would recommend trying it.
No matter which shell you learn - their syntax is very similar. Only built-in commands may slightly differ. But don't choose those old and unmaintained.
Bash is the most popular. But almost every command in bash works in zsh the same way. There are some exceptions of course.
AFAIK, every shell handles it the same way. But be warned - shells are stupid, they are not as smart as programming languages.
I saw zsh working on all Linuxes, FreeBSD and OpenSolaris.
See 4. Distros have zsh in their repos.
Why I prefer zsh (Z shell) to bash:
files matching like this: for file in ./**/*.java; do ... (I mean ./**/*.ext)
wants me to confirm when I do rm * :)
tab-autocompletion is a lot better, I can write dmdomi[tab] and it suggests dnddomainname. java wants class name as the first parameter, zsh will suggest all classes available in the package and all subpackages.
But you are not limited to zsh only. If something does not work for you, you just write it in bash or sh. This is what is "#!/bin/bash" on top of the script for. :-)
To start quickly, use my .zshrc config: http://www.rozne.geozone.pl/.zshrc The only thing you should change there is export LANG="pl_PL.UTF-8". You probably don't want Polish locale.
Shell scripts for any *nix shell are generally deceptively simple. Easy things are usually easy, sometimes hard things are easy, sometimes easy-seeming things are hard. No shell is particularly better than the others in this area but some are worse (I can't seriously recommend csh). Some will say that bash is the worst 'modern' shell, which may be true but you can't completely escape it anyway.
There's an argument to be made that using the most 'popular' shell is best for maintainability for the same reason Windows is best (and I'm not saying that it is): It's easy to find people you can hire who know how to use it. There are simply more people who have at least a passing familiarity with bash-specific features, say, than ksh or zsh. Finding people who actually understand what they're doing is another matter.
All shells have various gotchas, corner-cases and weird behaviors. Mostly it comes down to what you're used to. Shooting yourself in the foot is what I'd call a grand Unix tradition and no *nix shell can truly keep you safe.
Nearly every shell you'll see is highly portable to almost every platform. Even though this is true you won't necessarily be able to run the same (say) bash script on three different boxes unless you were careful about what utilities you used and which options you passed them. Writing portable shell scripts is hard for reasons having nothing to do with which shell they're written for.
Nearly every Linux uses bash by default and has most shells available. FreeBSD includes sh, csh and tcsh by default with bash and others in ports. Once upon a long time ago, Mac OS X used tcsh by default, but it now uses bash by default, and includes zsh along with most common shells. Beyond that I cannot comment.
Personally I use bash out of (mostly) inertia. If I weren't so familiar with it already I would use zsh instead.
bash is the standard and is very good at interactive use (good completion supporting many programs, history, readline support, many kinds of string expansion). It is also good at scripting, for a shell (arrays and hashes, quoting, string manipulation); though writing reliable scripts requires you to learn a lot more.
If you want your programs to be able to grow, work with elaborate data structures, and use some useful libraries, you should learn a language like python, ruby or perl. Most of those have interactive interpreters as well, not as convenient as a shell but useful for quick testing. IPython, for Python, is particularly useful; it lets you explore documentation very easily, can load and reload source, includes a debugger. It also includes some standard shell commands and can pass the rest to a standard shell by prefixing them with a !.
Thanks to being interactive most shells are easy enough to learn once you start using them exclusively
I believe bash, and the posix subset, is better known by a wide margin. But the languages I mentioned are as well known as many shells.
You can easily shoot yourself in the foot, convenience often makes undesirable things easy.
and 5. Portability of the shell itself shouldn't be a problem; you may need to recompile to get more modern features on some of the OSes you mention. Using a full-blown language with its own libraries will help smoothe the variation of your multiplicity of platforms.
Ok this is a 2 parter.
First of all, this may sound stupid, but are all Unix Commands on Linux Distros? Does any Linux Distro add any.
And commands like....pwd or......ssh, like what are these "written" in programming language wise. C?
Second part, What is a good book that lists and explains all the Unix commands (or at least most of them)
thanks
The biggest difference you will find
between the Unix (and its
derivatives) and Linux commands are
the options that exist for those
commands. Specifically, the GNU
versions tend to add more
functionality to their commands but
their use is not portable to non-GNU
systems like BSD or Unix.
A common example of this would be
the GNU sed -i option which allows
you to do an in-place edit on the
file. Non-GNU versions of sed
require you to first write to a temp
file then copy the temp file over
the original.
In order to help define what is
portable code and what is not, there
are the
POSIX
and
SUS
specifications. Any shell which
claims to be POSIX compliant
guarantees that any code written
against the POSIX standard will
perform identically. The bash
shell is a POSIX compliant shell but
that doesn't mean all bash
commands are POSIX compliant.
For your second question, yes, most
commands in binary form were
written in the C language.
Sometimes you'll find certain
commands that are written in
scripting languages such as perl,
python or bash. The rename
command is one example that exists
in both perl and binary form
written from C.
For your 3rd question on Book
Recommendations, see this SO
question
Not all linux distributions come with one and the same set of commands. There is, however, The Single UNIX® Specification, which lists canonical commands and utilities.
In short: The majority of programs is written in C.
Again, take a look at The Single UNIX® Specification to get an overview. Once you have opened a terminal with a shell, just press Tab - this will yield a list of commands on your machine.
$ # press
Display all 3358 possibilities? (y or n)
Most commands have man pages, too, if you want to know more about them.
$ man pwd
First off, I think it would be better to refer to shells here, since each *nix distro can come with numerous shells. To answer your questions:
Many commands are available on all shells, but of course each shell can add their own commands. Additionally, many programs can add their own shell accessible programs.
I found the O'Reilly book on learning unix shells to be extremely useful.
I learned much of the basics from http://linuxcommand.org/. Maybe that will help you and the price is right
And visiting it just now I see that the author has compiled it into book for free under a CC licence. Check out the description and a download link here.
1. Yes / C. 2. O'Reilly, and Apress
I. Common Commands
There is a core set of commands that true Unix and all of the Linux distros have.
For some time, the distributions have been evolving to install a set of packages, and since there are thousands and thousands to choose from, the idea of a "base system" has given way to that of a "default install". For example, some can be installed without X or graphics.
Some of the original Unix commands or rewrites of them are available with BSD licenses, and you will run into many of those on true Unix, *BSD, and on the Mac. There are also a set of GPL-licensed rewrites of the Unix commands. These tend to be quite elaborate, leading to confusing manpages, but sometimes they are considerably more useful than the original Unix program. Groff is a good example of a case where the original code has been completely superseded. In any case, Linux distros stick as much as possible to the GPL/GNU versions.
Language: The core commands are almost entirely written in C. As the C++ and C compilers have merged, a few packages are written in C++. A few are written in Posix shell code. As the distributions have grown in scope and size, more packages that are included are written in languages like Perl and Python. I presume you may see packages written in Ruby as well. (Ruby comes with the Mac now.)
II. Books
O'Reilly and Apress are two good sources for Unix and Linux publications.
III. Well, there wasn't a III but Mac OS X is built on a version of Unix. You can learn Unix perfectly well in the Terminal app on your Mac.
Just out of curiosity: What's the oldest code/package in a typical linux distro? Emacs? GCC?
Sun RPC is pretty old, and it's in the C library:
http://blogs.oracle.com/webmink/entry/old_code_and_old_licenses
In user space, there's a lot of code in coreutils that was written by RMS, which has not changed much since. The GNU project started out with lots of people writing free replacements to the UNIX core utilities (i.e. ls, cp, rm, etc), before the use of a window manager was even contemplated.
If you grab the source to the package, you'll find a lot of it.
CC the compiler.
With its built in back door for building the login command.
I would say Paul Vixie Cron and the SYS V utilities (init) are some of the oldest stand-alone packages. They are at long last replaced by upstart and other dependency- and event-based schemes.
I guess this will be gcc as it was one of the first products from GNU and existed long before linux came.
My guess would also be the gcc (GNU C Compiler).
Still in daily use, too. As well as Emacs (no VI flaming necessary, thanks).
So far as I know, the BSD fstab.h is the oldest code on a typical system; it is copyrighted 1980.