How to code in the unix spirit? (small single task tools) - linux

I have written a little script which retrieves pictures and movies from my camera and renames them based on their date and then copies them on my harddrive, managing conflicts automatically (same name? same size? same md5?)
Works pretty well.
But this is ONE script.
From time to time I need to check if a picture is already hidden somewhere in a volume, so I'd like to apply the "conflict manager" only. I guess if I had properly followed the unix spirit of tiny single-task tools, I could do that.
What are the best resources, best practices and your experience on the subject?
Thank you.
Edit : Although I'd love to read unix books and have a deep understanding of the subject, I am looking for the Great Principles first. Plus I tend to limit myself to online resources.

I would look at the book called The Art of Unix Programming.

I've found that most code doesn't start out being reusable, it evolves to be. Take your existing code and factor out the "conflict manager" portion into its own function or program, then call that program instead of having it be a part of your original application. After that you'll be able to reuse that part of your code that you have a need to reuse. Sometimes it's impossible to design software up front for reusability because you simply don't know which parts you'll want to reuse.
As for resources, it seems like the store shelves are packed with books for Linux desktop users and system administrators, but it's hard to find good Linux programming books. A few good ones:
Beginning Linux Programming
Professional Linux Programming
Linux Programming by Example: The Fundamentals
The Linux Programmer's Toolbox
Lastly, Eric Raymond has made The Art of Unix Programming available online for free.

Check out this book:
The Art of Unix Programming by Eric S. Raymond
http://www.amazon.com/UNIX-Programming-Addison-Wesley-Professional-Computing/dp/0131429019
Here is his website:
http://www.catb.org/~esr/writings/taoup/

Personally, whenever I see that the script I'm planning to write will be longer than a dozen lines, I use python instead of shell script. One neat trick with python scripting is that it's very easy to program in a style where you create both "unix spirit" command-line tools and libraries. E.g. for your "conflict manager", create a file (python module) and put the functionality in functions and/or classes, and then at the end you can put a python "main" function (the usual if __name__=='__main__': dance) where you parse command line options (use the builtin OptionParser module for this, it's very nice!) and use the functionality in the functions/classes.
This way you can use the utility both as a stand-alone command line program, or you can import the module in another python script and use the functionality defined there via functions/classes rather than parsing input.

Start with wikipedia (Dataflow programming)

The book Software Tools (amazon) by Kernighan and Plauger is a classic on this subject. I think it should be required reading for any serious student of software development.

- the art of UNIX programming - is quite a nice book ok "the unix way", in so far as one exists. OTOH if the way is "do as little work as gets your job done", you may already be there. :)

I think some of the keys for good gnu code, are:
Handling the system signals properly, like deattaching hard drive files if SIGTERM is received.
Proper use of pipes and standard input/output
Follwing common command line flag rules
I would also recommend this book. Pretty old, but I think is quite clear explaining the principles of unix.

Related

How to proceed with Linux source code customization?

I am a non CS/IT student, but having knowledge of C, Java, DS and Algorithms. Now-a-days I am focusing on operating system and had gained some of its concepts. But I want some practical knowledge of it. Merely writing algo code in java/c has no fun in doing. I have gone through many articles where they mentioned we can customize source code of Linux-kernel.
I want to start customizing the kernel as I move ahead in the learning of OS concepts and apply the same. It will make two goals achievable 1. I will gain practical idea of the operating system 2. I will have a project.
Problem which I face-
1. From where to get the source code? Which source code should I download? Also the documentation if possible.
https://www.kernel.org/
I went in there but there are so many of them which one will be better?
2. How will I customize the code once I have it?
Please give me suggestions with detail about how I should start this journey (of changing source code to customize Linux).
Moreover I am using Windows 8.
I recommend first reading several books on OSes and on programming. You need a broad CS culture (if possible get a CS degree)
I am a non CS/IT student,
You'll better become one, or else spend years of work to learn all the stuff a CS graduate student has learnt.
First, you need to be very familiar with Linux programming on user side (application programs). So read at least Advanced Linux Programming and study the source code of several programs, including shells (and some kind of servers). Read also carefully syscalls(2). Explore the state of your kernel (e.g. thru proc(5)...). Look into https://kernelnewbies.org/
I also recommend learning several programming languages. You should in particular read SICP, an excellent introduction to programming. Read also some book like programming language pragmatics. Read something about continuation and continuation passing style. Read the Dragon book. Read some Introduction to Algorithms. Read something about computer architecture and instruction set architecture
Merely writing algo code in java/c has no fun in doing.
But the kernel is also written in C (mostly) and full of algorithmic code. What makes you think you'll get more fun in it?
I want to start customizing the kernel as I move ahead in the learning of OS concepts and apply the same.
But why? Why don't you also consider studying and contributing to some user-level code
I would recommend first reading a good book on OSes in general, notably Operating Systems: Three Easy Pieces. Look also on OSdev.
At last, the general advice about kernel programming is don't. A common mistake is to try adding code inside the kernel to solve some issue that can and should be solved in user-land.
How will I customize the code once I have it?
You probably should not customize the kernel, but if you did you'll use familiar tools (a good source code editor like emacs or vim, a compiler and linker on the command line, a build automation tool like make). Patching the kernel is similar to patching some other free software. But testing your kernel is harder (because you'll often reboot).
You'll also find several books explaining the Linux kernel.
If you still want to customize the kernel you should first try to code some kernel module.
Moreover I am using Windows 8.
This is a huge mistake. You first need to be an advanced Linux user. So wipe out Windows from your computer, and install some Linux distribution -I recommend Debian- (and use only Linux, no more Windows). Become familiar with command line.
I seriously recommend to avoid working on the kernel as your first project.
I strongly recommend looking at some existing user-land free software project first (there are thousands of them, notably on github, e.g. choose some package in your distribution, study its source code, work on it, propose the patch to the community). Be able to build from source code a lot of things.
A wise man once said you "must act your way into right thinking, as you cannot think your way into right acting". In your case, you'll need to act as an experienced programmer would act, which means before we write any code, we need to answer some questions.
What do we want to change?
Why do we want to change it?
What are the repercussions of this change (ie what other functions - out of all the 10's of millions of lines of source code - call this function)?
After we've made the change, how are we going to compile it? In other words, there is a defined process for this. What is it?
After we compile our new kernel/module, how are we going to test it?
A good start, in addition to the answer that was just posted, would be to run LFS (Linux from Scratch). Get a successful install of that and use it as a starting point.
Now, since we're experienced programmers, we know that tinkering with a 10M+ line codebase is a recipe for trouble; we need a bit more direction than that. Here's a list of bugs that need to be fixed: https://bugzilla.kernel.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=7d
I, for one, would be glad to see the one called "AUFS hangs on fanotify" go away, as I use AUFS with Docker on a daily basis.
If, down the line, you decide you'd rather hack on something besides the kernel, there are plenty of other options.
From your question it follows that you've already gained some concepts of an operating system. However, if you feel that it's still insufficient, it is OK to spend more time on learning. An operating system (mainly, a kernel) has certain tasks to perform like memory management (or memory protection), multiprogramming, hardware abstraction and so on. Neither of the topics may be neglected - they are all as important. So, if you have some time, you may refer to such useful books as "Modern Operating Systems" by Andrew Tanenbaum. Special books like that will shed much light on all important aspects of a modern OS. Suffice it to say, Linux kernel itself was started by Linus Torvalds because of a strong inspiration by MINIX - an educational project by A. Tanenbaum.
Such a cumbersome project like an OS kernel (BSD, Linux, etc.) contains lots of code. Many people are collaborating to write or enhance whatever parts of the kernel. So, there is a common and inevitable need to use a version control system. So, if you have an intention to submit your code to the kernel in future, you also have to have hands on with version control. Particularly, Linux relies on Git SCM (software configuration management - a synonym for version control).
So, once you have some knowledge of Git, you can install it on your computer and download Linux source code: git clone https://github.com/torvalds/linux.git
Determine your goals at Linux kernel modification. What do you want to achieve? Perhaps, you have a network card which you suspect to miss some features in Linux? Take a look at the other vendors' drivers and make an attempt to fix the driver of interest to include the features. Of course, this will require some knowledge of the HW, and, if the features are HW dependent, you will unlikely succeed to elaborate your code without special knowledge. But, in general, - if you are trying to make an enhancement, it assumes that you are an experienced Linux user yourself. Otherwise, how will you understand that some fixes/enhancements/etc. are required? So, I can't help but agree with the proposal to postpone Windows 8 for a while and start using some Linux distribution (eg. Debian).
If you succeed to determine your goals (eg. if you find a paper describing some desired changes in Linux kernel or if you decide to enhance some device drivers / write your own), you will be able to try it hands on. However, you still might need some helpful books, but, in this case, some Linux-specific ones. Also, writing C code for the kernel itself will require one important detail - you will need to comply with a so called coding standard, otherwise Linux kernel maintainers will not be able to accept your patches.
So, I made an attempt to outline some tips based on your current question. Of course, the job of kernel development has far more broad prerequisites, but these are which are just obvious.

Platform for creating a visual programming language

I'm interested in creating a visual programming language which can aid non-programmers(like children) to write simple programs, much like Labview or Simulink allows engineers to connect functional blocks together without the knowledge of how they are internally built. Is this called programming by demonstration? What are example applications?
What would be an ideal platform which can allow me to do this(it can be a desktop or a web app)
Check out Google Blockly. Blockly allows a developer to create their own blocks, translations (generators) to virtually any programming language (or even JSON/XML) and includes a graphical interface to allow end users to create their own programs.
Brief summary:
Blockly was influenced by App Inventor, which itself was based off Scratch
App Inventor now uses Blockly (?!)
So does the BBC microbit
Blockly itself runs in a browser (typically) using javascript
Focused on (visual) language developers
language independent blocks and generators
includes a Block Factory - which allows visual programming to create new Blocks (?!) - I didn't find this useful myself...except for understanding
includes generators to map blocks to javascript/python
e.g. These blocks:
Generated this code:
See https://developers.google.com/blockly/about/showcase for more details
Best wishes - Andy
The adventure on which you are about to embark is the design and implementation of a visual programming language. I don't know of any good textbooks in this area, but there are an IEEE conference and refereed journal devoted to this field. Margaret Burnett of Oregon State University, who is a highly regarded authority, has assembled a bibliography on visual programming languages; I suggest you start there.
You might consider writing to Professor Burnett for advice. If you do, I hope you will report the results back here.
There is Scratch written by MIT which is much like what you are looking for.
http://scratch.mit.edu/
A restricted form of programming is dataflow (aka. flow-based) programming, where the application is built from components by connecting their ports. Depending on the platform and purpose, the components are simple (like a path selector) or complex (like an image transformator). There are several dataflow systems (just I've made two), some of them has no visual editor, some of them are just a part of a bigger system, and there're some which don't even mention the approach. (Did you think, that make, MS-Excel and Unix Shell pipes are some kind of this?)
All modern digital synths based on dataflow approach, there's an amazing visual example: http://www.youtube.com/watch?v=0h-RhyopUmc
AFAIK, there's no dataflow system for definitly educational purposes. For more information, you should check this site: http://flowbased.org/start
There is a new open source library out there: TUM.CMS.VPLControl. Get it here. This library may serve as a basis for your purposes.
There is Snap written by UC Berkeley. It is another option to understand VPL.
Pay attention on CoSpaces Edu. It is an online platform that enables the creation of virtual worlds and learning experiences whilst providing a more flexible approach to the learning curriculum.
There is visual coding named "CoBlocks".
Learners can animate and code their creations with "CoBlocks" before exploring and sharing them in mobile VR.
Also It is possible to use JavaScript or TypeScript.
If you want to go ahead with this, the platform that I suggest is the one used to implement Scratch (which already does what you want, IMHO), which is Squeak Smalltalk. The Squeak environment was designed with visual programming explicitly in mind. It's free, and Smalltalk syntax can learned in half an hour. Learning the gigantic class library may take just a little longer.
The blocks editor which was most support and development for microbit is microsoft makecode
Scratch is a horrible language to teach programming (i'm biased, but check out Pipes Visual Programming Language)
What you seem to want to do sounds a lot like Functional Block programming (as in functional block programming language IEC 61499 and other VPLs for mechatronics development). There is already a lot of research into VPLs so you might want to make sure that A) what your are trying to do has an audience and B) what you are trying to do can be done easily.
It sounds a bit negative in tone, but a good place to start to test the plausibility of your idea is by reading Davor Babic's short blog post at http://blog.davor.se/blog/2012/09/09/Visual-programming/
As far as what platform to use - you could use pretty much anything, just make sure it has good graphic libraries (You could use Java with Swing - if you like pain - or Python with TKinter) just depends what you are familiar with. Just keep in mind who you want to eventually launch the language to (if its iOS, then look at using Objective-C, etc.)

An Emacs alternative which exposes a text model to a scriptable environment?

everybody out there!
I'm currently seeking for a technical solution to create a nice literate programming environment. Unfortunately, most editors are too much hard coded, and their functionalities just cover most famous needs, and can't cleanly cover special needs.
I came to Emacs (later after some others), but I also came to numerous troubles with Emacs (I will not talk about these, this is not the topic).
However, there is one thing I like with Emacs and which was indeed matching what I was looking for: it exposes a full text model to a scriptable environment, and the overall UI is designed so it is well suited to either graphical UIs or text UIs (because it is mostly text based). And last but not least, this is scriptable with a kind of LISP, and LISP indeed seems a good choice to me, in the area of text manipulation and interpretation.
I've searched the web for a text editor which would expose a full text model to a scriptable environment, but I have not found anything. I guess this is not an everyday request on the web, so it is probably better to ask some humans about it, better than to ask a robot.
I was, but in short, I'm looking for: an editor which exposes a full text model [*], and which exposes this model to a script engine (preferably LISP, but I would enjoy Python as well, or any others after all).
[*] Talking about text model, I mean: text attributes (optionally font face), text visibility, text read-write property, and text content iteration, at a level as low as the character basis.
Have a nice day! :)
--
Yannick Duchêne
JEdit seems to be very scriptable with Java, BeanShell, Jython and other languages targeting the JVM. Most of its functionality is implemented with OSGI plugins. If you really like LISP, maybe you could even try with Clojure! :-)
Emacs, Climacs, Portable Hemlock (and to some extent Hemlock).
I am sure there are other editors around that exposes a full text model to a script engine that are NOT in "the emacs family", but I don't know them.
Oh, yes, there's the VMS editor framework, but I cannot recall its name.
What Vatine said, plus there's a very minimal Scheme editor built into Fluxus, which I extended with Emacs key-bindings (in my personal copy), so I know it would work as something close to a stubbed implementation (if you rip out all the OpenGL stuff).
Edit:
Looks like I was working with fluxus-0.8, which doesn't even seem to be on the site anymore. If you end up needing to go that low-level to start, let me know and I'll send it over.
Not sure if this is useful, but there is a long list of Emacs-like editors: http://www.finseth.com/emacs.html
Btw., Craig A. Finseth also wrote a book on implementing an Emacs-like editor: http://www.finseth.com/craft/
The Book as PDF.
Report of an (unsuccessfully) ending quest :
Although a possible technical choice I could figure will not work for me (see later), I still point it here, if this can ever be useful to someone running UNIX-Like (I'm running Windows).
Context and state of the “ art ” : near to all (or all) so called Emacsen and Emacs clone, has nothing to compare with Emacs. They just mimics terms like major mode an minor mode, mimics key-bindings, and most of time also, the UI look and feel. But the core is not there. I've learned these are called “ Emacs Ersatz ”.
Disclaimer : for some reasons, I have not tested Climax and Hemlock, so the latter comment does not apply to these.
EFuns : the last one I came to, was EFuns, but unfortunately, I could not compile it on Windows (I suspect something is wrong with the sources, some directory are missing in the archive). Interested parties may get it here : EFuns, an Emacs-like scripted in OCaml. Fortunately for UNIX-Like users, binaries are provided (not for Windows).
Implementations List : to complete the list Rainer Joswig pointed to, here is another one, shorter, but more up-to-date : [ Sorry I can't post this link, it seems I'm not allowed to post more than one link - I'm sorry for interested parties (sad) ]

Interactive programming language?

Is there a programming language which can be programmed entirely in interactive mode, without needing to write files which are interpreted or compiled. Think maybe something like IRB for Ruby, but a system which is designed to let you write the whole program from the command line.
I assume you are looking for something similar to how BASIC used to work (boot up to a BASIC prompt and start coding).
IPython allows you to do this quite intuitively. Unix shells such as Bash use the same concept, but you cannot re-use and save your work nearly as intuitively as with IPython. Python is also a far better general-purpose language.
Edit: I was going to type up some examples and provide some links, but the IPython interactive tutorial seems to do this a lot better than I could. Good starting points for what you are looking for are the sections on source code handling tips and lightweight version control. Note this tutorial doesn't spell out how to do everything you are looking for precisely, but it does provide a jumping off point to understand the interactive features on the IPython shell.
Also take a look at the IPython "magic" reference, as it provides a lot of utilities that do things specific to what you want to do, and allows you to easily define your own. This is very "meta", but the example that shows how to create an IPython magic function is probably the most concise example of a "complete application" built in IPython.
Smalltalk can be programmed entirely interactively, but I wouldn't call the smalltalk prompt a "command line". Most lisp environments are like this as well. Also postscript (as in printers) if memory serves.
Are you saying that you want to write a program while never seeing more code than what fits in the scrollback buffer of your command window?
There's always lisp, the original alternative to Smalltalk with this characteristic.
The only way to avoid writing any files is to move completely to a running interactive environment. When you program this way (that is, interactively such as in IRB or F# interactive), how do you distribute your programs? When you exit IRB or F# interactive console, you lose all code you interactively wrote.
Smalltalk (see modern implementation such as Squeak) solves this and I'm not aware of any other environment where you could fully avoid files. The solution is that you distribute an image of running environment (which includes your interactively created program). In Smalltalk, these are called images.
Any unix shell conforms to your question. This goes from bash, sh, csh, ksh to tclsh for TCL or wish for TK GUI writing.
As already mentioned, Python has a few good interactive shells, I would recommend bpython for starters instead of ipython, the advantage of bpython here is the support for autocompletion and help dialogs to help you know what arguments the function accepts or what it does (if it has docstrings).
Screenshots: http://bpython-interpreter.org/screenshots/
This is really a question about implementations, not languages, but
Smalltalk (try out the Squeak version) keeps all your work in an "interactive workspace", but it is graphical and not oriented toward the command line.
APL, which was first deployed on IBM 360 and 370 systems, was entirely interactive, using a command line on a modified IBM Selectric typewriter! Your APL functions were kept in a "workspace" which did not at all resemble an ordinary file.
Many, many language implementations come with pure command-line interactive interpreters, like say Standard ML of New Jersey, but because they don't offer any sort of persistent namespace (i.e., when you exit the program, all your work is lost), I don't think they should really count.
Interestingly, the prime movers behind Smalltalk and APL (Kay and Iverson respectively) both won Turing Awards. (Iverson got his Turing award after being denied tenure at Harvard.)
TCL can be programmed entirely interactivly, and you can cetainly define new tcl procs (or redefine existing ones) without saving to a file.
Of course if you are developing and entire application at some point you do want to save to a file, else you lose everything. Using TCLs introspective abilities its relatively easy to dump some or all of the current interpreter state into a tcl file (I've written a proc to make this easier before, however mostly I would just develop in the file in the first place, and have a function in the application to resources itself if its source changes).
Not sure about that, but this system is impressively interactive: http://rigsomelight.com/2014/05/01/interactive-programming-flappy-bird-clojurescript.html
Most variations of Lisp make it easy to save your interactive work product as program files, since code is just data.
Charles Simonyi's Intentional Programming concept might be part way there, too, but it's not like you can go and buy that yet. The Intentional Workbench project may be worth exploring.
Many Forths can be used like this.
Someone already mentioned Forth but I would like to elaborate a bit on the history of Forth. Traditionally, Forth is a programming language which is it's own operating system. The traditional Forth saves the program directly onto disk sectors without using a "real" filesystem. It could afford to do that because it didn't ran directly on the CPU without an operating system so it didn't need to play nice.
Indeed, some implementations have Forth as not only the operating system but also the CPU (a lot of more modern stack based CPUs are in fact designed as Forth machines).
In the original implementation of Forth, code is always compiled each time a line is entered and saved on disk. This is feasible because Forth is very easy to compile. You just start the interpreter, play around with Forth defining functions as necessary then simply quit the interpreter. The next time you start the interpreter again all your previous functions are still there. Of course, not all modern implementations of Forth works this way.
Clojure
It's a functional Lisp on the JVM. You can connect to a REPL server called nREPL, and from there you can start writing code in a text file and loading it up interactively as you go.
Clojure gives you something akin to interactive unit testing.
I think Clojure is more interactive then other Lisps because of it's strong emphasis of the functional paradigm. It's easier to hot-swap functions when they are pure.
The best way to try it out is here: http://web.clojurerepl.com/
ELM
ELM is probably the most interactive you can get that I know of. It's a very pure functional language with syntax close to Haskell. What makes it special is that it's designed around a reactive model that allows hot-swapping(modifying running code(functions or values)) of code. The reactive bit makes it that whenever you change one thing, everything is re-evaluated.
Now ELM is compiled to HTML-CSS-JavaScript. So you won't be able to use it for everything.
ELM gives you something akin to interactive integration testing.
The best way to try it out is here: http://elm-lang.org/try

Which Scripting language is best? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
For writing scripts for process automisation in Linux platform, which scripting language will be better? Shell script, Perl or Python or is there anything else? I am new to all of them. So, am just thinking to which one to go for?
The answer is: Whatever best fits the job!
My rule of thumb;
Bash - for a short script that might need a for loop to do something repetitively.
Perl - anything to do with some kind of text processing or file processing, especially if it's a one off. Just do a dirty nasty perl script and be done with it
Python - If it's something you might want to do again or something very like it. Then at least you have a chance of being able to reuse the script.
Go for all three of them, start with bash/awk/sed plus fileutils (grep, find, and so on) and then move up the abstraction hierarchy with perl and python.
That way you will be able to decide for yourself which one fits your needs best. I say start with bash and friends because they are ubiquitous, some machines will not have perl or python installed and you'll feel helpless there, especially in traditional unix land (ie, not linux)
When choosing a scripting language to help automate your linux / unix environment, the most important thing in my opinion is... your replacement :-)
By which I mean the next / other sysadmins who may have to maintain your scripts. I am currently working in an environment where the lead Unix guy is a real script head, but he has mainly restrained himself to using bash, with some perl and windows vbscript thrown in for good luck. At least it has forced me to brush up my perl.
While agreeing with the other comments here, my suggestion would be to master bash - where possible do as much as possible in bash, as most people know it, and can maintain / debug it. And it will be most portable. Use with sed & awk is particularly powerful.
When you have that mastered, you can come back here and ask "What scripting language should I learn after bash?" :-)
JB
I use Perl for anything beyond extremely simple scripts.
I also 'use warnings', 'use strict', avoid backticks, call system as 'system($command, #and_args)'. And because I like it to be maintainable: IPC::Run (for pipes), File::Fu (for filenames, tempfiles, etc), YAML (for configs or misc data), and Getopt::Helpful (so I can remember what the options were.)
I think it depends on how complex the tasks are you want to automate. Personally, I've always gone with shell-scripts, which enables you to call on awk, sed, grep, find, ls, cat, etc. which can be combined together to do pretty much anything you can achieve using perl or python. On the other hand, if the processes you want to automate are complex (e.g., not just a linear sequence of steps) then you'll probably find that writing the scripts in perl or python (or even ruby!) is much quicker and makes them easier to maintain.
I'd recommend bash, awk, and sed.
bash - http://tldp.org/LDP/abs/html/
awk - http://www.uga.edu/~ucns/wsg/unix/awk/
http://www.grymoire.com/Unix/Awk.html
sed - http://www.ce.berkeley.edu/~kayyum/unix_tips/sedtips.html
http://www.grymoire.com/Unix/Sed.html
Just some ideas.
Depends on the complexity and problem domain of the task at hand.
Bash scripts are quick and dirty for simple system automation tasks. For more complex things than moving files around and running commands, I'd personally say Perl is next in line as the defacto sys-admin goto automation tool. For more focus on code reuse and readability/maintainability I'd want to step it up it up to Python or Ruby.
PHP can also be used to automate tasks, however it is not widely accepted for this purpose in my experience.
It really comes down to what language you are most interested in learning, most can be used for automation, in addition to many other things.
I prefer shell scripts only for very small tasks. Writing robust shell scripts requires a lot of knowledge about possible pitfalls, which you only learn by doing. But learning even the basics will increase your productivity a lot!
If I need to have complex logic, I usually use Python. By complex I mean anything that has more than two if -statements =)
Perl is okay for its original purpose, but be warned that many of the perlisms you learn are not applicable anywhere else.
Python and Ruby are roughly equivalent. I'd recommend you learn one of them well and check out a tutorial on the other. I prefer Python but it really comes down to personal preference.
To summarize: Learn basics of shell scripts. Learn at least Python or Ruby well.
If you want minimalistic, compact and fast solution (faster than Python/Ruby) then -> go for LUA scripting language :-)
However Lua speed & code compactness is achieved by relativelly small Lua language core, so if you want "batteries included" (aka. very big "standard" libraries) then Lua is not for you. Otherwise, guys who come from C/C++ world very enjoys Lua speed :-)
p.s.
Lua vs Ruby 1.9 benchmark (you can look also Lua Vs Python 3):
http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=lua&lang2=yarv
I have been getting Python recommended all the time. It's supposed to let you do anything. For the small tasks i use shell scripts though.
I would usually say the one you know best which can achieve the results you want. Like all religious wars, and after learning a large number of languages, you realise that you can do most things in most languages (Note I did say most). I use Perl. It is maybe not as up to date as Python or Ruby, but it does have massive library support from CPAN. And I have not found anything I can't do in it yet. When I do I will look at other languages to find out which one can fill that gap.
If I was starting today, maybe I would pick Python or Ruby, but I don't know enough about them to make a judgement call. Do any of your friends/colleagues know scripting languages. This could help you massively as the support when learning a new language is very important.
Good luck
Well, it's like this:
Perl is not the most user friendly scripting language, but it has CPAN (Comprehensive Perl Archive Network), which contains thousands of libraries that implement almost anything you may think of, and Perl is really powerful when it comes to text processing. The disadvantage would be that perl code is kinda hard to maintain (if you don't know it very well).
Python is a scripting language that is becoming more and more popular among scripters. It doesn't have a community like CPAN (yet), but it's more readable, and it's easier to maintain. It's as fast as perl.
Ruby is the newest trend in scripting languages. Ruby is full OOP, which means that everything is an object. Its advantage is that the code is very readable, and it's pretty easy to learn, if you are a beginner. The main disadvantage is its execution speed, which kinda s*x.
That depends on which type of automation you are doing like if it is testing autoamtion Perl is suggested because Perl is much powerful extension modules via CPAN, an online Perl module inventory. If you only need a handy tool to complete a simple source file, awk is very convenient. If you are planning to use the scripts to automate a big project, Perl is a better choice with more features.
Again Python was designed from the start as an object-oriented language. Perl 5 has some o-o features added on, but it looks to me like an awkward retrofit. Python has well-implemented o-o features for multiple inheritance, polymorphism, and encapsulation.n summary, it seems to me that Python dominates Perl in most applications except for fairly short shell-script sorts of applications, and there they are roughly comparable.
If I had to pick one, it would have to be AWK. It's lightweight, has a small learning curve and has many useful functions like index and substr.
Depends on what you want to do, I regularly use all of them:
Shell for simple batching of commands with perhaps a loop or an if-statement.
Perl when I'm munching files and do some text replacement and souch things.
Python when need more logic.
Under *nix you should use the right tool for the right work, which can be hard for the beginner since it's so many things to learn (after some 15 years as a *nix user I still find new things). My recommendation is to look at all the languages quickly to see what they can do, and then start with using shell for everything, when your scripts gets clunky move them to something else.
Just write your commands one after the other, put it in a file and run this file with
promp> bash file
and you have your first automation. Then learn about bash variables, loops and control structures.
I second Python - powerful, simple, performant, and... actually quite fun, compared to perl or bash. Also if you know it, you'll find other uses, it's used in a lot of projects.
And not just as a "classic" scripting language, take for example the twisted project. That's true for Perl too I guess, but I like Python better order of magnitudes myself
Bottom line though is like has been said beofre, make sure you have the right tool for the job...
If you aim at having a simple script program "controlling" another (command-line, of course) program, then you should review Tcl/Tk, especially its dialect expect - they're simple and oriented towards that goal - it's very easy to create a script that controls ftp and even does a su with them!
Awk's very nice to process text files - not as powerful as perl, yet much more simple and straightforward (and without the horrible syntax).
Of course, your mileage may vary, so I guess the best answer would be to ask you: what do you want to write scripts for? And then: Are you familiar with any language script? The answers to these questions will point you to the scripting language you should use, according to the pros/cons of each one and their main target.
On Linux? Choose your poison, basically. I like Python, others Ruby, still others Perl. Pick one and go for it. :-)
I'd say Python - it has a very high readability, it is simple( no curly brackets, key words as close to english as possible etc.) and you can do almost everything in it, from simple to very complex things. It is also popular and fun to code.
This may sound a little odd, I had been using bash for over 10 years. I have started using PHP5 and it was difficult at first, but now I have a much better reusable code base.
I wouldn't recommend it as a starting point though!

Resources