In Linux command line, one can use either of two ways to pass options to commands. Either we can use the short option format which uses a single dash followed by a single letter, for example: -o or the long option format which uses two consecutive dashes followed by a word, for example: --option. But recently I came across some commands which in my thinking uses a 'hybrid' of both the formats, which uses a single dash followed by a word, for example: -option. Now I'm not talking about a commands where you can stick multiple short options together like ls -lisa. I'm talking about options where the word after the single dash is just one option and not multiple short options strung together.
I don't seem to understand why there's a third option. Because what I know about the Linux command line is you can have only a short form format or a long form format. Where did the third format came from?
It's actually confusing because sometimes you cannot be sure if the third format is really a dash followed by one option or a dash followed by multiple short options.
This is not a bash issue. All programs have their on way of handling the options/flags. There are many different styles:
the singe letter style with a single hyphen, for example:
ls -l
the mnemonic-style with double dashes, which seems a preference for GNU-stuff, for example, ls --size
the variable=value-style, for example dd if=file of=otherfile
options without dashes, as in tar cvzf arghive.tgz
You could even use a + instead of a - (as in date +%m).
etcetera.
It is important to understand that bash just passes these options to the programs/commands. So, in the programs you will generally see:
int main(int argc, char *argv[]){
(c-code example). In that case, argv[0] will point to the program-name (to simplify things a bit) and argv[1] will point to the first argument. Depending on the program, that may be different.
A quick scan through the built-in commands reveals that the built-ins always seem to use the minus-single letter (-a) for specifying options.
I think you are confusing which component does which part of the parsing.
The command line you type into bash gets parsed twice. First it gets parsed by bash. At this stage, spaces are used to separate the different parameters. Quotes and escapes are being taken into consideration. Wildcards are expanded, and $ variables are substituted.
At the end of this phase, we are left with a command line that has a list of strings, the first of which describes the command to be executed. At this point, bash calls execve, and passes it that list of strings.
The next phase of parsing is optional, and is up to each program to carry out. Most programs call getopt_long, a library function that parses options. The one and two dash convention you mention is applied by it (as well as it's older sibling, getopt).
It is, however, up to each program to parse its own parameters. Many programs use getopt_long, which is why you feel, correctly, that it is a standard. Some, however, do not. Those who do not follow their own way.
That's just how things are.
For your programs, you should try to use either getopt_long or some compatible solution, as that causes the least amount of confusion for users.
Related
In my problem statement I would like to replace a word with a prefix
sed 's/hello-world/'"$1"'-hello-world/g' test.sql
Here $1 is any prefix passed as parameter to the shell script
In this case in the first go it works absolutely fine.
Let's assume "prefix=new"
It replaces as new-hello-world which is a perfect output.
If i re-run the command again I get new-new-hello-world which is not intended
Run it again i would get new-new-new-hello-world which is not intended
How can we search and replace it as new-hello-world no matter how many times it is run? Using a regex is also fine.
To make it idempotent, just check first that it doesn't already match. eg:
sed "/$1-hello-world/!s/hello-world/$1-hello-world/g" test.sql
This is not particularly robust, and will fail if the original documents contains the line new-hello-world hello-world, but is probably sufficient for your needs. (You need to worry more about / characters in the prefix, so if you want a robust solution there's a fair bit of work to be done.)
I have a file whose name is -q (It looks like this is made by accident)
I want to see it’s content, so I tried these
$ cat '-q'
$ cat "-q"
$ cat $'-q'
But nothing worked.(All gives same error cat: invalid option -- 'q')
Is there any way to see it’s content?
sjsam's helpful answer offers viable solutions, but it's worth digging deeper:
Arguments passed to Unix utilities (such as cat) can be divided into two groups (using POSIX terminology):
options (e.g., -l; depending on the option's definition, possibly followed by an option-argument) - switches that modify the utility's basic behavior.
operands (e.g., a filename) - the data that the utility operates on.
POSIX requires all options, if any, to come before any operands, but GNU utilities relax this requirement to allow intermingling options and operands for convenience.
Either way, a - followed by a letter or digit generally implies an option.
So as to allow passing operands that happen to start with -, special option -- can be used to tell any utility: treat any subsequent arguments as operands, even if they look like options.
Since operand -q in this case looks like an option, preceding it with -- to mark it as an operand is a must.
As for how to pass string -q itself as an argument:
-q contains no shell metacharacters, so the shell passes it as-is to cat - no quoting needed.
Quoting ('-q', "-q", $'-q', \-q) does no harm in this case, but has absolutely no effect here, because the shell - after having applied any expansions - simply removes any quoting characters (a process called quote removal) before passing the word to cat (as Charles Duffy notes in comments on the question).
to create a file named -q do
touch -- "-q"
and to view its contents
cat -- '-q' # Used single quotes to treat anything inside as literal characters
should do it. -- means what follows should be treated as a positional parameter.
Pls take time to look at #rici's [ answer ] which reminds some substle aspects regarding single vs double quotes.
I once found a built-in command that would take a prefix as an argument and return all words that could complete that word. So for example,
>> COMMAND cali
California
calibrate
calibration
........
of course it would list a lot more words, in alphanumerical order. It was really useful, and optionally took a file other than the default to look in.
I'm not just trying to produce this behavior: there are obviously a million ways to use grep, sed, awk, perl, or INSERT TURING-COMPLETE LANGUAGE HERE to get this. I'm looking for the command.
Unfortunately it's hard to google something when you don't remember the name, but while it might not have been POSIX standard it was definitely a very common Linux utility, does anyone know what this was called?
Found it: it's called look, and it seems to have been around Unix since V7. (The man page is dated 1993!)
It does a binary search on the optional second argument to find all matches, defaulting to /usr/share/dict/words.
Not really a builtin command, but there is /usr/share/dict/* and grep:
$ grep -i '^Cali' /usr/share/dict/words
Caliban
Calibanism
caliber
calibered
calibogus
calibrate
calibration
calibrator
calibre
Caliburn
...
I am trying to add security of GET query to exec function.
If I remove escapeshellarg() function, it work fine. How to fix this issue?
ajax_command.php
<?php
$command = escapeshellarg($_GET['command']);
exec("/usr/bin/php-cli " . $command);
?>
Assume $_GET['command'] value is run.php -n 3
What security check I can also add?
You want escapeshellcmd (escape a whole command, or in your case, sequence of arguments) instead of escapeshellarg (escape just a single argument).
Notice that although you have taken special precautions, this code allows anyone to execute arbitrary commands on your server anyways, by specifying the whole php script in a -r option. Note that php.ini can not be used to restrict this, since the location of it can be overwritten with -c. In short (and with a very small error margin): This code creates a severe security vulnerability.
escapeshellarg returns a quoted value, so if it contains multiple arguments, it won't work, instead looking like a single stringesque argument. You should probably look at splitting the command up into several different parameters, then each can be escaped individually.
It will fail unless there's a file called run.php -n 3. You don't want to escape a single argument, you want to escape a filename and arguments.
This is not the proper way to do this. Have a single PHP script run all your commands for you, everything specified in command line arguments. Escape the arguments and worry about security inside that PHP file.
Or better yet, communicate through a pipe.
What idiom should one use in Bash scripts (no Perl, Python, and such please) to build up a command line for another program out of the script's arguments while handling filenames correctly?
By correctly, I mean handling filenames with spaces or odd characters without inadvertently causing the other program to handle them as separate arguments (or, in the case of < or > — which are, after all, valid if unfortunate filename characters if properly escaped — doing something even worse).
Here's a made-up example of what I mean, in a form that doesn't handle filenames correctly: Let's assume this script (foo) builds up a command line for a command (bar, assumed to be in the path) by taking all of foo's input arguments and moving anything that looks like a flag to the front, and then invoking bar:
#!/bin/bash
# This is clearly wrong
FILES=
FLAGS=
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags string
FLAGS="$FLAGS $ARG"
else
# Looks like a file, add it to the files string
FILES="$FILES $ARG"
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD="bar $FLAGS $FILES"
echo "Issuing: $CMD"
$CMD
(Note that this just an example; there are lots of other times one needs to do this and that to a bunch of args and then pass them onto other programs.)
In a naive scenario with simple filenames, that works great. But if we assume a directory containing the files
one
two
three and a half
four < five
then of course the command foo * fails miserably in its task:
foo: Handling four < five
foo: Handling one
foo: Handling three and a half
foo: Handling two
Issuing: bar four < five one three and a half two
If we actually allow foo to issue that command, well, the results won't be what we're expecting.
Previously I've tried to handle this through the simple expedient of ensuring that there are quotes around each filename, but I've (very) quickly learned that that is not the correct approach. :-)
So what is? Constraints:
I want to keep the idiom as simple as possible (not least so I can remember it).
I'm looking for a general-purpose idiom, hence my making up the bar program and the contrived example above instead of using a real scenario where people might easily (and reasonably) go down the route of trying to use features in the target program.
I want to stick to Bash script, I don't want to call out to Perl, Python, etc.
I'm fine with relying on (other) standard *nix utilities, like xargs, sed, or tr provided we don't get too obtuse (see #1 above). (Apologies to Perl, Python, etc. programmers who think #3 and #4 combine to draw an arbitrary distinction.)
If it matters, the target program might also be a Bash script, or might not. I wouldn't expect it to matter...
I don't just want to handle spaces, I want to handle weird characters correctly as well.
I'm not bothered if it doesn't handle filenames with embedded nul characters (literally character code 0). If someone's managed to create one in their filesystem, I'm not worried about handling it, they've tried really hard to mess things up.
Thanks in advance, folks.
Edit: Ignacio Vazquez-Abrams pointed me to Bash FAQ entry #50, which after some reading and experimentation seems to indicate that one way is to use Bash arrays:
#!/bin/bash
# This appears to work, using Bash arrays
# Start with blank arrays
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS+=("$ARG")
else
# Looks like a file, add it to the files array
FILES+=("$ARG")
fi
done
# Call bar with the flags and files
echo "Issuing (but properly delimited, not exactly as this appears): bar ${FLAGS[#]} ${FILES[#]}"
bar "${FLAGS[#]}" "${FILES[#]}"
Is that correct and reasonable? Or am I relying on something environmental above that will bite me later. It seems to work and it ticks all the other boxes for me (simple, easy to remember, etc.). It does appear to rely on a relatively recent Bash feature (FAQ entry #50 mentions v3.1, but I wasn't sure whether that was arrays in general of some of the syntax they were using with it), but I think it's likely I'll only be dealing with versions that have it.
(If the above is correct and you want to un-delete your answer, Ignacio, I'll accept it provided I haven't accepted any others yet, although I stand by my statement about link-only answers.)
Why do you want to "build up" a command? Add the files and flags to arrays using proper
quoting and issue the command directly using the quoted arrays as arguments.
Selected lines from your script (omitting unchanged ones):
if [[ ${ARG:0:1} == - ]]; then # using a Bash idiom
FLAGS+=("$ARG") # add an element to an array
FILES+=("$ARG")
echo "Issuing: bar \"${FLAGS[#]}\" \"${FILES[#]}\""
bar "${FLAGS[#]}" "${FILES[#]}"
For a quick demo of using arrays in this manner:
$ a=(aaa 'bbb ccc' ddd); for arg in "${a[#]}"; do echo "..${arg}.."; done
Output:
..aaa..
..bbb ccc..
..ddd..
Please see BashFAQ/050 regarding putting commands in variables. The reason that your script doesn't work is because there's no way to quote the arguments within a quoted string. If you were to put quotes there, they would be considered part of the string itself instead of as delimiters. With the arguments left unquoted, word splitting is done and arguments that include spaces are seen as more than one argument. Arguments with "<", ">" or "|" are not a problem in any case since redirection and piping is performed before variable expansion so they are seen as characters in a string.
By putting the arguments (filenames) in an array, spaces, newlines, etc., are preserved. By quoting the array variable when it's passed as an argument, they are preserved on the way to the consuming program.
Some additional notes:
Use lowercase (or mixed case) variable names to reduce the chance that they will collide with the shell's builtin variables.
If you use single square brackets for conditionals in any modern shell, the archaic "x" idiom is no longer necessary if you quote the variables (see my answer here). However, in Bash, use double brackets. They provide additional features (see my answer here).
Use getopts as Let_Me_Be suggested. Your script, though I know it's only an example, will not be able to handle switches that take arguments.
This for ARG in "$#" can be shortened to this for ARG (but I prefer the readability of the more explicit version).
See BashFAQ #50 (and also maybe #35 on option parsing). For the scenario you describe, where you're building a command dynamically, the best option is to use arrays rather than simple strings, as they won't lose track of where the word boundaries are. The general rules are: to create an array, instead of VAR="foo bar baz", use VAR=("foo" "bar" "baz"); to use the array, instead of $VAR, use "${VAR[#]}". Here's a working version of your example script using this method:
#!/bin/bash
# This is clearly wrong
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS=("${FLAGS[#]}" "$ARG") # FLAGS+=("$ARG") would also work in bash 3.1+, as Dennis pointed out
else
# Looks like a file, add it to the files string
FILES=("${FILES[#]}" "$ARG")
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD=("bar" "${FLAGS[#]}" "${FILES[#]}")
echo "Issuing: ${CMD[*]}"
"${CMD[#]}"
Note that in the echo command I used "${VAR[*]}" instead of the [#] form because there's no need/point to preserving word breaks here. If you wanted to print/record the command in unambiguous form, this would be a lot messier.
Also, this gives you no way to build up redirections or other special shell options in the built command -- if you add >outfile to the FILES array, it'll be treated as just another command argument, not a shell redirection. If you need to programmatically build these, be prepared for headaches.
getopts should be able to handle spaces in arguments correctly ("file name.txt"). Weird characters should work as well, assuming they are correctly escaped (ls -b).