I'm trying to do variable substitution in SCons, followed immediately by characters that can be part of a variable name. If VARIABLE='first' I want "$VARIABLE.name"to expand to first.name instead of raising an error. $(VARIABLE).name and ${VARIABLE}.name didn't seem to work. Am I missing something very obvious?
It's curly braces. Notably it is also possible to use attribute access in variable substitution.
SConstruct:
class Bag:
def __init__(self, d):
self.__dict__ = d
env = Environment(FOO='foo', BAR=Bag({'baz':'quux'}))
env.Command('dummy1', None, 'echo ${FOO}.bar')
env.Command('dummy2', None, "echo $BAR.baz")
Outputs
scons: Building targets ...
echo foo.bar
foo.bar
echo quux
quux
scons: done building targets.
Related
I have two functions in Bash. One is a generic run function, that accepts an input and evaluates it, while printing the command, and testing the exit code. This is used in a large script to ensure each command executes successfully before continuing.
The second one is a complex function, that is doing some Git history parsing. The problematic line is the only one shown.
I am calling this function from a for-loop, that iterates over a list of terms to search. The issue is that spaces are not being handled correctly, when between other words. I have tried running my script though shell-checking websites, and all of the suggestions seem to break my code.
function run() {
echo "> ${1}"
eval "${1}"
# Test exit code of the eval, and exit if non-zero
}
function searchCommitContents() {
run 'result=$(git log -S'"${1}"' --format=format:%H)'
# Do something with result, which is a list of matching SHA1 hashes for the commits
echo "${result}"
}
# Main
declare -a searchContents=('foo' 'bar' ' foo ' 'foo bar')
for i in "${searchContents[#]}"
do
searchCommitContents "${i}"
done
Here is the output I get:
> result=$(git log -Sfoo --format=format:%H)
<results>
> result=$(git log -Sbar --format=format:%H)
<results>
> result=$(git log -S foo --format=format:%H)
<results>
> result=$(git log -Sfoo bar --format=format:%H)
fatal: ambiguous argument 'bar': unknown revision of path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
I tried to add additional single and double-quotes to various areas of the code, such that the 'foo bar' string would not resolve to two different words. I also tried adding an escape to the dollar sign, like so: -s'"\${1}"' based on other questions on this site.
Why are you printing result=$(? It's an internal variable, it can be anything, there is no need for it in logs.
Print the command that you are executing, not the variable name.
run() {
echo "+ $*" >&2
"$#"
}
searchCommitContents() {
local result
result=$(run git log -s"${1}" --format=format:%H)
: do stuff to "${result}"
echo "$result"
}
issue with an input that has a space in the middle.
If you want quoted string, use printf "%q" or ${...#Q} for newer Bash, but I don't really enjoy both quoting methods and just use $*. I really like /bin/printf from GNU coreutils, but it's a separate process... while ${..#Q} is the fastest, it's (still) not enough portable for me (I have some old Bash around).
# compare
$ set -- a 'b c' d
$ echo "+ $*" >&2
+ a b c d
$ echo "+$(printf " %q" "$#")" >&2
+ a b\ \ c d
$ echo "+" "${##Q}" >&2
+ 'a' 'b c' 'd'
$ echo "+$(/bin/printf " %q" "$#")" >&2
+ a 'b c' d
See these lines:
> result=$(git log -Sfoo bar --format=format:%H)
fatal: ambiguous argument 'bar': unknown revision of path not in the working tree.
Specifically this: -Sfoo bar. It should be -S"foo bar" or -S "foo bar". Because to pass an argument with spaces, we need to quote the argument. But, each time the argument pass through a command/function layer, one layer of quote ('', "") is extracted. So, we need to nest the quote.
So in this line:
declare -a searchContents=('foo' 'bar' ' foo ' 'foo bar')
change 'foo bar' to '"foo bar"' or "'foo bar'" or "\"foo bar\"".
This is a case of 2 layers nested quotes. The more the layer, the trickier it gets. Here's an example of 4 layers quotes I once did.
In some of the code I was going through I found that if was using braces {} for someplace and parenthesis (()) for some other. Can someone tell me the exact meaning and where to use which one?
if [ "$1" = "--help" ]
if (( $# != 3 ))
The bracket [ is a built-in command of the shell; you can also call it as test:
if [ a = b ]
then ...
equals:
if test a = b
then ...
The syntax of the test command is rather text-oriented (see the bash man page for details at chapter CONDITIONAL EXPRESSIONS).
The braces {…} are shell syntax and used for grouping commands (without creating a subshell):
{ date; ls; echo $$; } > 1>&2
This will execute date, ls, and echo $$ and redirect all their output to stderr.
The parenthesis (…) are shell-syntax and used for creating a subshell:
(date; ls; echo $$) > 1>&2
Like above but the PID ($$) given out is that of the subshell.
The difference between grouping and subshell is delicate (and out of scope here).
The doubled brackets [[…]] are shell syntax but otherwise behave like the single bracket [ command. The only difference is for using < etc. for string comparison and locale support.
The doubled parentheses ((…)) are equivalent to using the let builtin shell command. They basically allow number-oriented expressions to be evaluated (ARITHMETIC EVALUATION). < and > sort numerically (instead of lexicographically) etc. Also, in some constructs like for ((i=0; i<10; i++)); do echo "$i"; done they are used as a fixed syntax.
Dollar-parenthesis $(…) result in the output of the command they enclose:
echo "$(date)" # a complicated way to execute date
Dollar-brackets $[…] are deprecated and should be replaced by dollar-double-parentheses.
Dollar-double-parentheses $((…)) result in the value of the numerical expression they enclose:
echo "$((4 + 3 * 2))" # should print 10
Dollar-braces ${…} result in the variable expansion they enclose. In the simplest case this is just a variable name, then they evaluate to the variable value:
a=foo
echo "${a}" # prints foo
This can (and often is) abbreviated by stripping the braces: $a
But it also can be more complex like ${a:-"today is $(date)"}. See Parameter Expansion in the bash man page for details.
Redirection-parenthesis <(…) and >(…) create a subprocess, a file descriptor its output/input is associated with, and a pseudo file associated with that descriptor. It can be used to pass the output of a program as a seeming file to another program: diff <(sleep 1; date) <(sleep 2; date)
Why doesn't this work???
#!/bin/ksh
# array testfunc()
function testfunc {
typeset -A env
env=( one="motherload" )
print -r $env
return 0
}
testfunc # returns: ( one=motherload )
typeset -A testvar # segfaults on linux, memfaults on solaris
testvar=$(testfunc) # segfaults on linux, memfaults on solaris
print ${testvar.one}
note: I updated the above script to print ${testvar.one} from print $testvar to show more precisely what I am trying to accomplish.
I am sure this has been asked before, but I am not sure what to search on and everything I have been trying to use for keywords is not bringing me any answers that relate to my problem.
ksh version:
linux: version sh (AT&T Research) 1993-12-28 s+
solaris: version sh (AT&T Research) 93s+ 2008-01-31
Update:
So another question is, this will run in ksh 93t+ without giving an error, but, it doesn't assign the array properly. I would I go about assigning an array from a function? I tried assigning the array like this also:
typeset -A testvar=$(testfunc)
print ${testvar.one}
But that also didn't work properly.
EDIT
So what is happening here?
typeset -A env=( one="motherload" two="vain" )
print ${env.one}
print ${env.two}
I thought this was how you defined associative arrays, maybe what I was looking at was old but who knows.... seems odd behaviour since this prints out "motherload" and "vain"
Your script works fine for me on Linux with ksh 93t+.
Since it's the same script and you're getting similar errors in two different environments, I would suspect stray characters in the file. Try one of these to show any stray characters that might be present:
hd filename
cat -v filename
hexdump -C filename
If it's simply a matter of DOS line endings, then this will fix that:
dos2unix filename
Edit:
Here's one way to create and populate an associative array in ksh:
$ typeset -A testvar
$ testvar=([one]="motherlode" [two]="vein" [waste]="tailings")
$ echo ${testvar[two]}
vein
$ testvar[ore]="gold"
$ echo ${!testvar[#]} # print the indices of the array
one two waste ore
$ typeset -p testvar # show the current definition of the array
typeset -A testvar=([one]="motherlode" [two]="vein" [waste]="tailings" [ore]="gold")
As you can see, ksh uses bracketed subscripts for arrays. Dotted notation is used for accessing members of a compound variable.
I don't believe ksh functions can return arrays. However, you can use the print technique you have in your function (but add square brackets around the index name) and use eval to do the assignment.
$ typeset -A testvar
$ eval "testvar=($(testfunc))"
or to append to an existing array:
$ eval "testvar+=($(testfunc))"
Unless your function is using associative arrays internally, you don't necessarily need to use them to build your output.
However, if you do, you can parse from the result of typeset -p:
$ result=$(typeset -p env)
$ result=${result#*\(}
$ result=${result%\)*}
$ print result
or iterate through the array:
$ for index in ${!env[#]}; do print -n "[$index]=${env[$index]} "; done; print
You may want to consult the documentation concerning discipline functions and type variables
Here is an alternative to getting any return value from a function using name reference. The value returned will be stored in a variable defined as the first positional argument of the function (not declaring the variable beforehand will work but the variable will be global):
#################################
# Example using compound variable
#################################
function returnCompound {
typeset -n returnVal="$1"
returnVal=( one="motherloadCompound" )
return 0
}
# Declaring the variable to keep it in this scope
# Useful for calling nested functions whitout messing
# with the global scope
typeset myNewCompoundVar
returnCompound myNewCompoundVar
echo "Compound: ${myNewCompoundVar.one}"
##################################
# Example using asssociative array
##################################
function returnMap {
typeset -n myNewMapVar="$1"
myNewMapVar=( [one]="motherloadMap" )
typeset nestedCompoundVar
returnCompound nestedCompoundVar
echo "Compound (Nested) from inside: ${nestedCompoundVar.one}"
return 0
}
# Declaring the variable to keep it in this scope
# Useful for calling nested functions whitout messing
# with the global scope
typeset myNewMapVar
returnMap myNewMapVar
echo "Associative array: ${myNewMapVar[one]}"
echo "Compound (Nested) from outside: ${nestedCompoundVar.one}"
Output:
Compound: motherloadCompound
Compound (Nested) from inside: motherloadCompound
Associative array: motherloadMap
Compound (Nested) from outside:
Important side notes:
Function declarations must be done using the function keyword or else the concept of local scope variable won't be taken under account. In which case the name of your reference variable and global variable might clash if they happen to be the same, resulting in a typeset: invalid self reference error. This can be tested by changing the declaration of the 'returnMap' function.
If you do not declare the return variable before the function call, the variable to which is assigned the return value will be created globally and not limited to the calling scope.
I'm feel like I'm missing something pretty obvious here, but I can't seem to figure out what's going on. I have a perl script that I'm calling from C code. The script + arguments is something like this:
my_script "/some/file/path" "arg" "arg with spaces" "arg" "/some/other/file"
When I run it in Windows, Perl correctly identifies it as 5 arguments, whereas when I ran it on the SunOS Unix machine, it identified 8, splitting the arg with spaces into separate args.
Not sure if it makes any difference, but in Windows I'm running it like:
perl my_script <args>
While in Unix I'm just running it as an executable like show above.
Any idea why Unix is not managing that argument properly?
Edit:
Here's the code for calling the perl script:
char cmd[1000];
char *script = "my_script";
char *argument = "\"arg1\" \"arg2\" \"arg3 with spaces\" \"arg4\" \"arg5\"";
sprintf( cmd, "%s %s >1 /dev/null 2>&1", script, arguments);
system( cmd );
That's not exactly it, as I build the argument string a little more dynamically, but that's the gist.
Also, here's my code for reading the arguments in:
($arg1, $arg2, $arg3, $arg4, $arg5) = #ARGV;
I know it's ridiculously naive, but this script will only be run from the C application, so nothing more complex should be required.
Presumably
system("my_script \"/some/file/path\" \"arg\" \"arg with spaces\" \"arg\" \"/some/other/file\");
causes everything to go through bash (because of the need to interpret the shebang line, eating up the quotes you pass). Again, presumably, the problem could be avoided by invoking perl directly rather than relying on the shell to find it (although this might be a problem if the perl on the path is different than the one provided on the shebang line).
Update:
Given your:
char *argument = "\"arg1\" \"arg2\" \"arg3 with spaces\" \"arg4\" \"arg5\"";
you might want to try:
char *argument = "\\\"arg1\\\" \\\"arg2\\\" \\\"arg3 with spaces\\\" \\\"arg4\\\" \\\"arg5\\\"";
Another update:
Thank you for accepting my answer, however, my whole theory might be wrong.
I tried the double-backwhacked version of argument above in GNU bash, version 4.0.28(2)-release (i686-pc-linux-gnu) and it ended up passing
[sinan#kas src]$ ./t
'"arg1"'
'"arg2"'
'"arg3'
'with'
'spaces"'
'"arg4"'
'"arg5"'
whereas the original argument worked like a charm. I am a little puzzled by this. Maybe the shell on the SUN isn't bash or maybe there is something else going on.
Did you forget the quotes on SunOS?
If you do
perl script arg1 arg2 "arg3 with spaces" arg4 arg5
you should be good. Otherwise, try switching shells.
Now that the question was updated, I still can't reproduce your results:
~% cat test.c
#include <stdio.h>
#include <stdlib.h>
int main(void){
char cmd[1000];
char *script = "./my_script";
char *arguments = "\"arg1\" \"arg2\" \"arg3 with spaces\" \"arg4\" \"arg5\"";
sprintf( cmd, "%s %s", script, arguments);
system( cmd );
return 0;
}
~% cat my_script
#!/usr/bin/perl
($arg1, $arg2, $arg3, $arg4, $arg5) = #ARGV;
print "arg1 = $arg1\n";
print "arg2 = $arg2\n";
print "arg3 = $arg3\n";
print "arg4 = $arg4\n";
print "arg5 = $arg5\n";
~% gcc test.c
~% ./a.out
arg1 = arg1
arg2 = arg2
arg3 = arg3 with spaces
arg4 = arg4
arg5 = arg5
~%
There is something else with your configuration.
Previous answer:
Unix shells interpret quoted arguments
as a single argument. You can do a
quick test:
for i in a b "c d e" f; do echo $i; done
The result is what you expect it to
be: "c d e" is treated like a single
argument.
I think you have a problem in your
script, in the argument handling
logic.
Is it possible that the SunOS kernel's handing of shebang interpreters is ludicrously bad? Try running it as "/path/to/perl script args" instead of "script args" and see if anything changes.
I am having some issues with word-splitting in bash variable expansion. I want to be able to store an argument list in a variable and run it, but any quoted multiword arguments aren't evaluating how I expected them to.
I'll explain my problem with an example. Lets say I had a function decho that printed each positional parameter on it's own line:
#!/bin/bash -u
while [ $# -gt 0 ]; do
echo $1
shift
done
Ok, if I go decho a b "c d" I get:
[~]$ decho a b "c d"
a
b
c d
Which is what I expect and want. But on the other hand if I get the arguments list from a variable I get this:
[~]$ args='a b "c d"'
[~]$ decho $args
a
b
"c
d"
Which is not what I want. I can go:
[~]$ echo decho $args | bash
a
b
c d
But that seems a little clunky. Is there a better way to make the expansion of $args in decho $args be word-split the way I expected?
You can use:
eval decho $args
You can move the eval inside the script:
#!/bin/bash -u
eval set -- $*
for i;
do
echo $i;
done
Now you can do:
$ args='a b "c d"'
$ decho $args
a
b
c d
but you'll have to quote the arguments if you pass them on the CL:
$ decho 'a b "c d"'
a
b
c d
Have you tried:
for arg in "$#"
do
echo "arg $i:$arg:"
let "i+=1"
done
Should yield something like:
arg 1: a
arg 2: c d
in your case.
Straight from memory, no guarantee :-)
hmmm.. eval decho $args works too:
[~]$ eval decho $args
a
b
c d
And I may be able to do something with bash arrays using "${array[#]}" (which works like "$#"), but then I would have to write code to load the array, which would be a pain.
It is fundamentally flawed to attempt to pass an argument list stored in a variable, to a command.
Presumably, if you have code somewhere to create a variable containing the intended args. for a command, then you can change it to instead store the args into an array variable:
decho_argv=(a b 'c d') # <-- easy!
Then, rather than changing the command "decho" to accommodate the args taken from a plain variable (which will break its ability to handle normal args) you can do:
decho "${decho_argv[#]}" # USE DOUBLE QUOTES!!!
However, if you are the situation where you are trying to take arbitrary input which is expected to be string fields corresponding to intended command positional arguments, and you want to pass those arguments to a command, then you should instead of using a variable, read the data into an array.
Note that suggestions which offer the use of eval to set positional parameters with the contents of an ordinary variable are extremely dangerous.
Because, exposing the contents of a variable to the quote-removal and word-splitting on the command-line affords no way to protect against shell metachars in the string in the variable from causing havoc.
E.g., imagine in the following example if the word "man" was replaced with the two words "rm" and "-rf" and the final arg word was "*":
Do Not Do This:
> args='arg1 ; man arg4'
> eval set -- $args
No manual entry for arg4
> eval set -- "$args" # This is no better
No manual entry for arg4
> eval "set -- $args" # Still hopeless
No manual entry for arg4
> eval "set -- '$args'" # making it safe also makes it not work at all!
> echo "$1"
arg1 ; man arg4