Recursive variable substitution with env.subst - scons

According to the scons documentation, the subst method will recursively interpolate construction variables. However, it does not seem to be recursive:
e = Environment(CPPDEFINES = ["FOOBAR=${foobar}"])
e["foo"] = 1
e["bar"] = "${foo + 1}"
e["foobar"] = "$${foo + ${bar}}"
# I want this to print "foobar: 3"
print "foobar:", e.subst("${foobar}")
e.Program("test.c")
Prints:
scons: Reading SConscript files ...
foobar: ${foo + 2}
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test.o -c -DFOOBAR=3 test.c
gcc -o test test.o
scons: done building targets.
foobar is correctly evaluated during compilation as a part of CPPDEFINES, but not in the print statement. How can I get subst to fully evaluate foobar?

Using the expression
e["foobar"] = "${foo + ${bar}}"
, as suggested by Kenny Ostrom, doesn't help either. It yields a syntax error because the subst method doesn't really handle nested braces too well.
The actual question is: Why do we see different outputs when using subst in the SConstruct directly, and when it gets used within a build command?
If we add
print "CPPDEFINES:", e.subst("$CPPDEFINES")
to the SConstruct, we see the same output ${foo + 2} for FOOBAR. The difference at build time is, that the internal variable $_CPPDEFFLAGS is declared in terms of the _defines method:
'_CPPDEFFLAGS': '${_defines(CPPDEFPREFIX, CPPDEFINES, CPPDEFSUFFIX, __env__)}'
(from a print e.Dump()). This _defines method runs all variables through subst_path a second time, such that one can use variables in include paths, for example.
So the subst method is doing the right thing, you just have to evaluate again:
print "foobar:", e.subst(e.subst("${foobar}"))
to get the same output.

Just to make it clear what dirkbaechle said; we can achieve this by
simply doing interpolation and evaluation in two separate steps (by calling subst twice). This allows us to have arbitrary complex expressions:
# Notice how we wrap foobar in $$
e = Environment(CPPDEFINES = ["FOOBARBAZ=$${${foobarbaz}}"])
e["foo"] = 1
e["bar"] = "($foo + 1)"
e["foobar"] = "($foo + $bar)"
e["foobarbaz"] = "($foobar + $foobar)"
print "foobarbaz after substituting once:", e.subst("$${${foobarbaz}}")
print "foobarbaz after substituting twice:", e.subst(e.subst("$${${foobarbaz}}"))
e.Program("test.c")
Prints:
scons: Reading SConscript files ...
foobarbaz after substituting once: ${((1 + (1 + 1)) + (1 + (1 + 1)))}
foobarbaz after substituting twice: 6
scons: done reading SConscript files.
scons: Building targets ...
gcc -o test.o -c -DFOOBARBAZ=6 test.c
gcc -o test test.o
scons: done building targets.
Thanks again, dirkbaechle!

Related

How to debug diverging test using QuickCheck

I have some parsing code using Megaparsec that I've written a simple property to test (it generates a random expression tree, pretty-prints it, and then checks that the result parses back to the original tree).
Unfortunately, there seems to be a bug and if I run the tests without any limits, I see the GHC process allocating more and more memory until either I kill it or the OOM killer gets there first.
Not a problem, I thought... but I can't for the life of me figure out what's causing the divergence. The property itself looks like this: (I've ripped out the proper testing and the shrinking code to try to minimise the code that actually runs)
prop_parse_expr :: Property
prop_parse_expr =
forAll arbitrary $
(\ pe ->
let str = prettyParExpr 0 pe in
counterexample ("Rendered to: " ++ show str) $
trace ("STARTING TEST: " ++ show str) $
case parse (expr <* eof) "" str of
Left _ -> trace "NOPE" $ False
Right _ -> trace "GOOD" $ True)
If I compile with profiling (using stack test --profile), I can run the resulting binary with RTS options. Ahah, I thought, and ran with -xc, thinking that I'd get a helpful stack trace when I sent a SIGINT to the stuck job. It seems not. Running with
./long/path/to/foo-test -j1 --test-seed 1 +RTS -xc
I see this output:
STARTING TEST: "0"
GOOD
STARTING TEST: "(x [( !0 )]) "
STARTING TEST: "({ 2 {( !0 )}} ) "
STARTING TEST: "{ 2{ ( x[0? {( 0) ,( x ) } :((0 )? (x ):0) -: ( -(^( x )) ) ]), 0**( x )} } "
STARTING TEST: "| (0? (x[({ 1{ (0)? x : ( 0 ) }} ) ]) :(~&( 0) ?( x):( (x ) ^( x ) )))"
STARTING TEST: "(0 )"
STARTING TEST: "0"
^C*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
Test.Framework.Improving.runImprovingIO,
called from Test.Framework.Providers.QuickCheck2.runProperty,
called from Test.Framework.Providers.QuickCheck2.runTest,
called from Test.Framework.Runners.Core.runSimpleTest,
called from Test.Framework.Runners.Core.runTestTree.go,
called from Test.Framework.Runners.Core.runTestTree,
called from Test.Framework.Runners.Core.runTests',
called from Test.Framework.Runners.Core.runTests,
called from Test.Framework.Runners.Console.defaultMainWithOpts,
called from Test.Framework.Runners.Console.defaultMainWithArgs,
called from Test.Framework.Runners.Console.defaultMain,
called from Main.main
<snip: 2 more identical backtraces>
*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
Test.Framework.Runners.Console.Utilities.hideCursorDuring,
called from Test.Framework.Runners.Console.Run.showRunTestsTop,
called from Test.Framework.Improving.runImprovingIO,
called from Test.Framework.Providers.QuickCheck2.runProperty,
called from Test.Framework.Providers.QuickCheck2.runTest,
called from Test.Framework.Runners.Core.runSimpleTest,
called from Test.Framework.Runners.Core.runTestTree.go,
called from Test.Framework.Runners.Core.runTestTree,
called from Test.Framework.Runners.Core.runTests',
called from Test.Framework.Runners.Core.runTests,
called from Test.Framework.Runners.Console.defaultMainWithOpts,
called from Test.Framework.Runners.Console.defaultMainWithArgs,
called from Test.Framework.Runners.Console.defaultMain,
called from Main.main
Can anyone tell me:
Why I see multiple STARTING TEST lines without GOOD or NOPE between them, despite the -j1?
How I get an actual stack trace that shows where the test is allocating all its memory?
Thanks for any ideas!
For anyone who finds this question, the problem with my code was that my arbitrary instance for expressions didn't constrain sizes properly, so sometimes tried to make enormous trees. See the "Generating Recursive Data Types" section of the QuickCheck manual for what I should have been doing!
I found that running commands like:
./long/path/to/foo-test -o3 +RTS -xc
helped me figure out what was going on. Strangely, the backtrace still shows several threads of execution. I don't really understand why, but at least I could then see that I was spending time in my "makeAnExpr" function. The trick is to tune the timeout (3 seconds above) so that it doesn't kill a test until it's well and truly stuck, but does stop it before it starts eating all your RAM!

unreadable quickcheck log file after a test routine

I made a test routine for a Haskell program with quickcheck. I declared it in my cabal file with :
Test-Suite routine_de_test
Type: exitcode-stdio-1.0
Hs-Source-Dirs: test
Main-is: Tests.hs
and launched it with :
cabal configure --enable-tests
cabal buil
cabal test
The tests are processed correctly and I was expecting to see details about the random value used for each test in the log file dist/test/ but when I open it, the file looks like this :
I tried to open the file with several encoding (UTF8, ISO-8859-15, ...) but nothing is changed.
Is it normal? Or is there something wrong?
Is it possible when performing quickcheck test from cabal to get the complete list of random values used for each tests?
It looks like the funny characters are simply backspaces, and quickcheck is simply reporting the number of tests it has performed so far by overwriting (0 tests) with (1 test) and then (2 tests) and then with (3 tests), etc.
Visually it will look fine when displayed to a terminal.
Update:
To report the random values used for a test the only way I know of is to write your test to explicitly display (or save to a file) the values used.
If your test is a pure function you can use the trace function from Debug.Trace. For instance, if you have this property:
prop_commutes :: Int -> Int -> Bool
prop_commutes a b = a + b == b + a
You can trace each invocation of prop_commutes by modifying like this:
import Debug.Trace
prop_commutes :: Int -> Int -> Bool
prop_commutes x y = a + b == b + a
where (a,b) = trace ("(a,b) = " ++ show (x,y)) (x,y)
and then quickCheck prop_commutes will emit lines like:
(x,y) = (20,-73)
(x,y) = (71,-36)
(x,y) = (2,-11)
...
in addition to its normal output.

I taught ghci to compile my StackOverflow posts. Can I make it slicker?

Haskell Stack Overflow layout preprocessor
module StackOverflow where -- yes, the source of this post compiles as is
Skip down to What to do to get it working if you want to play with this first (1/2 way down).
Skip down to What I would like if I witter on a bit and you just want to find out what help I'm seeking.
TLDR Question summary:
Can I get ghci to add filename completion to the :so command I defined in my ghci.conf?
Could I somehow define a ghci command that returns code for compilation instead of returning a ghci command, or
does ghci instead have a better way for me to plug in Haskell code as a
file-extension-specific pre-processor, so :l would work for .hs and .lhs files as usual, but use my handwritten preprocessor for .so files?
Background:
Haskell supports literate programming in .lhs source files, two ways:
LaTeX style \begin{code} and \end{code}.
Bird tracks: Code starts with > , anything else is a comment.
There must be a blank line between code and comments (to stop trivial accidental misuse of >).
Don't Bird tracks rules sound similar to StackOverflow's code blocks?
References: 1. The .ghci manual
2. GHCi haskellwiki
3. Neil Mitchell blogs about :{ and :} in .ghci
The preprocessor
I like writing SO answers in a text editor, and I like to make a post that consists of code that works,
but end up with comment blocks or >s that I have to edit out before posting, which is less fun.
So, I wrote myself a pre-processor.
If I've pasted some ghci stuff in as a code block, it usually starts with * or :.
If the line is completely blank, I don't want it treated as code, because otherwise
I get accidental code-next-to-comment-line errors because I can't see the 4 spaces I accidentally
left on an otherwise blank line.
If the preceeding line was not code, this line shouldn't be either, so we can cope with StackOverflow's
use of indentation for text layout purposes outside code blocks.
At first we don't know (I don't know) whether this line is code or text:
dunnoNow :: [String] -> [String]
dunnoNow [] = []
dunnoNow (line:lines)
| all (==' ') line = line:dunnoNow lines -- next line could be either
| otherwise = let (first4,therest) = splitAt 4 line in
if first4 /=" " --
|| null therest -- so the next line won't ever crash
|| head therest `elem` "*:" -- special chars that don't start lines of code.
then line:knowNow False lines -- this isn't code, so the next line isn't either
else ('>':line):knowNow True lines -- this is code, add > and the next line has to be too
but if we know, we should keep in the same mode until we hit a blank line:
knowNow :: Bool -> [String] -> [String]
knowNow _ [] = []
knowNow itsCode (line:lines)
| all (==' ') line = line:dunnoNow lines
| otherwise = (if itsCode then '>':line else line):knowNow itsCode lines
Getting ghci to use the preprocessor
Now we can take a module name, preprocess that file, and tell ghci to load it:
loadso :: String -> IO String
loadso fn = fmap (unlines.dunnoNow.lines) (readFile $ fn++".so") -- so2bird each line
>>= writeFile (fn++"_so.lhs") -- write to a new file
>> return (":def! rso (\\_ -> return \":so "++ fn ++"\")\n:load "++fn++"_so.lhs")
I've used silently redefining the :rso command becuase my previous attemts to use
let currentStackOverflowFile = .... or currentStackOverflowFile <- return ...
didn't get me anywhere.
What to do to get it working
Now I need to put it in my ghci.conf file, i.e. in appdata/ghc/ghci.conf
as per the instructions
:{
let dunnoNow [] = []
dunnoNow (line:lines)
| all (==' ') line = line:dunnoNow lines -- next line could be either
| otherwise = let (first4,therest) = splitAt 4 line in
if first4 /=" " --
|| null therest -- so the next line won't ever crash
|| head therest `elem` "*:" -- special chars that don't start lines of code.
then line:knowNow False lines -- this isn't code, so the next line isn't either
else ('>':line):knowNow True lines -- this is code, add > and the next line has to be too
knowNow _ [] = []
knowNow itsCode (line:lines)
| all (==' ') line = line:dunnoNow lines
| otherwise = (if itsCode then '>':line else line):knowNow itsCode lines
loadso fn = fmap (unlines.dunnoNow.lines) (readFile $ fn++".so") -- convert each line
>>= writeFile (fn++"_so.lhs") -- write to a new file
>> return (":def! rso (\\_ -> return \":so "++ fn ++"\")\n:load "++fn++"_so.lhs")
:}
:def so loadso
Usage
Now I can save this entire post in LiterateSo.so and do lovely things in ghci like
*Prelude> :so StackOverflow
[1 of 1] Compiling StackOverflow ( StackOverflow_so.lhs, interpreted )
Ok, modules loaded: StackOverflow.
*StackOverflow> :rso
[1 of 1] Compiling StackOverflow ( StackOverflow_so.lhs, interpreted )
Ok, modules loaded: StackOverflow.
*StackOverflow>
Hooray!
What I would like:
I would prefer to enable ghci to support this more directly. It would be nice to get rid of the intermediate .lhs file.
Also, it seems ghci does filename completion starting at the shortest substring of :load that determines
you're actually doing load, so using :lso instead of :so doesn't fool it.
(I would not like to rewrite my code in C. I also would not like to recompile ghci from source.)
TLDR Question reminder:
Can I get ghci to add filename completion to the :so command I defined in my ghci.conf?
Could I somehow define a ghci command that returns code for compilation instead of returning a ghci command, or
does ghci instead have a better way for me to plug in Haskell code as a
file-extension-specific pre-processor, so :l would work for .hs and .lhs files as usual, but use my handwritten preprocessor for .so files?
I would try to make a standalone preprocessor that runs SO preprocessing code or the standard literary preprocessor, depending on file extension. Then just use :set -pgmL SO-preprocessor in ghci.conf.
For the standard literary preprocessor, run the unlit program, or use Distribution.Simple.PreProcess.Unlit.
This way, :load and filename completion just work normally.
GHCI passes 4 arguments to the preprocessor, in order: -h, the label, the source file name, and the destination file name. The preprocessor should read the source and write to the destination. The label is used to output #line pragmas. You can ignore it if you don't alter the line count of the source (i.e. replace "comment" lines with -- comments or blank lines).

What is the difference between := and += in make file?

what is working difference in the below statements?
LDDIRS := -L$(ORACLE_LIB)
LDDIRS += -L$(ORACLE_LIB)
:= (Simply Expanded Variable ) The value is scanned for once and for all expanding any
references to other variables and functions, when variable is defined. e.g.
x:=foo
y:=$(x) bar
x:=later
so above is equivalent to
y:=foo bar
x:=later
+= is used for appending more text to variables e.g.
objects=main.o foo.o bar.o
objects+=new.o
which will set objects to 'main.o foo.o bar.o new.o'
= is for recursively expanded variable.The value is install verbatim; if it contains
reference to other variables these variables are expanded whenever this variable is
substituted.And this is known as recursive expansion.
"=" is for defining recursively expanded variable. The follow make file will print out "y is later bar"
x = foo
y = $(x) bar
x = later
all:;echo "y is" $(y)
":=" is for defining simply expanded variable, which is expanded once and for all. The following make file will print out "y is foo bar"
x := foo
y := $(x) bar
x := later
all:;echo "y is" $(y)
Also, as other people pointed earlier, you can get more details in Using Variables section of the GNU make manual.
Hope this helps :-)
GNU Make has three assignment operators, ":=" , "=", "?=" and one "+=" for appending to the variables.
":=" performs immediate evaluation of the right-hand side and stores an actual
string into the left-hand side.
e.g.:
x:=foo
y:=$(x) bar
x:=later
so above is equivalent to
y:=foo bar
x:=later
test above example
x := foo
y := $(x) bar
x := later
all:;echo "y is" $(y)
output
------
y is foo bar
"=" is like a formula definition; it stores the right-hand side in an
unevaluated form and then evaluates this form each time the left-hand
side is used.
e.g.:
x = foo
y = $(x) bar
x = later
all:;echo "y is" $(y)
output
------
y is later foo
"?=" Assign only if it's not set/doesn't have a value.
e.g.:
KDIR ?= "foo"
KDIR ?= "bar"
test:
echo $(KDIR)
Would print "foo"
"+=" is used for appending more text to variables.
e.g.:
objects=main.o foo.o bar.o
objects+=new.o
which will set objects to main.o foo.o bar.o new.o
:= Defines the variable here to be the left hand side, += adds the right hand side to the existing value of the variable. Compare := with = which evaluates the right hand side at the place of use (rather than in this particular line)
You can look at the manual here (Assuming that you are using GNU make)
From This website
for the syntax := Link to place on page
Simply expanded variables are defined by lines using ‘:=’ (see Setting Variables). The value of a simply expanded variable is scanned once and for all, expanding any references to other variables and functions, when the variable is defined. The actual value of the simply expanded variable is the result of expanding the text that you write. It does not contain any references to other variables; it contains their values as of the time this variable was defined.
for the syntax += Link to place on page
When the variable in question has not been defined before, ‘+=’ acts just like normal ‘=’: it defines a recursively-expanded variable. However, when there is a previous definition, exactly what ‘+=’ does depends on what flavor of variable you defined originally. See The Two Flavors of Variables, for an explanation of the two flavors of variables.
the := will set the value once to the variable, ie it wont be re-evaluated everytime make encouters that variable. Can make a huge difference in performance when compiling the code.
+= will simply add up a value to the variable.
The := is for assignation, in the same manner as =.
+= add a new value to the variable.

How to test my haskell functions

I just started with Haskell and tried to do write some tests first. Basically, I want to define some function and than call this function to check the behavior.
add :: Integer -> Integer -> Integer
add a b = a+b
-- Test my function
add 2 3
If I load that little script in Hugs98, I get the following error:
Syntax error in declaration (unexpected `}', possibly due to bad layout)
If I remove the last line, load the script and then type in "add 2 3" in the hugs interpreter, it works just fine.
So the question is: How can I put calls of my functions in the same script as the function definition? I just want to load the script and be able to check if it does what I expect it to...I don't want to type them in manually all the time.
Others have said how to solve your immediate problem, but for testing you should be using QuickCheck or some other automated testing library.
import Test.QuickCheck
prop_5 = add 2 3 == 5
prop_leftIdentity n = add 0 n == n
Then run quickCheck prop_5 and quickCheck prop_leftIdentity in your Hugs session. QuickCheck can do a lot more than this, but that will get you started.
(Here's a QuickCheck tutorial but it's out of date. Anyone know of one that covers QuickCheck 2?)
the most beginner friendly way is probably the doctest module.
Download it with "cabal install doctest", then put your code into a file "Add.hs" and run "doctest Add.hs" from the command line.
Your code should look like this, the formatting is important:
module Add where
-- | add adds two numbers
--
-- >>> add 2 3
-- 5
-- >>> add 5 0
-- 5
-- >>> add 0 0
-- 0
add :: Integer -> Integer -> Integer
add a b = a+b
HTH Chris
Make a top level definition:
add :: Integer -> Integer -> Integer
add a b = a + b
test1 = add 2 3
Then call test1 in your Hugs session.
How can I put calls of my functions in the same script as the function definition? I just want to load the script and be able to check if it does what I expect it to...I don't want to type them in manually all the time.
In short, you can't. Wrap it in a function and call it instead. Your file serves as a valid Haskell module, and having "flying" expression is not a valid way to write it.
You seem to come from a scripting language background, but don't try treating Haskell as one of them.
If you have ghc installed, then the runhaskell command will interpret and run the main function in your file.
add x y = x + y
main = print $ add 2 3
Then on the command line
> runhaskell Add.hs
5
Not sure, but hugs probably has a similar feature to the runhaskell command. Or, if you load the file into the hugs interpreter, you can simply run it by calling main.
I was trying to do the same thing and I just made a function that ran through all my test cases (using guards) and returned 1 if they all passed and threw an error if any failed.
test :: Num b => a->b
test x
| sumALL [1] /= 1 = error "test failed"
| sumALL [0,1,2] /= 3 = error "test failed"
...
| otherwise = 1

Resources