In a SCons build setup that uses quite some custom build actions, I had a lengthy action that was repeatedly retriggered because a target name was misspelled. Is it possible to configure SCons such that it the targets of a builder (or all builders) is really generated to prevent such cases?
For example, given
target = Command('some_target_file',
'some_input',
'echo foo > wrong_target_file'
)
with > scons --debug=explain will always result in
scons: building `some_target_file' because it doesn't exist
echo foo > wrong_target_file
without failure. While I would like to get an error in the spirit of
Error: target 'some_target_file' was not generated
I could emulate the desired behaviour using something like
dummy = Command('dummy', 'some_target_file', 'cat $SOURCE')
Default ([target, dummy])
resulting in
...
cat some_target_file
The system cannot find the file specified.
scons: *** [dummy] Error 1
scons: building terminated because of errors.
Related
While building my program it is important to distinguish between files that don't exists and files that are empty.
However, it appears that scons treats them the same and neglect to rebuild a target when a source file changed from one of these states to the other one.
Step by step example:
Step 0:
SConstruct
foo = Command('foo', [], 'echo $TARGET is not created here!')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)
Result:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
echo foo is not created here!
foo is not created here!
touch bar ; test -f foo
scons: *** [bar] Error 1
scons: building terminated because of errors.
My interpretation:
The command for foo fails to create the file but it doesn't raise and error so the command for bar is run. It checks if foo exists and returns an error. Build fails (everything as expected so far).
Step 1:
SConstruct:
foo = Command('foo', [], 'touch $TARGET')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)
Result:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
touch foo
touch bar ; test -f foo
scons: done building targets.
My interpretation:
foo is rebuilt because it has changed. This time it creates an empty file. bar is rebuild because it failed before. It succeeds this time. The build is successful (still as expected).
Step 2:
SConstruct
foo = Command('foo', [], 'echo $TARGET is not created here!')
bar = Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)
Result:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
echo foo is not created here!
foo is not created here!
scons: `bar' is up to date.
scons: done building targets.
My interpretation:
foo is rebuilt because it has changed again (was restores to the previous version). The file foo doesn't exist any longer because scons removes files before building them and the command fails to recreate it. bar is not rebuilt because scons doesn't seem to detect a change in the source file.
The build is successful while it shouldn't!
How can I force scons to rebuild bar in the last step?
The solution should scale well to "foo" commands that produce many files, list of which is generated programmatically in SConscript and cannot be hard-coded.
By the way, scons does now distinguish between emtpy and nonexistent, that's a fairly recent change (commit 3b7f8b4ce0, github.com/SCons/scons/pull/3833).
The default way SCons determines is file has hanged is to compare MD5 signatures of the previous and the current versions of the file.
The signature for a non-existent file is calculated from 0 bytes of data just like for an empty file so SCons doesn't see a difference between them. This is normally OK, especially that not creating the target file isn't an entirely legitimate use of SCons.
However, we can make it work by supplying different function that decides if files are different.
Such function in SCons is called a Decider.
There are three of them provided out of the box. The default one uses MD5. The second one uses timestamp. The third one uses MD5 but only if the timestamp is different.
In this case, timestamp could perhaps work because it is 0 for a non-existent file. However, it would generate false-positives when timestamp changes and the contents of the file do not.
Instead, we can supply our own decider which will do exactly what we want it to:
from os import path
env = DefaultEnvironment()
decider_env = env.Clone()
def decide_if_changed(dependency, target, prev_ni, repo_node=None):
csig = dependency.get_csig() # it has to be called every time or the value won't be in `prev_ni` for the next check
return not path.isfile(dependency.abspath) or not hasattr(prev_ni, 'csig') or prev_ni.csig != csig
decider_env.Decider(decide_if_changed)
foo = env.Command('foo', [], 'echo $TARGET is not created here!')
bar = decider_env.Command('bar', foo, 'touch $TARGET ; test -f $SOURCE')
Default(bar)
This custom decider is similar to the default implementation based on MD5 except it also reports a change if the file doesn't exist.
This should cover the problem described in the question.
The new decider is assigned to a clone of the default environment. This way we have control over which target uses it. In this case only bar uses the non-default decider.
If you're saying "how can I force SCons to do xyz?", then you're understanding of SCons is incomplete.
SCons will only build targets which are out of date.
Unless..
You use AlwaysBuild(target) see: https://scons.org/doc/production/HTML/scons-man.html#f-AlwaysBuild
It also seems like you never want foo to be removed before it's (re)built?
Then you should use Precious(target) see: https://scons.org/doc/production/HTML/scons-man.html#f-Precious
Also.. it's bad form to call a builder with an empty source.
How would SCons ever know if it's out of date?
For your example what causes foo to be (re)built?
It's said that scons uses MD5 signature as default decider to dertermine whether a source file needs re-compilation. E.g. I've got SConstruct as below:
Library('o.c')
And my o.c is:
$ cat o.c
/*commented*/
#include<stdio.h>
int f(){
printf("hello\n");
return 2;
}
Run scons and remove the comment line, run scons again. I expect that scons should not compile it again, but actually it's:
gcc -o o.o -c o.c
scons: done building targets.
If I change SConstruct file to add one line:
Decider('MD5').
Still same result.
My question is: how to make sure that for scons, when changing source file comments, they don't get re-built?
Thanks!
As you correctly stated, SCons uses the MD5 hashsum of a source file to decide whether it has changed or not (content-based), and a rebuild of the target seems to be required (since one of its dependencies changed).
By adding or changing a comment, the MD5 sum of the file changes...so the trigger fires.
If you don't like this behaviour, you can write and use your own Decider function which will omit comment changes to your likings. Please check section 6.1.4 "Writing Your Own Custom Decider Function" in the UserGuide to see how this can be done.
I'm new to Scons and I'm trying to figure out if I could use it for my use-case. I have a script whose main actoin is to take a single input and produces multiple output files in a given directory. However, it also needs one additional input and one additional output, as in
script --special-in some.foo --special-in some.bar input.foo output.dir/
The names of some.* files can be computed from the input file name (here input.foo). And the some.* files produced by one rule are consumed by other rules.
In the documentation I found that one can create custom builders as in
bld = Builder(action = 'foobuild $TARGETS - $SOURCES',
suffix = '.foo',
src_suffix = '.input',
emitter = modify_targets)
where the emitter adds the additional target and source. However, I couldn't find how should I distinguish the main source/target from the special ones, which need to be passed using specific options - I can't use $TARGETS and $SOURCES as in the above example. I could probably use a generator and index into source and target, but this seems a bit hacky. I there a better way?
From what you describe, you should be using both an emitter and a generator, just as you state at the end of your question. The "main" source/target will be the first element in the source/target lists. This doesn't seem hacky to me, but I may just be used to it...
Answers are always better with a working example...
Here is the SConstruct to do what you describe. I'm not exactly sure how you plan to compute some.foo and some.bar from input.foo, so in this example I compute input.bar and input.baz from input.foo, and just append output.dir to the list of targets.
import os
def my_generator(source, target, env, for_signature):
command = './script '
command += ' '.join(['--special-in %s' % str(i) for i in source[1:]])
command += ' '
command += ' '.join([str(t) for t in target])
return command
def my_emitter(target, source, env):
source += ['%s%s' % (os.path.splitext(
str(source[0]))[0], ext) for ext in ['.bar', '.baz']]
target += ['output.dir']
return target, source
bld = Builder(generator=my_generator,
emitter=my_emitter)
env = Environment(BUILDERS={'Foo':bld})
env.Foo('output.foo', 'input.foo')
When run on linux...
>> touch input.bar input.baz input.foo
>> echo "#\!/bin/sh" > script && chmod +x script
>> tree
.
├── input.bar
├── input.baz
├── input.foo
├── SConstruct
└── script
0 directories, 5 files
>> scons --version
SCons by Steven Knight et al.:
script: v2.3.4, 2014/09/27 12:51:43, by garyo on lubuntu
engine: v2.3.4, 2014/09/27 12:51:43, by garyo on lubuntu
engine path: ['/usr/lib/scons/SCons']
Copyright (c) 2001 - 2014 The SCons Foundation
>> scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
./script --special-in input.bar --special-in input.baz output.foo output.dir
scons: done building targets.
All dependencies/targets will be maintained, if you need to feed the outputs from one builder like this into another.
If this doesn't answer your question, please clarify what more you are trying to do.
I want to be able to call avrdude from SCons as a target. For example, running scons erase-device should run the avrdude command for doing so.
I'm attempting to do this by creating Builder objects that call avrdude and adding them to the environment.
# a string forming a base avrdude command that we can just add on to in the targets
avrdude_base = 'avrdude -p ' + env['MCU'] + ' -c ' + icspdevice
# target to erase everything--flash, EEPROM, and lock bits (but not fuse bits)
erase_dev = Builder(action = avrdude_base + ' -e')
env.Append(BUILDERS = {'EraseDevice' : erase_dev})
ed = env.EraseDevice()
eda = env.Alias('erase-device', ed)
env.AlwaysBuild(eda)
# target to write the AVR fuses and lock bits
write_fuse = Builder(action = avrdude_base + ' -U lfuse:w:' + lfuse + ':m -U hfuse:w:' + hfuse +
':m -U efuse:w:' + efuse + ':m -U lock:w:' + lockbits + ':m')
env.Append(BUILDERS = {'WriteFuses' : write_fuse})
wf = env.WriteFuses()
wfa = env.Alias('write-fuses', wf)
env.AlwaysBuild(wfa)
With this code, scons always exits saying that there is nothing to do. I think this is because, the way the code is shown, I don't give any source files to these Builders (env.EraseDevice() and env.WriteFuses()); therefore, SCons assumes that they don't need to be called.
So that's what I tried next. I just passed in an existing filename into those two Builders to make scons happy, even though it isn't needed. The problem now is that, regardless of whether I want to run scons write-fuses, scons erase-flash, or other targets that use avrdude, scons acts as though I'm trying to write the fuses. If, for example, the file name I passed in were foo.hex, then scons now thinks that it has to run the write-fuses target every time because scons thinks that 'avrdude' was supposed to generate an output file called foo, but that file is never generated.
Also, doing this means that I have to build the hex file before erasing the device or programming the fuse bits, which wouldn't normally be necessary.
How can I create targets in SCons that do not require any sources for input and that do not generate any output?
Thanks!
You're on the right track in saying that scons isn't running anything because it doesn't have any sources to turn into outputs, but the key is that scons wants to generate targets and it didn't think there were any to build.
One easy workaround is to give the erase-device and write-fuse commands dummy targets. These target files will never be generated, so if scons determines that this target needs to be built (because it was specified on the command line or is a dependency of something on the command line), scons will always run the appropriate avrdude ... command.
I think the use of Builders is adding extra complexity that you don't need. Builders are good for creating new source to target mappings, but you don't really need to involve files.
ed = env.Command('erase.dummy', [], avrdude_base + ' -e')
ed = env.EraseDevice()
env.AlwaysBuild(ed)
env.Alias('erase-device', ed)
...
As a side note, scons --tree=all is a nice way to see scons' calculated dependency tree. If you're mystified by what scons is doing, seeing the dependency tree can help with debug where your model diverges from scons.
I want to inject a "Cleanup" target which depends on a number of other targets finishing before it goes off and gzip's some log files. It's important that I not gzip early as this can cause some of the tools to fail.
How can I inject a cleanup target for Scons to execute?
e.g. I have targets foo and bar. I want to inject a new custom target called 'cleanup' that depends on foo and bar and runs after they're both done, without the user having to specify
% scons foo cleanup
I want them to type:
% scons foo
but have scons execute as though the user had typed
% scons foo cleanup
I've tried creating the cleanup target and appending to sys.argv, but it seems that scons has already processed sys.argv by the time it gets to my code so it doesn't process the 'cleanup' target that I manually append to sys.argv.
you shouldn't use _Add_Targets or undocumented features, you can just add your cleanup target to BUILD_TARGETS:
from SCons.Script import BUILD_TARGETS
BUILD_TARGETS.append('cleanup')
if you use this documented list of targets instead of undocumented functions, scons won't be confused when doing its bookkeeping. This comment block can be found in SCons/Script/__init__.py:
# BUILD_TARGETS can be modified in the SConscript files. If so, we
# want to treat the modified BUILD_TARGETS list as if they specified
# targets on the command line. To do that, though, we need to know if
# BUILD_TARGETS was modified through "official" APIs or by hand. We do
# this by updating two lists in parallel, the documented BUILD_TARGETS
# list, above, and this internal _build_plus_default targets list which
# should only have "official" API changes. Then Script/Main.py can
# compare these two afterwards to figure out if the user added their
# own targets to BUILD_TARGETS.
so I guess it is intended to change BUILD_TARGETS instead of calling internal helper functions
One way is to have the gzip tool depend on the output of the log files. For example, if we have this C file, 'hello.c':
#include <stdio.h>
int main()
{
printf("hello world\n");
return 0;
}
And this SConstruct file:
#!/usr/bin/python
env = Environment()
hello = env.Program('hello', 'hello.c')
env.Default(hello)
env.Append(BUILDERS={'CreateLog':
Builder(action='$SOURCE.abspath > $TARGET', suffix='.log')})
log = env.CreateLog('hello', hello)
zipped_log = env.Zip('logs.zip', log)
env.Alias('cleanup', zipped_log)
Then running "scons cleanup" will run the needed steps in the correct order:
gcc -o hello.o -c hello.c
gcc -o hello hello.o
./hello > hello.log
zip(["logs.zip"], ["hello.log"])
This is not quite what you specified, but the only difference between this example and your requirement is that "cleanup" is the step that actually creates the zip file, so that is the step that you have to run. Its dependencies (running the program that generates the log, creating that program) are automatically calculated. You can now add the alias "foo" as follows to get the desired output:
env.Alias('foo', zipped_log)
In version 1.1.0.d20081104 of SCons, you can use the private internal SCons method:
SCons.Script._Add_Targets( [ 'MY_INJECTED_TARGET' ] )
If the user types:
% scons foo bar
The above code snippet will cause SCons to behave as though the user had typed:
% scons foo bar MY_INJECTED_TARGET