Scons Install Builder with hard link support - scons

I am using Scons for building an environment which include propagating a partial directory structure including hard links.
So it looks like this, I compile some sources into $BIN directory, some of the created targets are hard links, and in some other module I am scanning the $BIN directory and triggering a builder which either strip the the binaries to the new target, but maintain hard links. So if there are two files which are hard links in source, it copies the first and create hard link to the other target for the second.
Now the problem is if I try to run this with -j4 for example, a race condition could happen, because at the moment the builder tries to create a hard link, the target which is created by strip is not done yet.
Below is my builder:
def GenerateM3Strip(target, source, env):
if type(source) is list:
source = source[0]
if type(target) is list:
target = target[0]
src_stat = os.stat(source.abspath)
if (src_stat.st_nlink > 1
and src_stat.st_ino in GenerateM3Strip.inodes_dict):
previous_target = GenerateM3Strip.inodes_dict[src_stat.st_ino]
if target.get_dir().get_path() in previous_target:
source = previous_target[target.get_dir().get_path()]
print('LinkTarget: %s %s' % (source.get_path(), target.get_path()))
try:
os.unlink(target.get_path())
except Exception:
pass
os.link(source.get_path(), target.get_path())
return
# We have to keep a dictionary of target directory and target pairs, to
# support separate hard links for different directory targets, for example:
# fsdata/default and fsdata/bench
GenerateM3Strip.inodes_dict[src_stat.st_ino] = {
target.get_dir().get_path(): target}
action_str = env.subst(cross + 'strip -o $TARGET $SOURCE',
source=source, target=target)
env.Execute(env.Action(action_str))
GenerateM3Strip.inodes_dict = {}

Related

Remove outdated intermediate files before the build

I have a project where a lot of the source files needs to be modified by a script before they are compiled.
The build process has 2 steps:
Run a script on the original sources to create intermediate sources.
Compile the intermediate sources.
It works fine when the original source files is modified or a new one is created.
In such cases SCons is able to build / rebuild the appropriate files.
However, when a source file is deleted, the corresponding intermediate file is not removed, which may end in successful build where it should fail due to missing source.
Example:
SConscript:
env = Environment()
source_files = ['main.cc.template', 'some-header.hh.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
env.Program('my-program', intermediates)
main.cc.template:
#include "some-header.hh"
int main() {
return get_number();
}
some-header.hh.template:
inline int get_number() {
return 0;
}
This compiles correctly but if you remove the file some-header.hh.template from the list and from the filesystem, it still compiles while it shouldn't.
You need to manually remove the intermediate file some-header.hh from the file system or else you'll get a false-positive result of build and subsequent tests.
I would like to automate the deletion process to prevent inevitable broken commits that will happen if I won't.
I've managed to create a dirty solution of the problem:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
old_intermediates = Glob('*.hh') + Glob('*.cc')
intermediates_to_delete = [x for x in old_intermediates if x not in intermediates]
for x in intermediates_to_delete:
x.remove()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This more or less works.
However, the files are removed too late and SCons seem to already be aware of their presence which causes the build error to origin from SCons and not the C++ compiler.
Because of that, the error message is less helpful. Also, I don't know if such operations are good for the stability of the SCons itself.
The error I'm getting is:
scons: *** [main.o] /home/piotr/tmp/some-header.hh: No such file or directory
Is there a clear way to delete outdated intermediate files?
Your approach is more or less correct.
SCons doesn't have any built-in mechanism to remove such dangling intermediate files; you need to write your own.
The error you're getting is caused by the fact you've used the SCons function Glob. It creates File nodes and makes SCons aware of existence of those files.
(Btw, the SCons function remove() is not designed to be called outside of a builder; you shouldn't do that.)
To avoid the problem, you need to delete the file before SCons has a chance to search for it.
You can just replace SCons function with standard Python library, like pathlib.
(It will require some tinkering to convert intermediates to pathlib objects too, but it won't be that much more code.)
A fixed solution:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
from pathlib import Path
old_intermediates = list(Path.cwd().glob('*.hh')) + list(Path.cwd().glob('*.cc'))
current_intermediates = [Path(x.get_path()).resolve() for x in intermediates]
intermediates_to_delete = [x for x in old_intermediates if x.resolve() not in current_intermediates]
for x in intermediates_to_delete:
print('unlink:', x)
x.unlink()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This gives the expected error message:
main.cc:1:10: fatal error: some-header.hh: No such file or directory
1 | #include "some-header.hh"
| ^~~~~~~~~~~~~~~~
compilation terminated.
scons: *** [main.o] Error 1

I am trying to add a recipe to my new meta-layer, but facing ERROR NOTHING FOUND

I want to create a recipe for this https://github.com/kuscsik/streamfs in my new layer (meta-example) and include it to image.
My layer is added in bblayers.conf :
~/rdk/build-raspberrypi-rdk-hybrid$ bitbake-layers show-layers
**layer path priority**
**meta-example /home/xyz/rdk/build-raspberrypi-rdk-hybrid/meta-example 6**
This is my path to layer.conf and content in layer.conf:
~/rdk/build-raspberrypi-rdk-hybrid/meta-example/conf$ vi layer.conf
#We have a conf and classes directory, add to BBPATH
BBPATH .=":${LAYERDIR}"
#We have recipes-* directories, add to BBFILES
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb\
${LAYERDIR}/recipes-*/*/*.bbapend"
BBFILE_COLLECTIONS += "example"
BBFILE_PATTERN_example = "^${LAYERDIR}/"
BBFILE_PRIORITY_example = "6"
Then i have created a directory(example) inside meta-example which contains streamfs_git.bb with contents as shown below:
~/rdk/build-raspberrypi-rdk-hybrid/meta-example/example$ vi streamfs_git.bb
DESCRIPTION = "First recipe"
HOMEPAGE = "https://github.com/kuscsik/streamfs"
LICENSE = "LGPL-2.1"
LIC_FILES_CHKSUM = "file://LICENSE;md5=fc178bcd425090939a8b634d1d6a9594"
inherit cmake pkgconfig
SRC_URI = "git://github.com/kuscsik/streamfs"
SRCREV = "${AUTOREV}"
S = "${WORKDIR}/git"
Then i run this command :~/rdk/build-raspberrypi-rdk-hybrid/meta-example/example$ bitbake streamfs_git
It shows me this error
WARNING: No bb files matched BBFILE_PATTERN_example '^/home/xyz/rdk/build-raspberrypi-rdk-hybrid/meta-example/'
**ERROR: Nothing PROVIDES 'streamfs_git'
I have even tried bitbake streamfs_git.bb and bitbake streamfs also, all are giving same error.
How can I fix the error? Do I have to add something in my layer.conf or .bb file or is there an error in any of my steps?
I've noticed two potential errors right away within your BBFILES declaration. You have
BBFILES += "${LAYERDIR}/recipes-//.bb
${LAYERDIR}/recipes-//.bbapend"
The first issue being that BBFILES is looking for any recipes in AND ONLY in a directory called 'recipes-'. I'm assuming (I could be wrong) that your recipes directory isn't called 'recipes-'. To ensure any recipes within a folder are getting parsed, make sure your bbfiles properly points to the directory that it resides in. In this case, perhaps change 'recipes-' to 'recipes-[directory-name]', although you can also use wildcards here, so something like 'recipes-*' would work as well, giving you the ability to parse any recipes within any directory named 'recipes-[anything]'.
Additionally, you seem to have an extraneous '/' in that same line, and no declaration of your recipe name. It's generally good practice in yocto keep your recipes in a subfolder of a recipes-* directory, though it's only for organizational reasons. You're also only pointing to a recipes named '.bb'. You should have either a wildcard, or the exact recipe name here. Also, don't forget to separate your multilined variables with a \ at the end of each line. It should look something like:
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
Moving to the second line, it's the same issues with lack of wildcards/direct definitions, although there also seems to be a typo in 'bbapend', which is just in spelling, should be 'bbappend'.
Ultimately, you should end up with something like this
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
${LAYERDIR}/recipes-*/*/*.bbappend"
There may be more issues, though work on this stuff for now, and I'll help out more if this doesn't solve your issue.

Example waf project with vala

there is a nice waf vala example here:
https://code.launchpad.net/~asabil/vala/vala-project-template.waf
and it shows a library and an application in vala. Unfortunately the program in this example does not actually USE the library (which defines method "hello"). When I try to call it from the program, I get compilation errors.
I am not able to modify the wscript's to load the library properly. What is the trick here? Thanks.
What I have added is this line in the program:
My.Lib.hello();
But it won't compile:
Waf: Entering directory `/home/lzap/work/shellmail/TEST/vala-template/_build_'
[1/6] valac: src/hello-gtk.vala -> _build_/default/src/hello-gtk.c
../src/hello-gtk.vala:16.9-16.10: error: The name `My' does not exist in the context of `Sample.create_widgets._lambda0_'
Waf: Leaving directory `/home/lzap/work/shellmail/TEST/vala-template/_build_'
Build failed: -> task failed (err #1):
{task: valac_task hello-gtk.vala -> hello-gtk.c}
I guess I need to change the program wscript:
#!/usr/bin/env python
def build(bld):
prog = bld(features='cc cprogram')
# symbolic name used to reference this object
prog.name = 'hello-gtk.program'
# name of the resulting program
prog.target = 'hello-gtk'
prog.source = 'hello-gtk.vala'
# libraries to link against
prog.uselib = 'GTK+'
# Vala packages to use
prog.packages = 'gtk+-2.0'
# Extra vapi dirs
#prog.vapi_dirs = '../my_lib'
# Enable threading
#prog.threading = True
The hello method is not an static method, but an instance method, so, you need to create a My.Lib instance first and then call the method.
var obj = new My.Lib();
obj.hello();
If this still failing, try to add using My; on hello-gtk.vala.

SCons manual build step

Is it possible to get SCons to remind me to perform a manual step using it's dependancy tracking?
My build uses the .swc output from a .fla, which you can't do using a command-line.
I tried something like:
env.Command(target, sources + SHARED_SOURCES,
Action(lambda target, source, env: 1, "Out of date: $TARGET"))
But with that method, I have to use Decider('make') or I get:
$ scons --debug=explain
scons: rebuilding `view_bin\RoleplaySkin.swc' because `view_src\RoleplaySkin.fla' changed
Out of date: view_bin\RoleplaySkin.swc
scons: *** [view_bin\RoleplaySkin.swc] Error 1
And, more importantly, SCons never realizes it's cache is out of date, so any change in the Environment or sources since it wrote the signature in .sconsign.dblite means it will allways try to rebuild (and therefore, always fail).
What about using the Precious method to protect the *.swc output before converting it into a *.fla?
How about creating your own RemindMe builder which reminds you and fails to build the target?
It would look something like this:
def remind_me(target, source, env):
os.remove(target.abspath) #we do not build, we destroy
print ("This is a friendly reminder, your $SOURCE is out of date, run manual build step")
return None
reminder = Builder(action = remind_me,
suffix = '.swc',
src_suffix = '.fla')
env = Environment(BUILDERS = {'RemindMe' : reminder})
#Run builder like this
swc_file = env.RemindMe('some_fla_file')
final_target = env.BuildWithSWC(some_other_target,swc_file)
This is however only a theory, I have never tried actually deleting the target instead of creating it. It might be worth a try at least.

How to use Scons to compile same objects in different environments with Glob?

I have a C++ project builds with Scons. At first I have only the optimized version to compile, it works fine. Then I also need a debug version, then I add another environment for it. Here is the Scons code:
env = Environment()
opt = env.Clone(CCFLAGS=['-pthread', '-O3', '-Wall'])
opt_objs = opt.Glob('src/*.cpp')
prog = opt.Program('prog', opt_objs)
dbg = env.Clone(CCFLAGS=['-pthread', '-Wall', '-g', '-O0'])
dbg_objs = dbg.Glob('src/*.cpp')
dbg_prog = dbg.Program('dbg_prog', dbg_objs)
With this code, I ran into error:
scons: *** Two environments with different actions were specified for the same target:
src/CometReadService.o
As you can see, those .o files targets created by opt.Glob('src/.cpp') and dbg.Glob('src/.cpp') exactly same name. By reading the document Multiple Construction Environments I know I can rename the object like "opt.Object('xxx-opt', 'xxx.c')", but however, it is Glob not Object. How can I solve this problem?
The scons manual describes how to use the VariantDir function (or argument when adding SConscripts) to set up different build directories. At its simplest, VariantDir separates the build output from the source files, but it can also be used to separate the build output of different environments.
env = Environment()
opt = env.Clone(CCFLAGS=['-pthread', '-O3', '-Wall'])
opt.VariantDir('gen-opt', 'src', duplicate=0)
opt_objs = opt.Glob('gen-opt/*.cpp')
prog = opt.Program('prog', opt_objs)
dbg = env.Clone(CCFLAGS=['-pthread', '-Wall', '-g', '-O0'])
dbg.VariantDir('gen-dbg', 'src', duplicate=0)
dbg_objs = dbg.Glob('gen-dbg/*.cpp')
dbg_prog = dbg.Program('dbg_prog', dbg_objs)
Using VariantDir can take some experimentation. For instance, note that the Glob argument has changed -- without the duplicate=0 parameter, the default behavior is for VariantDir to duplicate the source files in the build directory.

Resources