I have the following project structure:
/project
/src_dirs
/src_dir_1
/...
/include_dirs
/inc_dir_1
/...
/output
SConstruct
/Sconscripts
lib1_sconscript
lib2_sconscript
/...
objects/
/...
libs/
/...
The build process invoked from the output directory. so all the paths in the sconscripts, are relative to output directory. my sconscript files are auto generated. as you can see below, the paths to source files and to include files are relative paths. this is a demo sconscript file:
Import('libaEnv')
includes = ['../../project/include_dirs/inc_dir_1/be/cmn/inc', '../../project/include_dirs/inc_dir_1/be/be/cmn/api_inc']
directives = ['-CC','-c','-g']
code_switches = ['FAST_RUNG_MLK', 'EN_PRIORITY']
listDefines = code_switches
linkLibs = []
crt = libaEnv.Object(target='objects/crt.o', source='../../project/src_dirs/src_dir_1/crt.c', CPPDEFINES=listDefines, CPPFLAGS=directives, CPPPATH=includes)
ert = libaEnv.Object(target='objects/ert.o', source='../../project/src_dirs/src_dir_1/ert.c', CPPDEFINES=listDefines, CPPFLAGS=directives, CPPPATH=includes)
urt = libaEnv.Object(target='objects/urt.o', source='../../project/src_dirs/src_dir_1/urt.c', CPPDEFINES=listDefines, CPPFLAGS=directives, CPPPATH=includes)
liba = libaEnv.Library (target='libs/liba.a', source=[crt,ert,urt])
Return('liba')
I have seen that scons invokes the compiler with absolute paths to the source and the headers files. i have seen this by running scons with -verbose (i have also validate this by printing the command line in Action.py in the spawn method).
My scons version is: SCons 2.5.1 and i am running with python 2.7.
How can i force scons to invoke the compiler with relative paths only ?
One approach here is you can use top-relative addressing, as in:
includes = ['#project/include_dirs/inc_dir_1/be/cmn/inc', '#project/include_dirs/inc_dir_1/be/be/cmn/api_inc']
This still gets you absolute paths, but they'll be computed based on a valid starting point (# = "the directory containing the top-level SConsctruct", thus top-relative)
Related
I have a project where a lot of the source files needs to be modified by a script before they are compiled.
The build process has 2 steps:
Run a script on the original sources to create intermediate sources.
Compile the intermediate sources.
It works fine when the original source files is modified or a new one is created.
In such cases SCons is able to build / rebuild the appropriate files.
However, when a source file is deleted, the corresponding intermediate file is not removed, which may end in successful build where it should fail due to missing source.
Example:
SConscript:
env = Environment()
source_files = ['main.cc.template', 'some-header.hh.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
env.Program('my-program', intermediates)
main.cc.template:
#include "some-header.hh"
int main() {
return get_number();
}
some-header.hh.template:
inline int get_number() {
return 0;
}
This compiles correctly but if you remove the file some-header.hh.template from the list and from the filesystem, it still compiles while it shouldn't.
You need to manually remove the intermediate file some-header.hh from the file system or else you'll get a false-positive result of build and subsequent tests.
I would like to automate the deletion process to prevent inevitable broken commits that will happen if I won't.
I've managed to create a dirty solution of the problem:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
old_intermediates = Glob('*.hh') + Glob('*.cc')
intermediates_to_delete = [x for x in old_intermediates if x not in intermediates]
for x in intermediates_to_delete:
x.remove()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This more or less works.
However, the files are removed too late and SCons seem to already be aware of their presence which causes the build error to origin from SCons and not the C++ compiler.
Because of that, the error message is less helpful. Also, I don't know if such operations are good for the stability of the SCons itself.
The error I'm getting is:
scons: *** [main.o] /home/piotr/tmp/some-header.hh: No such file or directory
Is there a clear way to delete outdated intermediate files?
Your approach is more or less correct.
SCons doesn't have any built-in mechanism to remove such dangling intermediate files; you need to write your own.
The error you're getting is caused by the fact you've used the SCons function Glob. It creates File nodes and makes SCons aware of existence of those files.
(Btw, the SCons function remove() is not designed to be called outside of a builder; you shouldn't do that.)
To avoid the problem, you need to delete the file before SCons has a chance to search for it.
You can just replace SCons function with standard Python library, like pathlib.
(It will require some tinkering to convert intermediates to pathlib objects too, but it won't be that much more code.)
A fixed solution:
env = Environment()
source_files = ['main.cc.template']
def make_intermediate(env, source):
target = source[:-9] # Remove ".template" from the file name
return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])
# --- The new starts code here ---
from pathlib import Path
old_intermediates = list(Path.cwd().glob('*.hh')) + list(Path.cwd().glob('*.cc'))
current_intermediates = [Path(x.get_path()).resolve() for x in intermediates]
intermediates_to_delete = [x for x in old_intermediates if x.resolve() not in current_intermediates]
for x in intermediates_to_delete:
print('unlink:', x)
x.unlink()
# --- The new code ends here ---
env.Program('my-program', intermediates)
This gives the expected error message:
main.cc:1:10: fatal error: some-header.hh: No such file or directory
1 | #include "some-header.hh"
| ^~~~~~~~~~~~~~~~
compilation terminated.
scons: *** [main.o] Error 1
Building my workspace with colcon, some OSM files which are found in a directory "OSM" in a sub_package in the workspace are not found in built space. So when I go to the install space, the files are not there. I am not sure how to do this and if I should put it in the setup.py.
I tried putting this in the setup.py file in arguments of setup():
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
('.package_name/sub_package', glob('OSM_folder/*.osm')),
],
.
.
.
) # close setup()
but it did not work.
I am using ROS2 Galactic.
Directory structure:
package_name
┃
┣━━━━setup.py
┣━━━━package.xml
┣━━━━resource/
┣━━━━launch/
┗━━━━package_name
┗━sub_package_name
┗━OSM
┣━__init__.py
┗━some_osm_files.osm
I have the OSM directory in the built workspace but it has only the init.py file
I solved it. I am not sure if this is the right way of doing this or is there like another better/proper way or not, but here we go.
In the setup.py file, I added the line
(os.path.join('lib/python3.8/site-packages/package_name/sub_package/OSM'),glob(package_name+'/sub_package_name/OSM/*.osm')), in the data_files variable.
The first part of the new line which is os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM') determines the new location of the files in the install folder after building the workspace.
The second part which is glob(package_name+'/sub_package_name/OSM/*.osm') determines the files original location in the project workspace.
So the result is that it takes the files from the location mentioned in the second part and puts them in the location mentioned in the first part.
The resulting block is:
setup(
name=package_name,
version='0.0.0',
packages=[package_name, submodules, osm],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py')),
(os.path.join('lib/python3.8/site-packages/package_name/sub_package_name/OSM'), glob(package_name+'/sub_package_name/OSM/*.osm')),
],
.
.
.
)
I'm trying to make a python package and I have most of the things already setup by when I try to install the library from Github here, it installs everything except for the folder called champs and it's files
This is my File directory structure
LeagueYue
champs
-Lname_num.json
-Lname_Uname.json
-num_Uname.json
-__init__.py
-champion_files.py
-external.py
-match.py
-rank.py
-status.py
-summoner.py
-requirements.txt
-setup.py
All the files are installed except for the folder and the files inside champs
As this question answers:
There are 2 ways to add the static files:
1) Include_package_data=True + MANIFEST.in
A MANIFEST.in file in the same directory of setup.py, that looks like this:
include src/static/*
include src/Potato/*.txt
2) Package_data in setup.py
package_data = {
'static': ['*'],
'Potato': ['*.txt']
}
Specify the files inside the setup.py.
Two of the files could probably be derived at runtime from num_Uname.json, but that's fine.
I do not yet see a data_files directive in https://github.com/CharmingMother/LeagueLib/blob/async/setup.py
Thomas Cokelaer suggests using an expression like
datafiles = [(datadir, list(glob.glob(os.path.join(datadir, '*'))))]
and then
setup(
...
data_files = datafiles,
)
in http://thomas-cokelaer.info/blog/2012/03/how-to-embedded-data-files-in-python-using-setuptools/
In your case this could be as simple as:
data_files = [('', ['champs/num_Uname.json'])],
Martin Thoma explains you should access them using filepath = pkg_resources.resource_filename(__name__, path)
in How to read a (static) file from inside a Python package?
When I Read The Fine Manual, this setup.cfg alternative surfaces:
[options.data_files]
...
data = data/img/logo.png, data/svg/icon.svg
suggesting a line like . = champs/num_Uname.json or champs = num_Uname.json
I have the following project structure:
/prj
SConstruct
/src
/app
/lib1
/lib2
/...
'/prj/src/lib1' structure:
/lib1
/src
/test
SConscript
'lib1/SConscript':
SConscript('test/test1/SConscript',
exports = 'env',
variant_dir = '#build/release/lib1/test',
duplicate = 0)
and, finally, 'test' directory:
/test
/common
helpers.cpp
/test1
SConscript
main.cpp
In 'test/test1/SConscript' sources specified as:
Sources = ['../common/helpers.cpp', 'main.cpp']
Result:
scons: *** [build/release/lib1/common/helpers.o]
Source `build/release/lib1/common/helpers.cpp' not found,
needed by target `build/release/lib1/common/helpers.o'
As can be seen the problem is that scons tries to find out the source file 'helpers.cpp' in build directory, not in source one.
Some research shows that the problem raised when source file path begins with '../'. When all sources defined underneath 'SConscript' file all is Ok.
Scons v2.5.1 and v3.0.1 demonstrate the same behavior.
What I did wrong? I've found this answer where the author advised:
You could use ../test.cpp as filename
but I doing exactly the same. Is such scons behavior intended or this is a bug?
Your "code" in "lib1/SConscript":
SConscript('test/test1/SConscript',
exports = 'env',
variant_dir = '#build/release/lib1/test',
duplicate = 0)
uses the name of the given "test/test1/SConscript" and implicitly links the folder "test/test1" as "variant folder" (=variant_dir) to the target directory "#build/release/lib1/test". So if a required file can't be found in "#build/release/lib1/test", SCons tries to do an alternative "lookup" in "test/test1/". But this link is not automatically set for the "common" folder as well, and that's why the lookup of the "helpers.cpp" fails. This is the intended behaviour and correct in SCons.
A solution for your current problem would be to move the "test/test1/SConscript" one folder level higher, include the sources there as "common/helpers.cpp" and "test/main.cpp" and call this new SConscript from "lib1/SConscript" as:
SConscript('test/SConscript',
exports = 'env',
variant_dir = '#build/release/lib1/test',
duplicate = 0)
I have a C++ project builds with Scons. At first I have only the optimized version to compile, it works fine. Then I also need a debug version, then I add another environment for it. Here is the Scons code:
env = Environment()
opt = env.Clone(CCFLAGS=['-pthread', '-O3', '-Wall'])
opt_objs = opt.Glob('src/*.cpp')
prog = opt.Program('prog', opt_objs)
dbg = env.Clone(CCFLAGS=['-pthread', '-Wall', '-g', '-O0'])
dbg_objs = dbg.Glob('src/*.cpp')
dbg_prog = dbg.Program('dbg_prog', dbg_objs)
With this code, I ran into error:
scons: *** Two environments with different actions were specified for the same target:
src/CometReadService.o
As you can see, those .o files targets created by opt.Glob('src/.cpp') and dbg.Glob('src/.cpp') exactly same name. By reading the document Multiple Construction Environments I know I can rename the object like "opt.Object('xxx-opt', 'xxx.c')", but however, it is Glob not Object. How can I solve this problem?
The scons manual describes how to use the VariantDir function (or argument when adding SConscripts) to set up different build directories. At its simplest, VariantDir separates the build output from the source files, but it can also be used to separate the build output of different environments.
env = Environment()
opt = env.Clone(CCFLAGS=['-pthread', '-O3', '-Wall'])
opt.VariantDir('gen-opt', 'src', duplicate=0)
opt_objs = opt.Glob('gen-opt/*.cpp')
prog = opt.Program('prog', opt_objs)
dbg = env.Clone(CCFLAGS=['-pthread', '-Wall', '-g', '-O0'])
dbg.VariantDir('gen-dbg', 'src', duplicate=0)
dbg_objs = dbg.Glob('gen-dbg/*.cpp')
dbg_prog = dbg.Program('dbg_prog', dbg_objs)
Using VariantDir can take some experimentation. For instance, note that the Glob argument has changed -- without the duplicate=0 parameter, the default behavior is for VariantDir to duplicate the source files in the build directory.