I have been using cmake to build my projects out of source, which is really convenient as you avoid polluting your source directory with unnecessary files.
Assuming the CMakeLists.txt is in the current directory, this could be done as follows:
mkdir build
cd build
cmake ..
make
How can I do the same in scons?
In your SConstruct file, you use a variant dir:
SConscript("main.scons", variant_dir="build", duplicate=0)
Then in main.scons you set up everything as usual:
env = Environment()
env.Program(target='foo', source=Split('foo.c bar.c'))
It's possible to do this without hardcoding the variant dir into the SConstruct by (ab)using repositories, but that approach has its bugs. For the record, you would run the above as follows to build in another directory:
mkdir mybuild
cd mybuild
scons -Y .. -f ../main.scons
The easiest and most workable is to just use variant_dir. You then run this as usual from the top level source directory. All the build artefacts get produced in the build sub directory.
In response to JesperE's comment, here is how you could write the top level SConstruct to add an optionally named build directory:
AddOption('--build', default='build')
SConscript("main.scons", variant_dir=GetOption('build'), duplicate=0)
Then you would call this from the command line as follows, to create a build directory called "baz":
$ scons --build=baz
Related
I'm working on adding googletest unit testing to a large project.
We have a top level SConstruct, which calls SConscripts for each of the git submodules, which may further call other SConstructs contained in each component.
One of these SConscripts will build the googletest framework, which will create the 'gtest.h' file among others. I need to include 'gtest.h' in my source file (test1.cpp) which is in a different repo that has it's own SConscript.
How can I require the gtest.h in test SConscript file before test1.cpp consumes it?
Imagine something like:
Top level SConstruct:
env.SConscript('{path to Gtest}/SConscript')
env.SConscript('{path to my unit test}/SConscript')
SConscript for building Gtest:
env2 = env.Clone()
def buildGtest(target, source, env):
#Assuming I have a CMake file that does this in the directory...
subprocess.run(['cmake', '../'], cwd='build/')
subprocess.run(['make'], cwd='build/')
env2.Command(['gtest.h', 'build/libgtest.a', 'build/libgtest_main.a'], [], buildGtest)
SConscript for my unit test:
env2 = env.Clone()
env2.Require('{path to gtest}/build/include/gtest.h') # This doesn't seem to work
env2.Append(CPPPATH='{path to gtest}/build/include')
env2.Object(target = 'test1.o', source = 'test1.cpp')
You need to ensure that your CPPPATH includes the location where googletest generates the header file.
This is not great. You should specify at least one source file. I'm guessing you have a CmakeList.txt and a bunch of source files you should list here, so it will rebuild if they change.
env2.Command(['gtest.h', 'build/libgtest.a', 'build/libgtest_main.a'], [], buildGtest)
Additionally I'd change your buildGtest to the following.
buildGtest = ["cd build; cmake ../','cd build; make']
And get rid of your function.
env2.Command(['gtest.h', 'build/libgtest.a', 'build/libgtest_main.a'], [], buildGtest)
Then SCons will scan your source file and know that it requires that file.
You can check the dependency tree that SCons is aware of by running as
scons --tree=prune
It may be quite verbose, but you'll see all the dependencies.
You should also be able to run as
scons --tree=prune {path to my unit test}/test.o
Suppose I execute cargo new one --bin and cargo new two --bin then add the same dependency to each project's Cargo.toml and build them.
Now there are two absolutely identical sets of libraries:
/one/target/debug/deps/ *.rlib
/two/target/debug/deps/ *.rlib
They are same files and waste storage space, but really the problem is that I have to compile these libraries again for every project. It takes a very much time. There is the same problem with cargo install.
Can I specify a place to store compiled libraries to avoid recompilation?
Several Cargo projects might share the libraries by using the same target dir.
.cargo/config
Place a ".cargo" folder in a project and create a "config" file there containing:
[build]
target-dir = "/path/to/your/shared/target/dir"
On Unix this might look like:
mkdir ~/shared_rust_target
mkdir .cargo
echo "[build]" > .cargo/config
echo "target-dir = \"$HOME/shared_rust_target\"" >> .cargo/config
CARGO_TARGET_DIR
Set the CARGO_TARGET_DIR environment variable.
On Unix this might look like:
export CARGO_TARGET_DIR = "$HOME/shared_rust_target"
See this commit for some extra target-dir documentation.
In particular, prior to Cargo 1.9 you shouldn't build several projects into the same target dir concurrently. (Here's more on how Cargo 1.9 supports concurrent builds).
target-dir is also mentioned in the Cargo docs.
Should work, according to this issue.
P.S. It is now also possible to achieve crate reuse with workspaces.
P.S. https://docs.rs/cargo-hakari/latest/cargo_hakari/ helps with keeping some of dependencies compatible between projects.
Even if there is a way to do it, you probably don't want to. Just because you happen to be using the same libraries doesn't mean that they were compiled the same. For example, Cargo supports the concept of features, compilation time configuration that changes how the crate was compiled.
Likewise, you may need to support multiple versions of Rust, such as nightly and stable. Or perhaps you need to cross-compile for a different architecture. Each of those will produce different code.
Cargo will cache the build products of a single project, so I've found the overhead not really noticeable, and I compile a lot of projects from people asking questions on Stack Overflow! :-)
I managed to piece together code from a few answers, but mainly this one.
This Dockerfile should not just cache Cargo dependancy downloads, but also their compilations and the crates.io index. All the other answers I could find only cached the downloads or the index, not both.
FROM arm64v8/rust as builder
# Capture dependencies
COPY Cargo.toml Cargo.lock /app/
# We create a dummy main.rs to build deps
WORKDIR /app
RUN mkdir src && echo "fn main() {}" > src/main.rs
# RUN rustup install nightly && rustup default nightly
# This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
# Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release && rm src/main.rs
# Copy our sources
COPY ./src /app/src
# A bit of magic here!
# * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
# * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
RUN --mount=type=cache,target=/usr/local/cargo/registry \
set -e && \
# update timestamps to force a new build &&
touch /app/src/main.rs && \
cargo build --release
# Again, our final image is the same - a slim base and just our app
FROM debian:buster-slim as app
COPY --from=builder /app/target/release/app/app
CMD ["/app"]
Take note of the FROM arm64v8, if you are targeting x86, replace the builder and app FROMs with their respective x86 versions.
My project uses external packages. These packages are distributed in tarball and their contents is extracted when the tarballs are updated.
For example, I use boost. The corresponding external package is boost.tar.bz2. This package contains the header files and libraries. What I would like to do is to automatically extract the contents of this archive with scons when the tarball is updated.
I can achieve this with a Makefile using a "timestamp file". When the tarball is newer than the timestamp file, the archive is automatically extracted:
all: external-packages
external-packages: boost xml2
boost: .boost-timestamp
xml2: .xml2-timestamp
.boost-timestamp: boost.tar.bz2
#echo updating boost externals
#tar xjf boost.tar.bz2
#touch .boost-timestamp
.xml2-timestamp: xml2.tar.bz2
#echo updating xml2 externals
#tar xjf xml2.tar.bz2
#touch .xml2-timestamp
clean:
rm -rf .*-timestamp boost xml2
How can I achieve the same with scons?
I think your best bet is to have a look at the untar builder.
After you add the addition of the builder function/emitter in your site_scons folder your scons file might look like this:
env = Environment()
env..Append(BUILDERS = {'UnTar' : unTarBuilder})
external_package = "packages/boost.tar.bz2"
archive = env.UnTar(source=external_package)
You should however note that the untar builder doesn't take directories into the emitter, and thus does not delete these on a clean
The point with this builder is the emitter it will make sure that SCons knows that it can create the headers, so whenever something depends on them (someone include them) it will launch the untar builder whenever the tar file has changed.
I already could compile different targets or flavours(debug release), but the problem is when I make: make debug or make release this generate objects and the library in the respective folder folder.
This is the Makefile.am:
AM_CXXFLAGS = #AM_CXXFLAGS#
ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS}
lib_LIBRARIES = libInitDB.a
libInitDB_a_SOURCES = \
InitDB.cpp
.PHONY: debug release
debug:
make CXXFLAGS='$(CXX_DEBUG_FLAGS) $(CXXFLAGS)'
mkdir -p $(DEBUG_DIR)
mv $(lib_LIBRARIES) $(DEBUG_DIR)/$(lib_LIBRARIES)
mv *.o $(DEBUG_DIR)
release:
make CXXFLAGS='$(CXX_RELEASE_FLAGS) $(CXXFLAGS)'
mkdir -p $(RELEASE_DIR)
mv $(lib_LIBRARIES) $(RELEASE_DIR)/$(lib_LIBRARIES)
mv *.o $(RELEASE_DIR)
but the problem is when I make: make debug or make release again, as I move the objects and the library, that generates again the objects and the library that already stored in the debug or release folder.
Could someone help me to find how to avoid this and when I compile that search in the correct folder?
Instead of having one Makefile generating both flavours of your targets, you could have two separate build directories configured with different options.
For example:
mkdir debug
(cd debug && ../configure --enable-debug)
mkdir release
(cd release && ../configure --enable-release)
This way, you can go to either directory and recompile only what is needed simply typing make.
I just started learning to use SCons, anticipating that it solves some of my issues with make. I'm creating a source hierarchy to understand the basics of SCons.
Let's start with this folder structure:
test/foo: contains main.cpp, main.h
test/bar: contains its own main.cpp, main.h
test/common: contains utils.cpp and utils.h used by both foo and bar
test/external/moo: the source to some external library, containing 'configure' which produces 'Makefile' (not using SCons), so SCons needs to invoke 'make' after 'configure'; I suspect this part might be tricky when build dirs are used
test/build/debug: build dir for debug
test/build/release: build dir for release
Here's what I'd like to do:
Have two types of builds: debug/release where the only difference is that debug specifies -DDEBUG to g++
Use build dirs so that no .o files are created in my source tree. Let's call these build dirs "build/debug" and "build/release"
Be able to invoke ./configure and make on another project that does not use SCons, followed by linking libmoo.a it produces with my project
Have the builds be perfectly parallel (scons -j9 for an 8-core?)
Have some debug/release-independent way of specifying libraries to link. Something like:
env.Program(target='foo', source=['foo/main.cpp', '#build/(DEBUG_OR_RELEASE)/lib/libsomething.a'])
What would the very basic SConstruct/SConscript files to do the above look like? Even just pointers in the right directions would be great too!
Thanks in advance :-)
I do this for builds for multiple platforms (rather than debug/release) but the concept's the same. The basic idea is that you need 2 files in the project root - a SConstruct to set up the build directories (or "variant directories" as they are known in scons), then a SConscript that describes the actual build steps.
In the SConstruct file you'd specify the variant directory and its corresponding source directory:
SConscript(dirs='.',
variant_dir=variant_dir,
duplicate=False,
exports="env")
Now you want variant_dir to depend on a flag. You'd use AddOption or Variables to do this. Here's one example of a complete top-level SConstruct to do that:
# build with `scons --debug-build` for debug.
AddOption(
'--debug-build',
action='store_true',
help='debug build',
default=False)
env = Environment()
if GetOption('debug_build'):
env.ParseFlags('-DDEBUG')
variant_dir = 'build/debug'
else:
variant_dir = 'build/release'
SConscript(dirs='.',
variant_dir=variant_dir,
duplicate=False,
exports="env")
AddOption is easiest to use, but if you use Variables then you can cache the result between runs, rather than having to spell out "scons --debug-build" each time.
All the directory setup and associated cruft is in the SConstruct. Now the SConscript file is quite simple and doesn't need to worry about build directories at all.
Import('env')
env.Program(target='foo_prog', source=['foo/main.cpp', 'lib/libmoo.a'])
# foo_prog since foo already exists as the name of the directory...
This is about the simplest way I've found to set up different build directories without getting weird errors. It's also pretty flexible - you can add different platform builds just by modifying the "env" in the top-level script without having to alter the actual meat of the build.
The only spanner in the works in your question is the way to compile autoconf-style projects directly from SCons. The easiest way is probably with a couple of Command() calls, but SCons likes to know about the inputs and outputs of each step, so this can get hacky. Also, you have to rely on the autoconf build having a correct VPATH setup - some projects don't work if you try and compile outside the source tree. Anyway, a way to compile autoconf projects would be something like this:
import os
Import('env')
# get the path to the configure script from the "moo" source directory
conf = env.File('moo/configure').srcnode().abspath
# Create the "moo" build directory in the build dir
build_dir = env.Dir('.').path
moo_dir = os.path.join(build_dir, 'moo')
Mkdir(moo_dir)
# run configure from within the moo dir
env.Command('moo/Makefile', 'moo/Makefile.am',
conf, chdir=moo_dir)
# run make in the moo dir
env.Command('moo/libmoo.a', 'moo/Makefile',
'make', chdir=moo_dir)
env.Program(target='foo_prog', source=['foo/main.cpp', 'moo/libmoo.a'])
Running the configure step from the source directory while the current working directory is somewhere in the build hierarchy is awkward. The make step is less messy, but still needs to know about the current build directory. Since you specify "libmoo.a" as an output of the make step and libmoo.a as an input to the program, all the dependencies Just Work, so a parallel build works fine. Parallel builds only break down when you fudge dependencies too much.
I know this is an old question, I just want to add an alternative to:
be able to know the current variant in the sconscript file (not only in the parent)
and to be able to build multiple variants in a single scons command
In the sconstruct file (the parent), we define a ListVariable named variants with the list of the variants that we allow (eg. ['release', 'debug']).
Then to be able to know the current variant in the sconscript file, we just loop option we have defined and export it into the sconscript.
I use genv as variable name to notate global environment:
# sconstruct
opts = Variables()
opts.AddVariables(
ListVariable('variants', 'list of variants to build', 'all', names = ['debug','release']),
)
genv = Environment( options = opts )
for variant in genv['variants']:
SConscript('sconscript', exports=['genv', 'variant'], variant_dir='#build/'+variant, duplicate=False)
In the sconscript file we Clone de genv and we can use the variant variable to do our setup in the local environment env:
# sconscript
Import('*')
import os.path
env = genv.Clone()
if variant == 'debug':
env.Append( CPPFLAGS = ['/Zi'])
src = 'src/hello.cpp'
app,ext = os.path.splitext(os.path.basename(src))
obj = env.Object ('obj/'+app, src)
bin = env.Program('bin/'+app, obj)
Using a ListVariable allows us to call
scons variants=release
or
scons variants=debug
or
scons variants=all
This last command (and the default command) builds all the variants.
There's a good solution to define multiple build modes ('debug', 'release') in the SCons Wiki:
http://www.scons.org/wiki/SconstructMultiple
That's how the richq SConstruct file would look like:
#get the mode flag from the command line
#default to 'release' if the user didn't specify
mymode = ARGUMENTS.get('mode', 'release')
#check if the user has been naughty: only 'debug' or 'release' allowed
if not (mymode in ['debug', 'release']):
print "Error: expected 'debug' or 'release', found: " + mymode
Exit(1)
#tell the user what we're doing
print '**** Compiling in ' + mymode + ' mode...'
env = Environment()
if mode == 'debug':
env.Append(CPPDEFINES = ['DEBUG'])
variant_dir = 'build/debug'
else:
variant_dir = 'build/release'
SConscript(dirs = '.', variant_dir = variant_dir, duplicate = False, exports = "env")
You then call scons mode=release (or just scons as the release is the default mode), or scons mode=debug.