How does scons cache the dependencies that is scans for C code?
The context of the question is I want to use that same technique in other languages.
Right now I have to do something like this:
env.Depends(target = 'lib/PROG1.so', dependency = getDependencies('src/PROG1.xxx'))
where getDependencies() returns a list of dependencies. But that will scan for dependencies on every run. I want to cache those results and only rescan if the source file changes.
If you're using Depends() you're probably doing something wrong...
(this usually means you're not specifying a target or a source, or the scanner for your builder(s) are missing files)
SCons' builders define scanners which scan the source code looking for included files and save that info.
Additionally you specify to all builders the sources for the targets they build.
Generally files are already only re-scanned if they change.
There are some command line flags to SCons which can impact the scanning of source files (From the manpage: https://scons.org/doc/production/HTML/scons-man.html )
--implicit-cache
Cache implicit dependencies. This causes scons to use the implicit (scanned) dependencies from the last time it was run instead of scanning the files for implicit dependencies. This can significantly speed up SCons, but with the following limitations:
scons will not detect changes to implicit dependency search paths (e.g. CPPPATH, LIBPATH) that would ordinarily cause different versions of same-named files to be used.
scons will miss changes in the implicit dependencies in cases where a new implicit dependency is added earlier in the implicit dependency search path (e.g. CPPPATH, LIBPATH) than a current implicit dependency with the same name.
--implicit-deps-changed
Forces SCons to ignore the cached implicit dependencies. This causes the implicit dependencies to be rescanned and recached. This implies --implicit-cache.
--implicit-deps-unchanged
Force SCons to ignore changes in the implicit dependencies. This causes cached implicit dependencies to always be used. This implies --implicit-cache.
Or did you mean something else?
Related
I am using cargo build to build for many platforms.
Given a sha1 hash of one of the binaries I have released, I want to be able to tell exactly what input was used for the compiler (in case I need to reference it in a security audit).
I am using Docker to run cargo build, so I have hashes of each image that will be stored long term. The target dir can be many GB in size, so I do not want to be committing it to a Docker image if it is not needed.
I am not completely sure what data is contained in target.
Should I keep it, or is the Cargo.lock enough to understand what input was given to the compiler?
If a Cargo crate is deleted, is there an immutable reference I can always download (so no need to keep target which would contain the source code)?
Would running cargo build with verbose logs also be useful for a historical reference?
It's much cheaper in terms of storage space to store the vendored dependencies instead of storing the target folder. The sources, being essentially text files, can be further compressed to reduce the final size.
The added benefit is that you can actually use those to perform a reproducible build and even if a crate disappears upstream, you will still have it.
Just run cargo vendor and cargo will create a directory named vendor which will contain the sources of all your dependencies, including any transitive dependency. Then package those together with your app sources as a signed zip.
I am creating a simple linux kernel with buildroot and I am adding a small driver I've done myself, I created the Config.in file and drivername.mk to be able to select the driver in make menuconfig succesfully.
When executing make to build the image, the compilation goes correctly until my driver starts to compile, it looks to compile and create the image right but I get loooots of warnings saying that different files in ./lib/gcc/arm-buildroot-linux-uclibcgnueabihf/ are touched by more than one package: [u'host-gcc-initial', u'host-gcc-final'].
Anyone can explain me a bit about this issue and what is causing it? Do you need any more info to know what is happening? Is it safe to ignore them?
Thanks beforehand
Actually, doing a search on 'touched by more than one package', I found http://lists.busybox.net/pipermail/buildroot/2017-October/205602.html, where we find that this warning can safely be ignored if you're not doing a parallel build and aren't a kernel maintainer.
That said, if you're submitting code for inclusion in the Linux kernel, please be a good citizen and make sure you identify all of the things your code is dependent upon. (I'm not actually an active kernel hacker, so I don't know what method they're using for this right now.)
The basic idea is that there are a bunch of steps in compiling things that need to be done in a logical order. In a small project, we simply use dependencies that we know to put in because we also coded in that dependency. But with a project the size of the kernel, you can guarantee that not everyone does this. Some of them instead just specify dependencies if they're needed for things to build properly - if the default order works, things could go years before someone figures out that there was a missing dependency, causing them grief when they were trying to update just the one thing that was a missing dependency, and the other code not getting updated as a result.
When you're doing things in parallel, on the other hand, it becomes a lot more complicated. Now you really need to have every dependency specified, because there is no longer any inherent dependable order. Some people will probably still build serially, while others use two processing threads. I'll use 8. I've worked in groups that would be inclined to do 30, because they're on a 32 processor machine, and don't really need all of those during the off hours. Suddenly the fact that the file you needed from a directory that normally got processed 30 directories before yours is now getting processed at the same time as your file that needed it, because you didn't list the dependency and everything in those 30 directories that hasn't already been processed and isn't being processed has a dependency that's not yet finished its processing.
The scenario outlined is this:
Someone has built the Linux kernel from source code.
That person wants to change the build configuration.
They still have all of the object files and temporary files that were produced by the previous build operation.
Given all of that, what needs to be done to rebuild as few things as possible in order to save time?
I understand that these will trigger or necessitate a complete recompilation of the source code:
Running make clean.
Running make menuconfig.
make clean is an obvious course of action to avoid to achieve the desired goal because it deletes all object files, both those that would need to be rebuilt and those that could otherwise be left alone. I don't know why make menuconfig would cause the build system to recompile everything, but I've read on here that that is what it would do.
The problem I see with not having the second avenue open to me is that if I change the configuration manually with a text editor, the options that I change might require changes in other options that depend on them (e.g., IMA_TRUSTED_KEYRING depends on SYSTEM_TRUSTED_KEYRING) and I'd be working without an interface that would automatically make those required secondary changes.
It occurred to me that invoking scripts/kconfig/mconf, the program built and launched by make menuconfig, could possibly be a solution to the problems described in the previous paragraph since it was not stated that mconf is what makes the build system recompile everything. But, it possibly could be that very program, so I do not wish to try it until I know it won't do that.
Sooooo, how does one achieve the stated objective given the stated scenario?
I have a package where most of the .hs files are generated from some spec files by a Haskell processor (stored in a subdirectory).
At present this preprocessor is run manually, and so does not work particularly well with version control or when used as part of a larger stack.
I would like Cabal to run this processor, for example from Setup.hs, so that the package can be built using cabal build or by stack as part of a larger project without needing separate manual steps.
The input files are not 1-1 with the output .hs files: instead, a pile of .csv files are read in, and then a differently arranged pile of .hs files are output. I think this means I can't use cabal's preprocessor support.
For my present development purposes I have bodged a call to stack build && stack exec gen-exe at the start of Setup.hs. This runs sometimes too many times (for example it runs on register at the end of a build; and sometimes not enough (for example, it does not run if one of the input .csv files changes, so a manual clean is necessary).
Is this achieveable in cabal, and if so, what do I need to put where to run my build commands frequently enough and not too frequently?
(This is an inherited package... please don't blame me)
What is says really. We need a cache because most files are built using the same versions of every file, but any developer who is altering files will only be altering a few files, and generally they get altered a lot.
There's little point in writing that change into the cache specified with CacheDir() until it is approved for production, but there's a lot of point in copying stuff from CacheDir
But I can only see options to disable the cache entirely.
(I'd post this to the scons mailing lists but it's just come up with a completely illegible captcha)
I can think of 2 different options:
Instead of a CacheDir, how about using a Repository() for those files that almost never change?
Consider using the --implicit-deps-unchanged option as described in the SCons man pages, and here.
Here's a similar discussion.
How do you plan on toggling the read-only behavior? You'll need some logic to do that. Since its not possible to use the CacheDir in a read-only way, an alternative would be to use this same toggling logic to switch between using a Repository and the CacheDir.
Fwiw, scones has this now. Version 2.3.1
From: http://www.scons.org/doc/production/HTML/scons-man.html
scons can maintain a cache of target (derived) files that can be shared between multiple builds. When caching is enabled in a SConscript file, any target files built by scons will be copied to the cache. If an up-to-date target file is found in the cache, it will be retrieved from the cache instead of being rebuilt locally. Caching behavior may be disabled and controlled in other ways by the --cache-force, --cache-disable, --cache-readonly, and --cache-show command-line options. The --random option is useful to prevent multiple builds from trying to update the cache simultaneously.