I have build process where I have a set of input files I want to build slightly differently based on slow-to-extract meta-data I can find in the input files.
The simple approach would be to get this information from the file every time I run scons and build conditionally based on it but it means I need to rescan the files every time I start a build significantly slowing down the build.
I have two potential approaches I'm looking to explore:
A two stage build where I first run one scons file to extract the metadata into sidecar files. These sidecar files get picked up by a second scons project that generates the right build targets and operations based on the sidecar files.
(Ab)use a custom scanner for the input files to generate sidecar files with the metadata in and enable the implicit_cache to make sure scanning only happens when the input files changes.
What would be the most correct and scons-idiomatic way of accomplishing what I'm looking to do?
Related
I'm newbie in Yocto Project. I'm managing several projects which all of them have a version for each: development/debug and field/industrialization. While working with the build system, I've noticed the recurrent following scenario.
Let's assume workspace is clean, a fresh build.
Launch bitbake, minimal-image with certain linux-kernel device tree and defconfig parameters. Bitbake will take some time and output files will be created.
Now, change parameters in previously mentioned device tree and defconfig (imagine new peripherals are added). Relaunch bitbake, output files wil be created for this new compilation.
Now, here comes the trick. Reset device tree and defconfig files to the setting before compiling in step 2. Relaunch bitbake and it will be nearly instant. Output files are replaced by the ones created in step 2.
So, I know this is possible due to bitbake and the use of the sstate-cache, or that's what I suppose. I've been googling around for a while but info is not too clear. How does this exactly work? Is there any kind of signature created in compilation time with the inputs of config files and stored associated to the compilation? I'm concerned about this becase I really need to trust that what I'm sending to field is precisely the correct compilation and not the unsafe development version.
And related to this, which is the difference between launching a bitbake -c cleanall or hard deleting deploy and sstate-cache dirs?
Thanks in advance.
The information is available in the official manual:
https://www.yoctoproject.org/docs/3.0/overview-manual/overview-manual.html#shared-state-cache
I am running a pipeline on a SLURM-cluster, and for some reason a lot of smaller files (between 500 and 2000 bytes in size) named along the lines of slurm-XXXXXX.out (where XXXXXX is a number). I've tried to find out what these files are on the SLURM website, but I can't find any mention of them. I assume they are some sort of in-progress files that the system uses while parsing my pipeline?
If it matters, the pipeline I'm running is using snakemake. I know I've seen these types of files before though, without snakemake, but I they weren't a big problem back then. I'm afraid that clearing the working directory of these files after each step of the workflow will interrupt in-progress steps, so I'm not doing anything with them at the moment.
What are these files, and how can I suppress their output or, alternatively, delete them after their corresponding job is finished? Did I mess up my workflow somehow, and that's why they are created?
You might want to take a look at the sbatch documentation. The files that you are referring to are essentially SLURM logs as explained there:
By default both standard output and standard error are directed to a
file of the name "slurm-%j.out", where the "%j" is replaced with the
job allocation number.
You can change the filename with the --error=<filename pattern> and --output=<filename pattern> command line options. The filename_pattern can have one or more symbols that will be replaced as explained in the documentation. According to the FAQs, you should be able to suppress standard output and standard error by using the following command line options:
sbatch --output=/dev/null --error=/dev/null [...]
I have a package where most of the .hs files are generated from some spec files by a Haskell processor (stored in a subdirectory).
At present this preprocessor is run manually, and so does not work particularly well with version control or when used as part of a larger stack.
I would like Cabal to run this processor, for example from Setup.hs, so that the package can be built using cabal build or by stack as part of a larger project without needing separate manual steps.
The input files are not 1-1 with the output .hs files: instead, a pile of .csv files are read in, and then a differently arranged pile of .hs files are output. I think this means I can't use cabal's preprocessor support.
For my present development purposes I have bodged a call to stack build && stack exec gen-exe at the start of Setup.hs. This runs sometimes too many times (for example it runs on register at the end of a build; and sometimes not enough (for example, it does not run if one of the input .csv files changes, so a manual clean is necessary).
Is this achieveable in cabal, and if so, what do I need to put where to run my build commands frequently enough and not too frequently?
(This is an inherited package... please don't blame me)
I am build several large set of source files (targets) using scons. Now, I would like to know if there is a metric I can use to show me:
How many targets remain to be build.
How long it will take -- this to be honest, this is probably a no-go as it is really hard to tell!
How can I do that in scons?
There is currently no progress indicator built into SCons, and it's also not trivial to provide one. The problem is, that SCons doesn't build the complete DAG first, and then starts the build...such that you'd have a total number of targets to visit that you could use as a reference (=100%).
Instead, it makes up the DAG on the go... It looks at each target, and then expands the list of its children (sources and implicit dependencies like headers) to check whether they are up-to-date. If a child has changed, it gets rebuilt by applying the same "build step" recursively.
In this way, SCons crawls itself from the list of targets, as given on the command-line (with the "." dir being the default), down the DAG...where only the parts are ever visited, that are required for (or, in other words: have a dependency to) the requested targets.
This makes it possible for SCons to handle things like "header files, generated by a program that must be compiled first" in the first go...but it also means that the total number of targets/children to get visited changes constantly.
So, a standard progress indicator would continuously climb towards the 80%-90%, just to then fall back to 50%...and I don't think this would give you the information you're really after.
Tip: If your builds are large and you don't want to wait, do incremental builds and only build the library/program you're currently doing work on ("scons lib1"). This will still take into account all dependencies, but only a fraction of the DAG has to get expanded. So, you use less memory and get faster update times...especially if you use the "interactive" mode. In a project with a 100000 C files total, the update of a single library with 500 C files takes about 1s on my machine. For more infos on this topic check out http://scons.org/wiki/WhySconsIsNotSlow .
I have to build custom kernels often enough, that it would make sense for me to have a tool which would take some a short list of approx. 50-100 configuration options and apply them over the default stock config file.
Now, I surely could just take a fresh kernel config (call make nconfig -> save the file without any changes done). Then change the corresponding options and compile the thing. But, the nconfig/menuconfig UIs perform some dependencies - automatically setting the values of depending/relative options. So, my question is:
(How) can I easily tap into the dependency resolving mechanism? What I'd ideally like to get is the following: taking a fresh .config file. applying a single change. do something to change the values of related configuration options, if there are any to be cahnged.
Thanks in advance!