Background
Our build script use Install() and InstallAs() to install a number of .dylib and .so files to a "dist" directory.
Install(dist_dir, 'libfoo')
Install(dist_dir, 'libbar')
...
The problem
After the library files are copied to dist_dir, we would like to run a custom function on each of the files. That custom command is as followed:
def add_magic(lib_filename, arg1, arg2, arg3)
We appreciate any help to achieve our goals.
What have we tried so far?
We just started looked into creating custom builder via Builder()
We also looked at Command() builder
We also looked at the AddMethod() function to create pseudo-builder
Right now our bets are on the first two approaches, we are reading the user guide and working on simple examples. We are no where near the end and appreciate any tips/hint.
You can try something like this:
AddPostAction(target, action)
env.AddPostAction(target, action)
Arranges for the specified action to be performed after the specified
target has been built. The specified action(s) may be an Action
object, or anything that can be converted into an Action object (see
below).
When multiple targets are supplied, the action may be called multiple
times, once after each action that generates one or more targets in
the list.
Example:
installBarCmd = Install(dist_dir, 'libbar')
AddPostAction(installBarCmd, Action(...))
Related
I'm working on a Kotlin multi-platform project, and I need my JS tests to run on Node.js but with custom command line arguments (specifically I need node to run with the --expose-gc flag, because some tests need to trigger garbage collection).
Looking at the documentation for the Gradle Kotlin JS DSL I didn't find any mention of how to do that; does anyone know whether it's at all possible and how?
Unfortunately can not answer your question directly, but there is some suggestion to help you with reverse engineering.
Let's start from some example. We have Gradle tasks to run our project using webpack's dev server such as browserDevelopmentRun, browserProductionRun (not sure if multi-platform projects have it, but JS projects do). We can add:
println(tasks.named("browserProductionRun").get().javaClass)
to build.gradle.kts to find out the exact class used for this task. When we sync Gradle, it outputs:
org.jetbrains.kotlin.gradle.targets.js.webpack.KotlinWebpack_Decorated
Now we know the exact class of this task so we can investigate its API. The auto completion or navigating inside of the KotlinWebpack class helps us to find out that it has a helpful nodeArgs property to configure NodeJS arguments for it, so we can set them, for example:
tasks.named("browserProductionRun", org.jetbrains.kotlin.gradle.targets.js.webpack.KotlinWebpack::class).get().nodeArgs.add("--trace-deprecation")
Getting back to your question.
In your case I guess you need to investigate the browserTest task. Let's get some info about it by adding:
println(tasks.named("browserTest").get().javaClass)
to build.gradle.kts - a-ha - it seems to be of the org.jetbrains.kotlin.gradle.targets.js.testing.KotlinJsTest_Decorated type. Let's check what's inside. Open KotlinJsTest.kt somehow - for example by typing its name into the window being opened by CMD + Shift + O (make sure to select "All Places" here) or just by typing its name somewhere in build.gradle.kts and navigating inside it.
The only interesting thing I see inside this open class is the following block:
override fun createTestExecutionSpec(): TCServiceMessagesTestExecutionSpec {
val forkOptions = DefaultProcessForkOptions(fileResolver)
forkOptions.workingDir = npmProject.dir
forkOptions.executable = nodeJs.requireConfigured().nodeExecutable
val nodeJsArgs = mutableListOf<String>()
return testFramework!!.createTestExecutionSpec(
task = this,
forkOptions = forkOptions,
nodeJsArgs = nodeJsArgs,
debug = debug
)
}
So maybe it can work out to create your own extension of this class, override its createTestExecutionSpec method and provide nodeJsArgs as you need inside it. After that you'll be needing to declare another Gradle task to launch tests inside build.gradle.kts which will use this new extended class.
Disclaimer: I can achieve the behavior I’m looking for with Active Choices plugin, BUT I really want this to work in a Jenkinsfile and controlled with scm because it’s tedious to configure the Active Choices on each job we may need them on. And with it being separate from the Jenkinsfile creation, it’s then one job defined in multiple places. :(
I am looking to verify if this is possible, because I can’t get the syntax right, if it is possible. And I haven’t been able to find any examples online:
pipeline {
environment {
ARTIFACTS = lib.myfunc() // this works well
}
parameters {
choice(name: "Artifacts", choices: ARTIFACTS) // I can’t get this to work
}
}
I cannot use the function inline in the declaration of the parameter. The errors were clear about that, but it seems as though I should be able to do what I’ve written out above.
I am not home, so I do not have the exceptions handy, but I will add them soon. They did not seem very helpful while I was working on this yesterday.
What have I tried?
I’ve tried having the the function return a List Because it requires a list according to the docs, and I’ve also tried (illogically) returning a String in the precise syntax of a list of strings. (It was hacky, like return "['" + artifacts.join("', '") + "']" to look like ['artifact1.zip', 'artifact2.zip']
I also tried things like "$ARTIFACTS" and ${ARTIFACTS} in desperation.
the list of choices has to be supplied as String containing new line characters (\n): choices: 'TESTING\nSTAGING\nPRODUCTION'
I was tipped off by this article:
https://st-g.de/2016/12/parametrized-jenkins-pipelines
Related to a bug:
https://issues.jenkins.io/plugins/servlet/mobile#issue/JENKINS-40358
:shrug:
First, we need to understand that Jenkins starts running your pipeline code by presenting you with Parameters page. Once you've set up the parameters, and pressed Build, then a node is allocated, variables are set, and your code starts to run.
But in your pipeline, as presented above, you want to run some code to prepare the parameters.
This is not how Jenkins usually works. It's definitely not doing the following: allocating a node, setting the variables, running some of your code until parameters clause is reached, stopping all that, presenting you with GUI, and then continuing where it left off. Again, it's not how Jenkins works.
This is why, when writing a new pipeline, your first option to build it is Build and not Build with Parameters. Jenkins hasn't run your code yet; it doesn't have any idea if there are any parameters. When running for the first time, it will remember the parameters (and any choices, if were) as were configured for this (first) run, so in the second run you will see the parameters as configured in the first run. (Generally, in run number n you will see the result of configuration in run number n-1.)
There are a number of ways to overcome this.
If having a "somewhat recent" (and not "current and absolutely up-to-date") situation fits you, your code may need minor changes to work — second time. (I don't know what exactly lib.myfunc() returns but if it's a choice of Development/Staging/Production this might be good enough.)
If having a "somewhat recent" situation is an absolute no-no (e.g. your lib.myfunc() returns the list of git branches, and "list of branches as of yesterday" is unacceptable), then your only solution is ActiveChoice. ActiveChoice allows you to run some code before showing you the Build with Parameters GUI (with script approval etc.).
I have some log files generated after each file is compiled.
I am making SCons aware of these files by using an emitter attached to the builder that I'm using to compile that file.
Unfortunately, because I am deleting the empty log files after each build SCons recompiles the source files because the log files are missing.
I would like to ignore these 'side effect' files using SCons Ignore function.
In my emitter I am doing something like this:
def compiler_emitter(target, source, env):
target.append(env.File(source[0].name.split('.')[0] + env['ERRSUFFIX']))
env.Ignore(source[0], target[1])
return target, source
As a note I always pass only one file to my builder.
In my case Ignore function is not working.
What will be the best approach to solve this problem in a 'SCons way' ?
Try using env.SideEffect() instead of Ignore:
SideEffect(side_effect, target) , env.SideEffect(side_effect, target)
Declares side_effect as a side effect of building target. Both
side_effect and target can be a list, a file name, or a node. A side
effect is a target file that is created or updated as a side effect of
building other targets. For example, a Windows PDB file is created as
a side effect of building the .obj files for a static library, and
various log files are created updated as side effects of various TeX
commands. If a target is a side effect of multiple build commands,
scons will ensure that only one set of commands is executed at a time.
Consequently, you only need to use this method for side-effect targets
that are built as a result of multiple build commands.
Because multiple build commands may update the same side effect file,
by default the side_effect target is not automatically removed when
the target is removed by the -c option. (Note, however, that the
side_effect might be removed as part of cleaning the directory in
which it lives.) If you want to make sure the side_effect is cleaned
whenever a specific target is cleaned, you must specify this
explicitly with the Clean or env.Clean function.
http://scons.org/doc/production/HTML/scons-man.html
I'm trying to set up a build system involving a code generator. The exact files generated are unknown until after the generator is run, but I'd like to be able to run further build steps by pattern matching (run some program on all files with some extension). Is this possible?
Some of the answers here involving code generation seem to assume that the output is known or a listing of generated files is created. This isn't impossible in my case, but I'd like to avoid it since it makes things more complicated.
https://bitbucket.org/scons/scons/wiki/DynamicSourceGenerator seems to indicate that it's possible to add additional targets during Builder actions, but while I could get the build to run and list the generated files, any build steps introduced don't run.
https://bitbucket.org/scons/scons/wiki/NonDeterministicDependencies uses Scanners to add build steps. I put a glob(...) in a scanner, and it succeeds in detecting the generated files, but the files are inexplicably deleted before it actually runs the dependent step.
Is this use case possible? And why is SCons deleting my generated files?
A toy example
source (the file referenced in SConscript)
An example generator, constructs 3 files (not easily known to the build system) and puts them in the argument folder
echo "echo 1" > $1/gen1.txt
echo "echo 2" > $1/gen2.txt
echo "echo 3" > $1/gen3.txt
SConstruct
Just sets up a variant_dir
SConscript('SConscript', variant_dir='build')
SConscript
The goal is for it to:
"Compile" the generator (in this toy example, just copies a file called 'source' and adds execute permissions
Run the "compiled" generator ('source' is a script that generates files)
Perform some operation on each of those generated files by extension. This example just runs the "compile" copy operation on them (for simplicity).
env = Environment()
env.Append(BUILDERS = {'ExampleCompiler' :
Builder(action=[Copy('$TARGET', '$SOURCE'),
Chmod('$TARGET', 0755)])})
generator = env.ExampleCompiler('generator', 'source')
env.Append(BUILDERS = {'GeneratorRun' :
Builder(action=[Mkdir('$TARGET'),
'$SOURCE $TARGET'])})
generated_dir = env.GeneratorRun(Dir('generated'), generator)
Everything's fine up to here, where all the targets are explicitly known to the build system ahead of time.
Attempting to use this block of code to glob over the generated files causes SCons to delete (!!) the generated files:
for generated in generated_dir[0].glob('*.txt'):
generated_run = env.ExampleCompiler(generated.abspath + '.sh', generated)
Attempting to use an action to update the build tree results in additional actions not being run:
def generated_scanner(target, source, env):
for generated in source[0].glob('*.txt'):
print "scanned " + generated.abspath
generated_target = env.ExampleCompiler(generated.abspath + '.sh', generated)
Alias('TopLevelAlias', generated_target)
env.Append(BUILDERS = {'GeneratedOperation' :
Builder(action=[generated_scanner])})
dummy = env.GeneratedOperation(generated_dir[0].File('#dummy'), generated_dir)
Alias('TopLevelAlias', dummy)
The Alias operations are suggested in above dynamic source generator guide, but don't seem to do anything. The prints do execute and indicate that the action gets run.
Running some build pattern on special file extensions is possible with SCons. For C/CPP files this is the preferred scheme, for example:
env = Environment()
env.Program('main', Glob('*.cpp'))
The main task of SCons, as a build system, is to do the minimum amount of work such that all your targets are up-to-date. This makes things complicated for the use case you've described above, because it's not clear how you can reach a "stable" situation where no generated files are added and all targets are built.
You're probably better off by using a simple Python script directly...I really don't see how using SCons (or any other build system for that matter) is mission-critical in this case.
Edit:
At some point you have to tell SCons about the created files (*.txt in your example above), and for tracking all dependencies properly, the list of *.txt files has to be complete. This the task of the Emitter within SCons, which is responsible for returning the list of resulting target and source files for a Builder call. Note, that these files don't have to exist physically during the "parse" phase of SCons. Please also have a look at my answer to Scons: create late targets , which goes into some more detail.
Once you have a proper Emitter in place (see also https://bitbucket.org/scons/scons/wiki/ToolsForFools , "Using Emitters") you should be able to use the Glob('*.txt') call, which will detect and track your created files automatically.
Finally, on our page "Talks and Slides" ( https://bitbucket.org/scons/scons/wiki/TalksAndSlides ) you can find my talk from the PyCon FR.2014, "Why SCons is Not Slow", which explains shortly how SCons works internally. This might be helpful in understanding this problem better and coming up with a full solution.
Ive setup a GroovyResourceLoader and it seems to get requests for groovy scripts as necessary. I was just wondering is it specially used anywhere besides class loading ? Is there any benefit in simply wrapping a ClassLoader and loading *.groovy files there as opposed to using a GRL ? Are they just different ways to the same end ?
GroovyResourceLoader (GRL) is used by GroovyClassLoader (GCL) and at least since Groovy 1.8 indirectly through GroovyScriptEngine (GSE). But GSE loads it through a GCL as well.
But what GRL does is to "locate" the scripts and return an URL to the location. What GCL does is use the URL returned by a GRL to get the source and compile it to create the class, which is then available for loading.
GRL is a backend for GCL. So they are not different ways to the same end. True, you still have to do more things to actually execute the script code (unless it is precompiled), but "get the script source, compile it, make a Class out of it and finally execute it" are the steps you always have to do. In our GRL/GCL discussion, GRL does part of the first step, GCL itself does the third step. Step 2 is done by CompilationUnit inside GCL and the last step is yours to be done. There are other ways to complete these steps of course, but that's out of the scope for this discussion.