I want to define the scons build variables in external.py file like
external.py
mode=debug
toolchain=x86
This I want to read back these variables in the SConstruct file which is there in the same directory. Depending on the variable values I want to do some operations!
vars = Variables('external.py')
vars.Add('mode', 'Set the mode for debug or release', 'debug')
if ${RELEASE}=="debug"
#Do these!
elif ${RELEASE}=="release"
#Do that!
If external.py contains valid Python code then you can simply import it using the import keyword. You can then use the dir function to iterate over the names defined in the external module and add them to the SCons variables. You might also want to take a look at the getattr function.
The thing you are missing is Scons Environment with your variables.
vars = Variables('external.py')
vars.Add('mode', 'Set the mode for debug or release', 'debug')
env = Environment(variables = vars)
if env['mode'] == 'debug':
# do action1
elif env['mode'] == 'release':
# do action2
else:
# do action3
You can read more about using Scons here, and about your question here
Soumyajit answer is great but I would add that if you want to be able to override values from your file with the command line and restrain the allowed values for your variables you can do as follow:
# Build variables are loaded in this order:
# Command Line (ARGUMENTS) >> Config File (external.py) >> Default Value
vars = Variables(files='external.py', args=ARGUMENTS)
vars.Add(EnumVariable('mode', 'Build mode.', 'debug', allowed_values=('debug', 'release')))
env = Environment(variables = vars)
if env['mode'] == 'debug':
env.Append(CCFLAGS = [ '-g' ])
# whatever...
else:
env.Append(CCFLAGS = '-O2')
# whatever...
You can invoke you build script like this scons but also override specific variables without editing your config file by doing scons mode=release
If you specify a bad value for your variable you will get an error from Scons like:
$> scons mode=foo
scons: Reading SConscript files ...
scons: *** Invalid value for option mode: foo. Valid values are: ('debug', 'release')
Related
i have process that generates a value. I want to forward this value into an value output channel. but i can not seem to get it working in one "go" - i'll always have to generate a file to the output and then define a new channel from the first:
process calculate{
input:
file div from json_ch.collect()
path "metadata.csv" from meta_ch
output:
file "dir/file.txt" into inter_ch
script:
"""
echo ${div} > alljsons.txt
mkdir dir
python3 $baseDir/scripts/calculate.py alljsons.txt metadata.csv dir/
"""
}
ch = inter_ch.map{file(it).text}
ch.view()
how do I fix this?
thanks!
best, t.
If your script performs a non-trivial calculation, writing the result to a file like you've done is absolutely fine - there's nothing really wrong with this approach. However, since the 'inter_ch' channel already emits files (or paths), you could simple use:
ch = inter_ch.map { it.text }
It's not entirely clear what the objective is here. If the desire is to reduce the number of channels created, consider instead switching to the new DSL 2. This won't let you avoid writing your calculated result to a file, but it might mean you can avoid an intermediary channel, potentially.
On the other hand, if your Python script actually does something rather trivial and can be refactored away, it might be possible to assign a (global) variable (below the script: keyword) such that it can be referenced in your output declaration, like the line x = ... in the example below:
Valid output
values
are value literals, input value identifiers, variables accessible in
the process scope and value expressions. For example:
process foo {
input:
file fasta from 'dummy'
output:
val x into var_channel
val 'BB11' into str_channel
val "${fasta.baseName}.out" into exp_channel
script:
x = fasta.name
"""
cat $x > file
"""
}
Other than that, your options are limited. You might have considered using the env output qualifier, but this just adds some syntactic-sugar to your shell script at runtime, such that an output file is still created:
Contents of test.nf:
process test {
output:
env myval into out_ch
script:
'''
myval=$(calc.py)
'''
}
out_ch.view()
Contents of bin/calc.py (chmod +x):
#!/usr/bin/env python
print('foobarbaz')
Run with:
$ nextflow run test.nf
N E X T F L O W ~ version 21.04.3
Launching `test.nf` [magical_bassi] - revision: ba61633d9d
executor > local (1)
[bf/48815a] process > test [100%] 1 of 1 ✔
foobarbaz
$ cat work/bf/48815aeefecdac110ef464928f0471/.command.sh
#!/bin/bash -ue
myval=$(calc.py)
# capture process environment
set +u
echo myval=$myval > .command.env
I'm trying to write a Builder in SCons to call a command line executable with some arguments that are generated by a (series of) python functions.
cmdVars = Variables(None, ARGUMENTS)
cmdVars.AddVariables(
EnumVariable('DEBUG', 'help for debug', 'a', allowed_values=('a','b','c')),
PathVariable('CLI', 'path to cli exe', 'C:\...\blah.exe', PathVariable.PathIsFile)
)
env = Environment(variables = cmdVars)
def generateSomeExtraBitsDependingOnFlag(debug):
if (debug == 'a'):
return "-DDEBUG -DBlah myTextHere"
return ''
myBuilder = Builder(
action = '"$CLI" generateSomeExtraBitsDependingOnFlag("$DEBUG")'
)
<extra stuff to add myBuilder to env and call env.myBuilder>
The "$CLI" input is correctly substituted to output (when calling scons) something like C:\Program Files\...\blah.exe but the output from the function never appears, regardless of the DEBUG setting.
Prepending print(debug) inside the function prints $DEBUG whilst parsing the SConstruct file (so it's not surprising it doesn't match the if condition).
Do I just need to use a Generator or follow the instructions in chapter 18.4 (Builders That Execute Python Functions) to make this work? Is the section Writing Builders That Execute External Commands not what I want here?
I suspect that given understanding, the user guide is clear, but without already knowing the answer, the guide is a little opaque to me.
Try this:
cmdVars = Variables(None, ARGUMENTS)
cmdVars.AddVariables(
EnumVariable('DEBUG', 'help for debug', 'a', allowed_values=('a','b','c')),
PathVariable('CLI', 'path to cli exe', 'C:\...\blah.exe', PathVariable.PathIsFile)
)
env = Environment(variables = cmdVars)
def generateSomeExtraBitsDependingOnFlag(source, target, env, for_signature):
if (env['DEBUG'] == 'a'):
return "-DDEBUG -DBlah myTextHere"
return ''
env['generateSomeExtraBitsDependingOnFlag'] =generateSomeExtraBitsDependingOnFlag
myBuilder = env.Builder(
action = '"$CLI" ${generateSomeExtraBitsDependingOnFlag}'
)
env.Append(BUILDERS = {'myBuilder' : myBuilder})
env.myBuilder('dummy','input')
I want build files based on variables that are assigned in the SConstruct file. As in this example:
import os
env = Environment(ENV = os.environ)
def text_file_maker(target, source, env):
with open(str(target[0]), "w") as text_file:
text_file.write(env['my_text'])
return 0
env.Append( BUILDERS = {'Make_text' : Builder(action = text_file_maker)})
env.Make_text(target = 'sour.txt',
source = None,
my_text = 'lemon')
env.Make_text(target = 'sweet.txt',
source = None,
my_text = 'apple')
Running this script generates two text files with the content 'lemon' and 'apple'. If I'm running the script again, SCons correctly detects that the targets exist:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
Now if I'm changing one target, e.g:
env.Make_text(target = 'sweet.txt',
source = None,
my_text = 'orange')
and run the script, I'll get scons: '.' is up to date. and the content 'orange' is not written to sweet.txt.
I could specify a dependency:
Depends(['sweet.txt', 'sour.txt'], 'SConstruct')
but that forces a rebuild for any change in my SConstruct script.
How can I make SCons recognise changes in the SConstruct script that affects the targets?
This could be a very simple question, sorry if I missed something obvious.
You're example very closely resembles the answer which is in the manpage (Search for "Action Objects"):
https://scons.org/doc/production/HTML/scons-man.html
The third and succeeding arguments, if present, may either be a construction variable or a list of construction variables whose values will be included in the signature of the Action when deciding whether a target should be rebuilt because the action changed. The variables may also be specified by a varlist= keyword parameter; if both are present, they are combined. This is necessary whenever you want a target to be rebuilt when a specific construction variable changes. This is not often needed for a string action, as the expanded variables will normally be part of the command line, but may be needed if a Python function action uses the value of a construction variable when generating the command line.
def build_it(target, source, env):
# build the target from the 'XXX' construction variable
open(target[0], 'w').write(env['XXX'])
return 0
# Use positional arguments.
a = Action(build_it, '$STRINGIT', ['XXX'])
# Alternatively, use a keyword argument.
a = Action(build_it, varlist=['XXX'])
So in your case change:
env.Append( BUILDERS = {'Make_text' : Builder(action = text_file_maker)})
To
tfb_action = Action(text_file_maker, varlist=['my_text'])
env.Append( BUILDERS = {'Make_text' : Builder(action = tfb_action)})
Seems -u doesn't work on for me ( I am using scons-2.3.6).
To simplify the context, you can imagine my project structure like,
+root
+project
- bar.vcxproj (generated vs project)
-SConstruct
-bar.c
Inside SConstruct, I have put code like:
env_base = Environment()
...
env_base.StaticLibrary(target = 'bar', source = ['bar.c'])
...
If I execute command "scons" in root folder, everything works perfectly.
But If I execute command "scons -u" in project folder, scons can find my SConstruct up in root folder, but no file get compiled.
BTW : The reason for me to execute "scons -u" in project folder is because I want to put the generated vsproj in projet folder and use BuildCommandLine to compile the project.
I guess I didn't use "-u" correctly, what will be the elegant solution for my situation?
1st edit:
As bdbaddog asked, I have put the SConstruct here:
def BuildConfig(env, config):
env.Append(CCFLAGS = '/W 4')
env.Append(CCFLAGS = '/WX')
if config == "debug":
env.Append(CCFLAGS = '/DEBUG')
#env.Append(CCFLAGS = '-Zi /Fd${TARGET}.pdb')
env.Append(CCFLAGS = '/Z7')
elif config == "release":
pass
env_base = Environment()
lib = env_base.StaticLibrary(target = 'bar', source = ['bar.c'])
opts=Variables()
opts.Add('target', 'Compile Target (debug/release).', "debug")
# there is more in my project....
opts.Update(env_base) # update environment
# here I want to use my own command to build the project, so it can support different build option that is defined by me.
env_base['MSVSBUILDCOM'] = "scons -u target=$(Configuration)"
target = env_base["target"]
BuildConfig(env_base, env_base['target'])
env_base.MSVSProject(target = "project\\bar" + env_base['MSVSPROJECTSUFFIX'],
srcs = ["..\\bar.c"],
incs = [],
localincs = "",
resources = "",
misc = "",
buildtarget = lib,
variant = ['debug'],
auto_build_solution=0)
SCons only builds files under the current directory by default.
If you you wanted to only build files in a certain directory (for which there are rules that build the targets there), you can invoke SCons as follows:
scons the_target_directory_I_want_to_build
Though this may cause sources for targets in that directory to also be built.
Let's say I want to strip all the debug symbols in the shared libraries that I build whiling keeping the original file name.
I tried to add an command in the method:
def mySharedLibrary(self, *args, **kwargs):
# do some common work for every shared library like add a soname or append some lib files to LIBS parameter
target = SharedLibary(*args, **kwargs)
target = env.Command(target,target, "objcopy --strip-debug ${SOURCE}")
return target
I get this error: two different method was given to the same target,
I guess it's because the two targets returned by env.Command and SharedLibrary are exactly the same name.
Any ideas to do this?
Thanks in advance!
I had the same problem and got the same error. What I had to do was to create an intermediate target/library. The intermediate and final targets each had their own library name, so SCons doesnt get confused.
You could probably do something like the following:
env.SharedLibrary(target = 'namePreStrip', source = 'yourSource')
env.Command(target = 'name', source = 'namePreStrip', 'objcopy...')
I used objcopy to build a library out of several libraries. Here's the actual source code I implemented:
#
# Build an object file out of several other source files, objects, and libraries
# Optionally execute objcopy on the resulting library, depending if objcopyFlags
# has been populated
#
# env - SCons Environment used to build, Mandatory arg
# target - resulting library name, without LIBPREFIX and LIBSUFFIX, ej 'nsp2p',
# Mandatory arg
# sourceFiles - list of '.cc' files that will be compiled and included in the
# resulting lib, Optional arg
# objects - list of already compiled object files to be included in resulting lib,
# Optional arg
# libraries - list of libraries to be included in resulting lib, Optional arg
# objcopyFlags - list of flags to pass to objcopy command. objcopy will only
# be executed if this list is populated, Optional arg
#
# One of [sourceFiles, objects, or libraries] must be specified, else nothing
# will be performed
#
# Not using a custom builder because I dont like the way SCons prints the
# entire command each time its called, even if its not going to actually
# build anything AND I need more method args than provided by custom builders
#
def buildWholeArchive(self, env, target, sourceFiles, objects, libraries, objcopyFlags):
if len(sourceFiles) == 0 and len(objects) == 0 and len(libraries) == 0:
print "Incorrect use of buildWholeArchive, at least one of [sourceFiles | objects | librarires] must be specified, no build action will be performed"
return None
# Compile each source file
objNodes = []
if len(sourceFiles) > 0:
objNodes = env.Object(source = sourceFiles)
cmdList = []
cmdList.append(env['CXX'])
cmdList.append('-nostdlib -r -o $TARGET -Wl,--whole-archive')
for obj in objNodes:
cmdList.append(env.File(obj).abspath)
for obj in objects:
cmdList.append(env.File(obj).abspath)
for lib in libraries:
cmdList.append(lib)
cmdList.append('-Wl,--no-whole-archive')
cmd = ' '.join(cmdList)
libTarget = '%s%s%s' % (env['LIBPREFIX'], target, env['LIBSUFFIX'])
if len(objcopyFlags) > 0:
# First create the library, then run objcopy on it
objTarget = '%s%s_preObjcopy%s' % (env['LIBPREFIX'], target, env['LIBSUFFIX'])
preObjcopyTarget = env.Command(target = objTarget, source = [], action = cmd)
env.Depends(preObjcopyTarget, [objNodes, sourceFiles, objects, libraries])
objCmdList = [env['OBJCOPY']]
objCmdList.extend(objcopyFlags)
objCmdList.append('$SOURCE $TARGET')
objcopyCmd = ' '.join(objCmdList)
archiveTarget = env.Command(target = libTarget, source = preObjcopyTarget, action = objcopyCmd)
else:
# Just create the library
archiveTarget = env.Command(target = libTarget, source = [], action = cmd)
env.Depends(archiveTarget, [objNodes, sourceFiles, objects, libraries])
return archiveTarget
And here is how I called it:
sourceFiles = ['file1.cc', 'file2.cc']
libSource = []
if 'OcteonArchitecture' in env:
libSource.append(lib1)
libSource.append(lib2)
libSource.append(lib3)
objcopy = []
if 'OcteonArchitecture' in env:
objcopy.extend([
'--redefine-sym calloc=ns_calloc',
'--redefine-sym free=ns_free',
'--redefine-sym malloc=ns_malloc',
'--redefine-sym realloc=ns_realloc'])
archiveTarget = clonedEnv.buildWholeArchive(target = libName,
sourceFiles = sourceFiles,
objects = [],
libraries = libSource,
objcopyFlags = objcopy)
env.Alias('libMyLib', archiveTarget)