I'm trying to access a variable in a nextflow.config file during the execution of a pipeline. I want to supply image_standard as a string in run.nf and I want to receive eu.gcr.io/proj_name/image1:latest as an output. I figured out a way to obtain the content of the .config file within the nextflow script, but I don't know how to access this specific property.
This is my nextflow.config file:
process {
withLabel: image_standard {
container = "eu.gcr.io/proj_name/image1:latest"
}
withLabel: image_deluxe {
container = "eu.gcr.io/proj_name/image2:latest"
}
}
the run.nf
x = workflow.configFiles[0]
Properties properties = new Properties()
File propertiesFile = new File("${x}")
propertiesFile.withInputStream {
properties.load(it)
}
log.info "${properties.process}"
Which just prints the line:
{
You could try instead slurping the config file and selecting the process you want using the ProcessConfig class and the applyConfigSelector() method:
import nextflow.config.ConfigParser
import nextflow.script.ProcessConfig
def config_file = file("${baseDir}/nextflow.config")
def config = new ConfigParser().setIgnoreIncludes(true).parse(config_file.text)
def process = new ProcessConfig([:])
process.applyConfigSelector(config.process, 'withLabel:', 'image_standard')
println(process.container)
Related
I have a terraform script where I have to set up an Influxdb server and I want to create different databases based on the workspace name. Is it possible to create a map in the variables file to allocate a database name and look it up from a different variable within the same file?
Ex:
var file:
variable "influx_database" "test" {
name = "${lookup(var.influx_database_name, terraform.workspace)}
}
variable "influx_database_name" {
type = "map"
default = {
dump = "dump_database"
good = "good_database"
}
}
You can use local variable like below,
locals {
influx_database_name = "${lookup(var.influx_database_name, terraform.workspace)}"
}
variable "influx_database_name" {
type = "map"
default = {
default = "default_database"
dump = "dump_database"
good = "good_database"
}
}
output "influx_database_name" {
value = "${local.influx_database_name}"
}
local.influx_database_name is defined by workspace name.
In jenkins pipeline,
I was launch multiple URL in side groovy script, like as below
stages {
stage("Launch URL") {
steps {
script {
def url1 = "https://www.paypal.com/us/home".toURL().getText()
def url2 = "https://www.ebay.com".toURL().getText()
def url3 = "https://www.yahoo.com/".toURL().getText()
}
}
}
}
Is there a better way to do this one.
Is it possible to use one variable and execute all three urls?
In the inner part of your code, you can do the following in groovy:
def texts = ["https://www.paypal.com/us/home",
"https://www.ebay.com",
"https://www.yahoo.com/".collect {
it.toURL().text
}
where texts will be a List<String> containing the string content returned from each url respectively.
I would like to write a system groovy script which inspects the queued jobs in Jenkins, and extracts the build parameters (and build cause as a bonus) supplied as the job was scheduled. Ideas?
Specifically:
def q = Jenkins.instance.queue
q.items.each { println it.task.name }
retrieves the queued items. I can't for the life of me figure out where the build parameters live.
The closest I am getting is this:
def q = Jenkins.instance.queue
q.items.each {
println("${it.task.name}:")
it.task.properties.each { key, val ->
println(" ${key}=${val}")
}
}
This gets me this:
4.1.next-build-launcher:
com.sonyericsson.jenkins.plugins.bfa.model.ScannerJobProperty$ScannerJobPropertyDescriptor#b299407=com.sonyericsson.jenkins.plugins.bfa.model.ScannerJobProperty#5e04bfd7
com.chikli.hudson.plugin.naginator.NaginatorOptOutProperty$DescriptorImpl#40d04eaa=com.chikli.hudson.plugin.naginator.NaginatorOptOutProperty#16b308db
hudson.model.ParametersDefinitionProperty$DescriptorImpl#b744c43=hudson.mod el.ParametersDefinitionProperty#440a6d81
...
The params property of the queue element itself contains a string with the parameters in a property file format -- key=value with multiple parameters separated by newlines.
def q = Jenkins.instance.queue
q.items.each {
println("${it.task.name}:")
println("Parameters: ${it.params}")
}
yields:
dbacher params:
Parameters:
MyParameter=Hello world
BoolParameter=true
I'm no Groovy expert, but when exploring the Jenkins scripting interface, I've found the following functions to be very helpful:
def showProps(inst, prefix="Properties:") {
println prefix
for (prop in inst.properties) {
def pc = ""
if (prop.value != null) {
pc = prop.value.class
}
println(" $prop.key : $prop.value ($pc)")
}
}
def showMethods(inst, prefix="Methods:") {
println prefix
inst.metaClass.methods.name.unique().each {
println " $it"
}
}
The showProps function reveals that the queue element has another property named causes that you'll need to do some more decoding on:
causes : [hudson.model.Cause$UserIdCause#56af8f1c] (class java.util.Collections$UnmodifiableRandomAccessList)
Using Job-DSL we can configure a C# project in Jenkins.
The SonarQube tasks is giving us a hard time.
StepContext.metaClass.sonar = {
-> NodeBuilder nodeBuilder = new NodeBuilder()
stepNodes << nodeBuilder.'hudson.plugins.sonar.SonarRunnerBuilder' {
jdk('(Inherit From Job)')
usePrivateRepository(false)
}
}
How to set the path to the sonar-project.properties config file, using the Job-DSL script?
Final script
Thanks to #Bruno César, I added pathToSonarProjectProperties as parameter.
StepContext.metaClass.sonar = { String pathToSonarProjectProperties
-> NodeBuilder nodeBuilder = new NodeBuilder()
stepNodes << nodeBuilder.'hudson.plugins.sonar.SonarRunnerBuilder' {
jdk('(Inherit From Job)')
usePrivateRepository(false)
project(pathToSonarProjectProperties)
}
}
The sonar function is called with the relative-to-project-root path of sonar-project.properties:
sonar("Framework\\xxx\\xxx\\sonar-project.properties")
In SonarRunnerBuilder class there is a project attribute that represents the path to a file with properties for the project.
In the same way in which you set the JDK (jdk('(Inherit From Job)')) you can set the path property. In your example, try like this:
StepContext.metaClass.sonar = {
-> NodeBuilder nodeBuilder = new NodeBuilder()
stepNodes << nodeBuilder.'hudson.plugins.sonar.SonarRunnerBuilder' {
jdk('(Inherit From Job)')
usePrivateRepository(false)
project('${your.path.here}')
}
}
I am trying to write a generic program in Groovy that will get the SQL from config file along with other parameters and put them into file.
here is the program:
def config = new ConfigSlurper().parse(new File("config.properties").toURL())
Sql sql = Sql.newInstance(config.db.url, config.db.login, config.db.password, config.db.driver);
def fileToWrite = new File(config.copy.location)
def writer = fileToWrite.newWriter()
writer.write(config.file.headers)
sql.eachRow(config.sql){ res->
writer.write(config.file.rows)
}
in the config the sql is something like this:
sql="select * from mydb"
and
file.rows="${res.column1}|${res.column2}|${res.column3}\n"
when I run it I get
[:]|[:]|[:]
[:]|[:]|[:]
[:]|[:]|[:]
in the file. If I substitute
writer.write(config.file.rows)
to
writer.write("${res.column1}|${res.column2}|${res.column3}\n")
it outputs the actual results. What do I need to do different to get the results?
You accomplish this by using lazy evaluation of the Gstring combined with altering the delegate.
First make the Gstring lazy by making the values be the results of calling Closures:
file.rows="${->res.column1}|${->res.column2}|${-> res.column3}"
Then prior to evaluating alter the delegate of the closures:
config.file.rows.values.each {
if (Closure.class.isAssignableFrom(it.getClass())) {
it.resolveStrategy = Closure.DELEGATE_FIRST
it.delegate = this
}
}
The delegate must have the variable res in scope. Here is a full working example:
class Test {
Map res
void run() {
String configText = '''file.rows="${->res.column1}|${->res.column2}|${-> res.column3}"
sql="select * from mydb"'''
def slurper = new ConfigSlurper()
def config = slurper.parse(configText)
config.file.rows.values.each {
if (Closure.class.isAssignableFrom(it.getClass())) {
it.resolveStrategy = Closure.DELEGATE_FIRST
it.delegate = this
}
}
def results = [
[column1: 1, column2: 2, column3: 3],
[column1: 4, column2: 5, column3: 6],
]
results.each {
res = it
println config.file.rows.toString()
}
}
}
new Test().run()
The good news is that the ConfigSlurper is more than capable of doing the GString variable substitution for you as intended. The bad news is that it does this substitution when it calls the parse() method, way up above, long before you have a res variable to substitute into the parser. The other bad news is that if the variables being substituted are not defined in the config file itself, then you have to supply them to the slurper in advance, via the binding property.
So, to get the effect you want you have to parse the properties through each pass of eachRow. Does that mean you have to create a new ConfigSlurper re-read the file once for every row? No. You will have to create a new ConfigObject for each pass, but you can reuse the ConfigSlurper and the file text, as follows:
def slurper = new ConfigSlurper();
def configText = new File("scripts/config.properties").text
def config = slurper.parse(configText)
Sql sql = Sql.newInstance(config.db.url, config.db.login, config.db.password, config.db.driver);
def fileToWrite = new File(config.copy.location)
def writer = fileToWrite.newWriter()
writer.write(config.file.headers)
sql.eachRow(config.sql){ result ->
slurper.binding = [res:result]
def reconfig = slurper.parse(configText)
print(reconfig.file.rows)
}
Please notice that I changed the name of the Closure parameter from res to result. I did this to emphasize that the slurper was drawing the name res from the binding map key, not from the closure parameter name.
If you want to reduce wasted "reparsing" time and effort, you could separate the file.rows property into its own separate file. i would still read in that file text once and reuse the text in the "per row" parsing.