I want to load a config value (something like json, yaml, xml or ini) from a jenkins pipeline script. When I try to use org.yaml.snakeyaml.Yaml I get
Scripts not permitted to use new org.yaml.snakeyaml.Yaml
I know I can unlock org.yaml.snakeyaml.Yam, but the message tells me that this does not seem to be the standard way of loading config files.
Is there a way of loading config files that is already unlocked?
If anyone looking for yaml parser in jenkinsfile, I recommend the following
def yamlData = readYaml file: 'cae.yaml'
Reference : https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#code-readyaml-code-read-yaml-from-files-in-the-workspace-or-text
Try using the JsonSlurper:
def config = new JsonSlurper().parse(new File("config.json"))
Related
I'm using Python 3.8 and pytest for unit testing. I have tests similar to the following
def test_insert_record(az_sql_db_inst):
"""."""
foreign_key_id = 10
I don't like hard-coding "10" in the file, and would prefer this be stored in some external configuration file where other tests can also access the value. Does pytest provide a way to do this (to load the values from a properties or yml file)? I could just open a file on my own, e.g.
def test_insert_record(az_sql_db_inst):
"""."""
file = open('test.txt')
for line in file:
fields = line.strip().split()
foreign_key_id = fields[0]
but it seems like there is something that can automatically do that for me.
If you simply want to initialize some constants, a separated config.py would do the trick.
from config import foreign_key_id
And PyYAML package would be good if you prefer general-purpose text files.
However, if you want to specify expected test results or even different results depending on scenarios, #pytest.mark.parametrize may be a good option.
I am trying to simply load a file from a packages resource folder. I have the following project structure:
And have tried the following in an attempt to load each of the txt files to the Populator.groovy script:
File file = new File(Populator.class.getResource("/names/first-names.txt").getFile())
The above results in a FileNotFoundException if any methods are called from the file instance. The path returned is correct, and the file is indeed where the path specifies. I am also using very similar methods of extracting resources in above modules and no errors are occurring. Whats going on here?
Why not
File file = new File(Populator.class.getResource("/names/first-names.txt").toURI())
Not sure why you want it as a file though? Wouldn't an input stream do?
Based on this answer, I would like to iterate over all the properties in jenkins job and replace with their value in the job in build step.
e..g.
file = new File("folder/path/myfile.cs")
fileText = file.text;
fileText = fileText.replaceAll("parameter", "parametervalue");
file.write(fileText);
I saw an example to resolve one param like this:
build.buildVariableResolver.resolve("myparameter")
But I need
To iterate over all of them
To do it in Build step (after pulling from Source Code) so in the current build.
How can I do it?
The following is a sample groovy code that you can use with the scriptler plugin, that I suggest to use. This code can correctly read the Environment variables that are set in any Jenkins job:
import jenkins.model.Jenkins
def thr = Thread.currentThread()
def build = thr?.executable
def env = build.getEnvironment()
def buildNumber = env.get("BUILD_NUMBER")
In case you want to read the Build Parameters (you have a parameterized build and need access to those variables), you need to use the previous code and add the following:
def bparams = build.getBuildVariables()
def myBuildParam = bparams.get("MY_BUILD_PARAMETER")
after you read them, you can write them to your file, in a build step occurring after the git cloning.
I have used Spark to build a machine learning pipeline, which takes a job XML file as an input where users can specify data, features, models and their parameters. The reason for using this job XML input file is that users can simply modify their XML file to config the pipeline and do not need to re-compile from the source code. However, currently the Spark job is typically packaged into an uber-Jar file, and it seems that there is no way to provide additional XML inputs when the job is submitted to YARN.
I wonder if there are any solutions or alternatives?
I'd look into Spark-JobServer You can use it to submit your job to a Spark Cluster together with a configuration. You might have to adapt your XML to the JSON format used by the config or maybe encapsulate it somehow.
Here's an example on how to submit a job + config:
curl -d "input.string = a b c a b see" 'localhost:8090/jobs?appName=test&classPath=spark.jobserver.WordCountExample'
{
"status": "STARTED",
"result": {
"jobId": "5453779a-f004-45fc-a11d-a39dae0f9bf4",
"context": "b7ea0eb5-spark.jobserver.WordCountExample"
}
}
You should use the resources directory to place the xml file if you want it to be bundled with the jar. This is a basic Java/Scala thing.
Suggest reading: Get a resource using getResource()
To replace the xml in the jar without rebuilding the jar: How do I update one file in a jar without repackaging the whole jar?
The final solution that I used to solve this problem is:
Store the XML file in HDFS,
Pass in the file location of the XML file,
Use the InputStreamHDFS to directly read from HDFS:
val hadoopConf = sc.hadoopConfiguration
val jobfileIn:Option[InputStream] = inputStreamHDFS(hadoopConf, filename)
if (jobfileIn.isDefined) {
logger.info("Job file found in file system: " + filename)
xml = Some(XML.load(jobfileIn.get))
}
I'm having issues importing htmlunit (htmlunit.sf.net) into a groovy script.
I'm currently just using the example script that was on the web and it gives me unable to resolve class com.gargoylesoftware.htmlunit.WebClient
The script is:
import com.gargoylesoftware.htmlunit.WebClient
client = new WebClient()
html = client.getPage('http://www.msnbc.msn.com/')
println page.anchors.collect{ it.hrefAttribute }.sort().unique().join('\n')
I downloaded the source from the website and placed the com folder (and all its contents) where my script was located.
Does anyone know what issue I'm encountering? I'm not quite sure why it won't import it
You could use Grape to get the dependecy for you during script runtime. Easiest way to do it is to add a #Grab annotation to your import statement.
Like this:
#Grab('net.sourceforge.htmlunit:htmlunit:2.7')
import com.gargoylesoftware.htmlunit.WebClient
client = new WebClient()
// Added as HtmlUnit had problems with the JavaScript
client.javaScriptEnabled = false
html = client.getPage('http://www.msnbc.msn.com/')
println page.anchors.collect{ it.hrefAttribute }.sort().unique().join('\n')
There's only one problem. The page seems to be a little bit to much to chew off for HtmlUnit. When I ran the code I got OutOfMemoryException every time. I'd suggest downloading the html the normal way instead and then using something like NekoHtml or TagSoup to parse the html into XML and work with it that way.
This example uses TagSoup to work with html as xml in Groovy: http://blog.foosion.org/2008/06/09/parse-html-the-groovy-way/
you just need to download zip file, extract the jar file(s) and place them on the class path when compiling... You dont need the source
http://sourceforge.net/projects/htmlunit/files/htmlunit/2.8/htmlunit-2.8.zip/download