I have a vary simple/toy pipeline where I have a pyspark notebook that has an exit value, a set variable activity set to the exit value and a second notebook that is parameterized with the variable. It looks like the below.
I successfully set the variable, but I cannot seem to read this variable into the input_values notebook. The notebook looks like the below, also quite simple.
My pipeline parameters then look like the below. However my exit value just comes out as "hello "
I have also tried accessing the variable directly in the notebook (without parametrization) by f"hello {#variables("my_message")}", but this resulted in an error. When I try to look at documentation for this type of behavior it generally points to calling a notebook from another notebook so caller/callee notebooks, such as the examples from this documentation. This seems to be backwards for creating a DAG, so I do not want to do that. What I want to do seems possible and I saw an example of something similar here, but I can't seem to get it to work.
Can anyone point me to documentation that would explain how to accomplish what I want or is there something in the above that I am missing?
Edit:
I tried to set themsg parameter to #activity('exit_values').output.status.Output.result.exitValue, which is what I successfully set the my_message variable to and that did not work either.
Success
I don't feel very smart, but I will leave this up for anyone else that might be in the same position. I was setting my msg parameter on the pipeline, not the individual notebook. I successfully ran it by setting the parameter on the notebook.
I successfully ran it by setting the parameter on the notebook.
It is the correct way to pass the values to the Notebook parameters.
In the above the #variables('msg_var') is a should be given to in a dyamic content and not in default value. The pipeline parameters do not support dynamic content here.
Also, we have to give the value of the Notebook pipeline in Base parameters of Notebook activity by creating a parameter with same name as that.
Here, it supports dynamic content, so give the variable like below.
Notebook output:
Related
Is this possible and if so how? As a simple example I might have an R cell like the following
%r
x <- rbinom(10, 100, .5)
How can I access x from a Python cell?
A small excerpt from the Databricks website - mix-languages
When you invoke a language magic command, the command is dispatched to
the REPL in the execution context for the notebook. Variables defined
in one language (and hence in the REPL for that language) are not
available in the REPL of another language. REPLs can share state only
through external resources such as files in DBFS or objects in object
storage.
Some workarounds that I have seen on stackoverflow
Save in file in DBFS and read it back in python cell.
Other not tested/not recommended options.
2.1. Set environmental variable in one language and then read the same in another language.
2.2. Set value as a spark - conf and then read back in another language cell.
The file option works well. The other options I have not been able to pass along reliably. Maybe it is just for me.
Please validate and see if it works for you...
Disclaimer: I can achieve the behavior I’m looking for with Active Choices plugin, BUT I really want this to work in a Jenkinsfile and controlled with scm because it’s tedious to configure the Active Choices on each job we may need them on. And with it being separate from the Jenkinsfile creation, it’s then one job defined in multiple places. :(
I am looking to verify if this is possible, because I can’t get the syntax right, if it is possible. And I haven’t been able to find any examples online:
pipeline {
environment {
ARTIFACTS = lib.myfunc() // this works well
}
parameters {
choice(name: "Artifacts", choices: ARTIFACTS) // I can’t get this to work
}
}
I cannot use the function inline in the declaration of the parameter. The errors were clear about that, but it seems as though I should be able to do what I’ve written out above.
I am not home, so I do not have the exceptions handy, but I will add them soon. They did not seem very helpful while I was working on this yesterday.
What have I tried?
I’ve tried having the the function return a List Because it requires a list according to the docs, and I’ve also tried (illogically) returning a String in the precise syntax of a list of strings. (It was hacky, like return "['" + artifacts.join("', '") + "']" to look like ['artifact1.zip', 'artifact2.zip']
I also tried things like "$ARTIFACTS" and ${ARTIFACTS} in desperation.
the list of choices has to be supplied as String containing new line characters (\n): choices: 'TESTING\nSTAGING\nPRODUCTION'
I was tipped off by this article:
https://st-g.de/2016/12/parametrized-jenkins-pipelines
Related to a bug:
https://issues.jenkins.io/plugins/servlet/mobile#issue/JENKINS-40358
:shrug:
First, we need to understand that Jenkins starts running your pipeline code by presenting you with Parameters page. Once you've set up the parameters, and pressed Build, then a node is allocated, variables are set, and your code starts to run.
But in your pipeline, as presented above, you want to run some code to prepare the parameters.
This is not how Jenkins usually works. It's definitely not doing the following: allocating a node, setting the variables, running some of your code until parameters clause is reached, stopping all that, presenting you with GUI, and then continuing where it left off. Again, it's not how Jenkins works.
This is why, when writing a new pipeline, your first option to build it is Build and not Build with Parameters. Jenkins hasn't run your code yet; it doesn't have any idea if there are any parameters. When running for the first time, it will remember the parameters (and any choices, if were) as were configured for this (first) run, so in the second run you will see the parameters as configured in the first run. (Generally, in run number n you will see the result of configuration in run number n-1.)
There are a number of ways to overcome this.
If having a "somewhat recent" (and not "current and absolutely up-to-date") situation fits you, your code may need minor changes to work — second time. (I don't know what exactly lib.myfunc() returns but if it's a choice of Development/Staging/Production this might be good enough.)
If having a "somewhat recent" situation is an absolute no-no (e.g. your lib.myfunc() returns the list of git branches, and "list of branches as of yesterday" is unacceptable), then your only solution is ActiveChoice. ActiveChoice allows you to run some code before showing you the Build with Parameters GUI (with script approval etc.).
I am facing an issue that I could not understand how to resolve.
I created a test plan that need to connect DB and count the results.
The problem is that Jmeter not perform any validation afterwards, I created a JSSR223 in the JDBC request and just want to print the results and Jmeter not print.
I created another sampler to print the DB results and still Jmeter not printing.
Jmeter just passes this steps,
In the results tree I saw that it connects to DB and failed in the assertion, but why it passes the other steps? and just moving to debug sampler?
I can not print the results, I can not perform any debug since it is just black box.
can someone please advise?
you can see in yellow all the steps that Jmeter not performed and just not exists in the results tree.
enter image description here
Get used to check jmeter.log file, it normally contains information regarding what went wrong, you should be able to figure out the root cause by looking into the log file. If you are not - update your question with jmeter.log file contents (at least essential parts)
My expectation is that your ${Conv_sense} variable is not defined (or cannot be cast to Integer). Double check whether it is defined or not using Debug Sampler and View Results Tree listener combination.
Also don't refer JMeter Variables like ${Conv_sense} in Groovy scripts body, use vars.get('Conv_sense}') instead, otherwise it might conflict with Groovy GStringTemplate resulting in undefined behavior.
So in the website for the plugin, the change-log for Version 0.61 (Mar 30, 2016) mentions that they have exposed Jenkins and project bindings for use in a groovy script. I'm currently trying to set a parameter (multi select) and i want it to have different options based on the value of another parameter which is set before this one in the Jenkins job. However i'm not able to get the value of that other parameter in the script.
My problem is very similar to the one explained here
I've tried using ${param}, $(param), $param, bindings.variables.get('param') in the script, and none of it works
I've also tried to set that parameter in the Variable bindings section of the script but still no success.
Any ideas about the correct way to obtain the value?
I set the variable binding as param on the other field that you want to dynamically populate and used param (please note that i didnt use $ ${} or binding.get etc). It worked for me.
As a side note, Will active choice reactive parameter work in your case?
Is there a way to add a simulink block programmatically from within a function? The principle works by using add_block(...) via script in the basic matlab workspace. But if i try to add a block from within a function, Matlab seems to dont see the simulink model anymore. I get the error Invalid destination block specification.
Any clue what to do? Thanks.
EDIT: This add_block('built-in/Gain','Model/blk') worksin the command window, but it doesnt work inside a function.
Got it finally working, just had to change the current workspace. evalin('Base',...) does the trick.