groovy script not working in nifi executescript processor - groovy

I'm trying to execute something via executescript processor; a groovy code inside. In the code I'm trying to create a scala script which is to be executed on spark in a further processor.
// Get flow file
def flowFile = session.get()
if (!flowFile) return
// Create output directory
def userInputDir = flowFile.getAttribute("user.input.path")
def finalFolder = new File(userInputDir + "/" + innerDir)
try
{
if (!finalFolder.exists()) finalFolder.mkdirs()
// Write script
file = "spark.sqlContext.setConf(\"hive.exec.dynamic.partition\", \"true\")\n"
file = file + "spark.sqlContext.setConf(\"hive.exec.dynamic.partition.mode\", \"nonstrict\")\n"
file = file + "import org.apache.spark.sql._"
file = file + "\n"
file = file + "import java.io._"
file = file + "\n"
}
.
.
.
The rest other steps are adding some other spark specific commands to the script variable. Script is huge so skipping the full code paste.
Finally, closing with a catch
// Output file path
flowFile = session.putAttribute(flowFile, "generatedScript", scalaFile.getCanonicalPath())
session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e)
{
log.info("File: {}\n", finalFolder.file)
session.transfer(flowFile, REL_FAILURE)
}
The processor is not even beginning to start the groovy script execution and it fails with the error:
groovy.lang.MissingPropertyException: No such property: script for calss: javal.io.File
By the statement 'not even beginning to start' means that the previous queue is not empty and the processor throws the error. I'm guessing it's a syntax issue but I don't find any syntactical problems related in the script. I also tried running the script in local machine's groovy shell and have the same error there as well, but no syntax issue.
Googling the error got me the suggestion to include the imports in the script but even after including the relevant imports the error is the same.
Any clues?

You're referring to a variable "innerDir" which is not defined anywhere. Are you expecting the user to add a user-defined property to ExecuteScript called innerDir? If so, the innerDir variable in the script is a PropertyValue object, so you'd need to call getValue() on it to get the actual value of the property:
innerDir.value
Also you refer to scalaFile.getCanonicalPath() but scalaFile is not defined above, and getCanonicalPath() won't give you the contents of the script, is that what you meant?
I reworked the partial script above to assume that innerDir is a user-defined property and you write the contents of the file variable to a File pointed to by scalaFile; also I made it more Groovy by using heredoc instead of appending to the file variable:
// Get flow file
def flowFile = session.get()
if (!flowFile) return
// Create output directory
def userInputDir = flowFile.getAttribute("user.input.path")
def finalFolder = new File(userInputDir + "/" + innerDir?.value ?: '')
try
{
if (!finalFolder.exists()) finalFolder.mkdirs()
// Write script
file =
"""
spark.sqlContext.setConf("hive.exec.dynamic.partition", "true")
spark.sqlContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
import org.apache.spark.sql._
import java.io._
"""
scalaFile = new File(finalFolder, 'script.scala')
scalaFile.withWriter {w -> w.write(file)}
// Output file path
flowFile = session.putAttribute(flowFile, "generatedScript", scalaFile.canonicalPath)
session.transfer(flowFile, REL_SUCCESS)
}
catch(Exception e) {
log.info("File: {}\n", finalFolder.file)
session.transfer(flowFile, REL_FAILURE)
}

Related

Groovy code to read rabbitMQ working on Windows, not working on Linux

Need: Read from rabbitMQ with AMQPS
Problem: ConsumeAMQP is not working so I'm using groovy script that's working on windows and not working on linux. Error message is:
groovy.lang.MissingMethodException: No signature of method: com.rabbitmq.client.ConnectionFactory.setUri() is applicable for argument types: (String) values: [amqps://user:xxxxxxxXXXxxxx#c-s565c7-ag77-etc-etc-etc.mq.us-east-1.amazonaws.com:5671/virtualhost]
Possible solutions: getAt(java.lang.String), every(), every(groovy.lang.Closure)
Troubleshooting:
Developed code on python to test from my machine using pika lib and it's working with URL amqps. It reads from rabbitMQ. no connection issues.
put the python code on the nifi server (1.15.3) machine, installed python and pika lib, execute on the command line, it's working on the server and reads from rabbitMQ.
Develop groovy code to test from my windows apache nifi (1.15.3)` and it's working, it's reading from rabbitMQ client system.
Copy the code (copy past) to the nifi server, uploaded the .jar lib also. Not working with this error message. create a groovy file and execute the code. not working.
Can anyone help me?
NOTE: I want to use groovy code to output the results to the flowfile.
#Grab('com.rabbitmq:amqp-client:5.14.2')
import com.rabbitmq.client.*
import org.apache.commons.io.IOUtils
import java.nio.charset.*
// -- Define connection
def ConnectionFactory factory = new ConnectionFactory();
factory.setUri('amqps://user:password#a-r5t60-etc-etc-etc.mq.us-east-1.amazonaws.com:5671/virtualhost');
factory.useSslProtocol();
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
// -- Waiting for messages.");
boolean noAck = false;
int count = 0;
while(count<10) {
GetResponse response = channel.basicGet("db-user-q" , noAck)
if (response != null) {
byte[] body = response.getBody()
long deliveryTag = response.getEnvelope().getDeliveryTag()
def msg = new String(body, "UTF-8")
channel.basicAck(response.envelope.deliveryTag, false)
def flowFile = session.create()
flowFile = session.putAttribute(flowFile, 'myAttr', msg)
session.transfer(flowFile, REL_SUCCESS);
}
count++;
}
channel.close();
connection.close();
The following code is suspect:
def ConnectionFactory factory = new ConnectionFactory();
You don't need both def and a type ConnectionFactory. Just change it to this:
ConnectionFactory factory = new ConnectionFactory()
You don't need the semi-colon either. The keyword def is used for dynamic typing situations (or laziness), and specifying the type (ie ConnectionFactory) is for static typing situations. You can't have both. It's either dynamic or static typing. I suspect Groovy VM is confused by what type the object is hence why it can't figure out if setUri exists or not.

Soap UI: Groovy Test Step : How to call a Specific Method in a groovy script from another Groovy Script

In my project, I want to keep all groovy utilities test step under one test case and to call them again and again where ever is needed. Like reading the test data file etc. I would be able to achieve that if the below problem is resolved. I tried a lot of ways but couldn't make it. Any help is welcome!!
For Example
script 1: test1Script
def sayHellow(){
log.info "Hello!!"
}
Script 2 : test2Script
import groovy.lang.Binding
import groovy.util.GroovyScriptEngine
def groovyUtils = new com.eviware.soapui.support.GroovyUtils(context)
def projectPath = groovyUtils.projectPath
def scriptPath = projectPath + "\\GroovyScripts\\"
//GroovyShell shell = new GroovyShell()
//Util = shell.parse(new File(directoryName + "groovyUtilities.groovy"))
//groovyUtilities gu = new groovyUtilities(Util)
// Create Groovy Script Engine to run the script.
GroovyScriptEngine gse = new GroovyScriptEngine(scriptPath)
// Load the Groovy Script file
externalScript = gse.loadScriptByName("sayHello.groovy")
// Create a runtime instance of script
instance = externalScript.newInstance()
// Sanity check
assert instance!= null
// run the foo method in the external script
instance.sayhellowTest()
When I'm calling that method from another script, I'm getting below exception
groovy.lang.MissingPropertyException: No such log for class test1Script
The error shows that groovy runtime calls your method but it can't find the log property. I assume that this log variable is declared in the test1Script body, e.g. def log = ... In this case the variable becomes local to its declaration scope and it's not visible to the script functions. To make it visible, it can be annotated by #Field or it should be undeclared (doesn’t have type declaration, just log = ...). The latter, however, requires you to pass the variable value via so-called bindings when running the script as you run it. So given my assumptions above, you can annotate your log variable as a field and it should work:
//just for the sake of example it prints to stdout whatever it receives
#groovy.transform.Field
def log = [info: {
println it
}]
def sayHellow() {
log.info "Hello!!"
}
Now calling sayHellow from another script prints "Hello" to stdout:
...
instance.sayHellow()
It is very important to declare, context, testRunner, and Log variables in the called script.
script 1: sayHello.groovy
public class groovyUtilities {
def context
def testRunner
def log
def sayhellowTest() {
log.info "Hi i'm arpan"
}
}
script 2: RunAnotherGroovyScript.groovy
def groovyUtils = new com.eviware.soapui.support.GroovyUtils(context)
def projectPath = groovyUtils.projectPath
def scriptPath = projectPath + "\\GroovyScripts\\"
// Create Groovy Script Engine to run the script and pass the location of the directory of your script.
GroovyScriptEngine gse = new GroovyScriptEngine(scriptPath)
// Load the Groovy Script file
externalScript = gse.loadScriptByName("sayHello.groovy")
// Create a runtime instance of script
instance = externalScript.newInstance(context: context, log: log, testRunner: testRunner)
// Sanity check if the instance is null or not
assert instance != null
// run the foo method in the external script
instance.sayhellowTest()
Standoutput:
Hi i'm arpan
"I want to keep all groovy utilities test step under one test case and
to call them again and again where ever is needed. Like reading the
test data file etc."
OK, so to me this simply sounds like you have a library of reusable functions and want to be able to call them from any test you might be running.
I suppose you could store them with another test and then call them from the test you're currently running, but SoapUI comes with the neat feature in that you can store your common functions/libraries 'outside' of the SoapUI project.
I have lots of such Groovy libraries and I store mine under the bin/scripts folder of SoapUI. I typically call common functions from a Script assertion test step in the test I'm running. For example, I have a getUserDetails type test step. I can do all the usual assertions against the step like valid response code, SLA etc. I can then use a Script assertion test step. This type of step allows you to run a chunk of Groovy script. This is OK for specific cases, but you wouldn't want to paste in something common as you need to update every Script assertion if something changes. But you can call an 'external' groovy script. Also, the Script Assertion step is just a method that has log, context and message exchange passed into it, so no need to instantiate your own. Just pass them into you external groovy script...
So, as an illustration...
ValidateUser.groovy (stored in bin/scripts/groovyScripts/yourOrg/common)
package groovyScripts.yourOrg.common; // Package aligns with folder it's stored in.
Class ValidateUser {
def log = null;
def context = null;
def messageExchange = null;
// Constructor for the class
ValidateUser(logFromTestStep, contextFromTestStep, messageExchangeFromTestStep) {
// Assign log, context and messageExchange to the groovy class.
this.log = logFromTestStep;
this.context = contextFromTestStep;
this.messageExhange = messageExchangeFromTestStep;
}
def runNameCheck() {
// Performs some validation. You have access to the API response via
// this.messageExchange
log.info("Running the Name Check");
}
}
In the test step of interest, go to the assertions and create a 'Script Assertion' From here you can instantiate your external class and call some method. E.g.
def validateUserObject = new groovyScripts.yourOrg.common.ValidateUser(log, context, messageExchange);
validateUserObject.runNameCheck();
What I like about these external type scripts is that I can use any text editor I like. Also, when I make a change and press Save, SoapUI is monitoring the scripts folder for changes and reloads the script so no need to restart SoapUI.

Neo4j Java APOC query

I have my java app using Neo4j API. Everything works fine, except when I pass by args the path of my file to process.
This is the code working fine:
session.run("call
apoc.xml.import(\"http://www.example.com/myfile.xml\") YIELD node
RETURN node");
Since I need to create an app which is general purpose, I need to pass the file from the user input. When I pass by args the URL, I got the error because there must be something wrong in the string.
This is the code with problem:
session.run("call apoc.xml.import(\" " + file + " \" ) YIELD node RETURN node");
where file is the variable containing http://www.example.com/myfile.xml
Am I doing something wrong?
Try to pass the variable through the parameters:
Map<String, Object> params = new HashMap<>();
params.put( "file", file );
session.run("call apoc.xml.import($file) YIELD node RETURN node", params);
[ https://neo4j.com/docs/java-reference/3.4/tutorials-java-embedded/#tutorials-cypher-parameters-java ]

SOAPUI Assertion script writing twice to file

I am writing a groovy script where I am extracting text from a response and writing it to an output file on my system. The problem I encounter is when I run the groovy script that calls the test. The script assertions runs and logs the text twice to the file.
It seems to write to the file just before it leaves the assertion.
I have tried the following:
...
...
if (context.alreadyWritten == null || !context.alreadyWritten) {
inputFile.append (testString+ "\n")
log.info testString
} else {
log.info ('Already written!')
}
I have set the flag (context.alreadyWritten) to false in groovy before I execute the test step.
SOAPUI version : 5.3.0
I see there was an issue previous with Smartbear when appending to file in an assertion script. However the workaround was advised to resolve this:
if (context.alreadyWritten == null || !context.alreadyWritten) {
}
Which does not resolve my issue
When I log the result using log.info I see only one instance of the message been logged.
Any ideas.
Thanks
If you are using Script Assertion, try below:
//Define the file name, change as needed
def fileName = '/path/to/file.xml'
//check if you got the response
if (context.response) {
log.info 'Response available and not empty'
def file = new File(fileName)
if (!context?.alreadySaved) {
file.write(context.response)
context.alreadySaved = true
log.info 'response written to file'
} else {
log.info 'Response already written'
}
} else {
log.info 'there is no response to save'
}

Write to file via jenkins post-groovy script on slave

I'd like to do something very simple: Create/write to a file located in the remote workspace of a slave via the jenkins groovy post-build script plug-in
def props_file = new File(manager.build.workspace.getRemote() + "/temp/module.properties")
def build_num = manager.build.buildVariables.get("MODULE_BUILD_NUMBER").toInteger()
def build_props = new Properties()
build_props["build.number"] = build_num
props_file.withOutputStream { p ->
build_props.store(p, null)
}
The last line fails, as the file doesn't exist. I'm thinking it has something to do with the output stream pointing to the master executor, rather than the remote workspace, but I'm not sure:
Groovy script failed:
java.io.FileNotFoundException: /views/build_view/temp/module.properties (No such file or directory)
Am I not writing to the file correctly?
While writing onto slave you need to check the channel first and then you can successfully create a file handle and start reading or writing to that file:
if(manager.build.workspace.isRemote())
{
channel = manager.build.workspace.channel;
}
fp = new hudson.FilePath(channel, manager.build.workspace.toString() + "\\test.properties")
if(fp != null)
{
String str = "test";
fp.write(str, null); //writing to file
versionString = fp.readToString(); //reading from file
}
hope this helps!
Search for words The post build plugin runs on the manager and doing it as you say will fail if you are working with slaves! on the plugin page (the link to which you've provided) and see if the workaround there helps.
Does the folder /views/build_view/temp exist?
If not, you will need to do new File( "${manager.build.workspace.remote}/temp" ).mkdirs()

Resources