Groovy Imports Taking Several Seconds - groovy

I'm completely new to Groovy, so apologize in advance if I'm missing something obvious.
I'm trying to do some simple REST API scripting in Groovy, but first wanted to understand it's performance for requests/JSON parsing vs Python. I wrote the following script - and am seeing that the imports are taking ~7 seconds. Is there any way to 'include' those in the script, so it doesn't take so long on each run?
def now = new Date()
println now.format("yyyyMMdd-HH:mm:ss.SSS", TimeZone.getTimeZone('UTC'))
#Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7')
#Grab('oauth.signpost:signpost-core:1.2.1.2')
#Grab('oauth.signpost:signpost-commonshttp4:1.2.1.2')
import groovyx.net.http.RESTClient
import static groovyx.net.http.ContentType.*
for (i = 0; i <1; i++) {
def Client = new RESTClient("http://www.mocky.io/v2/59821b4a110000a9103964eb" )
def resp = Client.get(contentType: JSON)
def myResponseObject = resp.getData()
println myResponseObject.items[i].id
}
now = new Date()
println now.format("yyyyMMdd-HH:mm:ss.SSS", TimeZone.getTimeZone('UTC'))
I get this output:
~$ time groovy Requests.groovy
20170802-18:36:24.556
10
20170802-18:36:25.290
real 0m7.173s
user 0m4.986s
sys 0m0.329s
Just the first few lines of Grabs and imports are taking the majority of the runtime , and that's what I'd like to cut down.

It's not the import that takes time, but #Grab annotation which comes from Grape - a Groovy dependency management system. Those 3 lines:
#Grab('org.codehaus.groovy.modules.http-builder:http-builder:0.7')
#Grab('oauth.signpost:signpost-core:1.2.1.2')
#Grab('oauth.signpost:signpost-commonshttp4:1.2.1.2')
define your script dependencies. Those dependencies are 3rd party libraries provided as a JAR files. Some of them may even have their own dependencies which will be also downloaded to satisfy dependency you have defined (e.g. http-builder requires Apache's HTTP client and core lib).
Running this script takes some time (about 1 second on my laptop), because Groovy has to determine all dependencies and add them to the classpath to satisfy all imports. Keep in mind that your script uses a lot more dependencies than those 3 and all of them have to be resolved.
Using Grape is actually a compromise between using 3rd party libraries in the most easiest way and some overhead that is delegated to Groovy. Alternatively you could run your script with:
groovy -classpath ${GROOVY_CLASSPATH} Request.groovy
where ${GROOVY_CLASSPATH} contains paths to all JAR files you need to successfully run the script. And believe me - you will have to add at least 15 libraries instead of those 3 grapes. Then you will be able to remove all #Grab annotations (they are not needed in this case because you will satisfy groovy script with providing all libs in the classpath) and your script will execute in the blink of an eye - there will be no overhead caused by resolving and loading all dependencies.
Another alternative solution is to use Gradle to manage all dependencies and create so called "fat JAR" that contains all mandatory dependencies inside - in this case you will be able to run your program with java command and all imports will be in place without any dependencies resolving mechanism.
Final conclusion. Grape is a powerful Groovy's feature that has it's own limitations. It allows you to handle dependencies-hell pretty easily, but it comes with its own cost. I hope this answer will help you making a good choice.

Related

How to convert picocli groovy-grape script to native standalone app?

I have a made my first groovy CLI app with picocli. Now, I want it to be available for use without any JVM installed on the client machine, maybe with the use of GraalVM.
This is for an opensource project:
https://github.com/kchaitanya863/db2csv
Another easy option is to dockerize your script (read this blog about how to do it https://groovy-lang.gitlab.io/101-scripts/docker/basico-en.html)
If you want to build a linux executable you need to change your project:
convert to a gradle project (maven is also an option but gradle has a lot of plugins)
change your script to a class with a tipical main (and move it to the standard directory src/main/groovy/mypackage)
add some tasks into you build.gradle similar to these https://gitlab.com/snippets/1797638
You will need to:
statically compile your groovy script
make the args variable available after static compilation with
final String[] args = getProperty("args") as String[]
specify a reflection configuration file for the classes dynamically loaded/invoked using reflection by Groovy (this may be useful)
specify a reflection configuration file for the classes loaded/invoked using reflection by picocli. The picocli-codegen module provides a picocli.codegen.aot.graalvm.ReflectionConfigGenerator tool to generate the configuration file.
If your script has any #Grape dependencies, you may need to turn off the Grape dependency manager with -Dgroovy.grape.enabled=false and add all dependencies to the classpath manually instead
Credit: I got most of these tips from this article by Szymon Stepniak
If you want to use Graal with Groovy, check out this article:
https://e.printstacktrace.blog/graalvm-and-groovy-how-to-start/

How to run Java from Python [duplicate]

What is the best way to call java from python?
(jython and RPC are not an option for me).
I've heard of JCC: http://pypi.python.org/pypi/JCC/1.9
a C++ code generator for calling Java from C++/Python
But this requires compiling every possible call; I would prefer another solution.
I've hear about JPype: http://jpype.sourceforge.net/
tutorial: http://www.slideshare.net/onyame/mixing-python-and-java
import jpype
jpype.startJVM(path to jvm.dll, "-ea")
javaPackage = jpype.JPackage("JavaPackageName")
javaClass = javaPackage.JavaClassName
javaObject = javaClass()
javaObject.JavaMethodName()
jpype.shutdownJVM()
This looks like what I need.
However, the last release is from Jan 2009 and I see people failing to compile JPype.
Is JPype a dead project?
Are there any other alternatives?
You could also use Py4J. There is an example on the frontpage and lots of documentation, but essentially, you just call Java methods from your python code as if they were python methods:
from py4j.java_gateway import JavaGateway
gateway = JavaGateway() # connect to the JVM
java_object = gateway.jvm.mypackage.MyClass() # invoke constructor
other_object = java_object.doThat()
other_object.doThis(1,'abc')
gateway.jvm.java.lang.System.out.println('Hello World!') # call a static method
As opposed to Jython, one part of Py4J runs in the Python VM so it is always "up to date" with the latest version of Python and you can use libraries that do not run well on Jython (e.g., lxml). The other part runs in the Java VM you want to call.
The communication is done through sockets instead of JNI and Py4J has its own protocol (to optimize certain cases, to manage memory, etc.)
Disclaimer: I am the author of Py4J
Here is my summary of this problem: 5 Ways of Calling Java from Python
http://baojie.org/blog/2014/06/16/call-java-from-python/ (cached)
Short answer: Jpype works pretty well and is proven in many projects (such as python-boilerpipe), but Pyjnius is faster and simpler than JPype
I have tried Pyjnius/Jnius, JCC, javabridge, Jpype and Py4j.
Py4j is a bit hard to use, as you need to start a gateway, adding another layer of fragility.
Pyjnius docs and Github.
From the github page:
A Python module to access Java classes as Python classes using JNI.
PyJNIus is a "Work In Progress".
Quick overview
>>> from jnius import autoclass
>>> autoclass('java.lang.System').out.println('Hello world')
Hello world
>>> Stack = autoclass('java.util.Stack')
>>> stack = Stack()
>>> stack.push('hello')
>>> stack.push('world')
>>> print stack.pop()
world
>>> print stack.pop()
hello
If you're in Python 3, there's a fork of JPype called JPype1-py3
pip install JPype1-py3
This works for me on OSX / Python 3.4.3. (You may need to export JAVA_HOME=/Library/Java/JavaVirtualMachines/your-java-version)
from jpype import *
startJVM(getDefaultJVMPath(), "-ea")
java.lang.System.out.println("hello world")
shutdownJVM()
I'm on OSX 10.10.2, and succeeded in using JPype.
Ran into installation problems with Jnius (others have too), Javabridge installed but gave mysterious errors when I tried to use it, PyJ4 has this inconvenience of having to start a Gateway server in Java first, JCC wouldn't install. Finally, JPype ended up working. There's a maintained fork of JPype on Github. It has the major advantages that (a) it installs properly and (b) it can very efficiently convert java arrays to numpy array (np_arr = java_arr[:])
The installation process was:
git clone https://github.com/originell/jpype.git
cd jpype
python setup.py install
And you should be able to import jpype
The following demo worked:
import jpype as jp
jp.startJVM(jp.getDefaultJVMPath(), "-ea")
jp.java.lang.System.out.println("hello world")
jp.shutdownJVM()
When I tried calling my own java code, I had to first compile (javac ./blah/HelloWorldJPype.java), and I had to change the JVM path from the default (otherwise you'll get inexplicable "class not found" errors). For me, this meant changing the startJVM command to:
jp.startJVM('/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/MacOS/libjli.dylib', "-ea")
c = jp.JClass('blah.HelloWorldJPype')
# Where my java class file is in ./blah/HelloWorldJPype.class
...
I've been integrating a lot of stuff into Python lately, including Java. The most robust method I've found is to use IKVM and a C# wrapper.
IKVM has a neat little application that allows you to take any Java JAR, and convert it directly to .Net DLL. It simply translates the JVM bytecode to CLR bytecode. See http://sourceforge.net/p/ikvm/wiki/Ikvmc/ for details.
The converted library behaves just like a native C# library, and you can use it without needing the JVM. You can then create a C# DLL wrapper project, and add a reference to the converted DLL.
You can now create some wrapper stubs that call the methods that you want to expose, and mark those methods as DllEport. See https://stackoverflow.com/a/29854281/1977538 for details.
The wrapper DLL acts just like a native C library, with the exported methods looking just like exported C methods. You can connect to them using ctype as usual.
I've tried it with Python 2.7, but it should work with 3.0 as well. Works on Windows and the Linuxes
If you happen to use C#, then this is probably the best approach to try when integrating almost anything into python.
I'm just beginning to use JPype 0.5.4.2 (july 2011) and it looks like it's working nicely...
I'm on Xubuntu 10.04
I'm assuming that if you can get from C++ to Java then you are all set. I've seen a product of the kind you mention work well. As it happens the one we used was CodeMesh. I'm not specifically endorsing this vendor, or making any statement about their product's relative quality, but I have seen it work in quite a high volume scenario.
I would say generally that if at all possible I would recommend keeping away from direct integration via JNI if you can. Some simple REST service approach, or queue-based architecture will tend to be simpler to develop and diagnose. You can get quite decent perfomance if you use such decoupled technologies carefully.
Through my own experience trying to run some java code from within python in a manner similar to how python code runs within java code in python, I was unable to find a straightforward methodology.
My solution to my problem was by running this java code as BeanShell scripts by calling the BeanShell interpreter as a shell command from within my python code after editing the java code in a temporary file with the appropriate packages and variables.
If what I am talking about is helpful in any manner, I am glad to help you share more details of my solutions.

How to use Spock from the Command Line

I am writing a set of groovy scripts to be used as part of a Jenkins Pipeline Library. Currently I am using plain old JUnit to test them but would like to switch to Spock. I simply run the tests from the command line by invoking the following groovy script.
import groovy.util.AllTestSuite
import junit.textui.TestRunner
System.setProperty(AllTestSuite.SYSPROP_TEST_DIR, "./tests")
System.setProperty(AllTestSuite.SYSPROP_TEST_PATTERN, "**/*Test.groovy")
TestRunner.run(AllTestSuite.suite())
I am trying to figure what the equivalent script would be to run Spock specifications. My first attempt was to switch the SYSPROP_TEST_PATTERN to "**/*Spec.groovy. I have one ...Spec.groovy file written and sitting under ./tests that looks like this:
#Grab(group='org.spockframework', module='spock-core', version='1.0-groovy-2.3')
import spock.lang.*
class UtilsSpec extends Specification {
def "Just testing"() {
expect:
1 + 1 == 2
}
}
When I invoke my groovy script though I get:
java.lang.RuntimeException: Don't know how to treat
/SourcCode/jenkins/pipeline-utils/tests/JustTestingSpec.groovy as a
JUnit test
That makes sense. I need to be using Sputnik but I've looked at the Spock and Sputnik source, and the Spock example project but these all assume you are using maven or gradle. I can't figured out the right way to invoke Sputnik directly. Any ideas?
Even though what you ask is possible and BalRog has already suggested a solution, in the long run it is better if you just use Gradle or Maven to run your tests from command line.
All Jenkins tutorials you will encounter will talk about Maven and/or Gradle and thus it would make much more sense to use a build system than custom scripts.

Grape or normal import

Does it make a difference, how to integrate GPars in my projects?
Can I either do
#Grab(group='org.codehaus.gpars', module='gpars', version='1.0.0')
or just (provided I have all the required jars in my build path)
import groovyx.gpars.*
?
It's depending on how you organize your project and what is your build system.
If it's a plain Groovy script, using a #Grab annotation and import statements together will be working for you.
#Grab just tells the system to manage dependencies, and you still need import statements.
But if it's a bigger project, using Gradle is the better way.

Is there any global flag for Groovy static compilation?

I know that since Groovy 2.0 there are annotations for static compilation.
However it's ease to omit such annotation by accident and still run into troubles.
Is there any way to achieve the opposite compiler behaviour, like compile static all project files by default and compile dynamic only files chosen by purpose with some kind #CompileDynamic annotation for example?
I have found some (I believe recently introduced) feature which allows doing so with Gradle.
In build.gradle file for the project containing groovy sources we need to add following lines:
compileGroovy {
configure(groovyOptions) {
configurationScript = file("$rootDir/config/groovy/compiler-config.groovy")
}
}
or compileTestGroovy { ... for applying the same to test sources. Keep in mind that neither static compilation nor type checking works well with Spock Framework though. Spock by its nature utilizes dynamic 'groovyness' a lot.
Then on a root of the project create folder config/groovy/ and a file named compiler-config.groovy within. The content of the file is as follows:
import groovy.transform.CompileStatic
withConfig(configuration) {
ast(CompileStatic)
}
Obviously path and name of the configurationScript may vary and it's up to you. It shouldn't rather go to the same src/main/groovy though as it would be mixing totally separate concerns.
The same may be done with groovy.transform.TypeChecked or any other annotation, of course.
To reverse applied behaviour on certain classes or methods then #CompileDynamic annotation or #TypeChecked(TypeCheckingMode.SKIP) respectively may be used.
I'm not sure how to achieve the same when no Gradle is in use as build tool. I may update this answer in the future with such info though.
Not at this time, but there is an open Jira issue here you can follow to watch progress for this feature
There was also a discussion about methods for doing this on the Groovy developers list

Resources