I am trying to load 2 groovy scripts from a remote location (say S3). The first script is straight forward. It names a package a.b and the file itself is called c.dsl.
package a.b;
int i = 10;
public void anotherfunc() {};
private void anotheractivityFunc() { int k = 9;};
The second groovy script has the following code in it (it is named s.dsl)
package c.d;
// Notice the import of the other script file here
import a.b.c;
int i = 10;
c myself;
public void func() {};
private void activityFunc()
{
int k = 9;
c nbn;
};
This script is trying to create an object of the class defined in the first script. I run this through with this code
CustomClassLoader loader = new CustomClassLoader(GroovyDSLTest.class.getClassLoader(), configuration);
GroovyShell shell = new GroovyShell(loader, configuration);
Script script = shell.parse(scriptText, "s.dsl");
CustomClassLoader is defined with this function
public Class loadClass(name, ...)
{
try
{
return super.loadClass(name, lookupScriptFiles, preferClassOverScript, resolve);
}
catch (ClassNotFoundException e)
{
if (name.equals("a.b.c"))
// Notice how I handle the import a.b.c from the second groovy file
return parseClass(<text from c.dsl>, "c.dsl");
throw e;
}
}
I get a compile error when I do this. It loads the a.b.c file cleanly. But when I try to make a member variable "c" it fails.
s.dsl: 1: [Static type checking] - The variable [myself] is undeclared.
# line 8, column 3.
c myself;
^
s.dsl: 1: [Static type checking] - Cannot find matching method c.d.s#c(java.lang.Object). Please check if the declared type is right and if the method exists.
# line 8, column 1.
c myself;
^
Why is it not able to find c? I was able to print the classloader as it loads classes and it did load a.b.c. Changing the above to use a.b.c instead of just "c" does not work as well. What is the name of the class that groovy generates from the script file. I want to be able to access this class and create objects of it.
How about using #BaseScript?
You can create a class that you want to import:
abstract class Something extends Script {
...
}
And #BaseScript can help you to import it: #groovy.transform.BaseScript Something something
See: http://docs.groovy-lang.org/2.5.2/html/gapi/groovy/transform/BaseScript.html
Grrrr... I figured this out.
a) one needs to call in loadClass setClassCacheEntry();
b) Also naming a class "c" does not seem to work. I renamed it to MyClass and it seems to work
Related
I'm running into a problem with GroovyScriptEngine - it seems not to be able to work with inner classes. Anyone know whether there's some limitation in GroovyScriptEngine or a workaround?
I have a directory with these two files:
// MyClass.groovy
public class MyClass {
MyOuter m1;
MyOuter.MyInner m2;
}
and
// MyOuter.groovy
public class MyOuter {
public static class MyInner {}
}
I have a following test class:
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import groovy.util.GroovyScriptEngine;
public class TestGroovyScriptEngine {
public static void main(String[] args) throws MalformedURLException, ClassNotFoundException {
final File myGroovySourceDir = new File("C:/MyGroovySourceDir");
final URL[] urls = { myGroovySourceDir.toURL() };
GroovyScriptEngine groovyScriptEngine = new GroovyScriptEngine(urls,
Thread.currentThread().getContextClassLoader());
Class<?> clazz = groovyScriptEngine.getGroovyClassLoader().loadClass("MyClass");
}
}
When I run it I get the following compilation error:
Exception in thread "main" org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
C:\MyGroovySourceDir\MyClass.groovy: 3: unable to resolve class MyOuter.MyInner
# line 3, column 2.
MyOuter.MyInner m2;
^
1 error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:983)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:633)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:582)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:354)
at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:87)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:323)
at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:320)
at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:318)
at groovy.util.GroovyScriptEngine$ScriptClassLoader.doParseClass(GroovyScriptEngine.java:248)
at groovy.util.GroovyScriptEngine$ScriptClassLoader.parseClass(GroovyScriptEngine.java:235)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:307)
at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:811)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:767)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:836)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:824)
I would have expected a "clean compile", but the inner class seems to be causing problems.
My groovy classes compile fine at the command line using groovyc, or in Eclipse.
You have faced an edge case here. To clarify what happens let's define the initial conditions:
you have a Java (or Groovy) class that gets executed inside JVM
you have two Groovy classes that get loaded outside of the JVM
The problem you have described does not exist if you put these two Groovy classes inside the same path you execute your Java class from - in this case IDE takes care to compile these Groovy classes and put them to the classpath of a JVM that gets started to run your Java test class.
But this is not your case and you are trying to load these two Groovy classes outside the running JVM using GroovyClassLoader (which extends URLClassLoader btw). I will try to explain in the simplest possible words what happened that adding field of type MyOuter does not throw any compilation error, but MyOuter.MyInner does.
When you execute:
Class<?> clazz = groovyScriptEngine.getGroovyClassLoader().loadClass("MyClass");
Groovy class loader goes to script file lookup part, because it was not able to find MyClass in the current classpath. This is the part responsible for it:
// at this point the loading from a parent loader failed
// and we want to recompile if needed.
if (lookupScriptFiles) {
// try groovy file
try {
// check if recompilation already happened.
final Class classCacheEntry = getClassCacheEntry(name);
if (classCacheEntry != cls) return classCacheEntry;
URL source = resourceLoader.loadGroovySource(name);
// if recompilation fails, we want cls==null
Class oldClass = cls;
cls = null;
cls = recompile(source, name, oldClass);
} catch (IOException ioe) {
last = new ClassNotFoundException("IOException while opening groovy source: " + name, ioe);
} finally {
if (cls == null) {
removeClassCacheEntry(name);
} else {
setClassCacheEntry(cls);
}
}
}
Source: src/main/groovy/lang/GroovyClassLoader.java#L733-L753
Here URL source = resourceLoader.loadGroovySource(name); it loads the full file URL to the source file and here cls = recompile(source, name, oldClass); it executes class compilation.
There are several phases involved in Groovy class compilation. One of them is Phase.SEMANTIC_ANALYSIS which analyses class fields and their types for instance. At this point ClassCodeVisitorSupport executes visitClass(ClassNode node) for MyClass class and following line
node.visitContents(this);
starts class contents processing. If we take a look at the source code of this method:
public void visitContents(GroovyClassVisitor visitor) {
// now let's visit the contents of the class
for (PropertyNode pn : getProperties()) {
visitor.visitProperty(pn);
}
for (FieldNode fn : getFields()) {
visitor.visitField(fn);
}
for (ConstructorNode cn : getDeclaredConstructors()) {
visitor.visitConstructor(cn);
}
for (MethodNode mn : getMethods()) {
visitor.visitMethod(mn);
}
}
Source: src/main/org/codehaus/groovy/ast/ClassNode.java#L1066-L108
we will see that it analyses and process class properties, fields, constructors and methods. At this phase it resolves all types defined for these elements. It sees that there are two properties m1 and m2 with types MyOuter and MyOuter.MyInner accordingly, and it executes visitor.visitProperty(pn); for them. This method executes the one we are looking for - resolve()
private boolean resolve(ClassNode type, boolean testModuleImports, boolean testDefaultImports, boolean testStaticInnerClasses) {
resolveGenericsTypes(type.getGenericsTypes());
if (type.isResolved() || type.isPrimaryClassNode()) return true;
if (type.isArray()) {
ClassNode element = type.getComponentType();
boolean resolved = resolve(element, testModuleImports, testDefaultImports, testStaticInnerClasses);
if (resolved) {
ClassNode cn = element.makeArray();
type.setRedirect(cn);
}
return resolved;
}
// test if vanilla name is current class name
if (currentClass == type) return true;
String typeName = type.getName();
if (genericParameterNames.get(typeName) != null) {
GenericsType gt = genericParameterNames.get(typeName);
type.setRedirect(gt.getType());
type.setGenericsTypes(new GenericsType[]{ gt });
type.setGenericsPlaceHolder(true);
return true;
}
if (currentClass.getNameWithoutPackage().equals(typeName)) {
type.setRedirect(currentClass);
return true;
}
return resolveNestedClass(type) ||
resolveFromModule(type, testModuleImports) ||
resolveFromCompileUnit(type) ||
resolveFromDefaultImports(type, testDefaultImports) ||
resolveFromStaticInnerClasses(type, testStaticInnerClasses) ||
resolveToOuter(type);
}
Source: src/main/org/codehaus/groovy/control/ResolveVisitor.java#L343-L378
This method gets executed for both MyOuter and MyOuter.MyInner classes. It is worth mentioning that class resolving mechanism only checks if given class is available in the classpath and it does not load or parse any classes. That is why MyOuter gets recognized when this method reaches resolveToOuter(type). If we take a quick look at its source code we will understand why it works for this class:
private boolean resolveToOuter(ClassNode type) {
String name = type.getName();
// We do not need to check instances of LowerCaseClass
// to be a Class, because unless there was an import for
// for this we do not lookup these cases. This was a decision
// made on the mailing list. To ensure we will not visit this
// method again we set a NO_CLASS for this name
if (type instanceof LowerCaseClass) {
classNodeResolver.cacheClass(name, ClassNodeResolver.NO_CLASS);
return false;
}
if (currentClass.getModule().hasPackageName() && name.indexOf('.') == -1) return false;
LookupResult lr = null;
lr = classNodeResolver.resolveName(name, compilationUnit);
if (lr!=null) {
if (lr.isSourceUnit()) {
SourceUnit su = lr.getSourceUnit();
currentClass.getCompileUnit().addClassNodeToCompile(type, su);
} else {
type.setRedirect(lr.getClassNode());
}
return true;
}
return false;
}
Source: src/main/org/codehaus/groovy/control/ResolveVisitor.java#L725-L751
When Groovy class loader tries to resolve MyOuter type name it reaches
lr = classNodeResolver.resolveName(name, compilationUnit);
which locates script with a name MyOuter.groovy and it creates a SourceUnit object associated with this script file name. It is simply something like saying "OK, this class is not in my classpath at the moment, but there is a source file I can see that once compiled it will provide a valid type of name MyOuter". This is why it finally reaches:
currentClass.getCompileUnit().addClassNodeToCompile(type, su);
where currentClass is an object associated with MyClass type - it adds this source unit to MyClass compilation unit, so it gets compiled with the MyClass class. And this is the point where resolving
MyOuter m1
class property ends.
In the next step it picks MyOuter.MyInner m2 property and it tries to resolve its type. Keep in mind - MyOuter got resolved correctly, but it didn't get loaded to the classpath, so it's static inner class does not exist in any scope, yet. It goes through the same resolving strategies as MyOuter, but any of them works for MyOuter.MyInner class. And this is why ResolveVisitor.resolveOrFail() eventually throws this compilation exception.
Workaround
OK, so we know what happens, but is there anything we can do about it? Luckily, there is a workaround for this problem. You can run your program and load MyClass successfully only if you load MyOuter class to Groovy script engine first:
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import groovy.util.GroovyScriptEngine;
public class TestGroovyScriptEngine {
public static void main(String[] args) throws MalformedURLException, ClassNotFoundException {
final File myGroovySourceDir = new File("C:/MyGroovySourceDir");
final URL[] urls = { myGroovySourceDir.toURL() };
GroovyScriptEngine groovyScriptEngine = new GroovyScriptEngine(urls,
Thread.currentThread().getContextClassLoader());
groovyScriptEngine.getGroovyClassLoader().loadClass("MyOuter");
Class<?> clazz = groovyScriptEngine.getGroovyClassLoader().loadClass("MyClass");
}
}
Why does it work? Well, semantic analysis of MyOuter class does not cause any problems, because all types are known at this stage. This is why loading MyOuter class succeeds and it results in Groovy script engine instance knows what MyOuter and MyOuter.MyInner types are. So when you next load MyClass from the same Groovy script engine it will apply different resolving strategy - it will find both classes available to the current compilation unit and it wont have to resolve MyOuter class based on its Groovy script file.
Debugging
If you want to examine this use case better it is worth to run a debugger and see analyze what happens at the runtime. You can create a breakpoint at line 357 of ResolveVisitor.java file for instance, to see described scenario in action. Keep in mind one thing though - resolveFromDefaultImports(type, testDefaultImports) will try to lookup MyClass and MyOuter classes by applying default packages like java.util, java.io, groovy.lang etc. This resolve strategy kicks in before resolveToOuter(type) so you have to patiently jump through them. But it is worth it to see and get a better understanding about how things work. Hope it helps!
I'm not sure if this is already possible but is it possible to use #CompileStatic on just plain groovy scripts without wrapping the script in a class?
I'm using the GroovyClassLoader to parse and compile scripts dynamically and so it would be nice if scripts could use #CompileStatic
You can add a compiler customizer:
configuration.addCompilationCustomizers(
new ASTTransformationCustomizer(CompileStatic))
If you were on the command line you could provide a --configscript config.groovy. This answer has an example of how the script should look like:
withConfig(configuration) {
ast(groovy.transform.CompileStatic)
}
You can add the #CompileStatic annotation to a class or a method.
It does not work for a whole script but you can add it to some methods or some classes of your script.
You have to add the import groovy.transform.CompileStatic in the imports of your script or class.
As an example you can try the following script in a GroovyConsole, commenting/uncommenting the method annotation:
import groovy.transform.CompileStatic
#CompileStatic
void loop() {
def d1 = new Date();
int[] t = new int[32*1024*1024]
(0 .. 32*1024*1024-1).each {
it -> t[it] = it
}
def d2 = new Date();
println d2.getTime() - d1.getTime()
}
loop()
I want to write a groovy DSL with syntax:
returnValue when booleanCondition
I want to use compilation customizers to transform this expression to a typical if return statement using AST transformations.
For this script:
2 when 1 == 1
I get exception message:
MultipleCompilationErrorsException: startup failed:
Script1.groovy: 1: expecting EOF, found '1' # line 1, column 8.
I don't understand why my compilation customizer is not called at all?
I need it to be called before compilation so I can make it into a valid groovy code.
If the script contains valid groovy code, then my compilation customizer is called.
My code:
class MyDslTest {
public static void main(String[] args) {
String script = '''2 when 1 == 1
'''
def compilerConfig = new CompilerConfiguration()
compilerConfig.addCompilationCustomizers(new MyCompilationCustomizer())
GroovyShell groovyShell = new GroovyShell(compilerConfig)
groovyShell.evaluate(script)
}
}
class MyCompilationCustomizer extends CompilationCustomizer {
MyCompilationCustomizer() {
super(CompilePhase.CONVERSION)
}
#Override
void call(SourceUnit source, GeneratorContext context, ClassNode classNode) throws CompilationFailedException {
println 'in compilation customizer'
}
}
The problem is that your code is not syntactically valid. A compilation customizer will not workaround that: to be able to get an AST, on which the customizer will work, you have to produce syntactically correct code. One option is to use a different AntlrParserPlugin, but in general I don't recommend to do it because it will modify the sources before parsing, and therefore create a mismatch between the AST and the actual source.
I'm currently creating a Groovy DSL that looks like the this:
types {
A "http://some.namespace.here"
B "..."
}
testMethod(A a, B b) { ... }
The user defines several types and some methods on these types. What I need to do is to create A and B as ClassNodes in the AST so that Groovy can find them.
To do so, I created the following AST transformation:
#GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)
class GenerateClassesTransformer implements ASTTransformation {
#Override
public void visit(ASTNode[] nodes, SourceUnit source) {
ClassNode a = createClass("A")
ClassNode b = createClass("b")
ModuleNode module = sourceUnit.getAST()
module.addClass(a)
module.addClass(b)
module.getClasses().each {
println "class: " + it
}
}
private ClassNode createClass(String className) {
new AstBuilder().buildFromSpec {
classNode(className, ClassNode.ACC_PUBLIC) {
classNode Object
interfaces { classNode GroovyObject }
mixins { }
}
}.first()
}
}
I then add the AST transformation to the Groovy shell:
CompilerConfiguration cc = new CompilerConfiguration();
cc.setScriptBaseClass("groovy.util.DelegatingScript");
cc.addCompilationCustomizers(new ASTTransformationCustomizer(new GenerateClassesTransformer()));
GroovyShell sh = new GroovyShell(loader, binding, cc);
// ...
When I run the code and do NOT reference A and B, everything works out fine. However, as soon as I use A and B, Groovy cannot find the classes (although they are on the AST). Instead, I get:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
C:\...\test.groovy: 23: unable to resolve class A
# line 23, column 6.
A a = null
^
Any help would be greatly appreciated!
Update: ClassLoader
I'm using a custom URLClassLoader, since I'm adding a couple of JARs to the classpath:
URL[] urls = this.getExtraJarUrls(); // read JARs from a directory relative to the DSL script
ClassLoader base = Launcher.class.getClassLoader();
loader = new URLClassLoader(urls, base);
So I want to add methods to JDK classes like InputStream, File, etc. I'm trying to figure out what is the best way to do that, but it seems there are several options for doing it. One way is do this by adding methods into the metaClass property on the Class like so:
InputStream.metaClass.copy = { OutputStream out ->
long total = 0
byte[] buffer = new byte[8096]
int len
while ((len = read(buffer)) >= 0) {
out.write(buffer, 0, len)
total += len
}
out.flush()
return delegate
}
Another way is using dynamic mixins like this:
class EnhancedInputStream {
static {
InputStream.metaClass.mixin( EnhancedInputStream )
}
public InputStream copy( OutputStream out ) {
long total = 0
byte[] buffer = new byte[8096]
int len
while ((len = mixinOwner.read(buffer)) >= 0) {
out.write(buffer, 0, len)
total += len
}
out.flush()
return mixinOwner
}
}
So the first question is do dynamic Mixins replace the use of using metaClass + Closure to create mixins? The examples of dynamic mixins don't really discuss scoping rules in any detail that I can find. Which leads me to the next point.
You can see in the first code sample using metaClass I used delegate to get access to the this pointer of the class I was adding methods to. What is the equivalent way to do that using dynamic Mixins? All examples I've found are stateless (pointless really). I found one example mentioning a special member mixinOwner that could be used in place of delegate. Is that true?
Second you'll see I used a static block in EnhancedInputStream to add the mixin dynamically to InputStream. When using metaClass what is the best way to add those? Another static block with import statement?
I suppose I really want just a compile time Mixin where I can define the #Mixin on the source of the mixin instead of destination because I didn't write the destination. Like
#MixinInto(File)
public class EnhancedFileMixin {
public void zip( File output ) {
// .....
}
}
But that doesn't appear to exist in Groovy land. So what's the best approach to reach this using metaClass or dynamic mixins?
I guess the nearest to #MixinInto would be the magic package convention. I couldn't mix it into a interface, but i managed to mix it into a FileInputStream, if that suits your case. I guess you can add state using the MetaClass which comes in the constructor.
To write a class to be mixed into InputStream. It needs to be:
In the package groovy.runtime.metaclass.java.io
Named FileInputStreamMetaClass (exactly)
Compiled and put into the classpath
Extend DelegatingMetaClass
It can only intercept the GroovyObject methods, so it is not so straightforward. If you are in for a pure dynamic groovy, it is great:
package groovy.runtime.metaclass.java.io
class FileInputStreamMetaClass extends DelegatingMetaClass {
FileInputStreamMetaClass(MetaClass meta) {
super(meta)
println "built FileInputStreamMetaClass"
}
Object invokeMethod(Object delegate, String method, Object[] args) {
switch (method) {
case "copy":
return "i'm copying stuff"
default:
return super.invokeMethod(delegate, method, args)
}
}
}
Compiling:
$ groovyc FileInputStreamMetaClass.groovy
$ groovy -cp . InputTest.groovy
A test:
InputStream input = new FileInputStream("/tmp/test.tmp")
assert input.copy() == "i'm copying stuff"
A bit cumbersome.
I'd go for Extensions any time of the day. Three files:
// file META-INF/services/org.codehaus.groovy.runtime.ExtensionModule
moduleName=InputExtensionModule
moduleVersion=0.1
extensionClasses=InputStreamExtension
The extension:
class InputStreamExtension {
static String copy(InputStream input) {
"copying stuff, doc"
}
}
The test:
def input = new FileInputStream("/tmp/test.tmp")
assert input.copy() == "copying stuff, doc"
Compile and run:
$ groovyc InputStreamExtension.groovy
$ groovy ISExtensionTest.groovy
And i think the extension is the perfect place to use the static { mixin } block. With some changes:
class InputStreamExtension {
static {
InputStream.mixin InputStreamMixin
}
static String copy(InputStream input) { "copying stuff, doc" }
}
#Category(InputStream)
class InputStreamMixin {
Object pop() {
"input mixin pop"
}
}
A new test:
def input = new FileInputStream("/tmp/test.tmp")
assert input.copy() == "copying stuff, doc"
assert input.pop() == "input mixin pop"
Well I finally figured it out on my own. Essentially the this reference refers to the instance of the Mixin which doesn't do us much good. However, you can use the "as" keyword to convert that to the instance of the target class. For example:
class MyMixin {
static {
File mixin MyMixin
}
File foo() {
return this as File
}
}
File f = new File()
println( f.foo().equals( f ) )
As for mixinOwner and owner references that the jira bug refers to. They don't exist. This is the only way to get a reference to the instance that the mixin was added to.
I wrote up a longer blogpost about it because I thought this was important information for future Groovy programmers since there is zero official docs discussing this.
http://wrongnotes.blogspot.com/2013/06/groovy-mixins-and-this-pointer.html
I am glad you asked this question. To answer a very important question:
I suppose I really want just a compile time Mixin where I can define the #Mixin on the source of the mixin instead of destination because I didn't write the destination.
You cannot achieve this by #Mixin but we do have something in Groovy which will help you out. It is called #Category. Let me go through your example again to show you how you can actually effectively use this in category. Have a look at the below script:
#Category(InputStream)
class InputStreamCategory{
def copy(OutputStream out){
long total = 0
byte[] buffer = new byte[8096]
int len
while ((len = this.read(buffer)) >= 0) {
out.write(buffer, 0, len)
total += len
}
out.flush()
return this
}
}
class MyUtil{
def str = 'This is a dummy String!!!!'
InputStream inputS = new ByteArrayInputStream(str.bytes)
OutputStream outputS = new ByteArrayOutputStream()
def copy(){
use(InputStreamCategory){
inputS.copy(outputS)
println "Printing Output Stream: " + outputS
}
}
}
def util = new MyUtil()
util.copy()
//Prints:
Printing Output Stream: This is a dummy String!!!!
Explanation:-
#Category(InputStream) sets the this in InputStreamCategory and in your util class you just use the newly added method copy to InputStream. The benefit for using category is that you get hold of the caller object in this case inputS. The first parameter passed into a category always refers to the caller. You can have different categories for different implementations like FileCategory etc and then create an Utility Class to use those Categories. Therefore, you would end up with utilities like zip, copy, etc.
You could get detail information about the same from the api. I also highly recommend going through Category and Mixin Transformations.
To answer the first question:-
do dynamic Mixins replace the use of using metaClass + Closure to create mixins?
No they do not. metaClass implementation does not create a Mixin. It just adds another method in the metaData registry about the class while runtime. You get an handle to the delegate. On the other hand #Mixin gives you the ability to inherit pre-defined properties.