I need to call a number of methods in parallel and wait for results. Each relies on different resources, so they may return at different times. I need to wait until I receive all results or time out after a certain amount of time.
I could just spawn threads with a reference to a shared object via a method call, but is there a better, more groovy way to do this?
Current Implementation:
Executors exec = Executors.newFixedThreadPool(10);
for (obj in objects) {
def method = {
def result = new ResultObject(a: obj, b: obj.callSomeMethod())
result
} as Callable<ResultObject>
callables << method
}
List<Future<ResultObject>> results = exec.invokeAll(callables)
for (result in results) {
try{
def searchResult = result.get()
println 'result retrieved'
} catch (Exception e)
{
println 'exception'
e.printStackTrace()
}
}
}
A Groovier solution is to use GPars - a concurrency library written in Groovy.
import static groovyx.gpars.GParsExecutorsPool.withPool
withPool {
def callable = {obj -> new ResultObject(a: obj, b: obj.callSomeMethod())}.async()
List<ResultObject> results = objects.collect(callable)*.get()
}
AbstractExecutorService.invokeAll(Collection<? extends Callable<T>> tasks, long timeout, TimeUnit unit)
The groovy part would be using closures as Callable
Related
I'm trying to do something rather simple. I would like to wrap the whole method code into an additional closure block that would measure the execution time. Right now I'm getting a really not helpful error message:
Error:Groovyc: NPE while processing Test.groovy
Annotation:
#Retention(RetentionPolicy.SOURCE)
#Target([ElementType.METHOD])
#GroovyASTTransformationClass(["WithTimingASTTransformation"])
public #interface WithTiming {
}
My wrapping closure:
class Benchmark {
static def measureTime(Closure cl) {
def start = System.currentTimeMillis()
def result = cl()
def time = System.currentTimeMillis() - start
println "it took $time"
result
}
}
My Transformation:
#GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)
class WithTimingASTTransformation implements ASTTransformation {
#Override
void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {
MethodNode method = astNodes[1]
method.code = wrap(method)
}
private Statement wrap(MethodNode method) {
def newBlock = new BlockStatement()
newBlock.addStatement(
new ExpressionStatement(
new StaticMethodCallExpression(
new ClassNode(Benchmark),
'measureTime',
new ArgumentListExpression(
new ClosureExpression(new Parameter[0], method.code)
))))
newBlock
}
}
I'm really stuck here and don't know how can I debug the problem.
There is an answer on a similar topic (wrapping whole method body into a try/catch block here). This works fine but my case is slightly different.
In my case similar NPE was coming from:
java.lang.NullPointerException
at org.codehaus.groovy.classgen.asm.ClosureWriter.createClosureClass(ClosureWriter.java:194)
at org.codehaus.groovy.classgen.asm.ClosureWriter.getOrAddClosureClass(ClosureWriter.java:159)
at org.codehaus.groovy.classgen.asm.ClosureWriter.writeClosure(ClosureWriter.java:90)
at org.codehaus.groovy.classgen.AsmClassGenerator.visitClosureExpression(AsmClassGenerator.java:673)
Whereas:
if (parameters == null || expression.getVariableScope() == null) {
parameters = Parameter.EMPTY_ARRAY;
} else if (parameters.length == 0) {
// let's create a default 'it' parameter
Parameter it = new Parameter(ClassHelper.OBJECT_TYPE, "it", ConstantExpression.NULL);
parameters = new Parameter[]{it};
Variable ref = expression.getVariableScope().getDeclaredVariable("it");
if (ref != null) it.setClosureSharedVariable(ref.isClosureSharedVariable());
}
and line 194 (as of https://github.com/groovy/groovy-core/commit/a52d0d3c5dd1cbb342992d36235171718a563c8b) is:
Variable ref = expression.getVariableScope().getDeclaredVariable("it");
Thus you need to define a VariableScope for your ClosureExpression. I had to add tracing into org.codehaus.groovy.ast.ClosureWriter to find this, because there is an issue with exception display on stage of Class Generation - both in IntelliJ Idea and in Groovy Console - it does not show proper lines of code.
Furthermore, I think that either ClosureWriter or ClosureExpression constructor can be fixed to work aligned by default - without this NPE. I will possibly submit an issue to Groovy Jira for this.
Now I am able to inject closure expression in my code. But struggling to call this closure.
Getting:
groovy.lang.MissingMethodException: No signature of method: com.a9ae0b01f0ffc.VSMSGEN.implementation.T_visa_recon_generator$_convert_vts_log_to_ctf_closure2.call() is applicable for argument types: () values: []
Long story short, after some iterations my method-wrapping AST looks like this:
BlockStatement bs = new BlockStatement()
ClosureExpression closureExp = new ClosureExpression( methodNode.parameters, methodNode.code )
closureExp.variableScope = new VariableScope() // <- this does the trick!
bs.addStatement new ExpressionStatement( new StaticMethodCallExpression( new ClassNode( TransactionUtil ), 'wrap', new ArgumentListExpression( closureExp ) ) )
methodNode.code = bs
The line closureExp.variableScope = new VariableScope() avoids the NPE in ClosureWriter.java:194 and the whole thing runs like a charm!
Hope it helps someone...
I call def activateReward by Akka actors and execution OracleClient.rewardActivate(user) sometimes is very slow (the database is outside of my responsibility and belongs to another company).
When database is slow the thread pool is exhausted and can not effectively allocate more threads to run callbacks future.onComplete because callbacks and futures works in the same execution context.
Please advise how to execute code in the callback asynchronously from threads which allocated for futures OracleClient.rewardActivate(user)
class RewardActivatorHelper {
private implicit val ec = new ExecutionContext {
val threadPool = Executors.newFixedThreadPool(1000)
def execute(runnable: Runnable) {threadPool.submit(runnable)}
def reportFailure(t: Throwable) {throw t}
}
case class FutureResult(spStart:Long, spFinish:Long)
def activateReward(msg:Msg, time:Long):Unit = {
msg.users.foreach {
user =>
val future:Future[FutureResult] = Future {
val (spStart, spFinish) = OracleClient.rewardActivate(user)
FutureResult(spStart, spFinish)
}
future.onComplete {
case Success(futureResult:FutureResult) =>
futureResult match {
case res:FutureResult => Logger.writeToLog(Logger.LogLevel.DEBUG,s"started:${res.spStart}finished:${res.spFinish}")
case _ => Logger.writeToLog(Logger.LogLevel.DEBUG, "some error")
}
case Failure(e:Throwable) => Logger.writeToLog(Logger.LogLevel.DEBUG, e.getMessage)
}
}
}
}
You can specify the execution context explicitly instead of implicitly for the onComplete callback by doing something along these lines:
import java.util.concurrent.Executors
import scala.concurrent.duration.Duration
object Example extends App {
import scala.concurrent._
private implicit val ec = new ExecutionContext {
val threadPool = Executors.newFixedThreadPool(1000)
def execute(runnable: Runnable) {threadPool.submit(runnable)}
def reportFailure(t: Throwable) {throw t}
}
val f = Future {
println("from future")
}
f.onComplete { _ =>
println("I'm done.")
}(scala.concurrent.ExecutionContext.Implicits.global)
Await.result(f, Duration.Inf)
}
This will of course not solve the underlying problem of a database not keeping up, but might be good to know anyway.
To clarify: I let the onComplete callback be handled by the standard global execution context. You might want to create a separate one.
I need to intercept method calls on predefined Java classes. For example, lets say I need to intercept String class split method, how do I do this?
I tried this which works, but I doesn’t want end user to change their code by wrapping their calls in with proxy block.
Is there any way this can be achieved with Groovy?
If what you want to do is intercept a call to a specific method you can do something like this...
// intercept calls to the split method on java.lang.String
String.metaClass.split = { String arg ->
// do whatever you want to do
}
If what you want to do is intercept a call to a specific method and do some stuff in addition to invoking the original (like to wrap the real method with some of your own logic) you can do something like this:
// get a reference to the original method...
def originalSplit = String.metaClass.getMetaMethod('split', [String] as Class[])
// now add your own version of the method to the meta class...
String.metaClass.split = { String arg ->
// do something before invoking the original...
// invoke the original...
def result = originalSplit.invoke(delegate, arg)
// do something after invoking the original...
// return the result of invoking the original
result
}
I hope that helps.
you want to use MetaClass for that see doc
ExpandoMetaClass.enableGlobally()
//call 'enableGlobally' method before adding to supplied class
String.metaClass.split = { regex ->
println "calling split from $delegate with $regex"
delegate.split regex, 22
}
To intercept all method calls in a class override Groovy's invokeMethod. Example:
class Test {}
Test.metaClass.foo = {"foo() called"}
Test.metaClass.static.bar = {"bar() called"}
Test.metaClass.invokeMethod = { name, args ->
handleInterception(name, args, delegate, false)
}
Test.metaClass.static.invokeMethod = { name, args ->
handleInterception(name, args, delegate, true)
}
def handleInterception(name, args, delegate, isStatic) {
def effDelegate = isStatic ? delegate : delegate.class
println ">> Entering ${delegate.class.name}.$name() with args: $args"
def metaMethod = effDelegate.metaClass.getMetaMethod(name, args)
if (!metaMethod) {
println "-- Method not found: $name($args)"
return
}
try {
def result = metaMethod.invoke(delegate, args)
println "<< Leaving ${delegate.class.name}.$name() with result: $result"
return result
} catch (ex) {
println "-- Exception occurred in $name: $ex.message"
throw ex
}
}
new Test().foo("1", 2)
Test.bar(2)
new Test().onTheFly(3)
Code taken from Roshan Dawrani's post at groovyconsole.appspot.com.
Output:
>> Entering Test.foo() with args: [1, 2]
-- Method not found: foo([1, 2])
>> Entering java.lang.Class.bar() with args: [2]
<< Leaving java.lang.Class.bar() with result: bar() called
>> Entering Test.onTheFly() with args: [3]
-- Method not found: onTheFly([3])
Other options:
Custom MetaClass implementing invokeMethod
Implementing the Interceptor Interface. Read more in this tutorial
I am doing some integration tests with Spock with 3rd party apps. Now I am struggling with a problem that I am not sure wether I am approaching the issue properly or not.
In one of the tests I am connecting to a 3rd party service to get some information in an array. Then each of these items are passed to another method to process them individually.
def get3rdPartyItems = {
[item1, item2, item3]
}
def processItem = { item ->
//do something with item
}
get3rdPartyItems.each {
processItem(it)
}
Then I have a test that connects to real 3rd party service using the method get3rdPartyItems() in which I am testing that processItem is called as many times as items has returned the method get3rdPartyItems().
What I am trying to do is to save one of the items as #Shared variable to write another test to know that the item is processed properly as I don't want to mock the content retrieved from the 3rd party service as I want real data.
Basically, this is what I am doing:
#Shared def globalItem
MyClass.metaClass.processItem = { i ->
if (!globalItem)
globalItem = i
//And now I would need to call the original method processItem
}
Any clue how to achieve this? I am probably overheading too much so I am open to change the solution.
Not sure if this is what you want, as it's hard to see your existing structure from the code and the code isn't runnable as-is, but given this class:
class MyClass {
def get3rdPartyItems = {
['item1', 'item2', 'item3']
}
def processItem( item ) {
println item
//do something with item
}
def run() {
get3rdPartyItems().each {
processItem( it )
}
}
}
You can do this:
def globalItem
def oldProcessItem = MyClass.metaClass.getMetaMethod("processItem", Object)
MyClass.metaClass.processItem = { item ->
if (!globalItem) {
println "Setting global item to $item"
globalItem = item
}
oldProcessItem.invoke( delegate, item )
}
def mc = new MyClass()
new MyClass().run()
Just as a matter of concision, that should be the way of passing the parameters to the metamethod in case you pass multiple parameters:
def globalItem
def oldProcessItem = MyClass.metaClass.getMetaMethod("processItem", ["",[:]] as Object[])
MyClass.metaClass.processItem = { String p1, Map p2 ->
if (!globalItem) {
println "Setting global item to $item"
globalItem = p2
}
oldProcessItem.invoke( delegate, [p1,p2] as Object[] )
}
def mc = new MyClass()
new MyClass().run()
Java Future has cancel method, which can interrupt the thread, which runs the Future task. For example, if I wrap an interruptible blocking call in a Java Future I can interrupt it later.
Scala Future provides no cancel method. Suppose I wrap an interruptible blocking call in a Scala Future. How can I interrupt it?
This is not yet a part of the Futures API, but may be added as an extension in the future.
As a workaround, you could use the firstCompletedOf to wrap 2 futures - the future you want to cancel and a future that comes from a custom Promise. You could then cancel the thus created future by failing the promise:
def cancellable[T](f: Future[T])(customCode: => Unit): (() => Unit, Future[T]) = {
val p = Promise[T]
val first = Future firstCompletedOf Seq(p.future, f)
val cancellation: () => Unit = {
() =>
first onFailure { case e => customCode}
p failure new Exception
}
(cancellation, first)
}
Now you can call this on any future to obtain a "cancellable wrapper". Example use-case:
val f = callReturningAFuture()
val (cancel, f1) = cancellable(f) {
cancelTheCallReturningAFuture()
}
// somewhere else in code
if (condition) cancel() else println(Await.result(f1))
EDIT:
For a detailed discussion on cancellation, see Chapter 4 in the Learning concurrent programming in Scala book.
I haven't tested this, but this expands on the answer of Pablo Francisco Pérez Hidalgo. Instead of blocking waiting for the java Future, we use an intermediate Promise instead.
import java.util.concurrent.{Callable, FutureTask}
import scala.concurrent.{ExecutionContext, Promise}
import scala.util.Try
class Cancellable[T](executionContext: ExecutionContext, todo: => T) {
private val promise = Promise[T]()
def future = promise.future
private val jf: FutureTask[T] = new FutureTask[T](
new Callable[T] {
override def call(): T = todo
}
) {
override def done() = promise.complete(Try(get()))
}
def cancel(): Unit = jf.cancel(true)
executionContext.execute(jf)
}
object Cancellable {
def apply[T](todo: => T)(implicit executionContext: ExecutionContext): Cancellable[T] =
new Cancellable[T](executionContext, todo)
}
By cancelling I guess you would like to violently interrupt the future.
Found this segment of code: https://gist.github.com/viktorklang/5409467
Did a few tests and seems to work fine!
Enjoy :)
I think it is possible to reduce the complexity of the implementations provided by making use of the Java 7 Future interface and its implementations.
Cancellable can build a Java future which is the one to be cancelled by its cancel method. Another future can wait for its completion thus becoming the observable interface which is itself immutable in state:
class Cancellable[T](executionContext: ExecutionContext, todo: => T) {
private val jf: FutureTask[T] = new FutureTask[T](
new Callable[T] {
override def call(): T = todo
}
)
executionContext.execute(jf)
implicit val _: ExecutionContext = executionContext
val future: Future[T] = Future {
jf.get
}
def cancel(): Unit = jf.cancel(true)
}
object Cancellable {
def apply[T](todo: => T)(implicit executionContext: ExecutionContext): Cancellable[T] =
new Cancellable[T](executionContext, todo)
}