Sleep in future blocking thread causes RejectedExecutionException - multithreading

If I do Thread.sleep, I get RejectedExecutionException, otherwise, the code works fine.
Is it not possible to make the thread sleep for some time?
import java.util.concurrent.{Executors, TimeUnit}
import scala.concurrent.{ExecutionContext, Future, blocking}
import scala.util.{Failure, Success}
val executorService = Executors.newFixedThreadPool(2)
implicit val exec = ExecutionContext.fromExecutorService(executorService)
Future { blocking{
println("Thread sleeps...")
Thread.sleep(100) // <<<< causes exception on 'shutdown'
println("Thread running again...")
}} onComplete {
case Success(_) => println("Done ABC")
case Failure(exception) => println(exception)
}
exec.shutdown()
exec.awaitTermination(1000, TimeUnit.MILLISECONDS)
outputs:
Thread sleeps...
Thread running again...
java.util.concurrent.RejectedExecutionException: Task scala.concurrent.impl.CallbackRunnable#79834f9b rejected from java.util.concurrent.ThreadPoolExecutor#35267234[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:157)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:23)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

By removing onComplete, I see there is no exception. I had registered a callback by using the onComplete, but the operation ExecutorService.shutdown must not be allowing any new tasks to be accepted.
If the future has completed, the function should be applied immediately or scheduled asynchronously. This didn't work because of the shutdown of executor service in my case.

If I do Thread.sleep, I get RejectedExecutionException, otherwise, the code works fine.
This problem is completely unrelated to Thread.sleep—since a Future only schedules its callbacks once its result is available, your code has a race-condition between starting the computations associated with the Future, and shutting down the ExecutionContext.
Is it not possible to make the thread sleep for some time?
This is a completely different question to the one above—if you want to use a Timer-facility, you can use that and complete the Future once your Timer triggers (after a delay).
For a discussion around timers, see: How to implement a delayed future without java.util.Timer?

Related

nodejs: what's the correct way to monitor a task queue?

I know I can write a while (true) loop to monitor the queue, but it will cause the CPU 100% problem.
I can sleep some seconds inside the while (true) loop, but it's NOT efficient.
In C language, I can wait for a semaphore inside the while (true) loop. When a task added into the queue, release the semaphore so that the while (true) loop can do its job. After the queue is empty, it can set the semaphore, and wait for it.
Is there similar way to do this in Nodejs?
Imagine we have this taskQueue:
// Tasks will be added to the array randomly
const tasks = [];
Note: the taskQueue above is something completely different than the internal NodeJS micro/macro task queue, that I'm referring to throughout this post.
A way of constantly monitoring this array would be to schedule a 'micro-task' or 'macro-task' that parses the array.
As an example:
function handleTasks() {
if (tasks.length) {
// Alternatively loop and pop all the current tasks in queue
const task = tasks.pop();
// Do something with the task
}
setImmediate(handleTasks)
}
setImmediate(handleTasks)
The setImmediate function will add a task to the internal macro-task queue.
The JS micro- and macro-tasks do not block the main thread and will only be executed when the event-loop picks it off the internal micro/macro task queue.
In NodeJS there are 4 ways of scheduling a function in a non-blocking way. Which way you pick is based on how much priority you'd want to give to the function.
Ordered by highest priority first the ways to do this are:
process.nextTick(handleTask)
new Promise((resolve) => { resolve() }).then(handleTask)
setImmediate(handleTask) / setTimeout(handleTask, 0)
setTimeout(handleTask, 1) # Every timeout value bigger than 0
Be aware that executing this function with the highest priority recursively could slow down the rest of your code.
Depending on how important clearing this taskQueue is, I'd generally suggest to use setTimeout with a reasonable value (as high as you can afford) to prevent affecting performance of your application. (Same goes for any other function that schedules itself on the micro/macro task queue.)
Questions
I know I can write a while (true) loop to monitor the queue, but it
will cause the CPU 100% problem.
In JavaScript the functions cannot be preempted, meaning that their execution cannot be halted somewhere in the middle.
The consequence is that once a function start, it will have to finish before another line of code (somewhere else) can be executed.
Therefore an infinite while-loop will not work.
I can sleep some seconds inside the while (true) loop, but it's NOT
efficient.
while(true) {
await timeout(1000);
// Do sth
}
Is actually syntactic sugar for
timeout(1000).then(() => {
// Do sth
timeout(1000).then(() => {
// Do sth
// ...etc
})
})
Using await inside a loop is considered a bad-practice, but could work since it just schedules each next iteration on the micro-task queue.
In C language, I can wait for a semaphore inside the while (true) loop. When a
task added into the queue, release the semaphore so that the while
(true) loop can do its job. After the queue is empty, it can set the
semaphore, and wait for it.
There is no such thing as a semaphore in JS. Something that might achieve a similar effect could be a callback function.
Example:
function heavyLoadTask() {
// Do sth
resumeExecution = () => {
// What to do when execution is resumed
}
}
// Somewhere else the execution could be resumed like this;
if (typeof resumeExecution === "function"){
resumeExecution();
}
Recommended reading
https://javascript.info/event-loop
https://nodejs.dev/learn/understanding-process-nexttick
https://nodejs.dev/learn/understanding-setimmediate

Play Framework: new thread is created without limit when invoking async future in blocking way

I have a following code:
import scala.concurrent.ExecutionContext.Implicits.global
def index = Action {
Ok(Await.result(callSync, 10.seconds).body)
}
def callSync = {
WS.url("http://yahoo.jp").get
}
Basically WS.url will return Future[ws.Response] so in the code above I wanted to monitor the behaviour of this service when invoked in blocking manner. In my action, I am waiting for the result then displaying the response body back. I am attempting this with 2000 concurrent users with 20sec ramp. Problem is that above code creates new threads in massive amount that play instance shuts down the the error "java.lang.OutOfMemoryError : unable to create new native Thread". This is totally not expected behaviour. I am using the default execution context, so this pool will only have core + 1 threads. Why is above creating massive amount of threads?
Await.result wraps the blocking wait for a result with scala.concurrent.blocking which informs the ExecutionContext that it is blocking. The default ExecutionContext is backed by a fork-join pool which would then starve quickly since it only has got as many threads as there is cores and instead it will spawn a new thread to keep the number of available threads for non-blocking operations the same.
Do this instead:
import play.api.libs.concurrent.Promise
def index = Action.async {
Future.firstCompletedOf(List(
callsync.map(x => Ok(x.body)),
Promise.timeout(Ok("an error occurred"), 10.seconds)
))
}

Interrupt parallel Stream execution

Consider this code :
Thread thread = new Thread(() -> tasks.parallelStream().forEach(Runnable::run));
tasks are a list of Runnables that should be executed in parallel.
When we start this thread, and it begins its execution, then depending on some calculations we need to interrupt (cancel) all those tasks.
Interrupting the Thread will only stop one of exections. How do we handle others? or maybe Streams should not be used that way? or you know a better solution?
You can use a ForkJoinPool to interrupt the threads:
#Test
public void testInterruptParallelStream() throws Exception {
final AtomicReference<InterruptedException> exc = new AtomicReference<>();
final ForkJoinPool forkJoinPool = new ForkJoinPool(4);
// use the pool with a parallel stream to execute some tasks
forkJoinPool.submit(() -> {
Stream.generate(Object::new).parallel().forEach(obj -> {
synchronized (obj) {
try {
// task that is blocking
obj.wait();
} catch (final InterruptedException e) {
exc.set(e);
}
}
});
});
// wait until the stream got started
Threads.sleep(500);
// now we want to interrupt the task execution
forkJoinPool.shutdownNow();
// wait for the interrupt to occur
Threads.sleep(500);
// check that we really got an interruption in the parallel stream threads
assertTrue(exc.get() instanceof InterruptedException);
}
The worker threads do really get interrupted, terminating a blocking operation. You can also call shutdown() within the Consumer.
Note that those sleeps might not be tweaked for a proper unit test, you might have better ideas to just wait as necessary. But it is enough to show that it is working.
You aren't actually running the Runnables on the Thread you are creating. You are running a thread which will submit to a pool, so:
Thread thread = new Thread(() -> tasks.parallelStream().forEach(Runnable::run));
In this example you are in lesser terms doing
List<Runnable> tasks = ...;
Thread thread = new Thread(new Runnable(){
public void run(){
for(Runnable r : tasks){
ForkJoinPool.commonPool().submit(r);
}
}
});
This is because you are using a parallelStream that delegates to a common pool when handling parallel executions.
As far as I know, you cannot get a handle of the Threads that are executing your tasks with a parallelStream so may be out of luck. You can always do tricky stuff to get the thread but probably isn't the best idea to do so.
Something like the following should work for you:
AtomicBoolean shouldCancel = new AtomicBoolean();
...
tasks.parallelStream().allMatch(task->{
task.run();
return !shouldCancel.get();
});
The documentation for the method allMatch specifically says that it "may not evaluate the predicate on all elements if not necessary for determining the result." So if the predicate doesn't match when you want to cancel, then it doesn't need to evaluate any more. Additionally, you can check the return result to see if the loop was cancelled or not.

Blocking calls in Akka Actors

As a newbie, I am trying to understand how actors work. And, from the documentation, I think I understand that actors are objects which gets executed in sync mode and also that actor execution can contain blocking/sync method calls, e.g. db requests
But, what I don't understand is that if you write an actor that has some blocking invocations inside (like a blocking query execution), it will mess up the whole thread pool (in the sense that cpu utilization will go down, etc.), right ? I mean, from my understanding, there is no way for JVM to understand whether it can switch that thread to someone else, if/when the actor makes a blocking call.
So, given the nature of concurrency, shouldn't it be obvious that Actors should not be doing any blocking calls, ever?
If that is the case, what is the recommended way of doing a non-blocking/async call, let's say a web service call that fetches something and sends a message to another actor when that request is completed? Should we simply use something like within the actor:
future map { response => x ! response.body }
Is this the proper way of handling this?
Would appreciate it if you can clarify this for me.
It really depends on the use-case. If the queries do not need to be serialized, then you can execute the query in a future and send the results back to the sender as follows:
import scala.concurrent.{ future, blocking}
import akka.pattern.pipe
val resFut = future {
blocking {
executeQuery()
}
}
resFut pipeTo sender
You could also create a dedicated dispatcher exclusively for the DB calls and use a router for actor creation. This way you can also easily limit the number of concurrent DB requests.
Really great intro "The Neophyte's Guide to Scala Part 14: The Actor Approach to Concurrency" http://danielwestheide.com/blog/2013/02/27/the-neophytes-guide-to-scala-part-14-the-actor-approach-to-concurrency.html.
Actor receives message, wraps blocking code to future, in it's Future.onSuccess method - sends out results using other async messages. But beware that sender variable could change, so close it (make a local reference in the future object).
p.s.: The Neophyte's Guide to Scala - really great book.
Updated: (added sample code)
We have worker and manager. Manager sets work to be done, worker reports "got it" and starts long process ( sleep 1000 ). Meanwhile system pings manager with messages "alive" and manager pings worker with them. When work done - worker notifies manager on it.
NB: execution of sleep 1000 done in imported "default/global" thread pool executor - you can get thread starvation.
NB: val commander = sender is needed to "close" a reference to original sender, cause when onSuccess will be executed - current sender within actor could be already set to some other 'sender' ...
Log:
01:35:12:632 Humming ...
01:35:12:633 manager: flush sent
01:35:12:633 worker: got command
01:35:12:633 manager alive
01:35:12:633 manager alive
01:35:12:633 manager alive
01:35:12:660 worker: started
01:35:12:662 worker: alive
01:35:12:662 manager: resource allocated
01:35:12:662 worker: alive
01:35:12:662 worker: alive
01:35:13:661 worker: done
01:35:13:663 manager: work is done
01:35:17:633 Shutdown!
Code:
import akka.actor.{Props, ActorSystem, ActorRef, Actor}
import com.typesafe.config.ConfigFactory
import java.text.SimpleDateFormat
import java.util.Date
import scala.concurrent._
import ExecutionContext.Implicits.global
object Sample {
private val fmt = new SimpleDateFormat("HH:mm:ss:SSS")
def printWithTime(msg: String) = {
println(fmt.format(new Date()) + " " + msg)
}
class WorkerActor extends Actor {
protected def receive = {
case "now" =>
val commander = sender
printWithTime("worker: got command")
future {
printWithTime("worker: started")
Thread.sleep(1000)
printWithTime("worker: done")
}(ExecutionContext.Implicits.global) onSuccess {
// here commander = original sender who requested the start of the future
case _ => commander ! "done"
}
commander ! "working"
case "alive?" =>
printWithTime("worker: alive")
}
}
class ManagerActor(worker: ActorRef) extends Actor {
protected def receive = {
case "do" =>
worker ! "now"
printWithTime("manager: flush sent")
case "working" =>
printWithTime("manager: resource allocated")
case "done" =>
printWithTime("manager: work is done")
case "alive?" =>
printWithTime("manager alive")
worker ! "alive?"
}
}
def main(args: Array[String]) {
val config = ConfigFactory.parseString("" +
"akka.loglevel=DEBUG\n" +
"akka.debug.lifecycle=on\n" +
"akka.debug.receive=on\n" +
"akka.debug.event-stream=on\n" +
"akka.debug.unhandled=on\n" +
""
)
val system = ActorSystem("mine", config)
val actor1 = system.actorOf(Props[WorkerActor], "worker")
val actor2 = system.actorOf(Props(new ManagerActor(actor1)), "manager")
actor2 ! "do"
actor2 ! "alive?"
actor2 ! "alive?"
actor2 ! "alive?"
printWithTime("Humming ...")
Thread.sleep(5000)
printWithTime("Shutdown!")
system.shutdown()
}
}
You are right to be thinking about the Thread Pool if you are considering doing blocking calls in Akka. The more blocking you do, the larger the Thread Pool you will need. A completely Non-Blocking system only really needs a pool of threads equal to the number of CPU cores of your machine. The reference configuration uses a pool of 3 times the number of CPU cores on the machine to allow for some blocking:
# The core pool size factor is used to determine thread pool core size
# using the following formula: ceil(available processors * factor).
# Resulting size is then bounded by the core-pool-size-min and
# core-pool-size-max values.
core-pool-size-factor = 3.0
source
But you might want to increase akka.default-dispatcher.fork-join-executor.core-pool-size-factor to a higher number if you do more blocking, or make a dispatcher other than the default specifically for blocking calls with a higher fork-join-executor.core-pool-size-factor
WRT what is the best way to do blocking calls in Akka. I would recommend scaling out by making multiple instances of the actors that do blocking calls and putting a router infront of them to make them look like a single actor to the rest of your application.

Kill Spring scheduled thread

Is there anyway to timeout a scheduled task (kill thread) in Spring if the task takes to long or even hangs because of remote resource unavailability
In my case, tasks can take too long or even hang because they're based on HtmlUnitDriver (Selenium) sequence of steps, but from time to time it hangs and I would like to be able to set a time limit for the thread to execute. Something like 1 minute at most.
I setup a fixed rate execution of 5 minutes with an initial delay of 1 minute.
Thanks in advance
I did the same some time ago following this example: example
The basic idea is to put your code in a class implementing Callable or Runnable, then create a FutureTask wherever you are going to invoque your thread with the Callable or Runnable class as parameter. Define an executor , submit your futureTask to the executor, and now you are able to execute the thread for x time inside a try catch block, if your thread ends with an timeoutException you will know that it took too long.
Here is my code:
CallableServiceExecutor callableServiceExecutor = new CallableServiceExecutor();
FutureTask<> task = new FutureTask<>(callableServiceExecutor);
ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(task);
Boolean exito = true;
try {
result = task.get(getTimeoutValidacion() , TimeUnit.SECONDS);
} catch (InterruptedException e) {
exito = false;
} catch (ExecutionException e) {
exito = false;
} catch (TimeoutException e) {
exito = false;
}
task.cancel(true);
executor.shutdown();
See: How to timeout a thread
The short answer is that there is not easy or reliable way to kill a thread due to the limitations of Java's thread implementation. The ExecutorService#shutdown() is sort of a hack and heavy. Its best to deal with this in the task itself e.g. like at the network request level if your making a REST request to timeout on the socket.
Or better if you do some sort of message passing ala Actor model (see Akka) you can send a message from "supervisor" for the Actor to die. Also avoiding blocking by using something like Netty will help.

Resources