Bamboo Deployment Wait Until No Quartz Jobs With Grails - memory-leaks

Question

I'm using Bamboo to build, test, and deploy commits to a server. However, every once in a while I get a failed deployment on the Tomcat end. Bamboo doesn't know this happened, so I have to run a manual build when the application doesn't start properly.
As far as I can tell, it's due to running Quartz tasks. Is there a way to safely deploy the WAR but wait until all jobs are done? Is better to build the plan to first undeploy before deploying?
I've pulled my Catalina logs to highlight the issue:
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-5] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-6] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-8] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [quartzScheduler_Worker-10] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:14 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/emdb] appears to have started a thread named [Thread-212] but has failed to stop it. This is very likely to create a memory leak.
Sep 12, 2012 12:01:15 PM org.apache.catalina.startup.HostConfig checkResources
INFO: Undeploying context [/emdb]
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb\WEB-INF] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [....webapps\emdb] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:16 PM org.apache.catalina.startup.ExpandWar delete
SEVERE: [....webapps\emdb] could not be completely deleted. The presence of the remaining files may cause problems
Sep 12, 2012 12:01:17 PM org.apache.catalina.startup.HostConfig deployWAR

Answer 1

You are right, Quartz scheduler is not stopping threads created to run jobs (worker thread pool). You need to call QuartzScheduler.shutdown(boolean) method when application shuts down. I'm surprised that Grails Quartz plugin is not doing it for you.
There are other reasons for this to happen. You might have some running jobs and Quartz hangs waiting for them to finish (even when proper shutdown was called). Tomcat is impatient and kills the whole thing in the middle. Check out (uncodumented?) org.quartz.scheduler.interruptJobsOnShutdown and org.quartz.scheduler.interruptJobsOnShutdownWithWait options.
Note that ShutdownHookPlugin won't help you since you are not shutting down the whole JVM, only undeploying one application.
Also the error message lists Thread-212 as one of the causes. This is most likely your custom thread (always name threads!) which also has to be interrupted.
See also
Quartz Scheduler not stopping after shutdown