Zeppelin Interpreter throws null pointer exception - apache-spark

I am running few notebooks in EMR zeppelin.
EMR version 5.29.0
Spark 2.4.4 and Zeppelin 0.8.2
At times there won't be sufficient memory to run a notebook since multiple notebooks are running parallel. Hence, it will throw an error like below.
INFO [2021-03-30 06:14:51,383] ({pool-2-thread-99} Paragraph.java[jobRun]:381) - Run paragraph [paragraph_id: 20210325-112126_1343087125, interpreter: pyspark, note_id: 2G34X6W7Z, user: admin]
ERROR [2021-03-30 06:14:51,385] ({pool-2-thread-99} Job.java[run]:190) - Job failed
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:141)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:169)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:136)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:312)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:408)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:62)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:133)
... 13 more
ERROR [2021-03-30 06:14:51,385] ({pool-2-thread-99} NotebookServer.java[afterStatusChange]:2308) - Error
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:141)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:169)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:136)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:312)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:408)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:62)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:133)
... 13 more
WARN [2021-03-30 06:14:51,386] ({pool-2-thread-99} NotebookServer.java[afterStatusChange]:2316) - Job 20210325-112126_1343087125 is finished, status: ERROR, exception: java.lang.RuntimeException: java.lang.NullPointerException, result: %text java.lang.NullPointerException
at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:62)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:133)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:169)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:136)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:312)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:408)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:315)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
INFO [2021-03-30 06:14:51,401] ({pool-2-thread-99} VFSNotebookRepo.java[save]:196) - Saving note:2G34X6W7Z
INFO [2021-03-30 06:14:51,775] ({pool-2-thread-99} SchedulerFactory.java[jobFinished]:120) - Job 20210325-112126_1343087125 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-iwb::2G34X6W7Z-shared_session
Once this issue occurs, even if the memory is available at a later stage the notebook would not run until the zeppelin service is restarted. We have tried to run the notebook when the entire yarn resource was free, but the notebook throws the same error. The issue persists until the interpreter/zeppelin is restarted. We cannot run the same notebook until then.

Related

Apache Spark Cluster\Windows: Getting Connection refused: no further information from worker node

I need help please getting spark working in windows cluster of 3 nodes. I am able to download and configure and run the master node and worker nodes. the worker nodes are registered successfully with the master. Im able to see both worker nodes in the Master UI. When I try to submit a job using:
spark-submit --master spark://IP:7077 hello_world.py
Spark continuously try to start multiple executors but the all failed with code exit 1 and it doesnt stop until I kill it. when I check the log in the UI for each worker Im seeing the following error:
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:424)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:413)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:444)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:442)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more
**Caused by: java.io.IOException: Failed to connect to <Master DNS>/<Master IP>:56785
at **org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: <Master DNS>/<Master IP>:56785
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Im using Spark: spark-3.3.1-bin-hadoop3
Please help.
Thanks
The application to run

Getting "org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Interpreter process is not running null"

Hi I am on Docker on mac[K8 enabled] and trying to deploy Zeppelin on K8 by following https://zeppelin.apache.org/docs/0.9.0-SNAPSHOT/quickstart/kubernetes.html.
After deploying the zeppelin server on K8, I am trying to run the Spark example but getting following exception:
org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Interpreter process is not running
null
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:134)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:281)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:412)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:72)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:180)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Interpreter process is not running
null
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:163)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:131)
... 13 more
Spark interpreter status is green
Interpreter is binded
What should I check to know why it is failing?

Not able to use Cassandra Interpreter in Apache Zeppelin

On executing cassandra code in Zeppelin notebook I get the following error on first run-
java.io.IOException: Invalid argument
at java.io.WinNTFileSystem.canonicalize0(Native Method)
at java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:428)
at java.io.File.getCanonicalPath(File.java:618)
at org.fusesource.scalate.util.ClassPathBuilder$$anonfun$getClassPathFrom$3.apply(ClassPathBuilder.scala:147)
at org.fusesource.scalate.util.ClassPathBuilder$$anonfun$getClassPathFrom$3.apply(ClassPathBuilder.scala:142)
at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)
at org.fusesource.scalate.util.ClassPathBuilder$.getClassPathFrom(ClassPathBuilder.scala:142)
at org.fusesource.scalate.util.ClassPathBuilder.addPathFrom(ClassPathBuilder.scala:68)
at org.fusesource.scalate.util.ClassPathBuilder.addPathFromContextClassLoader(ClassPathBuilder.scala:73)
at org.fusesource.scalate.support.ScalaCompiler.generateSettings(ScalaCompiler.scala:121)
at org.fusesource.scalate.support.ScalaCompiler.<init>(ScalaCompiler.scala:59)
at org.fusesource.scalate.support.ScalaCompiler$.create(ScalaCompiler.scala:42)
at org.fusesource.scalate.TemplateEngine.createCompiler(TemplateEngine.scala:231)
at org.fusesource.scalate.TemplateEngine.compiler$lzycompute(TemplateEngine.scala:221)
at org.fusesource.scalate.TemplateEngine.compiler(TemplateEngine.scala:221)
at org.fusesource.scalate.TemplateEngine.compileAndLoad(TemplateEngine.scala:757)
at org.fusesource.scalate.TemplateEngine.compileAndLoadEntry(TemplateEngine.scala:699)
at org.fusesource.scalate.TemplateEngine.liftedTree1$1(TemplateEngine.scala:419)
at org.fusesource.scalate.TemplateEngine.load(TemplateEngine.scala:413)
at org.fusesource.scalate.TemplateEngine.load(TemplateEngine.scala:471)
at org.fusesource.scalate.TemplateEngine.layout(TemplateEngine.scala:573)
at org.apache.zeppelin.cassandra.DisplaySystem$NoResultDisplay$.<init>(DisplaySystem.scala:369)
at org.apache.zeppelin.cassandra.DisplaySystem$NoResultDisplay$.<clinit>(DisplaySystem.scala)
at org.apache.zeppelin.cassandra.EnhancedSession.<init>(EnhancedSession.scala:40)
at org.apache.zeppelin.cassandra.InterpreterLogic.<init>(InterpreterLogic.scala:98)
at org.apache.zeppelin.cassandra.CassandraInterpreter.open(CassandraInterpreter.java:231)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
And on the second run I get-
java.lang.NoClassDefFoundError: Could not initialize class org.apache.zeppelin.cassandra.DisplaySystem$NoResultDisplay$
at org.apache.zeppelin.cassandra.EnhancedSession.<init>(EnhancedSession.scala:40)
at org.apache.zeppelin.cassandra.InterpreterLogic.<init>(InterpreterLogic.scala:98)
at org.apache.zeppelin.cassandra.CassandraInterpreter.open(CassandraInterpreter.java:231)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616)
at org.apache.zeppelin.scheduler.Job.run(Job.java:188)
at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Apache Zeppelin version-0.8.2
Could it be because of JAR file versions inside \interpreter\cassandra ?
Appreciate any help.
i am also facing similar issue and found a detailed analysis on why it is occuring . please follow the link to know root cause of the issue
And It was reported already at issues.apache.org click here for report

Getting java.lang.NoSuchMethodError while using acrDeploy method in jenkins with Azure Container Service Plugin

java.lang.NoSuchMethodError: com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.getClient()Lio/kubernetes/client/ApiClient;
at com.microsoft.jenkins.acs.commands.KubernetesDeploymentCommandBase$KubernetesDeployWorker.getMasterHost(KubernetesDeploymentCommandBase.java:222)
at com.microsoft.jenkins.acs.commands.KubernetesDeploymentCommandBase$KubernetesDeployWorker.call(KubernetesDeploymentCommandBase.java:193)
at com.microsoft.jenkins.acs.commands.KubernetesDeploymentCommandBase$KubernetesDeployWorker.call(KubernetesDeploymentCommandBase.java:132)
at hudson.FilePath.act(FilePath.java:1162)
at com.microsoft.jenkins.acs.commands.KubernetesDeploymentCommandBase.doExecute(KubernetesDeploymentCommandBase.java:78)
at com.microsoft.jenkins.acs.commands.AKSDeploymentCommand.execute(AKSDeploymentCommand.java:32)
at com.microsoft.jenkins.acs.commands.AKSDeploymentCommand.execute(AKSDeploymentCommand.java:20)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.acs.ACSDeploymentBuilder.perform(ACSDeploymentBuilder.java:56)
at com.microsoft.jenkins.acs.ACSDeploymentContext$ExecutionImpl.run(ACSDeploymentContext.java:141)
at com.microsoft.jenkins.acs.ACSDeploymentContext$ExecutionImpl.run(ACSDeploymentContext.java:125)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
Page generated: 23-Jan-2020 16:28:06 UTCREST APIJenkins ver. 2.204.1

Not able to reconnect to outlook MailBox using spring integration and imap idle

I am using spring integration and i am getting below exception when the server keeps IDLE for sometime. I am not able to reproduce the below exception.
Caused by: javax.mail.FolderClosedException: * BYE JavaMail Exception: java.net.SocketException: Connection timed out
at com.sun.mail.imap.IMAPFolder.handleIdle(IMAPFolder.java:3199)
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:3043)
at com.sun.mail.imap.IMAPFolder.idle(IMAPFolder.java:2995)
at org.springframework.integration.mail.ImapMailReceiver.waitForNewMessages(ImapMailReceiver.java:175)
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:220)
... 10 common frames omitted
The below exeption is thrown from java mail 1.5
Please help me in reproducing this issue and fix for this issue. Thanks
The server will time out the connection if it's idle for too long. You need to poke the server every 20 - 30 minutes to keep the connection alive, e.g., using Folder.getMessageCount. (Of course, the connection can still break for all sorts of other reasons, so you have to be prepared to handle that in any case.)
It's just a warning log; the adapter will reconnect after this exception.
If you want to suppress the log message, reduce the log level for that category to ERROR.
EDIT
It will reconnect 10 seconds after the failure, unless you have changed reconnectDelay ...
#Test
public void testIdleReconnects() throws Exception {
ImapMailReceiver receiver = spy(new ImapMailReceiver("imap:foo"));
receiver.setBeanFactory(mock(BeanFactory.class));
receiver.afterPropertiesSet();
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver);
Folder folder = mock(Folder.class);
willThrow(new FolderClosedException(folder, "test")).given(receiver).waitForNewMessages();
ThreadPoolTaskScheduler taskScheduler = new ThreadPoolTaskScheduler();
taskScheduler.initialize();
adapter.setTaskScheduler(taskScheduler);
adapter.afterPropertiesSet();
final CountDownLatch latch = new CountDownLatch(3); << exit test after 3
adapter.setApplicationEventPublisher(e -> {
latch.countDown();
});
adapter.start();
assertTrue(latch.await(60, TimeUnit.SECONDS));
}
If you are not seeing logs similar to these, you should edit your question with complete configuration and logs - you should also try setting the javamail property mail.debug to true to see the activity.
This is what the above test produces for me...
2017-02-26 12:59:43,977 [ThreadPoolTaskScheduler-1] DEBUG: ImapIdleChannelAdapter - waiting for mail
2017-02-26 12:59:43,982 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - error occurred in idle task
javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-02-26 12:59:43,984 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - Failed to execute IDLE task. Will attempt to resubmit in 10000 milliseconds.
java.lang.IllegalStateException: Failure in 'idle' task. Will resubmit.
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:296)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
... 10 more
2017-02-26 12:59:53,989 [ThreadPoolTaskScheduler-1] DEBUG: ImapIdleChannelAdapter - waiting for mail
2017-02-26 12:59:53,990 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - error occurred in idle task
javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-02-26 12:59:53,991 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - Failed to execute IDLE task. Will attempt to resubmit in 10000 milliseconds.
java.lang.IllegalStateException: Failure in 'idle' task. Will resubmit.
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:296)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
... 10 more
2017-02-26 13:00:03,997 [ThreadPoolTaskScheduler-1] DEBUG: ImapIdleChannelAdapter - waiting for mail
2017-02-26 13:00:03,997 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - error occurred in idle task
javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-02-26 13:00:03,998 [ThreadPoolTaskScheduler-1] WARN : ImapIdleChannelAdapter - Failed to execute IDLE task. Will attempt to resubmit in 10000 milliseconds.
java.lang.IllegalStateException: Failure in 'idle' task. Will resubmit.
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:296)
at org.springframework.integration.mail.ImapIdleChannelAdapter$ReceivingTask.run(ImapIdleChannelAdapter.java:243)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.FolderClosedException: test
at org.springframework.integration.mail.ImapIdleChannelAdapter$IdleTask.run(ImapIdleChannelAdapter.java:277)
... 10 more
As you can see, we attempt to reconnect every 10 seconds.

Resources