I use multiple threads to query data, these threads common to the same Session, will occur as follows
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:47)
at com.tss.storage.LuceneTask.run(LuceneTask.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:211)
at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:277)
at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:91)
at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:132)
... 8 more
Can anyone help me?
Thanks!
ExecutorService run this:
RangeCondition range = range("time").lower(start).upper(end);
String param = search().filter(range).build();
Stopwatch start = Stopwatch.createStarted();
ResultSet result = null;
try {
Session session = storageClient.getSession();
result = session.execute("SELECT * FROM demo.tweets WHERE expr(tweets_idx, ?)", param);
} catch (Exception e) {
e.printStackTrace();
}
Related
Transformer throws exception for returning null. I'm getting the message payload and doing my business logic in transformer. Then, sending response to fileoutput channel. I've tried using .handle method too instead of transformer, but getting one way message exception.
EDIT
#Bean
IntegrationFlow integrationFlow() {
return IntegrationFlows.from(this.sftpMessageSource()).channel(fileInputChannel()).
handle(service, "callMethod").channel(fileOutputChannel()).
handle(orderOutMessageHandler()).get();
}
EDIT 2
[ERROR] 2020-06-14 14:49:48.053 [task-scheduler-9] LoggingHandler - java.lang.AbstractMethodError: Method org/springframework/integration/sftp/session/SftpSession.getHostPort()Ljava/lang/String; is abstract
at org.springframework.integration.sftp.session.SftpSession.getHostPort(SftpSession.java)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.getHostPort(CachingSessionFactory.java:295)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.copyFileToLocalDirectory(AbstractInboundFileSynchronizer.java:496)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.copyIfNotNull(AbstractInboundFileSynchronizer.java:400)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.transferFilesFromRemoteToLocal(AbstractInboundFileSynchronizer.java:386)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.lambda$synchronizeToLocalDirectory$0(AbstractInboundFileSynchronizer.java:349)
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:437)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:348)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:265)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:66)
at org.springframework.integration.endpoint.AbstractFetchLimitingMessageSource.doReceive(AbstractFetchLimitingMessageSource.java:45)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:167)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:250)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:359)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:328)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$1(AbstractPollingEndpoint.java:275)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$2(AbstractPollingEndpoint.java:272)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The transformer is designed to always return a reply because it is a transformation operation. Therefore you can’t return null from your method . You get one-way error probably because your handle method is void.
I'm using Kafka and Spark to stream my data updates into my Hbase tables.
But I keep getting the OffsetOutOfRangeException, here's my code:
new KafkaStreamBuilder()
.setStreamingContext(streamingContext)
.setTopics(topics)
.setDataSourceId(dataSourceId)
.setOffsetManager(offsetManager)
.setConsumerParameters(
ImmutableMap
.<String, String>builder()
.putAll(kafkaConsumerParams)
.put("group.id", groupId)
.put("metadata.broker.list", kafkaBroker))
.build()
)
.build()
.foreachRDD(
rdd -> {
rdd.foreachPartition(
iter -> {
final Table hTable = createHbaseTable(settings);
try {
while (iter.hasNext()) {
String json = new String(iter.next());
try {
putRow(
hTable,
json,
settings,
barrier);
} catch (Exception e) {
throw new RuntimeException("hbase write failure", e);
}
}
} catch (OffsetOutOfRangeException e) {throw new RuntimeException(
"encountered OffsetOutOfRangeException: ", e);
}
});
});
I set my streaming job to run every 5 mins, and every time, after my consumers finish one batch of streaming, it'll write the latest markers and checkpoints to S3. And when next time, before the streaming job runs, it'll read the previous checkpoints and markers from S3, and start from there.
Here's the exception stacktrace:
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreachPartition$1.apply(JavaRDDLike.scala:219)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:925)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: kafka.common.OffsetOutOfRangeException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:86)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.handleFetchErr(KafkaRDD.scala:188)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:197)
at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:212)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:30)
What I had done:
I've checked and markers and checkpoints are both working as expected.
So, I'm a bit lost here, how could this exception happen and what the possible/reasonable fix could it be?
Thanks!
I want to get an appropriate exception message from Spark JDBC driver.
My test case:
Using Spark ver2.0.2 to access DB through JDBC Driver
At first locked DB, then try write to DB (write mode are append and overwrite)
Get exception message
Exception message I got:
case1 append: Cause already initialized <-Not good
case2 overwrite: DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033, SQLERRMC=00C9008E;00000210;DSN00009.DRITEMRI.00000001, DRIVER=4.19.56 <-Good
My question:
How can I get an exception message like DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033~ in 'case1 append'?
I guss a reason is function savePartitions (this function is called when Spark execute saveTable) doesn't show good exception. But I don't know how to fix it.
Here is detail exception message of Spark shell
case1 append:
scala> prodtbl.write.mode("Append").jdbc(url3,"DB2.D_ITEM_INFO",prop1)
17/04/03 17:50:26 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.IllegalStateException: Cause already initialized
at java.lang.Throwable.setCause(Throwable.java:365)
at java.lang.Throwable.initCause(Throwable.java:341)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:241)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:300)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:299)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
case2 overwrite:
scala> prodtbl.write.mode("Overwrite").jdbc(url3,"DB2.D_ITEM_INFO",prop1)
com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-913, SQLSTATE=57033, SQLERRMC=00C9008E;00000210;DSN00009.DRITEMRI.00000001, DRIVER=4.19.56
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.kd.a(Unknown Source)
at com.ibm.db2.jcc.am.fp.c(Unknown Source)
at com.ibm.db2.jcc.am.fp.d(Unknown Source)
at com.ibm.db2.jcc.am.fp.b(Unknown Source)
at com.ibm.db2.jcc.t4.bb.i(Unknown Source)
at com.ibm.db2.jcc.t4.bb.c(Unknown Source)
at com.ibm.db2.jcc.t4.p.b(Unknown Source)
at com.ibm.db2.jcc.t4.vb.h(Unknown Source)
at com.ibm.db2.jcc.am.fp.jb(Unknown Source)
at com.ibm.db2.jcc.am.fp.a(Unknown Source)
at com.ibm.db2.jcc.am.fp.c(Unknown Source)
at com.ibm.db2.jcc.am.fp.executeUpdate(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.dropTable(JdbcUtils.scala:94)
at org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:422)
... 48 elided
So initCause described as below:
public Throwable initCause(Throwable cause)
Initializes the cause of this throwable to the specified value. (The cause is the throwable that caused this throwable to get thrown.)
This method can be called at most once. It is generally called from within the constructor, or immediately after creating the throwable. If this throwable was created with Throwable(Throwable) or Throwable(String,Throwable), this method cannot be called even once.
An example of using this method on a legacy throwable type without other support for setting the cause is:
try {
lowLevelOp();
} catch (LowLevelException le) {
throw (HighLevelException)
new HighLevelException().initCause(le); // Legacy constructor
}
Parameters:
cause - the cause (which is saved for later retrieval by the getCause() method). (A null value is permitted, and indicates that the cause is nonexistent or unknown.)
Returns:
a reference to this Throwable instance.
Throws:
IllegalArgumentException - if cause is this throwable. (A throwable cannot be its own cause.)
IllegalStateException - if this throwable was created with Throwable(Throwable) or Throwable(String,Throwable), or this method has already been called on this throwable.
And it means the cause was something not originated to initialize with initCase method and you haven't received what you waited for.
I tried looking here and here and here.
I am using the dsl flow. And I would like to be able to see the console log printed out of the job i'm building within the job I'm running. I tried looking around for examples and I couldn't seem to find what I was looking for. I apologize if this question is not using the correct terminology or that it's been asked in different ways. I just want to find the answer of how to do this.
A = build("Main Suites", SUITE: "qa_smoketests", OS: "mac")
below I tried to do this but it didn't seem to work
OUTPUT = A.build.doConsoleText()
out.println(OUTPUT)
UPDATE:
I tried the 2nd method to update it on the fly and I keep getting an issue with the url? strangely if I click the url link in the error it takes me to the appropriate page. I did have to modify things b/c the Job name has spaces in it. I also edited my error log for this post to say MYHOSTURL instead of my actual Jenkins host url.
ERROR: Failed to run DSL Script
java.util.concurrent.ExecutionException: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://MYHOSTURL/job/Main%20Suites/159/consoleText
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at java_util_concurrent_Future$get$7.call(Unknown Source)
at com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure6.doCall(FlowDSL.groovy:440)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at groovy.lang.Closure.call(Closure.java:415)
at groovy.lang.Closure.call(Closure.java:428)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1379)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1351)
at org.codehaus.groovy.runtime.dgm$170.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.cloudbees.plugins.flow.FlowDelegate.parallel(FlowDSL.groovy:438)
at sun.reflect.GeneratedMethodAccessor10240.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:723)
at com.cloudbees.plugins.flow.FlowDelegate.invokeMethod(FlowDSL.groovy)
at hudson.util.spring.ClosureScript.invokeMethod(ClosureScript.java:83)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:72)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
at Script1.run(Script1.groovy:12)
at Script1$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at Script1$run.call(Unknown Source)
at com.cloudbees.plugins.flow.FlowDSL.executeFlowScript(FlowDSL.groovy:84)
at com.cloudbees.plugins.flow.FlowRun$FlyweightTaskRunnerImpl.run(FlowRun.java:219)
at hudson.model.Run.execute(Run.java:1759)
at com.cloudbees.plugins.flow.FlowRun.run(FlowRun.java:155)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
at hudson.model.OneOffExecutor.run(OneOffExecutor.java:43)
Caused by: org.codehaus.groovy.runtime.InvokerInvocationException: java.io.IOException: Server returned HTTP response code: 401 for URL: http://MYHOSTURL/job/Main%20Suites/159/consoleText
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:97)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at groovy.lang.Closure.call(Closure.java:415)
at groovy.lang.Closure.call(Closure.java:409)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: http://MYHOSTURL/job/Main%20Suites/159/consoleText
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)
at sun.net.www.protocol.http.HttpURLConnection$getInputStream.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at Script1.responseFrom(Script1.groovy:82)
at Script1$responseFrom.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at Script1.printConsoleOutputTextChunksFrom(Script1.groovy:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at Script1$_run_closure2.doCall(Script1.groovy:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at Script1$_run_closure2.doCall(Script1.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:39)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy:427)
at sun.reflect.GeneratedMethodAccessor8673.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:903)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at com.cloudbees.plugins.flow.FlowDelegate$_parallel_closure5_closure7.doCall(FlowDSL.groovy)
at sun.reflect.GeneratedMethodAccessor8672.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
... 9 more
}
Finished: FAILURE
UPDATE 2:
So I was able to get the first method working by modifying the responseFrom function to handle a page that needs authentication.
InputStream responseFrom(String url)
{
// See Using java.net.URLConnection to fire and handle HTTP requests
// https://stackoverflow.com/a/2793153/1744774
URL myURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection)myURL.openConnection();
String userCredentials = "username:password";
String encoding = new sun.misc.BASE64Encoder().encode(userCredentials.getBytes());
String basicAuth = "Basic " + encoding;
connection.setRequestProperty("Authorization", basicAuth);
connection.setRequestProperty("Accept-Charset", StandardCharsets.UTF_8.name());
return connection.getInputStream();
}
However I'm still having issues getting the 2nd method to work. I guess it's with the function that is being used to get the build number from the downstream job.
java.util.concurrent.ExecutionException:
org.codehaus.groovy.runtime.InvokerInvocationException:
groovy.lang.MissingMethodException: No signature of method: static
java.lang.Integer.parseUnsignedInt() is applicable for argument types:
(java.lang.String) values: [164]
Interesting question. I can't comprehend the downvote.
The first of your links describes how you should be able to do it: via "repeated GET requests".
The second is about setting the result of an upstream job according to the content of a downstream job's console output. This can be achieved easier by something else but parsing the console output.
The third is about the opposite direction: passing output from an upstream job to a downstream job.
I'm going to try the first and will update this answer accordingly then.
UPDATE 1
The following updates the upstream's log once the downstream build has finished.
// From: Output the console text of a Jenkins job that my Job is running
// http://stackoverflow.com/a/31573477/1744774
final String JENKINS_HOST = 'http://<your Jenkins host>'
final String DOWNSTREAM_NAME = '<your downstream job's name>'
final String SEPARATOR = new String(new char[8]).replace('\0', '-')
final int DOWNSTREAM_BUILD_NO = build(DOWNSTREAM_NAME).getBuild().getNumber()
println("${SEPARATOR} Begin console HTML output of ${DOWNSTREAM_NAME} #${DOWNSTREAM_BUILD_NO} ${SEPARATOR}")
printConsoleOutputHtmlFrom(
"${JENKINS_HOST}/job/${DOWNSTREAM_NAME}/${DOWNSTREAM_BUILD_NO}/console")
println("${SEPARATOR} Begin console text output of ${DOWNSTREAM_NAME} #${DOWNSTREAM_BUILD_NO} ${SEPARATOR}")
printConsoleOutputTextFrom(
"${JENKINS_HOST}/job/${DOWNSTREAM_NAME}/${DOWNSTREAM_BUILD_NO}/consoleText")
println("${SEPARATOR} End console output of ${DOWNSTREAM_NAME} ${SEPARATOR}")
import org.xml.sax.InputSource
void printConsoleOutputHtmlFrom(String consoleUrl)
{
println(xPathExpression('//pre[contains(#class, "console-output")]')
.evaluate(new InputSource(responseFrom(consoleUrl))))
}
import java.nio.charset.StandardCharsets
void printConsoleOutputTextFrom(String consoleTextUrl)
{
BufferedReader log = new BufferedReader(
new InputStreamReader(responseFrom(consoleTextUrl), StandardCharsets.UTF_8.name()));
while ((line = log.readLine()) != null)
{
println(line)
}
}
InputStream responseFrom(String url)
{
// See Using java.net.URLConnection to fire and handle HTTP requests
// http://stackoverflow.com/a/2793153/1744774
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Accept-Charset", StandardCharsets.UTF_8.name());
return connection.getInputStream();
}
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;
XPathExpression xPathExpression(String expression)
{
// See How to read XML using XPath in Java
// http://stackoverflow.com/a/2811101/1744774
XPathFactory xPathFactory = XPathFactory.newInstance();
XPath xPath = xPathFactory.newXPath();
return xPath.compile(expression);
}
References:
Build Flow Plugin
com.cloudbees.plugins.flow.JobInvocation (returned by build())
hudson.model.Run (returned by JobInvocation.getBuild())
UPDATE 2
The following updates the upstream's log on the fly.
// From: Output the console text of a Jenkins job that my Job is running
// http://stackoverflow.com/a/31573477/1744774
final String JENKINS_HOST = 'http://<your Jenkins host>'
final String DOWNSTREAM_NAME = '<your downstream job's name>'
final String SEPARATOR = new String(new char[8]).replace('\0', '-')
import java.util.concurrent.TimeUnit
parallel(
{ build(DOWNSTREAM_NAME) },
{
final String THIS_BUILD_NAME = build.getProject().getName()
final int THIS_BUILD_NO = build.getNumber()
final int DOWNSTREAM_BUILD_NO = downstreamBuildNoFrom(
"${JENKINS_HOST}/job/${THIS_BUILD_NAME}/${THIS_BUILD_NO}/consoleText")
println("${SEPARATOR} Begin console output of ${DOWNSTREAM_NAME} #${DOWNSTREAM_BUILD_NO} ${SEPARATOR}")
printConsoleOutputTextChunksFrom(
"${JENKINS_HOST}/job/${DOWNSTREAM_NAME}/${DOWNSTREAM_BUILD_NO}/consoleText", 3L, TimeUnit.SECONDS)
println("${SEPARATOR} End console output of ${DOWNSTREAM_NAME} #${DOWNSTREAM_BUILD_NO} ${SEPARATOR}")
})
int downstreamBuildNoFrom(String thisBuildConsoleTextUrl)
{
int downstreamBuildNo = -1
while (downstreamBuildNo <= 0)
{
TimeUnit.SECONDS.sleep(1L)
BufferedReader log = new BufferedReader(
new InputStreamReader(responseFrom(thisBuildConsoleTextUrl), StandardCharsets.UTF_8.name()));
while ((line = log.readLine()) != null)
{
if (line.matches('.* Build .* started'))
{
final int buildNoIdx = line.indexOf('#') + 1
downstreamBuildNo = Integer.parseUnsignedInt(
line.substring(buildNoIdx, line.indexOf(' ', buildNoIdx)))
break
}
/*if (log.lines().anyMatch(t -> t.matches(".* Build .* started")))
{
// in case of Java >=8
}*/
}
}
return downstreamBuildNo
} // downstreamBuildNoFrom()
import java.nio.charset.StandardCharsets
void printConsoleOutputTextChunksFrom(String consoleTextUrl, long updateInterval, TimeUnit updateIntervalUnit)
{
final int LINE_BREAK_LENGTH = System.getProperty("line.separator").length()
int charsPrintedSoFar = 0
while (true) // effectively a do { ... }
{
int charsPrintedThisTime = 0
BufferedReader chunk = new BufferedReader(
new InputStreamReader(responseFrom(consoleTextUrl), StandardCharsets.UTF_8.name()))
chunk.skip(charsPrintedSoFar)
while ((line = chunk.readLine()) != null)
{
println(line)
charsPrintedThisTime = line.length() + LINE_BREAK_LENGTH
charsPrintedSoFar += charsPrintedThisTime
}
updateIntervalUnit.sleep(updateInterval)
if (charsPrintedThisTime == 0)
break
}
} // printConsoleOutputTextChunksFrom()
InputStream responseFrom(String url)
{
// See Using java.net.URLConnection to fire and handle HTTP requests
// http://stackoverflow.com/a/2793153/1744774
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Accept-Charset", StandardCharsets.UTF_8.name());
return connection.getInputStream();
}
This worked for me:
myjob=build job: 'job path', parameters: [
...
], propagate: true, wait: true
echo "${myjob.getResult()}"
println myjob.getRawBuild().getLog()
getResult is just 'SUCCESS/FAILURE' but
getRaWBuild().getLog() has the output of the called job.
We've a Java application which periodically insert rows into the Oracle DB. This is a multi-threaded application. All threads barring one gets stuck periodically. We're thinking of upgrading the Oracle JDBC Driver, but I've a feeling it might show up again. Just wanted to get some info on if its an error with our code or something else. I've both the stacktrace and the parts of code below. We see locked periodically in the thread info. Do give us some info as to what could me wrong.
----Code----
LogEventBatchPreparedStatementUpdater statementUpdater = new LogEventBatchPreparedStatementUpdater(logEvents);
// _jdbcTemplate.batchUpdate(INSERT_SQL, statementUpdater);
Connection connection = null;
PreparedStatement preparedStatement = null;
try
{
connection = _dataSource.getConnection();
connection.setAutoCommit(false);
preparedStatement = connection.prepareStatement(INSERT_SQL);
for (int i = 0; i < statementUpdater.getBatchSize(); i++)
{
statementUpdater.setValues(preparedStatement, i);
preparedStatement.addBatch();
}
preparedStatement.executeBatch();
connection.commit();
}
catch (SQLException e)
{
_Log.error("Error inserting log line batch",e );
}
finally
{
try
{
preparedStatement.close();
connection.close();
}
catch (SQLException e)
{
_Log.error("Error inserting log line batch",e );
}
}
----Stack Trace----
"Thread-258 " daemon prio=6 tid=0x09437400 nid=0x2300 runnable [0x0f55f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:369)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891)
at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:109
3)
- locked <0x1ce417c0> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.ja
va:2047)
at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.jav
a:1940)
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePrepare
dStatement.java:3899)
- locked <0x18930c00> (a oracle.jdbc.driver.OraclePreparedStatement)
- locked <0x1ce3f9f0> (a oracle.jdbc.driver.OracleConnection)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingSt
atement.java:294)
at ************.insertLogEventBatch(JdbcL
ogEventBatchDao.java:61)
at ************.DBLogEventBatchProcessor.processLo
gLineBatch(DBLogEventBatchProcessor.java:30)
at ************.LogLineBatcher.processLogLineBatch
(LogLineBatcher.java:274)
at ************.LogLineBatcher.processBatchBasedOn
Time(LogLineBatcher.java:192)
at ************.LogLineBatcher.manageBatch(LogLine
Batcher.java:178)
at ************.LogLineBatcher.access$000(LogLineB
atcher.java:24)
at ************.LogLineBatcher$1.run(LogLineBatche
r.java:152)
at java.lang.Thread.run(Unknown Source)
The fact that the thread state is RUNNABLE, and that it is trying to read from a socket, imply that it is simply waiting for a response from the database. So the thing to investigate is what the database session is waiting on. If you can identify the session in the V$SESSION view, the EVENT column will indicate what it is waiting on. Seems like there could potentially be a lock wait in the database.
FYI, where the thread dump says "locked", e.g. locked <0x1ce417c0>, that is just telling yu that the thread has acquired a lock; I believe the hex code is the ID of the object on which the lock is held.
Here is some useful information on interpreting thread dumps.