In Scylladb while using sstable loader to load data, we are facing the following error-
root#scylla-chronicle-s-1:~# sstableloader -d 10.110.68.9 /var/lib/scylla/data/connections/by_date-4ce6c340f5dd11e980df000000000002/snapshots/1658173631835
Using /etc/scylla/scylla.yaml as the config file
===== Using optimized driver!!! =====
WARN 19:51:22,937 Error creating netty channel to /10.110.117.172:9042
com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /10.110.117.172:9042
at com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:218) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
ERROR 19:51:22,942 Unexpected error while executing task
java.lang.NullPointerException: null
at com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:838) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.SessionManager.removePool(SessionManager.java:437) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.SessionManager.onDown(SessionManager.java:525) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.Cluster$Manager.onDown(Cluster.java:2033) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.Cluster$Manager.access$1200(Cluster.java:1393) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.Cluster$Manager$5.runMayThrow(Cluster.java:1988) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) ~[scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_292]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
at com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [scylla-driver-core-3.7.1-scylla-2-shaded.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]
WARN 19:51:22,948 Error creating pool to /10.110.117.172:9042
The error you posted indicates that the machine where you are running sstableloader doesn't have network connectivity to the IP.
You need to make sure you are connecting to an IP that is reachable by clients. If the cluster is running on GKE, there's a good chance that you need to setup host networking so the cluster is accessible from outside GKE.
Usually, the pods are accessible via a Kubernetes service configured on GKE. See Accessing Scylla on a Kubernetes cluster for details. Cheers!
Related
I have setup a Hadoop cluster with 1 name node and 2 data nodes. I've also installed Yarn and Spark on top of that in the name node.
I notice that whenever I try run the example jar here:
spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_*.jar 10
I will always get the no route to host exception:
Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:558)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:277)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:926)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:925)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:925)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:957)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.io.IOException: Failed to connect to lnx-pen205/xx.xx.xx.xx:9222
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:288)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: lnx-pen205/xx.xx.xx.xx:9222
Caused by: java.net.NoRouteToHostException: No route to host
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
I noticed that the port being used will be randomly assigned during the runtime, the example .jar will work if for example I set the spark.driver.port as 9222 then opening said port with the firewall. But then if any other session is started (for example, pyspark shell), it wouldn't start as the port is already in use.
My question is: How do I allow connections to the ports dynamically defined by Spark/Yarn? I read somewhere that I should disable the firewall, but that does not sound like a good idea.. Thanks in advance.
There's spark.driver.port as well as spark.driver.blockManager.port. Both are starting ranges to spark.port.maxRetries (default 16).
So, you'll need to open at least 32 ports for these.
I did some testing with dynamic Spark ports in Mesos + Docker a few years ago - https://stackoverflow.com/a/56486271/2308683
I am running a on-prem k8s cluster and am running juypterhub on it.
I can successfully submit the job to an yarn queue, however the job will fail because users notebook pod IP is not resolvable and therefore it can’t talk back to the spark driver running on said pod and I get an error like:
Caused by: java.io.IOException: Failed to connect to podIP:33630 at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:287)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:230)
at
org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202) at
org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
java.net.UnknownHostException: pod-ip
I believe I’m missing something in my setup that will allow yarn to talk back to the spawned notebook pods on the kubernetes cluster.
Any help or hints are greatly appreciated .
For now I am passing the spark driver the internal Kubernetes Pod IP of juypterhub by setting:
"spark.driver.host" to str(socket.gethostbyname(socket.gethostname())). Could I change this to something else in the notebook I am running? I am not too sure what to change it to.
Thanks!
I'm trying to deploy ignite so that I can use the shared RDD/Dataframe cache for my spark cluster. I've followed the spark install instructions and choose to deploy into my existing yarn cluster running spark. I'm using HDP to deploy spark.
I've already verified that Resource Manager and History server are listening on the ports below and I can telnet to each port. What am I doing wrong? Am I not deploying this the way it is intended?
I'm running:
yarn jar ignite-yarn-2.6.0.jar ./ignite-yarn-2.6.0.jar ../../../cluster.properties
Error below:
18/09/24 22:13:38 INFO client.RMProxy: Connecting to ResourceManager at dev01clus02.dna.local/172.31.31.5:8050
18/09/24 22:13:38 INFO client.AHSProxy: Connecting to Application History server at dev01clus02.dna.local/172.31.31.5:10200
Exception in thread "main" java.lang.RuntimeException: Failed update ignite.
at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:243)
at org.apache.ignite.yarn.IgniteProvider.getIgnite(IgniteProvider.java:93)
at org.apache.ignite.yarn.IgniteYarnClient.getIgnite(IgniteYarnClient.java:194)
at org.apache.ignite.yarn.IgniteYarnClient.main(IgniteYarnClient.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:675)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.ignite.yarn.IgniteProvider.updateIgnite(IgniteProvider.java:220)
... 9 more
It looks like a piece of insfrastructure, provided by GridGain for Apache Ignite project, does not work currently. I'll raise the issue.
In the meantime, you can provide IGNITE_PATH property (in config, system properties or env) pointed to unzipped Apache Ignite 2.6 distribution directory to avoid downloading attempts altogether.
I created a microservice using jhipster (version of generator-jhipster is 5.0.1). It contains the following components:
Microservice1
Jhipster Registry
Keycloak as auth server
If I start the microservice using the generated docker-compose config (app.yml), it starts all the components without any issue.
But if I start the components individually using their respective docker-compose configurations
eg:
docker-compose -f jhipster-registry.yml up -d
docker-compose -f postgres.yml up -d
docker-compose -f keycloak.yml up -d
then I see the following exception in the microservice.
2018-09-14T05:56:33.860533100Z
2018-09-14 05:56:33.859 WARN 1 --- [freshExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: java.net.ConnectException: Connection refused (Connection refused)
2018-09-14 05:56:33.859 ERROR 1 --- [freshExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_MS1/ms1:9d9ed2e17bb03d945b64e18e4d49ba5f - was unable to refresh its cache! status = Cannot execute request on any known server
2018-09-14T05:56:33.860794800Z
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$6.execute(EurekaHttpClientDecorator.java:137)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.getApplications(EurekaHttpClientDecorator.java:134)
at com.netflix.discovery.DiscoveryClient.getAndStoreFullRegistry(DiscoveryClient.java:1051)
at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:965)
at com.netflix.discovery.DiscoveryClient.refreshRegistry(DiscoveryClient.java:1471)
at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1438)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-14T05:56:33.861228500Z
2018-09-14 05:56:34.735 ERROR 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error
2018-09-14T05:56:34.736291200Z
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused (Connection refused)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529)
at com.netflix.discovery.shared.transport.jersey.AbstractJerseyEurekaHttpClient.sendHeartBeat(AbstractJerseyEurekaHttpClient.java:102)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.MetricsCollectingEurekaHttpClient.execute(MetricsCollectingEurekaHttpClient.java:73)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:118)
at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:79)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:846)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1399)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
... 27 common frames omitted
2018-09-14T05:56:34.737447400Z
2018-09-14 05:56:34.738 WARN 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: java.net.ConnectException: Connection refused (Connection refused)
2018-09-14 05:56:34.739 ERROR 1 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient : DiscoveryClient_MS1/ms1:9d9ed2e17bb03d945b64e18e4d49ba5f - was unable to send heartbeat!
2018-09-14T05:56:34.741411800Z
com.netflix.discovery.shared.transport.TransportException: Cannot execute request on any known server
at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:112)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:846)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1399)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-14T05:56:34.742030100Z
The microservice is unable to register with the registry. Can you please let me know what the reason is and also the fix.
Thanks in advance :)
While debugging (using remote debug in eclipse) I found that while using separate docker compose files, service was trying to connect to localhost whereas single docker compose file case service was trying to connect to jhipster-registry.
Reason:
There are some minor differences for jhipster registry configuration in docker compose files.
app.yml uses docker-config folder in central-config folder whereas jhipster-registry uses localhost-config folder in central-config folder.
app.yml
jhipster-registry:
extends:
file: jhipster-registry.yml
service: jhipster-registry
environment:
- SPRING_CLOUD_CONFIG_SERVER_NATIVE_SEARCH_LOCATIONS=file:./central-config/docker-config/
central-server-config/docker-config/application.yml
eureka:
client:
service-url:
defaultZone: http://admin:${jhipster.registry.password}#jhipster-registry:8761/eureka/
====================================
jhipster-registry.yml
jhipster-registry:
image: jhipster/jhipster-registry:v3.3.0
....
environment:
- ...
- SPRING_CLOUD_CONFIG_SERVER_NATIVE_SEARCH_LOCATIONS=file:./central-config/localhost-config/
central-server-config/localhost-config/application.yml
eureka:
client:
service-url:
defaultZone: http://admin:${jhipster.registry.password}#localhost:8761/eureka/
Making jhipster-registry.yml to use docker-config folder value instead of localhost-config folder value helped in solving this issue.
Note:- no idea what role "native search location" property plays here and how and why it impacts the service as this is a jhipster registry config property. I will update when I figure it out.
I am trying to write a HDInsight Spark application which reads streaming data from an Azure EventHub. I am using a Zeppelin notebook with the Livy interpreter.
I need to import the dependency
com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.2
and to do that I add it to the
livy.spark.jars.packages
property of the Livy interpreter. However, this breaks my code. Even without the line
import org.apache.spark.eventhubs._
I still get a failure. (I don't use wildcard imports usually, but this is just a proof of concept application)
The error I am getting is
org.apache.zeppelin.livy.LivyException: Session 8 is finished, appId: application_[NUMBER], log: [ ApplicationMaster RPC port: -1, queue: default, start time: 1533304077387, final status: UNDEFINED, tracking URL: http://[LIVY_SERVER_HOSTNAME]:8088/proxy/application_[NUMBER]/, user: livy, 18/08/03 13:47:57 INFO ShutdownHookManager: Shutdown hook called, 18/08/03 13:47:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-[id],
YARN Diagnostics: , Application killed by user.]
at org.apache.zeppelin.livy.BaseLivyInterpreter.createSession(BaseLivyInterpreter.java:300)
at org.apache.zeppelin.livy.BaseLivyInterpreter.initLivySession(BaseLivyInterpreter.java:184)
at org.apache.zeppelin.livy.LivySharedInterpreter.open(LivySharedInterpreter.java:57)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.livy.BaseLivyInterpreter.getLivySharedInterpreter(BaseLivyInterpreter.java:165)
at org.apache.zeppelin.livy.BaseLivyInterpreter.open(BaseLivyInterpreter.java:139)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:493)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I suspect this is really not a problem with Livy, or Zeppelin, but just some configuration I have set wrongly, or that I need to change from the default settings, possibly to do with downloading the jar.
Any help would be appreciated