I've tried modifying the log4j file in the conf folder, but after I do that I get the error when I run my file
log4j:ERROR Could not instantiate class [org.apache.log4j.CosnsoleAppender].
java.lang.ClassNotFoundException: org.apache.log4j.CosnsoleAppender
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
This happened specifically when I changed the log4j.properties.template file to log4j.properties.
I also did
sc.setLogLevel("ERROR")
but then I still get log output.
INFO SecurityManager: Changing view acls to:
INFO SparkEnv: Registering MapOutputTracker
INFO Executor: Starting executor ID driver on host localhost
You have a typo in org.apache.log4j.CosnsoleAppender, should be ConsoleAppender.
Related
I encounter following error each time I try to run the blank application from Spring Tool Suite Eclipse (with JHipster plugins installed). What am I doing wrong? Running aplication via "yarn start" seems to work without problems. STS Version 3.9.4. JHipster version 5. Also I am getting error that Java is not 1.8 (it's 10) - but I thought this is the correct Java version for release 5.
2018-06-21 15:32:53.124 WARN 10648 --- [ restartedMain] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8761/config/bookstore/dev/master": Connection refused: connect; nested exception is java.net.ConnectException: Connection refused: connect
2018-06-21 15:32:53.132 INFO 10648 --- [ restartedMain] com.bookstore.BookstoreApp : The following profiles are active: dev,swagger
2018-06-21 15:32:54.976 ERROR 10648 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Error processing condition on org.springframework.cloud.netflix.ribbon.eureka.RibbonEurekaAutoConfiguration
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:64)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$TrackedConditionEvaluator.shouldSkip(ConfigurationClassBeanDefinitionReader.java:441)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:128)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:117)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:328)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:233)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:273)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:93)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:694)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
at com.bookstore.BookstoreApp.main(BookstoreApp.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration] from ClassLoader [sun.misc.Launcher$AppClassLoader#764c12b6]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:659)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:556)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:541)
at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:726)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:667)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:635)
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1489)
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1012)
at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanTypeForNonAliasDefinition(BeanTypeRegistry.java:164)
at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.addBeanType(BeanTypeRegistry.java:153)
at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.updateTypesIfNecessary(BeanTypeRegistry.java:203)
at org.springframework.boot.autoconfigure.condition.BeanTypeRegistry.getNamesForType(BeanTypeRegistry.java:115)
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.collectBeanNamesForType(OnBeanCondition.java:265)
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getBeanNamesForType(OnBeanCondition.java:254)
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchingBeans(OnBeanCondition.java:196)
at org.springframework.boot.autoconfigure.condition.OnBeanCondition.getMatchOutcome(OnBeanCondition.java:81)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition$MemberOutcomes.getConditionOutcome(AbstractNestedCondition.java:191)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition$MemberOutcomes.<init>(AbstractNestedCondition.java:184)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition$MemberConditions.lambda$getMatchOutcomes$0(AbstractNestedCondition.java:162)
at java.util.Map.forEach(Unknown Source)
at java.util.Collections$UnmodifiableMap.forEach(Unknown Source)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition$MemberConditions.getMatchOutcomes(AbstractNestedCondition.java:162)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition$MemberMatchOutcomes.<init>(AbstractNestedCondition.java:81)
at org.springframework.boot.autoconfigure.condition.AbstractNestedCondition.getMatchOutcome(AbstractNestedCondition.java:65)
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:47)
... 20 common frames omitted
Caused by: java.lang.NoClassDefFoundError: io/micrometer/core/instrument/binder/hystrix/HystrixMetricsBinder
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethods(Unknown Source)
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:641)
... 45 common frames omitted
Caused by: java.lang.ClassNotFoundException: io.micrometer.core.instrument.binder.hystrix.HystrixMetricsBinder
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 49 common frames omitted
2018-06-21 15:32:54.985 WARN 10648 --- [ restartedMain] o.s.boot.SpringApplication : Unable to close ApplicationContext
java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration] from ClassLoader [sun.misc.Launcher$AppClassLoader#764c12b6]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:659)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:556)
at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:541)
at org.springframework.util.ReflectionUtils.getUniqueDeclaredMethods(ReflectionUtils.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryMethod(AbstractAutowireCapableBeanFactory.java:726)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineTargetType(AbstractAutowireCapableBeanFactory.java:667)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.predictBeanType(AbstractAutowireCapableBeanFactory.java:635)
at org.springframework.beans.factory.support.AbstractBeanFactory.isFactoryBean(AbstractBeanFactory.java:1489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:420)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:390)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:511)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:503)
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1198)
at org.springframework.boot.SpringApplication.getExitCodeFromMappedException(SpringApplication.java:889)
at org.springframework.boot.SpringApplication.getExitCodeFromException(SpringApplication.java:875)
at org.springframework.boot.SpringApplication.handleExitCode(SpringApplication.java:861)
at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:810)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at com.bookstore.BookstoreApp.main(BookstoreApp.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.NoClassDefFoundError: io/micrometer/core/instrument/binder/hystrix/HystrixMetricsBinder
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethods(Unknown Source)
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:641)
... 23 common frames omitted
Caused by: java.lang.ClassNotFoundException: io.micrometer.core.instrument.binder.hystrix.HystrixMetricsBinder
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 27 common frames omitted
It has 2 issues. The first one - it tries to connect to Jhipster Registry. Read more here https://www.jhipster.tech/jhipster-registry/.
Just clone it and run using ./mvnw.
Second issue I resolved by commenting following exclude
compile ("org.springframework.boot:spring-boot-starter-actuator") {
// exclude module: 'micrometer-core'
}
So, I have a structured streaming application that is reading in 3 csv files from S3 on a 5 minute trigger interval. A basic groupby is performed on the dataframe and then using the foreach output I asynchronously write out the records to an external data store. After about 4-5 intervals, the spark job crashes and this is what I find in the logs: Does anyone have a solution for this? If not, what would your first steps to debug this be?
17/06/19 17:46:36 INFO SparkContext: Created broadcast 1 from broadcast at StreamFromS3.scala:65
17/06/19 17:47:31 WARN ClosureCleaner: Expected a closure; got com.ignitionone.es.ForEachOuput
17/06/19 17:48:19 WARN FileStreamSource: Listed 348653 file(s) in 47382.367306 ms
17/06/19 17:50:51 WARN FileStreamSource: Listed 348653 file(s) in 51272.664258 ms
17/06/19 17:55:50 WARN FileStreamSource: Listed 348707 file(s) in 50868.757278 ms
17/06/19 18:00:51 WARN FileStreamSource: Listed 348747 file(s) in 51256.287437 ms
17/06/19 18:00:51 ERROR ApplicationMaster: RECEIVED SIGNAL TERM
17/06/19 18:00:51 ERROR ApplicationMaster: Exception from Reporter thread.
org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: Application attempt appattempt_1497453419814_0060_000001 doesn't exist in ApplicationMasterService cache.
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:439)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy19.allocate(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)
at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:260)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:458)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException): Application attempt appattempt_1497453419814_0060_000001 doesn't exist in ApplicationMasterService cache.
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:439)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy18.allocate(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
... 9 more
I am using apache spark , and I want to mute the java.net.BindException: Address already in use. exception that is been thrown when I run my spark-submit command. log4j properties is set in a separate file that is posted bellow.
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)
at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1988)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1979)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)
at com.mypackage.myclass.MyUtil$class.build_context(MyUtil.scala:500)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/03/14 11:13:01 WARN Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
here is my log4j.properties
log4j.rootCategory=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.com.jiwire=INFO
log4j.logger.com.nd=INFO
Any idea how do I do that ?
The error is caused because Spark tries to host its web interface on a port that is already in use.
It is possible that you are running two instances of Spark simultaneously, making the first instance acquire the port.
It is usually a bad practice to suppress such errors.
Read here for more information about Spark Configuration.
While connecting Spark(1.4.0 built for Hadoop 2.6.0) with hive(Version 1.1.0) tables exception is coming due to derby version mismatch. Is there any way to both(spark & hive) can use same derby version ? ...Any other way to fix this ?
Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1#e0d0b81, see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 88 more
Caused by: java.sql.SQLException: Database at /home/saurabh/Softwares/hive1.1/bin/metastore_db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.11.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
... 85 more
Caused by: ERROR XSLAN: Database at /home/saurabh/Softwares/hive1.1/bin/metastore_db has an incompatible format with the current version of the software. The database was created by or upgraded by version 10.11.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown Source)
at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source)
... 85 more
15/07/01 17:29:49 INFO SparkUI: Stopped Spark web UI at http://192.168.17.155:4040
15/07/01 17:29:49 INFO DAGScheduler: Stopping DAGScheduler
This is how I solved it:
rm -rf "${APPDATA//\\//}/../Local/Temp/spark-"*
rm -rf /c/tmp/hive
rm -rf ./metastore_db
I deleted from git bash the:
local spark directories (not mandatory but good)
the temporary hive directory (mandatory)
the metastore_db (it is mandatory to delete the metastore_db/*.loc I deleted the whole thing to be sure )
I am not sure it is exactly applicable to your case, but it reset the derby database so that I can use it.
My spark is using derby-10.12.1.1.jar and hive was using derby-10.10.2.0.jar so went to /usr/local/spark/jars/ and copied derby-10.12.1.1.jar and pasted in /usr/local/hive/lib
Above step solved my problem.
This is due to incompatibility in derby version between spark & hive. In my case, spark was having derby 10.12 while hive was using 10.14. I copied derby-10.14.1.0.jar from /hive/lib directory to spark/jars directory & issue got resolved.
I am obfuscating JAR and that JAR also contains some resource files. I do not want them to get obfuscated or get corrupted. How to ask proguard to ignore resource file such as public key store file?
-adaptresourcefilenames
-adaptresourcefilecontents
We have to options for resources file but from both these options my public key store file get corrupted into output jar and throwing exception -
Exception in Application start method
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.javafx.main.Main.a(Main.java:642)
at com.javafx.main.Main.main(Main.java:805)
Caused by: java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherIm
pl.java:403)
at com.sun.javafx.application.LauncherImpl.access$000(LauncherImpl.java:
47)
at com.sun.javafx.application.LauncherImpl$1.run(LauncherImpl.java:115)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.AssertionError: java.security.KeyStoreException: Uninitiali
zed keystore
at de.schlichtherle.license.LicenseNotary.getPublicKey(Unknown Source)
at de.schlichtherle.license.LicenseNotary.verify(Unknown Source)
at de.schlichtherle.license.LicenseManager.verify(Unknown Source)
at de.schlichtherle.license.LicenseManager.verify(Unknown Source)
at com.sun.javafx.application.LauncherImpl$5.run(LauncherImpl.java:319)
at com.sun.javafx.application.PlatformImpl$5.run(PlatformImpl.java:206)
at com.sun.javafx.application.PlatformImpl$4.run(PlatformImpl.java:173)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.access$100(WinApplication.java:29
)
at com.sun.glass.ui.win.WinApplication$3$1.run(WinApplication.java:73)
... 1 more
Caused by: java.security.KeyStoreException: Uninitialized keystore
at java.security.KeyStore.isKeyEntry(Unknown Source)
... 17 more
Both options accept arguments to filter the files that you want to adapt:
cfr. ProGuard manual > Usage > -adaptresourcefilenames / -adaptresourcefilecontents