I managed several spark jobs to compute RDD and in the end, I would like to put some of these data in an Ignite Cache. Unfortunately, I have got an error :
Java.lang.ClassCastException: org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl cannot be cast to org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy
[info] at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.equals(GatewayProtectedCacheProxy.java:1715)
[info] at scala.collection.mutable.FlatHashTable$class.findElemImpl(FlatHashTable.scala:131)
[info] at scala.collection.mutable.FlatHashTable$class.containsElem(FlatHashTable.scala:124)
[info] at scala.collection.mutable.HashSet.containsElem(HashSet.scala:40)
[info] at scala.collection.mutable.HashSet.contains(HashSet.scala:57)
[info] at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:87)
[info] at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitExternalizable(SerializationDebugger.scala:142)
[info] at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:104)
[info] at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:206)
[info] at org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
[info] at org.apache.spark.serializer.SerializationDebugger$.find(SerializationDebugger.scala:67)
[info] at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:41)
[info] at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
[info] at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
[info] at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:400)
[info] at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:393)
[info] at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:162)
[info] at org.apache.spark.SparkContext.clean(SparkContext.scala:2326)
[info] at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:371)
[info] at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:370)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[info] at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[info] at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
[info] at org.apache.spark.rdd.RDD.map(RDD.scala:370).........
So my question is how to put data coming from spark rdd in specific Ignite Cache, in our case, an Ignite Cache with 3rd persistence cache store implemented on Postgres?
Related
I'm trying to run my appium test on appcenter.
After uploading test to appcenter and its bout to execute my test i get the error bellow
java.io.FileNotFoundException: src/test/resources/properties/Config.properties (No such file or directory)at java.io.FileInputStream.open(Native Method
Note: I successfully run the test on my local with any issues but on trying to execute on app center i get the above error
Something went wrong while running your tests
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building run-test 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # run-test ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # run-test ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # run-test ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # run-test ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.20:test (default-test) # run-test ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running appiumTest.TestCase.Rough
Logging xtc events to event_stream.log
log4j:WARN No appenders could be found for logger (devpinoyLogger).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:ERROR Could not read configuration file [src/test/resources/properties/log4j.properties].
java.io.FileNotFoundException: src/test/resources/properties/log4j.properties (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:372)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:403)
at appiumTest.base.TestBase.Capabilities(TestBase.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:62)
at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:340)
at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:294)
at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:176)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
at org.testng.TestRunner$$Lambda$45/479397964.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.testng.TestRunner.privateRun(TestRunner.java:770)
at org.testng.TestRunner.run(TestRunner.java:591)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
at org.testng.SuiteRunner.run(SuiteRunner.java:304)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
at org.testng.TestNG.runSuites(TestNG.java:1032)
at org.testng.TestNG.run(TestNG.java:1000)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
log4j:ERROR Ignoring configuration file [src/test/resources/properties/log4j.properties].
Feb 08, 2020 10:53:57 AM io.appium.java_client.remote.AppiumCommandExecutor$1 lambda$0
INFO: Detected dialect: W3C
In the cloud
java.io.FileNotFoundException: src/test/resources/properties/Config.properties (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at appiumTest.base.TestBase.setUp(TestBase.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:133)
at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:62)
at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:340)
at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:294)
at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:176)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
at org.testng.TestRunner$$Lambda$45/479397964.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.testng.TestRunner.privateRun(TestRunner.java:770)
at org.testng.TestRunner.run(TestRunner.java:591)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:402)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:396)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
at org.testng.SuiteRunner.run(SuiteRunner.java:304)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1180)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1102)
at org.testng.TestNG.runSuites(TestNG.java:1032)
at org.testng.TestNG.run(TestNG.java:1000)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 56.019 s <<< FAILURE! - in appiumTest.TestCase.Rough
[ERROR] setUp(appiumTest.TestCase.Rough) Time elapsed: 55.821 s <<< FAILURE!
java.lang.NullPointerException
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] Rough>TestBase.setUp:98 ยป NullPointer
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 2
[INFO]
[ERROR] There are test failures.
Please refer to target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # run-test ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: target/run-test-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-failsafe-plugin:2.20:integration-test (default) # run-test ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! The file encoding for reports output files should be provided by the POM property ${project.reporting.outputEncoding}.
[INFO]
[INFO] --- maven-failsafe-plugin:2.20:verify (default) # run-test ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2020-02-08T10:54:00+01:00
[INFO] Final Memory: 16M/211M
[INFO] ------------------------------
My Test Scrip code on eclipse
hope the above issue is resolved.
if not then do not provide absolute path of file starts from 'src' folder because we do not give src package to app center for execution. we prepare upload folder and submit this package to app center. please see your upload package and create path.
upload package must have resources/properties/log4j.properties.
for example: try like this
String path =
System.getProperty("user.dir")+"//test-classes//Resources//testdata.xlsx";
I'm building spark 2.4.3 to make it compatible to latest hadoop 3.2.0.
The source code is downloaded from https://www.apache.org/dyn/closer.lua/spark/spark-2.4.3/spark-2.4.3.tgz
Build command is ./build/mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.2.0 -DskipTests clean package
The build result is:
[INFO] Spark Project Parent POM ........................... SUCCESS [ 1.761 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 1.221 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 0.551 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 0.608 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 1.558 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 0.631 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 0.444 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 2.501 s]
[INFO] Spark Project Core ................................. SUCCESS [ 13.536 s]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 0.549 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 1.614 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 3.332 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 14.271 s]
[INFO] Spark Project SQL .................................. SUCCESS [ 13.008 s]
[INFO] Spark Project ML Library ........................... SUCCESS [ 7.923 s]
[INFO] Spark Project Tools ................................ SUCCESS [ 0.187 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 6.664 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 1.285 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 4.824 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 3.020 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 1.558 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 1.411 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 1.573 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 1.702 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 5.969 s]
[INFO] Spark Avro ......................................... SUCCESS [ 0.702 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:32 min
[INFO] Finished at: 2019-07-31T18:56:24+08:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "hadoop-3.2" could not be activated because it does not exist.
According to my expectation, an all-in-one compress file like spark-2.4.3-bin-hadoop3.2.tgz would be generated under build directory, just like the binary file that can be downloaded from official site, https://www.apache.org/dyn/closer.lua/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz.
How can I remove the warning The requested profile "hadoop-3.2" could not be activated because it does not exist, what does it mean?
Caution: What you are trying to do could result in very unstable environment if you don't know what you are doing.
That being said, spark 2.4.x stable release does not have profile hadoop-3.2, it has hadoop-3.1.
You will need to pull code from master to achieve what you want to achieve.
If your sole intention is to make spark 2.4.3 compatible with hadoop 3.2, you could look at profile in master along with relevant changes and cherrypick those into your own workspace.
I was installing Apache Zeppelin with Spark and while running the maven install command I get following error for Zeppelin: web Application
[ERROR] error Command failed with exit code 1.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Zeppelin ........................................... SUCCESS [ 50.394 s]
[INFO] Zeppelin: Interpreter .............................. SUCCESS [ 31.632 s]
[INFO] Zeppelin: Zengine .................................. SUCCESS [ 24.134 s]
[INFO] Zeppelin: Display system apis ...................... SUCCESS [ 19.607 s]
[INFO] Zeppelin: Spark dependencies ....................... SUCCESS [01:33 min]
[INFO] Zeppelin: Spark .................................... SUCCESS [ 29.058 s]
[INFO] Zeppelin: Markdown interpreter ..................... SUCCESS [ 5.796 s]
[INFO] Zeppelin: Angular interpreter ...................... SUCCESS [ 4.361 s]
[INFO] Zeppelin: Shell interpreter ........................ SUCCESS [ 4.827 s]
[INFO] Zeppelin: Livy interpreter ......................... SUCCESS [ 50.227 s]
[INFO] Zeppelin: HBase interpreter ........................ SUCCESS [ 11.682 s]
[INFO] Zeppelin: Apache Pig Interpreter ................... SUCCESS [ 10.991 s]
[INFO] Zeppelin: PostgreSQL interpreter ................... SUCCESS [ 5.541 s]
[INFO] Zeppelin: JDBC interpreter ......................... SUCCESS [ 6.663 s]
[INFO] Zeppelin: File System Interpreters ................. SUCCESS [ 6.304 s]
[INFO] Zeppelin: Flink .................................... SUCCESS [ 13.449 s]
[INFO] Zeppelin: Apache Ignite interpreter ................ SUCCESS [ 5.955 s]
[INFO] Zeppelin: Kylin interpreter ........................ SUCCESS [ 4.915 s]
[INFO] Zeppelin: Python interpreter ....................... SUCCESS [ 6.109 s]
[INFO] Zeppelin: Lens interpreter ......................... SUCCESS [ 11.360 s]
[INFO] Zeppelin: Apache Cassandra interpreter ............. SUCCESS [ 58.287 s]
[INFO] Zeppelin: Elasticsearch interpreter ................ SUCCESS [ 9.617 s]
[INFO] Zeppelin: BigQuery interpreter ..................... SUCCESS [ 5.584 s]
[INFO] Zeppelin: Alluxio interpreter ...................... SUCCESS [ 9.001 s]
[INFO] Zeppelin: Scio ..................................... SUCCESS [ 48.425 s]
[INFO] Zeppelin: web Application .......................... FAILURE [28:26 min]
[INFO] Zeppelin: Server ................................... SKIPPED
[INFO] Zeppelin: Packaging distribution ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 37:14 min
[INFO] Finished at: 2017-02-01T16:21:39+05:30
[INFO] Final Memory: 224M/1792M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:yar
n (yarn install) on project zeppelin-web: Failed to run task: 'yarn install --no
-lockfile --https-proxy=http://sg0227823:***#tulsa-proxy.sabre.com:80 --proxy=ht
tp://sg0227823:***#tulsa-proxy.sabre.com:80' failed. (error code 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureExc
eption
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :zeppelin-web
As #lambda has mentioned in comments, it was fixed by his PR-2016 github.com/apache/zeppelin/pull/2016 but it seems the issue is back in the latest version of Zeppelin.
I resolved this by installing system-wide npm and nodejs:
$ sudo yum install nodejs npm
(notice if you're using RHEL, both packages are available only through non-default EPEL yum repository https://fedoraproject.org/wiki/EPEL).
but then I was getting
"Node version 0.10.48 is not supported, please use Node.js 4.0 or higher." when manually ran 'yarn install --no-lockfile' (don't confuse Hadoop's yarn command with this yarn command - https://yarnpkg.com/en/ which is used by nodejs). This in its turn got resolved by installing latest stable node.js
$ sudo npm install n -g
$ sudo n stable
ps. It was hard to diagnose what exactly was wrong with yarn command at first as maven just spits 'error code 1', but you can debug by running that same command manually, for example, ./zeppelin-web/node/yarn/dist/bin/yarn install --no-lock (assuming you're in the root of zeppelin codebase). So you get output of exact problem.
I am trying to build the spark to be able to run the programs, however it does not seem to work.
This is what happens when I try to run sample program in spark:
hduser_#ankit-sve14137cnb:/usr/local/spark$ ./bin/run-example SparkPi 10
Failed to find Spark examples assembly in /usr/local/spark/lib or /usr/local/spark/examples/target
You need to build Spark before running this program
hduser_#ankit-sve14137cnb:/usr/local/spark$ sudo build/mvn -e -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package
Using `mvn` from path: /usr/bin/mvn
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO]
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM
[INFO] Spark Project Test Tags
[INFO] Spark Project Launcher
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Unsafe
[INFO] Spark Project Core
[INFO] Spark Project Bagel
[INFO] Spark Project GraphX
[INFO] Spark Project Streaming
[INFO] Spark Project Catalyst
[INFO] Spark Project SQL
[INFO] Spark Project ML Library
[INFO] Spark Project Tools
[INFO] Spark Project Hive
[INFO] Spark Project Docker Integration Tests
[INFO] Spark Project REPL
[INFO] Spark Project YARN Shuffle Service
[INFO] Spark Project YARN
[INFO] Spark Project Assembly
[INFO] Spark Project External Twitter
[INFO] Spark Project External Flume Sink
[INFO] Spark Project External Flume
[INFO] Spark Project External Flume Assembly
[INFO] Spark Project External MQTT
[INFO] Spark Project External MQTT Assembly
[INFO] Spark Project External ZeroMQ
[INFO] Spark Project External Kafka
[INFO] Spark Project Examples
[INFO] Spark Project External Kafka Assembly
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 1.6.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) # spark-parent_2.10 ---
[INFO] Deleting /usr/local/spark/target
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) # spark-parent_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) # spark-parent_2.10 ---
[INFO] Add Source directory: /usr/local/spark/src/main/scala
[INFO] Add Test Source directory: /usr/local/spark/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) # spark-parent_2.10 ---
[INFO] Dependencies classpath:
/root/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) # spark-parent_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) # spark-parent_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) # spark-parent_2.10 ---
[INFO] Executing tasks
main:
[mkdir] Created dir: /usr/local/spark/target/tmp
[INFO] Executed tasks
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) # spark-parent_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default) # spark-parent_2.10 ---
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) # spark-parent_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) # spark-parent_2.10 ---
[INFO] Building jar: /usr/local/spark/target/spark-parent_2.10-1.6.1-tests.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) # spark-parent_2.10 ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (default) # spark-parent_2.10 ---
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) # spark-parent_2.10 ---
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) # spark-parent_2.10 ---
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Test Tags 1.6.1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) # spark-test-tags_2.10 ---
[INFO] Deleting /usr/local/spark/tags/target
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) # spark-test-tags_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) # spark-test-tags_2.10 ---
[INFO] Add Source directory: /usr/local/spark/tags/src/main/scala
[INFO] Add Test Source directory: /usr/local/spark/tags/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) # spark-test-tags_2.10 ---
[INFO] Dependencies classpath:
/root/.m2/repository/org/scala-lang/scala-reflect/2.10.5/scala-reflect-2.10.5.jar:/root/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/root/.m2/repository/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.jar:/root/.m2/repository/org/scalatest/scalatest_2.10/2.2.1/scalatest_2.10-2.2.1.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) # spark-test-tags_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # spark-test-tags_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /usr/local/spark/tags/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) # spark-test-tags_2.10 ---
[INFO] Using zinc server for incremental compilation
[error] Required file not found: scala-compiler-2.10.5.jar
[error] See zinc -help for information about locating necessary files
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 3.566 s]
[INFO] Spark Project Test Tags ............................ FAILURE [ 0.466 s]
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Docker Integration Tests ............. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External MQTT Assembly ............... SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.128 s
[INFO] Finished at: 2016-04-02T22:46:45+05:30
[INFO] Final Memory: 38M/223M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-test-tags_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-test-tags_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 20 more
Caused by: Compile failed via zinc server
at sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136)
at sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86)
at scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303)
at scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119)
at scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99)
at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
... 21 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-test-tags_2.10
I do have the latest maven:
hduser_#ankit-sve14137cnb:/usr/local$ mvn -version
Apache Maven 3.3.3
Maven home: /usr/share/maven
Java version: 1.8.0_77, vendor: Oracle Corporation
Java home: /home/ankit/Downloads/jdk1.8.0_77/jre
Default locale: en_IN, platform encoding: UTF-8
Below are the paths mentioned in my bashrc file:
export JAVA_HOME=/home/ankit/Downloads/jdk1.8.0_77
export HADOOP_HOME=/home/ankit/Downloads/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
#export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native"
export CASSANDRA_HOME =$CASSANDRA_HOME:/home/hduser_/cassandra
#export PATH = $PATH:$CASSANDRA_HOME/bin
export SCALA_HOME = $SCALA_HOME:/usr/local/scala
export PATH = $SCALA_HOME/bin:$PATH
I am new to SOF,could someone please advise?
In Maven 3 if you just had a failed download and have fixed it (e.g. by uploading the jar toa repository) it will cache the failure. To force a refresh add -U to the command line. Try refreshing and let me know how it'll go.
If you have already failed build once you need to force refresh with maven 3 : The command should be (note the -U option):
mvn -U -DskipTests clean package
LOVE RQB!!, I want to get the source building. I have cloned the repo with git and I believe I have downloaded and have all of the bits installedbut when I run "mvn clean install" it goes through some machinations, then finally fails with the following
Failed tests:
[INFO] testJsList(com.redspr.redquerybuilder.js.client.GwtTestBasics)
Here is the output from the stack:
[INFO] Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.325 sec <<< FAILURE!
[INFO] testJsList(com.redspr.redquerybuilder.js.client.GwtTestBasics) Time elapsed: 0.047 sec <<< FAILURE!
[INFO] junit.framework.AssertionFailedError: Remote test failed at 127.0.0.1 / Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19
[INFO] expected: <22>, actual: <14>
[INFO] at junit.framework.Assert.fail(Assert.java:193)
[INFO] at junit.framework.Assert.failNotEquals(Assert.java:198)
[INFO] at junit.framework.Assert.assertEquals(Assert.java:94)
[INFO] at junit.framework.Assert.assertEquals(Assert.java:43)
[INFO] at com.redspr.redquerybuilder.js.client.GwtTestBasics.testJsList(GwtTestBasics.java:192)
[INFO] at com.redspr.redquerybuilder.js.client.__GwtTestBasics_unitTestImpl.doRunTest(__GwtTestBasics_unitTestImpl.java:7)
[INFO] at junit.framework.TestCase.runTest(TestCase.java:62)
[INFO] at com.google.gwt.junit.client.GWTTestCase.runBare(GWTTestCase.java:188)
[INFO] at com.google.gwt.junit.client.GWTTestCase.__doRunTest(GWTTestCase.java:129)
[INFO] at com.google.gwt.junit.client.impl.GWTRunner.runTest(GWTRunner.java:390)
[INFO] at com.google.gwt.junit.client.impl.GWTRunner.doRunTest(GWTRunner.java:318)
[INFO] at com.google.gwt.junit.client.impl.GWTRunner.access$9(GWTRunner.java:312)
[INFO] at com.google.gwt.junit.client.impl.GWTRunner$TestBlockListener.onSuccess(GWTRunner.java:107)
[INFO] at com.google.gwt.junit.client.impl.GWTRunner$TestBlockListener.onSuccess(GWTRunner.java:1)
[INFO] at com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:232)
[INFO] at com.google.gwt.http.client.Request.fireOnResponseReceived(Request.java:258)
[INFO] at com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:412)
[INFO] at sun.reflect.GeneratedMethodAccessor62.invoke(Unknown Source)
[INFO] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.lang.reflect.Method.invoke(Method.java:606)
[INFO] at com.google.gwt.dev.shell.MethodAdaptor.invoke(MethodAdaptor.java:103)
[INFO] at com.google.gwt.dev.shell.MethodDispatch.invoke(MethodDispatch.java:71)
[INFO] at com.google.gwt.dev.shell.OophmSessionHandler.invoke(OophmSessionHandler.java:172)
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.reactToMessagesWhileWaitingForReturn(BrowserChannelServer.java:338)
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.invokeJavascript(BrowserChannelServer.java:219)
[INFO] at com.google.gwt.dev.shell.ModuleSpaceOOPHM.doInvoke(ModuleSpaceOOPHM.java:136)
[INFO] at com.google.gwt.dev.shell.ModuleSpace.invokeNative(ModuleSpace.java:571)
[INFO] at com.google.gwt.dev.shell.ModuleSpace.invokeNativeObject(ModuleSpace.java:279)
[INFO] at com.google.gwt.dev.shell.JavaScriptHost.invokeNativeObject(JavaScriptHost.java:91)
[INFO] at com.google.gwt.core.client.impl.Impl.apply(Impl.java)
[INFO] at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:249)
[INFO] at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
[INFO] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[INFO] at java.lang.reflect.Method.invoke(Method.java:606)
[INFO] at com.google.gwt.dev.shell.MethodAdaptor.invoke(MethodAdaptor.java:103)
[INFO] at com.google.gwt.dev.shell.MethodDispatch.invoke(MethodDispatch.java:71)
[INFO] at com.google.gwt.dev.shell.OophmSessionHandler.invoke(OophmSessionHandler.java:172)
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.reactToMessages(BrowserChannelServer.java:293)
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:547)
[INFO] at com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:364)
[INFO] at java.lang.Thread.run(Thread.java:744)
This is a bug in the unit test of RedQueryBuilder 0.6.0 due to the unit test setting the time in UTC then testing the days, hours, minutes are the same in local time... Only true in the UK and Lisbon :( Fix should be in 0.7.0. Thanks for finding and raising the bug.