ERROR [WRITE-/10.10.35.30] 2013-06-19 23:15:56,907 CassandraDaemon.java (line 175) Exception in thread Thread[WRITE-/10.10.35.30,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:79)
at org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:66)
at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:341)
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:143)
When going through know issues i found this
The native library snappy-1.0.4.1-libsnappyjava.so for Snappy compression is included in the snappy-java-1.0.4.1.jar file. When the JVM initializes the JAR, the library is added to the default temp directory. If the default temp directory is mounted with a noexec option, it results in the above exception.
i added JVM_OPTS=-Dorg.xerial.snappy.tempdir=/tmp in cassandra.in.sh and it still didnot work.
i also tried specifying the temp directory directly
./bin/cassandra -Dorg.xerial.snappy.tempdir=/tmp
On the same machine cassandra version 1.0.12 works fine.
Any help will be appreciated.
The problem is that there is no FreeBSD library included in the snappy JAR file that comes with Cassandra. Install the archivers/snappy-java port, delete the snappy-java JAR file that came with Cassandra, and copy /usr/local/share/java/classes/snappy-java.jar into Cassandra's lib directory.
Same problem happened when trying to enable snappy compression for apache kafka 0.8 on FreeBSD but the solution was the same. Just copy /usr/local/share/java/classes/snappy-java.jar to the kafka/src/core/target/scala-2.8.0 directory, restart kafka and enjoy!
Related
How can I load XML files in Spark 2.0?
val rd = spark.read.format("com.databricks.spark.xml").load("C:/Users/kumar/Desktop/d.xml")
I'm getting error com.databricks.spark.xml not available.
java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.xml. Please find packages at https://cwiki.apache.org/confluence/display/SPARK/Third+Party+Projects
at org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource(DataSource.scala:148)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass$lzycompute(DataSource.scala:79)
at org.apache.spark.sql.execution.datasources.DataSource.providingClass(DataSource.scala:79)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:325)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:132)
... 48 elided
ClassNotFoundException means that you need a fat jar which you could include the package in your build.sbt and make the jar by sbt assembly. you may have a try.
If can not work. add the jar into $SPARK_HOME/jars and have a try.
Alternatively, you can add the jar file into your spark shell. Download the spark-xml_2.10-0.2.0.jar jar file and copy into the spark's class path and add the jar file in your spark shell using the :cp command as
:cp spark-xml_2.10-0.2.0.jar
/*
jar file will get imported into the spark shell
now you can use this jar file anywhere in your code inside the spark shell.
*/
val rd = spark.read.format("com.databricks.spark.xml").load("C:/Users/kumar/Desktop/d.xml")
I just upgraded from Spark 2.0.2 to Spark 2.1.0 (by downloading the prebuilt version for Hadoop 2.7&later). No Hive is installed.
Upon launch of the spark-shell, the metastore_db/ folder and derby.log file are created at the launch location, together with a bunch of warning logs (which were not printed in the previous version).
Closer inspection of the debug logs shows that Spark 2.1.0 tries to initialise a HiveMetastoreConnection:
17/01/13 09:14:44 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
Similar debug logs for Spark 2.0.2 do not show any initialisation of HiveMetastoreConnection.
Is this intended behaviour? Could it be related to the fact that spark.sql.warehouse.dir is now a static configuration shared among sessions? How do I avoid this, since I have no Hive installed?
Thanks in advance!
From Spark 2.1.0 documentation pages:
When not configured by the hive-site.xml, the context automatically
creates metastore_db in the current directory and creates a directory
configured by spark.sql.warehouse.dir, which defaults to the directory
spark-warehouse in the current directory that the Spark application is
started. Note that the hive.metastore.warehouse.dir property in
hive-site.xml is deprecated since Spark 2.0.0. Instead, use
spark.sql.warehouse.dir to specify the default location of database in
warehouse.
Since you do not have Hive installed, you will not have a hive-site.xml config file, and this must be defaulting to the current directory.
If you are not planning to use HiveContext in Spark, you could reinstall Spark 2.1.0 from source, rebuilding it with Maven and making sure you omit -Phive -Phive-thriftserver flags which enable Hive support.
For future googlers: the actual underlying reason for the creation of metastore_db and derby.log in every working directory is the default value of derby.system.home.
This can be changed in spark-defaults.conf, see here.
This happen also with Spark 1.6. You can change the path by adding in Spark submit extra options:
-Dderby.system.home=/tmp/derby
(or by derby.properties, there are several ways to change it).
Im trying to start cassandra but dont see any response.
hduser#vagrant:~/cassandra/apache-cassandra-2.1.12-src/bin$ cassandra -f
hduser#vagrant:~/cassandra/apache-cassandra-2.1.12-src/bin$
I have downloaded cassandra in this folder -
/home/hduser/cassandra/apache-cassandra-2.1.12-src
It looks like you downloaded the source distribution of cassandra, try downloading the binary ('bin') distribution instead from the download page.
Otherwise, you can compile the source distribution using Apache Ant by running ant from the root directory of the extracted archive and then running cassandra per usual.
I am trying to use log4j for my Java desktop application that i am developing using Netbeans IDE 6.9.1. I have log4j.properties file in META-INF folder for logging during development. I also tried to put it along with the executable jar(after installation) but both of them did not work. It also throws exception when i call the method.
PropertyConfigurator.configure(filepath);
and it always throws this exception irrespective of the location of log.properties file
java.io.FileNotFoundException: META-INF\log4j.properties (The system cannot find the path specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at java.io.FileInputStream.<init>(FileInputStream.java:66)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:306)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:324)
at fi.xmldation.common.SharedMethods.readSettingsFile(SharedMethods.java:43)
Is it a bug in the IDE or I am doing something wrong?
If you run your program with an IDE such as NetBeans, check an extra time that the file is in the output directory of the IDE. Usually, log4j looks for this file on the classpath, so you must make sure that NetBeans actually copies it there.
Your IDE will most likely set it's own output directory as part of the classpath, so i think you have (at least) two options here: either to add the directory where you've put log4j.properties to the classpath, or to make sure it is copied to the IDE:s output directory. Good luck!
it worked if i load the properties from the properties file
PropertyConfigurator.configure((new Properties()).load(new FileInputStream ("log4j.properties")));
I'm trying to install xuggler Java libraries in Tomcat (version 5.5) on fedora-release-7-3
Should I install the binaries available for download on xuggler website or build my own (http://www.xuggle.com/xuggler/downloads/build.jsp)?
I took the easy step first and installed the readymade binaries downloaded from http://www.xuggle.com/xuggler/downloads/ in usr/local/xuggler folder on my Linux server and then copied the jar files from share/java/jars folder to Tomcat's $CATALINA_HOME/shared/lib directory (as recommended by
http://wiki.xuggle.com/Frequently_Asked_Questions#I_get_an_.22UnsatisfiedLinkError.22_when_I_run_Xuggler-based_Applications_in_Tomcat
These are some 6 .jar files, including xuggle-xuggler.jar
After restarting Tomcat, I'm still getting "java.lang.UnsatisfiedLinkError: no xuggle-xuggler in java.library.path" exception when my Java code attempts to invoke some xuggler method such as the one to find video duration of an flv file.
What am I doing wrong? Any help is very much appreciated!
Make sure that Tomcat can see the environment variables you have to set up with XUGGLE. In particular LD_LIBRARY_PATH.
Putting the xuggle environmental variables in the startup.sh is probably one of easiest ways to get this going. Also don't forget to put the xuggle jar only in the shared / common lib or you may run into classloader issues when you attempt to use xuggler in multiple webapps.