I'm trying to build Spark 1.2.0 on ubuntu but i'm getting dependency issues.
I basically download the files extract the folder and run sbt/sbt/assembly
sbt = 0.13.6
scala = 2.10.4
sbt.ResolveException: unresolved dependency: org.apache.spark#spark-
network-common_2.10;1.2.0: configuration not public in
org.apache.spark#spark-network-common_2.10;1.2.0: 'test'. It was
required from org.apache.spark#spark-network-shuffle_2.10;1.2.0 test
This sbt issue seems to explain it: this would be a consequence of trying to get a test->test dependency if the same version has been resolved out of a public Maven repository.
A workaround would be using git SHA versioning or SNAPSHOT for non final builds of that test dependency, but we won't know more unless we get an idea of how you got to a 'bad' ivy cache state.
TL;DR : try clearing your cache of spark artefacts before building.
Edit: This is fixed in sbt 0.13.10-RC1 https://github.com/sbt/sbt/pull/2345 Please update
Related
I am trying to add the maven dependency in Databricks Analytics platform
Getting error: Unknown error while checking maven dependencies: Cannot download some libraries due to transient Maven issue. Please try again later.
This issue looks strange. AsFor a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket for the same.
Meanwhile, you can create the library manually:
Step1: You can download the jar file from MVN repository.
Step2: Create library by selecting "Library Source" as "Upload" and "Library Type" as "Jar" and select the jar file downloaded.
Here are the steps to create library manually by following the below steps:
I have an offline pyspark cluster (no internet access) where I need to install graphframes library.
I have manually downloaded the jar from here added in $SPARK_HOME/jars/ and then when I try to use it I get the following error:
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term typesafe in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term scalalogging in value com.typesafe,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.typesafe.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access type LazyLogging in value com.slf4j,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.slf4j.
Which is the correct way to offline install it with all the dependencies?
I manage to install the graphframes libarary. First of all I found the graphframes dependencies witch where:
scala-logging-api_xx-xx.jar
scala-logging-slf4j_xx-xx.jar
where xx is the proper versions for scala and the jar version. Then I installed them in the proper path. Because I work in an Cloudera machine the proper path is:
/opt/cloudera/parcels/SPARK2/lib/spark2/jars/
If you can not place them in this directory in your cluster (because you have no root rights and your admin is super lazy) you can simply add in your spark-submit/ spark-shell
spark-submit ..... --driver-class-path /path-for-jar/ \
--jars /../graphframes-0.5.0-spark2.1-s_2.11.jar,/../scala-logging-slf4j_2.10-2.1.2.jar,/../scala-logging-api_2.10-2.1.2.jar
This works for Scala. In order to use graphframes for python you need to
download graphframes jar and then through shell
#Extract JAR content
jar xf graphframes_graphframes-0.3.0-spark2.0-s_2.11.jar
#Enter the folder
cd graphframes
#Zip the contents
zip graphframes.zip -r *
And then add the zipped file in your python path in spark-env.sh or your bash_profile
with
export PYTHONPATH=$PYTHONPATH:/..proper path/graphframes.zip:.
Then opening the shell/submitting (again with the same arguments as with scala) importing graphframes works normaly
This link was extremely useful for this solution
I've been transitioning a portlet from Liferay 6.2 to a bundle for Liferay 7. When I deploy it, it can't load, because of a missing resource:
$ blade sh diag 1084
diag 1084
org.mycompany.caasd.portal-ldap-sync [1084]
Unresolved requirement: Import-Package: com.liferay.portal.kernel.service;
version="[1.20.0,2.0.0)"
When I use Felix Gogo shell to see what's available, I don't see any kernel bundles. Is that a bundle I should be expecting to be installed, install myself, include in my bundle, or am I just not thinking about this the right way?
The com.liferay.portal.kernel version for CE GA4 is 2.32.1. So if you were developing for CEGA4, with a gradle project, you'd configure the dependency in your project's build.gradle like this:
dependencies {
compileOnly group: "com.liferay.portal", name: "com.liferay.portal.kernel", version: "2.32.1"
}
You can find this by looking in the portal-kernel.jar file's MANIFEST/MANIFEST.MF or if you have the liferay src for your version, in any of its own modules build.gradle files. In my tomcat bundle, the portal-kernel.jar if in tomcat/lib/ext.
Unless I've misunderstood your question, that should get you what you want.
See the docs here for more information
When you build your bundle, bnd will either
inspect your classpath and detect the package versions for you
use whatever versions you manually provide in Import-Package
In your case, it seams it somehow detected that the version of com.liferay.portal.kernel.service package is 1.20.0. It therefore generated the range "equal to or greater than current version but lower than next major version" which in your case is expressed as [1.20.0,2.0.0). That may have been because you had old jar on claspath or behacause you had wrong Import-Package statement.
This may compile just fine as long as you are nor using functionality that was added/changed in the newer version. At runtime though, the actual package version is higher (something like 2.32.1) and therefore it does not meet your requirement. As OSGi runtime can not resolve your bundle requirement, the bundle is left in "istalled" state!
To solve that you have two options:
install your bundle on older Liferay version (where com.liferay.portal.kernel.service package is between 1.20.0 and 2.0.0)
recompile your bundle making sure the classpath contains only those jars in which com.liferay.portal.kernel.service package has version that will generate a version range in which you runtime package version fits.
I have run a command spark-shell --jars /home/krishnamahi/graphframes-0.4.0-spark2.1-s_2.11.jar and it threw me an error
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term typesafe in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term scalalogging in value com.typesafe,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.typesafe.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access type LazyLogging in value com.slf4j,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.slf4j.
I am using Spark Version 2.1.1, Scala Version 2.11.8, JDK Version 1.8.0_131, CentOS7 64-bit, Hadoop 2.8.0. Can anyone please tell me what additional command should I give for perfect run of program? Thanks in advance.
If you want to play with GraphFrames use --packages command-line option of spark-shell instead.
--packages Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.
For graphframes-0.4.0-spark2.1-s_2.11.jar that'd be as follows:
$SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.4.0-spark2.1-s_2.11
which I copied verbatim from How to section of GraphFrames project.
That way you don't have to search for all the (transitive) dependencies of GraphFrames library as Spark will do it for you automatically.
I have installed raw Hadoop, with all components Hive, Pig, Spark of latest versions. It then worked for me. I used Cent OS 7. Order for installing Hadoop with components is
Anaconda3/Python3 (Since Spark 2.x doesn't support Python 2)
Hadoop
Pig
Hive
Hbase
Spark
All the components should be in a single go, in same terminal. After Spark installation, restart the system.
I am having trouble building the Aerospark connector # https://github.com/aerospike/aerospark
I have created fresh Ubuntu box, installed JDK, SBT, Maven, Scala 2.10, and then followed the steps at the github page above. During the build i get this error and I am not sure which is the most direct way to build this properly....
[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in
the following: /opt/astools/aerospark/lib/aerospike-helper-java-
1.0.6.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
/root/.m2/repository/com/aerospike/aerospike-helper-java/1.0.6/aerospike-
helper-java-1.0.6.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
/root/.ivy2/cache/com.aerospike/aerospike-client/jars/aerospike-client
3.3.1.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
Are there any updated instructions for building this - the git link in the instructions seems outdated incidentally.
As an aside, has anyone tried this with PySpark?
There was an issue with the build file which affected ubuntu. Can you pull the latest changes and go through the build process again?