There are some pre installed java libraries in azure databricks https://docs.databricks.com/release-notes/runtime/6.6.html#installed-java-and-scala-libraries-scala-211-cluster-version
Is there a way to uninstall such libraries?
I have a library conflict. I need to use another version of the spring-core library.
Databricks includes a number of default Java and Scala libraries. You can replace any of these libraries with another version by using a cluster-scoped init script to remove the default library jar and then install the version you require.
Important Note: Removing default libraries and installing new versions may cause instability or completely break your Databricks cluster. You should thoroughly test any new library version in your environment before running production jobs.
Refer: Databricks - Replace a default library jar.
Related
The PostgreSQL used to provide EDB package for Linux x86_64 till version 10.15 like postgresql-10.15-1-linux-x64-binaries.tar.gz.
Now we want to upgrade PostgreSQL to version 13.1 and EDB package for it is not available.
Hence we want to create the package our own using documentation link. But we are able to compile package but it is not including all required files such and header and libs for SSL, tar.
How to generate a complete relocatable package like postgresql-10.15-1-linux-x64-binaries.tar to run PostgreSQL as standalone software on RHEL6/7/8 SUSE, SLES12/15 Linux versions.
or
Are there any special instructions to generate PostgreSQL 13.1 package like postgresql-10.15-1-linux-x64-binaries.tar?
I recommend that you use the PGDG packages.
Building from source is of course also an option, but I recommend that you do not try to create standalone binary packages. The proper way for that would be linking the software statically, which is not supported by the build process and would require you to have static libraries for all referenced software.
You could attempt to add all linked shared libraries to the software distribution, but you would have to make sure that these libraries are used instead of the ones on the default shared library path, which is difficult.
It seems to me that you are trying to re-invent the wheel here, and the wheel in this case is a docker container. Use that and stop worrying.
Im getting issues while using spark3.0 for reading elastic.
My elasticsearch version 7.6.0
I used elastic jar of the same version.
Please suggest a solution.
Spark 3.0.0 relies on Scala 2.12, which is not yet supported by Elasticsearch-hadoop. This and a few further issues prevent us from using Spark 3.0.0 together with Elasticsearch. If you want to compile it yourself, there is a pull-request on elasticsearch-hadoop (https://github.com/elastic/elasticsearch-hadoop/pull/1308) which should at least allow using scala 2.12. Not sure if it will fix the other issues as well.
It's officially released for spark 3.0
Enhancements:
https://www.elastic.co/guide/en/elasticsearch/hadoop/7.12/eshadoop-7.12.0.html
Maven Repository:
https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-30_2.12/7.12.0
It is not official for now, but you can compile the dependency on
https://github.com/elastic/elasticsearch, the steps are
git clone https://github.com/elastic/elasticsearch.git
cd elasticsearch-hadoop/
vim ~/.bashrc
export JAVA8_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
source ~/.bashrc
./gradlew elasticsearch-spark-30:distribution --console=plain
and finally you can find .jar package in folder: "elasticsearch-hadoop\spark\sql-30\build\distributions", elasticsearch-spark-30_2.12-8.0.0-SNAPSHOT.jar is the es packages
Currently, I have installed Spark 1.5.0 version on AWS using spark-ec2.sh script.
Now, I want to upgrade my Spark version to 1.5.1. How do i do this? Is there any upgrade procedure or do i have to build it from scratch using the spark-ec2 script? In that case i will lose all my existing configuration.
Please Advise
Thanks
1.5.1 has identical configuration fields with the 1.5.0, I am not aware of any automation tools, but upgrade should be trivial. C/P $SPARK_HOME/conf should suffice. Back up the old files, nevertheless.
I am a new user to Maven, as I am trying to use it to build apache spark on amazon EC2 VMs. I have mannually installed java version 1.7.0 on the VMs. However as I was running the Maven, the following error occurs:
Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) on project spark-core_2.10: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.0:testCompile failed. CompileFailed
As I think the java version mismatch is the potential reason, causing the compiling problem. I opened up the pom file of the spark for maven tool, it has declared java related version in two seperate places:
<java.version>1.6</java.version>
and
<aws.java.sdk.version>1.8.3</aws.java.sdk.version>
What are the differences between these two versions?
Which one should be edited to solve the jave version mismatch?
It's two different things
<java.version>1.6</java.version>
is the java version used and
<aws.java.sdk.version>1.8.3</aws.java.sdk.version>
is the AWS SDK for Java version used.
The minumum requirement of AWS SDK 1.9 is Java 1.6+ so there is no compatibility issues.
I have tried installing Groovy plugin via STS's dashboard install feature in both 2.7 (the version I downloaded initially which was the latest version at the time). My colleague recommended downloading 2.5.2, the version he is using, so I did; however the Groovy plugin installer still fails. It starts with:
Cannot complete the install because of a conflicting dependency.
Software being installed: Groovy-Eclipse Feature 2.5.2.xx-20110808-1400-e36 (org.codehaus.groovy.eclipse.feature.feature.group 2.5.2.xx-20110808-1400-e36)
Windows 7 64B
groovy v 1.8.2
grails v 1.3.7
jvm 1.6.0_26
Greatly appreciate any hints/recommendations/ideas. THANK YOU!
Make sure that your STS install is in a directory where the current user has write permissions. Also, there is a problem with installing into the Program Files directory (it is not really writable, even if you think it is).
The reason for this problem is the feature patch that comes with Groovy-Eclipse must be installed into the same directory as the rest of STS (an Eclipse limitation). This patch is the thing that patches the jdt compiler so that it can also compile groovy code.