Apache log4j zookeeper uses log4j 1.2 which is vulnerable to RCE.
To rectify this issue we planned to exclude log4j 1.2 and include log4j 2.17.1 core and log4j 2.17.1 api in the dependency
It doesnt help. Can somebody please suggest how to exclude jars from third party libraries
Error:
Getting this errror :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:50)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:61)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:125)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.jmx.HierarchyDynamicMBean
at java.net.URLClassLoader.findClass(URLClassLoader.
We tried this ..
<dependencies>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.5.1-alpha</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
<version>2.17.1</version>
</dependency>
</dependencies>
I believe I figured it out but I haven't tested this for long enough.
Considering this was applied to a v3.6.1 Zookeeper server, a summary of what needs to be done is:
Delete old log4j libraries from Zookeeper
log4j-1.2.17.jar
log4j-1.2.17.LICENSE.txt (That's obviously not necessary)
Add recent log4j libraries that has the fix for the log4shell vulerability.
A log4j2 bridge that is backward compatible with log4j1.x: log4j-1.2-api-2.17.1.jar
Necessary log4j libraries: log4j-api-2.17.1.jar & log4j-core-2.17.1.jar
Modify Zookeeper's server environment options file (i.e. /zookeeper/conf/server_jvm.properties) by adding the following lines
-Dlog4j.configuration=/incorta/IncortaAnalytics/IncortaNode/zookeeper/conf/log4j.properties (A pointer for log4j2 to the existing log4j1.x configuration file, see the reference below for more details)
-Dzookeeper.jmx.log4j.disable=true (Disable Zookeeper's JMX dependency on log4j1.x. Thanks to Piotr for that tip he mentioned for this question)
What this does is that it keeps the sl4j libraries shipped with Zookeeper because changing those to a version that is log4j2 compatible wasn't a pleasant experience for me.
And instead, I upgraded log4j1.x libraries to log4j2 while having the log4j bridge library too to enable Zookeeper's outdated slf4j libraries to use the recent log4j2 ones.
Reference
Update: Using JDK 11, we faced a weird error where our Zookeeper client couldn't connect to Zookeeper, and the solution was to remove the slf4j-log4j12 binder from our classpath.
Zookeeper is apparently trying to directly access Log4j 1.2 internal classes, which no longer exist in log4j-1.2-api (cf. source code).
You can:
either set the system property zookeeper.jmx.log4j.disable to true
or upgrade to a newer version (e.g. 3.5.9), which will detect the absence of the HierarchyDynamicMBean class automatically.
You should upgrade anyway since the alpha version you are using has several security vulnerabilities: cf. Maven Repository.
The following dependency configuration seems to have worked for me:
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.7.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.17.1</version>
</dependency>
Related
I want to add <JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json" charset="UTF-8"/> to log4j2.xml for spark configuration.
but I don't have JsonTemplateLayout dependency. how can I add this dependency to spark?
You need to make sure that both log4j-core and log4j-json modules are included in your project. To use log4j with Apache Spark, you need to add the following dependencies to your project:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.12.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-json</artifactId>
<version>2.12.1</version>
</dependency>
Note that the version numbers above are the latest version of log4j, you may need to adjust the version numbers appropriately depending on the version of log4j you are using.
I'm using the HDP version 2.6.3 with the 2.2 version of Spark (not HDP cloud) and I'm trying to write to s3 from an IntelliJ project. I have no problems writing to the s3 bucket from the shell on one of my data nodes, but when I try to test my app on my local machine in IntelliJ I get an error (ERROR MetricsSystem: Sink class org.apache.spark.metrics.sink.MetricsServlet cannot be instantiated) after adding the Hadoop-aws jar dependency to my pom file. Does anyone know if there is any nuance to how you need to add this dependency? If I put the dependency above the spark dependencies in my pom I get different errors with missing spark classes, so it seems to matter what order you put it in.
I had the same problem, solved it by excluding the libraries of Jackson from the Hadoop's dependencies.
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion> <!-- declare the exclusion here -->
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion> <!-- declare the exclusion here -->
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion> <!-- declare the exclusion here -->
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</exclusion>
</exclusions>
</dependency>
I am creating an application using following dependencies, which are all the latest version so far.
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.M5</version>
</parent>
`<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.5</</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.3.0</version>
</dependency>`
I get this exception
"Caused by: java.lang.NoSuchMethodError: com.datastax.driver.core.TypeCodec.getJavaType()Lcom/google/common/reflect/TypeToken;"
After doing some research, I found the reason is that spark-cassandra-connector_2.10 jar also contains com.datastax.driver class files and it also has TypeCodeC.class but this is different from TypeCodeC.class file in cassandra-driver-core
I have 2 solutions so far.
use maven-shade-plugin to exclude class files from the jar. However, this requires a lot of extra work. And for some reason, only I compile the project to a jar and add this jar as a dependency in my project, then it works. I don't think this is a good solution
I remove /com/datastax/driver folder and files directly from the jar. Use this command
zip -d /Users/cicidi/.m2/repository/com/datastax/spark/spark-cassandra-connector_2.10/2.0.5/spark-cassandra-connector_2.10-2.0.5.jar /com/datastax/driver/*
And it works! Then you need to can add this jar to your project, instead of using maven.(you can use maven on your local, but won't work if you pull the jar from again. )
I don't find any answer on the internet. I know there will be some smart guys fixed this issue. But before that, I am posting this answer to help whoever want to fix this problem immediately.
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>2.0.0-M1</version>
</dependency>
has no version conflict with
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.3.0</version>
</dependency>
According to the question "Why is the Cassandra Java Driver embedded in Spark Cassandra Connector artifacts?" in https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md . It's difficult to use these two libraries together. But I try to avoid the problem by explicitly specify the dependencies in my pom.
There are some points to pay attention to:
The cassandra-driver-* must be put before the spark-cassandra-connector. And the 3.1.4 may only work fine with 2.0.0-M1.
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.1.4</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-mapping</artifactId>
<version>3.1.4</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-extras</artifactId>
<version>3.1.4</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>2.0.0-M1</version>
</dependency>
I have a Jetty 9 based application that I inherited. I am trying to put in some logging functionality. I am using slf4j with log4j2. I have added the appropriate jars for this using Maven:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.22</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.7</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.7</version>
</dependency>
Unfortunately, the application keeps failing to find the log4j2.xml file:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
I have placed copies of that XML file in the webapps folder, the main folder under my server, and in the folder where the libraries are -- all places on the web application's classpath. The application still doesn't find it.
Can someone please advise on how I can make this application find the XML file?
Please take a look at the Log4j2 manual page for web applications.
By default Log4j2 looks in the WEB-INF folder of your web application.
I have a groovy application that uses groovy version 2.2.1. My groovy app was previously running fine but has recently started throwing this exception:
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.app.Main.main(Main.groovy:83)Caused by: java.lang.ClassNotFoundException: org.codehaus.groovy.runtime.typehandling.ShortTypeHandling
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
The ShortTypeHandling class was not even introduced until groovy 2.3.0. How can it be referenced in a groovy app running version 2.2.1? I can solve this problem by replacing the groovy-all-2.2.1.jar with groovy-all-2.3.0.jar in my pom but that doesn't root cause the issue.
ShortTypeHandling was introduced in groovy-all-2.3.0.jar so the quick fix was to replace the older groovy-all-x.x.x.jar with groovy-all-2.3.0.jar. This solved the runtime ShorTypeHandling ClassNotFoundException but also created new problems by introducing a new groovy-all.jar dependency in the application.
The real issue was how the groovy compiler was being invoked via maven. Because I introduced spock which required groovy 2.0, I needed to update the maven entries for the groovy-eclipse-compiler dependency. Here are the updated maven entries for working with groovy 2.x:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
<!-- Java version -->
<source>1.7</source>
<target>1.7</target>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.8.0-01</version>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<!-- Groovy version -->
<version>2.1.8-01</version>
</dependency>
</dependencies>
</plugin>
With this in place, I could leave my groovy-all dependency the way I originally had it for the working/fully tested application like this:
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<!-- If possible, its better if this matches 2.1.8 in the plugin definition -->
<!-- but 2.2.1 worked fine here and allowed me to keep the original pom definition -->
<version>2.2.1</version>
</dependency>
The application runtime no longer references the ShortTypeHandling class and everything worked as it previously did.
You have to add (If you are using Gradle)
compile 'org.codehaus.groovy:groovy-backports-compat23:2.4.5'
I've just had this after updating the groovy-eclipse Feature in Eclipse (in order to try and fix intermittent save issues caused by https://jira.codehaus.org/browse/GRECLIPSE-1519). Specifically in my case, my Groovy JUnit tests were throwing this exception.
In light of the suggestions above, I checked my Eclipse settings, and it was using Groovy 2.3.4.xx whereas my Maven POM was specifying 2.1.8.xx. I went to Window -> Preferences -> Groovy -> Compiler and clicked "Switch to 2.1.8.xx...", restarting Eclipse when prompted, and this fixed it.
I've solved this issue by adding this dependency on my POM:
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-backports-compat23</artifactId>
<version>2.4.5</version>
</dependency>
Then, it works like a charm.
Matthew Wise's solution worked for me, but in addition to restarting eclipse, I also had to do a project -> clean for it to recompile with the new compiler.
(I would have commented on his answer, but stack overflow has this stupid rule that you can't comment until you get more reputation)
I faced similar issue in our project. Surprisingly groovy version was not an issue.
I was building the project with older version of gradle than the expected gradle version for the project. That resolved the error.
Add following dependency to your pom.xml
<dependency>
<groupId>org.codehaus.groovy.maven.runtime</groupId>
<artifactId>gmaven-runtime-default</artifactId>
<version>1.0-rc-3</version>
<exclusions>
<exclusion>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
</exclusion>
</exclusions>
</dependency>
Kuldeep Singh