I am using CentOS Linux release 7.9.2009 (Core) and this is the GC log file configuration
JAVA_OPTS_GC="-XX:+UseG1GC -XX:+DisableExplicitGC -Xmx8192m -Xms8192m -Xss512k -XX:MaxGCPauseMillis=100 -XX:ParallelGCThreads=5 -XX:+UnlockCommercialFeatures -verbose:gc -XX:+PrintGCCause -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=50M -Xloggc:absolute_path"
and passing above in java jar command , JDK version used is 8, please share your thoughts on this
Related
Heap dump genrated for the process but process not crashed. what could be the reason of heap dump genration and how can we identify it ?
Below are the jvm arguments.
-XX:CompressedClassSpaceSize=528482304 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=1977595776 -XX:MaxHeapSize=4294967296 -XX:MaxMetaspaceSize=536870912 -XX:MetaspaceSize=268435456 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:-UsePerfData -XX:+UseStringDeduplication
we need to idenify why heap dump is genrated without the OOM ?
I am trying to use the G1GC garbage collector for spark job but I get a
Error: Invalid argument to --conf: -XX:+UseG1GC
I tried using these options but haven't been able to get it working
spark-submit --master spark://192.168.60.20:7077 --conf -XX:+UseG1GC /appdata/bblite-codebase/test.py
and
spark-submit --master spark://192.168.60.20:7077 -XX:+UseG1GC /appdata/bblite-codebase/test.py
What is the correct way to call a G1GC collector from spark?
JVM options should be passed as spark.executor.extraJavaOptions / spark.driver.extraJavaOptions, ie.
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC"
This is how you can configure garbage collection setting in both driver and executor.
spark-submit --master spark://192.168.60.20:7077 \
--conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" \
/appdata/bblite-codebase/test.py
Starting with Spark 2.4.3, this will not work for the driver extraJavaOptions, which will produce an error of
Conflicting collector combinations in option list; please refer to the release notes for the combinations allowed
This is because the default spark-defaults.conf includes
spark.executor.defaultJavaOptions -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:+UseParallelGC -XX:InitiatingHeapOccupancyPercent=70
spark.driver.defaultJavaOptions -XX:OnOutOfMemoryError='kill -9 %p' -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled
which already includes a GC setting, and setting two GC options causes it to complain with this error. So you may need:
--conf "spark.executor.defaultJavaOptions=-XX:+UseG1GC"
--conf "spark.driver.defaultJavaOptions=-XX:+UseG1GC"
and also adding other defaults you'd like to propagate over.
Alternatively, you can edit the defaults in spark-defaults.conf to remove the GC defaults for driver/executor and force it to be specified in extraJavaOptions, depending on your use cases.
I submitted my jar file using this:
spark-submit \
--class Hello \
--master spark://master:7077 \
--num-executors 6 \
--conf spark.executor.extraJavaOptions="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedOops" \
first.jar
You could see that I added the -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedOops part as recommended in the official documentation. However, when I look at the slaves' stdoutand stderrin the web UI , I could see nothing about Garbage Collection.
Is it the log4j setting preventing showing of gc log? I only have the log4j.properties.template file inside my spark's conf directory.
Any suggestions on what's wrong? Thank you.
According to https://databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html - the GC Logs are written into:
$SPARK_HOME/work/$app_id/$executor_id/stdout
Try configuring your spark app with the settings suggested on this page and see if it works as expected.
you can add these jvm param to
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=20M
to
spark.executor.extraJavaOptions
it will start logging gc logs to
$SPARK_HOME/work/$app_id/$executor_id/gc.log.0.current
In Spark 1.2.0, when I attempt to start the Spark SQL thrift server in cluster mode, I get the following output:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Spark Command: /usr/java/latest/bin/java -cp ::/home/tpanning/Projects/spark/spark-1.2.0-bin-hadoop2.4/sbin/../conf:/home/tpanning/Projects/spark/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:/home/tpanning/Projects/spark/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/home/tpanning/Projects/spark/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/home/tpanning/Projects/spark/spark-1.2.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar -XX:MaxPermSize=128m -Xms512m -Xmx512m org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --deploy-mode cluster --master spark://xd-spark.xdata.data-tactics-corp.com:7077 spark-internal
========================================
Jar url 'spark-internal' is not in valid format.
Must be a jar file path in URL format (e.g. hdfs://host:port/XX.jar, file:///XX.jar)
Usage: DriverClient [options] launch <active-master> <jar-url> <main-class> [driver options]
Usage: DriverClient kill <active-master> <driver-id>
Options:
-c CORES, --cores CORES Number of cores to request (default: 1)
-m MEMORY, --memory MEMORY Megabytes of memory to request (default: 512)
-s, --supervise Whether to restart the driver on failure
-v, --verbose Print more debugging output
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
The "spark-internal" argument seems to be a special flag to tell spark-submit that the class to be run is part of Spark's libraries, so it doesn't need to distribute a jar. But for some reason, this doesn't seem to be working here.
I filed this as SPARK-5176 and it will be addressed with an error message that explains that the Thrift server can not run in cluster mode.
I encounter weird issue this with jboss as 7 running with jvm 1.7._029. When I turned OFF following from standalone.conf, jboss takes 7x longer to deploy as well as 7x times longer to serve request.
JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
Request that takes normally 2sec now takes around 15 - 20 sec! If I turned on above option than both jboss deployment and request serving becomes faster. We usually have it turned on in Dev so we never experienced it until we turned it off in staging.
Xms and Xmx settings have no effect on above obervations.
I did some profiling with VisualVM. Only a single instance of jboss and only one client thread making request. Heap profiling indicates that when debugger is turned on It preallocates good chunk of heap out of Xms setting upon boot. i.e lets say Xms is 1g then it says "Heap Size" 1G and "Used Heap" 750mb. From then on it starts jboss quickly and it flacuate very little when request is made sequentially.
While running with debugger turned off, initial "Heap Size" is still at 1G but "Used Heap" is pretty low and then it tries to allocate more memory midway during deployment and eventually after a while it completes deployment. Upon new request it does same thing again. i.e allocate and de-allocates memory at runtime.
java -XX:+PrintCommandLineFlags -XX:+PrintGCDetails -version
-XX:InitialHeapSize=268435456 -XX:MaxHeapSize=4294967296 -XX:+PrintCommandLineFlags -XX:+PrintGCDetails -XX:+UseCompressedOops -XX:+UseParallelGC
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
Heap
PSYoungGen total 76800K, used 2642K [0x00000007aaa80000, 0x00000007b0000000, 0x0000000800000000)
eden space 66048K, 4% used [0x00000007aaa80000,0x00000007aad14878,0x00000007aeb00000)
from space 10752K, 0% used [0x00000007af580000,0x00000007af580000,0x00000007b0000000)
to space 10752K, 0% used [0x00000007aeb00000,0x00000007aeb00000,0x00000007af580000)
ParOldGen total 174592K, used 0K [0x0000000700000000, 0x000000070aa80000, 0x00000007aaa80000)
object space 174592K, 0% used [0x0000000700000000,0x0000000700000000,0x000000070aa80000)
PSPermGen total 21504K, used 2119K [0x00000006fae00000, 0x00000006fc300000, 0x0000000700000000)
object space 21504K, 9% used [0x00000006fae00000,0x00000006fb011da0,0x00000006fc300000)
JBoss Process:
/usr/bin/java -D[Standalone] -server -XX:+UseCompressedOops -XX:+**TieredCompilation** -Dprogram.name=standalone.sh -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n -Dcom.sun.management.jmxremote.port=11090 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xms1024m -Xmx2048m -XX:MaxPermSize=512M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000