Running a distributed application on Spark/YARN, I get the following error that kills an executor, and eventually kills the entire application:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f4f85ab41b1, pid=3309, tid=0x00007f4f90a4e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libSalience6.so+0x7631b1] lxaArrayTrie::Get(std::string const&) const+0x71
#
# Core dump written. Default location: /data/hadoop/yarn/local/usercache/koverse/appcache/application_1537930191769_0049/container_e08_1537930191769_0049_01_000016/core or core.3309
#
# An error report file with more information is saved as:
# /data/hadoop/yarn/local/usercache/koverse/appcache/application_1537930191769_0049/container_e08_1537930191769_0049_01_000016/hs_err_pid3309.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
There is a segmentation fault in libSalience6.so. So far so good!
But neither the core dump or logs files are where they say they are.
This error occurred on slv004 of a cluster, so the yarn application directory
/data/hadoop/yarn/local/usercache/koverse/appcache/application_1537930191769_0049
exists on that node. But a the container directory does not exist, and a find detects no log files.
Any ideas where this log file might be?
You probably have log aggregation enabled. If that's the case, log files are preserved in HDFS in TFile format.
You can check these logs using Application history server. This web app is accessible on port 8188 by default.
Try this
yarn logs -applicationId application_1537930191769_0049
Related
I am getting the following error message when I start logstash in Windows : Error: Could not find or load main class Stack\Elastic
Starting logstash with the command:
bin\logstash -e "input { stdin { } } output { stdout { } }"
JDK is installed properly on my system. java -version gives the following :
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
I am using logstash version 7.6.1.
The solution is rather funny. I have a folder named ELK Stack inside which I have logstash folder. The spacing between "ELK" and "Stack" was causing me the issue even I though I properly routed to the logstash folder and fired the above command. I did not face such issues while starting elastic search and kibana.
I've been at it for days but could not solve my problem.
I am running:
mpiexec -hostfile ~/machines -nolocal -pernode mkdir -p $dstpath where $dstpath points to current directory and "machines" is a file containing:
node01
node02
node03
node04
This is the error output:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:
* not finding the required libraries and/or binaries on
one or more nodes. Please check your PATH and LD_LIBRARY_PATH
settings, or configure OMPI with --enable-orterun-prefix-by-default
* lack of authority to execute on one or more specified nodes.
Please verify your allocation and authorities.
* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
Please check with your sys admin to determine the correct location to use.
* compilation of the orted with dynamic libraries when static are required
(e.g., on Cray). Please check your configure cmd line and consider using
one of the contrib/platform definitions for your system type.
* an inability to create a connection back to mpirun due to a
lack of common network interfaces and/or no route found between
them. Please check network connectivity (including firewalls
and network routing requirements).
--------------------------------------------------------------------------
[node01:06177] 1 more process has sent help message help-errmgr-base.txt / failed-daemon-launch
[node01:06177] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06181] [[6417,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
I have 4 machines, node01 to node04. In order to log into these 4 nodes, I have to first log in to node00. I am trying to run some distributed graph functions. The graph software is installed in node01 and is supposed to be synchronised to the other nodes using mpiexec.
What I've done:
Made sure all passwordless login are setup, every machine can ssh to any other machine with no issues.
Have a hostfile in the home directory.
echo $PATH gives /home/myhome/bin:/home/myhome/.local/bin:/usr/include/openmpi:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
echo $LD_LIBRARY_PATH gives
/usr/lib/openmpi/lib
This has previously worked before, but it just suddenly started giving these errors. I got my administrator to install fresh machines but it still gave such errors. I've tried doing it one node at a time but it gave the same errors. I'm not entirely familiar with command line at all so please give me some suggestions. I've tried reinstalling OpenMPI from source and from sudo apt-get install openmpi-bin. I'm on Ubuntu 16.04 LTS.
You should focus on fixing:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
I am trying to deploy a phoenix app on a Ubuntu Server with EXRM.
The release runs perfectly and the website is accessible but when I ping the release it says the
Node 'myapp#myhost' not responding to pings.
vm.args file
## Name of the node
-sname pxblog
## Cookie for distributed erlang
-setcookie pxblog
## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart
## Enable kernel poll and a few async threads
##+K true
##+A 5
## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096
## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10
Updated vm.args (Solved)
## Name of the node
-sname pxblog#localhost
## Cookie for distributed erlang
-setcookie pxblog
## Heartbeat management; auto-restarts VM if it dies or becomes unresponsive
## (Disabled by default..use with caution!)
##-heart
## Enable kernel poll and a few async threads
##+K true
##+A 5
## Increase number of concurrent ports/sockets
##-env ERL_MAX_PORTS 4096
## Tweak GC to run more often
##-env ERL_FULLSWEEP_AFTER 10
Check the vm.args file. Look for a line similar to this:
## Name of the node
-name test#127.0.0.1
I suspect the name you'll find there is "myapp#myhost". Try changing it to yourappname#localhost or yourappname#127.0.0.1. NB: I do not mean you should put the literal string yourappname there. Substitute the name of your app.
I have novice to cassandra and tried my hands to install cassandra-2.1.2 on centos 7.0.
After complete installation execute cqlsh command and created few keyspace(s) and column family.
Which seems to me in first glance its working perfectly.
But later onwards i realized below issues:
1- when i execute "service cassandra status" command, i got below error:
Output:Cassandra dead but pid file exists.
I googled the above issue and found some links
http://www.datastax.com/support-forums/topic/dse-dead-but-pid-file-exists
https://baioradba.wordpress.com/2014/06/13/how-to-install-cassandra-on-centos-6-5/
and found that I had same configuration mentioned in above links but the same error still persists.
Please tell me the root cause and how to resolve it.
2- Second issue is in the cassandra.log file.
When I analysed the cassandra.log file there was an expection as :
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Below is the complete log:
12:01:40.816 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:133) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:110) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554) [apache-cassandra-2.1.3.jar:2.1.3]
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Fatal configuration error; unable to start. See log for stacktrace.
I again searched the same issue in google and but the links were not that useful as they contained the java class code for cassandra.config .
Again please tell the root cause and how to resolve it?
Thanks in advance.
rm /var/run/cassandra.pid
Run ps -ef | grep cassandra
Kill the pid of the cassandra process.
Start cassandra
fix this issue, Edit the cassandra-env.sh:
sudo vi /etc/cassandra/conf/cassandra-env.sh
increase heap size for cassandra .. this should resolve your issue
Check if you have enough memory to start cassandra service with this command:
cat /proc/meminfo
I was running Hortonworks VM with Virtualbox, and I had a lot of Hadoop components started which needed a lot of memory, so for me the solution was to stop unnecessary Hadoop components and add some extra memory to the virtual machine.
From https://github.com/apache/cassandra/blob/cassandra-2.1/examples/client_only/README.txt#L43-L49 :
cassandra.yaml can be on the classpath as is done here, can be
specified (by modifying the script) in a location within the classpath
like this: java -Xmx1G
-Dcassandra.config=/path/in/classpath/to/cassandra.yaml ... or can be retrieved from a location outside the classpath like this: ...
-Dcassandra.config=file:///path/to/cassandra.yaml ... or ... -Dcassandra.config=http://awesomesauce.com/cassandra.yaml ...
So you probably had a misconfigured startup option.
Remove the pid file. Try
rm /var/run/cassandra.pid
I have build JavaFx code, deployed and wrapped into self contained JavaFX application using Ant build.xml on Windows 7, x64 Desktop Machine. Please see below for more system information -
Operating System: Windows 7 Professional 64-bit (6.1, Build 7600) (7600.win7_gdr.110622-1503)
Language: English (Regional Setting: English)
System Manufacturer: INTEL_
System Model: DH61WW__
BIOS: BIOS Date: 02/18/11 15:38:44 Ver: 04.06.04
Processor: Intel(R) Core(TM) i3-2100 CPU # 3.10GHz (4 CPUs), ~3.1GHz
Memory: 4096MB RAM
Available OS Memory: 4074MB RAM
When I am running my native bundle JavaFX application on any Desktop x64 Machine, it works absolutely fine. But when I am run the same native bundle JavaFX application on Virtual Machine, it gets started/launched successfully. But on clicking any button or closing the stage, it shows (Not Responding) and throws a fatal error. And finally application gets crashed.
Virtual Machine's system Information -
Operating System: Windows Web Server 2008 R2 64-bit (6.1, Build 7601) Service Pack 1 (7601.win7sp1_gdr.110622-1506)
Language: English (Regional Setting: English)
System Manufacturer: Microsoft Corporation
System Model: Virtual Machine
BIOS: BIOS Date: 05/05/08 20:35:56 Ver: 08.00.02
Processor: Intel(R) Core(TM)2 Quad CPU Q9400 # 2.66GHz, ~2.7GHz
Memory: 3072MB RAM
Available OS Memory: 3072MB RAM
Page File: 2457MB used, 3683MB available
You can download the attached Error File. For quick look -
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6dd5fe2d, pid=4016, tid=4292
#
# JRE version: 7.0_10-b18
# Java VM: Java HotSpot(TM) Client VM (23.6-b04 mixed mode windows-x86 )
# Problematic frame:
# C [glass.dll+0xfe2d] Java_com_sun_glass_events_KeyEvent__1getKeyCodeForChar+0x134d
#
# Core dump written. Default location: C:\Users\apancholi\Desktop\JavaFXJavaHeapMemoryFixes\JavaHeapSample-201302250\app\hs_err_pid4016.mdmp
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Please suggest what is happening here?
FYI: I am passing these VM arguments : -Xms500m -Xmx1g -XX:NewRatio=2 -XX:MaxPermSize=250m
There are next possibilities:
there is an issue with virtual environment
Windows Web Server 2008 R2 64-bit may not be supported, see list of the supported platforms here: http://www.oracle.com/technetwork/java/javafx/downloads/supportedconfigurations-1506746.html
In any case it's hardly possible to find out crash reason for non-developers of the product. Please, file an issue at http://javafx-jira.kenai.com
I also had that problem in my application then I install "Desktop Experience" under Server Manager in Windows 2008 R2. then I could solve my problem
Here are the steps to do it:
Go to Server Manager
Expand Features and Click add features
select Desktop Experience and install it
It will fix your problem