I am trying to install CDH5 (cloudera) on a single node(64-bit machine) running ubuntu 14.04 LTS. I have successfully installed hdfs (namenode and datanode are running fine). After configuring all the properties for deploying yarn, when I try to start resource manager or node manager it gives gives following errors:
ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Returning, interrupted : java.lang.InterruptedException
2015-06-29 14:02:44,265 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2015-06-29 14:02:44,264 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted
2015-06-29 14:02:44,265 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2015-06-29 14:02:44,265 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:278)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:983) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1083)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1215)
Caused by: java.io.IOException: Problem in starting http server. Server handlers failed
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:841)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:273)
... 4 more
Any kind of help will be appreciated.
Related
We have a 15 node cassandra setup across 3 DC's.
Using cassandra 3.0.9
One of the nodes in one of our DCs has died with the below startup error
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF 111111-111111111-11111111
What we have tried -
Bootstrap replacing the node -https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsReplaceNode.html
This works for a while and then the bootstrap process hangs with the SAME error in the logs -
CassandraDaemon.java:709 - Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF
Provision a new blank node,
https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html
and try to join this to the cluster (with the intention of removing the dead node after). It refuses to start, with the same error.
has anyone ever ran across this before?
Please help me how I can stop Cassandra update automatically?
At present when I install Cassandra through opscentre I get an error.
Unable to restart DSE service. See /var/log/cassandra/system.log and /var/log/cassandra/output.log on the target node for details.
system.log
ERROR [main] 2018-03-28 07:58:26,123 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135) ~[na:1.8.0_162]
at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405) ~[na:1.8.0_162]
at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:471) ~[dse-core-5.1.0.jar:5.1.0]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:599) ~[cassandra-all-3.10.0.1652.jar:3.10.0.1652]
at com.datastax.bdp.DseModule.main(DseModule.java:93) [dse-core-5.1.0.jar:5.1.0]
But when I update my OS with the command **yum update**, it updates DSE 5.1.0 to 5.1.7. After updating dse start. I am not able to understand why its happening?
On other node dse 5.1 working perfectly.
OS: centos 7
I have a listener project implemented on java that consumes the data from kafka and pushes into the Vanilla Chronicle queue. I am using the java libraries of jna, jna-platform version 4.4.0 and chronicle queue version 3.6.4. When I run the project on Windows it successfully consumes and pushes the data into the queue. However when I run it on "Ubuntu 16.04.3 LTS" that is located on Azure VM while pushing the data into the queue it fails. And on also when I run it on my local ubuntu that is in VMware it also succesfully runs. Anyone has an idea what the problem is? The execution log is below:
*WARN [main] (net.openhft.affinity.Affinity:149):
Linux JNA-based affinity not usable because it failed to load! Reason: java.lang.ExceptionInInitializerError
at net.openhft.affinity.Affinity.isLinuxJNAAffinityUsable(Affinity.java:113)
at net.openhft.affinity.Affinity.<clinit>(Affinity.java:52)
at net.openhft.affinity.AffinitySupport.getThreadId(AffinitySupport.java:26)
at net.openhft.chronicle.VanillaChronicle$VanillaAppenderImpl.startExcerpt(VanillaChronicle.java:628)
at net.openhft.chronicle.VanillaChronicle$VanillaAppenderImpl.startExcerpt(VanillaChronicle.java:615)
at com.myproject.queue.DataQueueManager.add(DataQueueManager.java:55)
at com.myproject.listener.myprojectListener.initKafka(myprojectListener.java:67)
at com.myproject.listener.myprojectListener.start(myprojectListener.java:43)
at com.myproject.listener.myprojectListener.main(myprojectListener.java:35)
Caused by: java.lang.IllegalStateException: sched_getaffinity(0, (8) , cpuset) failed; errno=22
at net.openhft.affinity.impl.LinuxHelper.sched_getaffinity(LinuxHelper.java:61)
at net.openhft.affinity.impl.LinuxJNAAffinity.getAffinity(LinuxJNAAffinity.java:62)
at net.openhft.affinity.impl.LinuxJNAAffinity.<clinit>(LinuxJNAAffinity.java:49)
... 9 more
Caused by: com.sun.jna.LastErrorException: [22] H"
at com.sun.jna.Native.invokeInt(Native Method)
at com.sun.jna.Function.invoke(Function.java:390)
at com.sun.jna.Function.invoke(Function.java:323)
at com.sun.jna.Library$Handler.invoke(Library.java:236)
at net.openhft.affinity.impl.$Proxy7.sched_getaffinity(Unknown Source)
at net.openhft.affinity.impl.LinuxHelper.sched_getaffinity(LinuxHelper.java:56)
... 11 more
07.09.2017 12:58:28.631 WARN [main] (net.openhft.affinity.Affinity:149):
Posix JNA-based affinity not usable because it failed to load! Reason: java.lang.ExceptionInInitializerError
at net.openhft.affinity.Affinity.isPosixJNAAffinityUsable(Affinity.java:99)
at net.openhft.affinity.Affinity.<clinit>(Affinity.java:56)
at net.openhft.affinity.AffinitySupport.getThreadId(AffinitySupport.java:26)
at net.openhft.chronicle.VanillaChronicle$VanillaAppenderImpl.startExcerpt(VanillaChronicle.java:628)
at net.openhft.chronicle.VanillaChronicle$VanillaAppenderImpl.startExcerpt(VanillaChronicle.java:615)
at com.myproject.queue.DataQueueManager.add(DataQueueManager.java:55)
at com.myproject.listener.myprojectListener.initKafka(myprojectListener.java:67)
at com.myproject.listener.myprojectListener.start(myprojectListener.java:43)
at com.myproject.listener.myprojectListener.main(myprojectListener.java:35)
Caused by: java.lang.IllegalStateException: sched_getaffinity((4) , &(allocated#0x7f2ba436c7e0 (4 bytes) (com.sun.jna.ptr.IntByReference#a437470b)) ) errorNo=22
at net.openhft.affinity.impl.PosixJNAAffinity.getAffinity(PosixJNAAffinity.java:125)
at net.openhft.affinity.impl.PosixJNAAffinity.<clinit>(PosixJNAAffinity.java:64)
... 9 more
Caused by: com.sun.jna.LastErrorException: [22] ý
at com.sun.jna.Native.invokeInt(Native Method)
at com.sun.jna.Function.invoke(Function.java:390)
at com.sun.jna.Function.invoke(Function.java:323)
at com.sun.jna.Library$Handler.invoke(Library.java:236)
at net.openhft.affinity.impl.$Proxy8.sched_getaffinity(Unknown Source)
at net.openhft.affinity.impl.PosixJNAAffinity.getAffinity(PosixJNAAffinity.java:117)
... 10 more
07.09.2017 12:58:28.633 INFO [main] (net.openhft.affinity.Affinity:61):
Using dummy affinity control implementation
07.09.2017 12:58:28.634 ERROR [main] (com.myproject.listener.myprojectListener:71):
java.lang.UnsupportedOperationException*
Caused by: com.sun.jna.LastErrorException: [22] H"
Error code 22 from the linux source is EINVAL [1].
Looking at the man page for the function sched_getaffinity shows the reason for EINVAL being returned:
EINVAL (sched_getaffinity() and, in kernels before 2.6.9, sched_setaffinity()) cpusetsize is smaller than the size of the affinity mask used by the kernel.
This could be an issue in the library, but is more likely due to the runtime environment and the way that the VM infrastructure is reporting/using CPU masks.
As Peter has pointed out, the value of setting process affinity in a virtualised environment is questionable.
1) http://elixir.free-electrons.com/linux/v4.4.84/source/include/uapi/asm-generic/errno-base.h#L25
I use Spark 1.6.1, Hadoop 2.6.4 and Mesos 0.28 on Debian 8.
While trying to submit a job via spark-submit to a Mesos cluster a slave fails with the following in stderr log:
I0427 22:35:39.626055 48258 fetcher.cpp:424] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/ad642fcf-9951-42ad-8f86-cc4f5a5cb408-S0\/hduser","items":[{"action":"BYP$
I0427 22:35:39.628031 48258 fetcher.cpp:379] Fetching URI 'hdfs://xxxxxxxxx:54310/sources/spark/SimpleEventCounter.jar'
I0427 22:35:39.628057 48258 fetcher.cpp:250] Fetching directly into the sandbox directory
I0427 22:35:39.628078 48258 fetcher.cpp:187] Fetching URI 'hdfs://xxxxxxx:54310/sources/spark/SimpleEventCounter.jar'
E0427 22:35:39.629243 48258 shell.hpp:93] Command 'hadoop version 2>&1' failed; this is the output:
sh: 1: hadoop: not found
Failed to fetch 'hdfs://xxxxxxx:54310/sources/spark/SimpleEventCounter.jar': Failed to create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was e$
Failed to synchronize with slave (it's probably exited)
My Jar file contains hadoop 2.6 binaries
The path to spark executor/binary is via an hdfs:// link
My jobs don't appear in the framework tab, but they do appear in the driver with the status 'queued' and they just sit there till I shut down the spark-mesos-dispatcher.sh service.
I was seeing a very similar error and I figured out my problem was that hadoop_home wasn't set in the mesos agent.
I added to /etc/default/mesos-slave (path may be different on your install) on each mesos-slave the following line: MESOS_hadoop_home="/path/to/my/hadoop/install/folder/"
EDIT: Hadoop has to be installed on each slave, the path/to/my/haoop/install/folder is a local path
I am running Linux in VirtualBox and am having an issue that I did not encounter on my machine with Linux as the primary OS.
When launching the neo4j service through sudo ./neo4j start in /opt/neo4j-community-2.3.1/bin I get a timeout with the message Failed to start within 120 seconds. Neo4j Server may have failed to start, please check the logs
my log from /opt/neo4j-community-2.3.1/data/graph.db/messages.log says:
http://pastebin.com/wUA715QQ
and data/log/console.log says:
2016-01-06 02:07:03.404+0100 INFO Successfully started database
2016-01-06 02:07:03.603+0100 INFO Successfully stopped database
2016-01-06 02:07:03.604+0100 INFO Successfully shutdown Neo4j Server
2016-01-06 02:07:03.608+0100 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception. Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:67)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:234)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:97)
at org.neo4j.server.CommunityBootstrapper.start(CommunityBootstrapper.java:48)
at org.neo4j.server.CommunityBootstrapper.main(CommunityBootstrapper.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.security.auth.FileUserRepository#9ab182' was successfully initialized, but failed to start. Please see attached cause exception.
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:462)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:194)
... 3 more
Caused by: java.nio.file.AccessDeniedException: /opt/neo4j-community-2.3.1/data/dbms/auth
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.neo4j.server.security.auth.FileUserRepository.loadUsersFromFile(FileUserRepository.java:208)
at org.neo4j.server.security.auth.FileUserRepository.start(FileUserRepository.java:73)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
... 5 more
Any idea why the server won't start?
Check the permissions on /opt/neo4j-community-2.3.1/data/dbms/auth
See the line that says:
Caused by: java.nio.file.AccessDeniedException: /opt/neo4j-community-2.3.1/data/dbms/auth