Cassandra dead but pid file exists - cassandra

I have novice to cassandra and tried my hands to install cassandra-2.1.2 on centos 7.0.
After complete installation execute cqlsh command and created few keyspace(s) and column family.
Which seems to me in first glance its working perfectly.
But later onwards i realized below issues:
1- when i execute "service cassandra status" command, i got below error:
Output:Cassandra dead but pid file exists.
I googled the above issue and found some links
http://www.datastax.com/support-forums/topic/dse-dead-but-pid-file-exists
https://baioradba.wordpress.com/2014/06/13/how-to-install-cassandra-on-centos-6-5/
and found that I had same configuration mentioned in above links but the same error still persists.
Please tell me the root cause and how to resolve it.
2- Second issue is in the cassandra.log file.
When I analysed the cassandra.log file there was an expection as :
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Below is the complete log:
12:01:40.816 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:133) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:110) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554) [apache-cassandra-2.1.3.jar:2.1.3]
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Fatal configuration error; unable to start. See log for stacktrace.
I again searched the same issue in google and but the links were not that useful as they contained the java class code for cassandra.config .
Again please tell the root cause and how to resolve it?
Thanks in advance.

rm /var/run/cassandra.pid
Run ps -ef | grep cassandra
Kill the pid of the cassandra process.
Start cassandra

fix this issue, Edit the cassandra-env.sh:
sudo vi /etc/cassandra/conf/cassandra-env.sh
increase heap size for cassandra .. this should resolve your issue

Check if you have enough memory to start cassandra service with this command:
cat /proc/meminfo
I was running Hortonworks VM with Virtualbox, and I had a lot of Hadoop components started which needed a lot of memory, so for me the solution was to stop unnecessary Hadoop components and add some extra memory to the virtual machine.

From https://github.com/apache/cassandra/blob/cassandra-2.1/examples/client_only/README.txt#L43-L49 :
cassandra.yaml can be on the classpath as is done here, can be
specified (by modifying the script) in a location within the classpath
like this: java -Xmx1G
-Dcassandra.config=/path/in/classpath/to/cassandra.yaml ... or can be retrieved from a location outside the classpath like this: ...
-Dcassandra.config=file:///path/to/cassandra.yaml ... or ... -Dcassandra.config=http://awesomesauce.com/cassandra.yaml ...
So you probably had a misconfigured startup option.

Remove the pid file. Try
rm /var/run/cassandra.pid

Related

sstableloader remote bulk upload

I'm trying to figure out how to upload data from a snapshot and why I'm getting this error on the bulk upload.
The local machine is trying to connect to cassandra.mydomain.com. The cassandra.yaml is the yaml from the remote server. I'm getting the same error with and without specifying --conf-path
Thanks for any advice.
cassandra version 3.11.2
~/deploy/cassandra/bin/sstableloader -d cassandra.mydomain.com --conf-path /tmp/cassandra.yaml /local/.data/cassandra/data/test/timeserie_time_daily-dd247b092e883bffbfce8621eff3cc3e/snapshots/1634621703263
10:10:50.138 [main] DEBUG o.a.c.config.YamlConfigurationLoader - Loading settings from file:/tmp/cassandra.yaml
Exception in thread "main" org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Found[cassandra.yaml]. Please prefix the file with [file:///] for local files and [file://<server>/] for remote file
s. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:80)
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100)
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:262)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:180)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:151)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:53)
at o
rg.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
As the exception states, you need to provide the correct URL to the cassandra.yaml.
If you're using a YAML that's on the local machine, you need to prefix it with file:///. For example:
$ sstableloader -f file:///path/to/cassandra.yaml
-d node1
ks_name/table_name
If you're specifying a YAML that's on a remote machine, you need to prefix it with file://host/. For example:
$ sstableloader -f file://hostname_or_ip/path/to/cassandra.yaml
-d node1
ks_name/table_name

Error while trying to cleanup Cassandra node

I am facing issue while launching cleanup command with nodetool.
Cleanup did work fine until now. I did'nt find any modification on my configuration. I have no clue on what could have change.
nodetool > cleanup
error: Expecting URI in variable: [cassandra.config]. Found[cassandra.yaml]. Please prefix the file with [file:///] for local files and [file://<server>/] for remote files. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Found[cassandra.yaml]. Please prefix the file with [file:///] for local files and [file://<server>/] for remote files. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:80)
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100)
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:262)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:180)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:151)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:281)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:288)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:255)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:169)
Any idea ?
Regards,
Nicolas
Nodetool uses cassandra.yaml to to find the number of concurrent compactors. Since you have cassandra.config set its using that cassandra.yaml, but the cassandra.config has an invalid value so the nodetool is choking on it.
I did find a solution that works for me (but i think it's specific to my prod environment).
Somewhere, something should have used /etc/alternative/cassandra symlink, which was heading to my empty default cassandra configuration.
After i correct it :
[root#myhost ~]# ll /etc/alternatives/
total 116
lrwxrwxrwx 1 root root 30 Oct 2 12:40 cassandra -> /etc/cassandra/my-cassandra-instance
That done, i was able to cleanup my cassandra clusters.
Didn't find time to work this case much more.
Anyway thanks for the clues, it was a context issue.
Nicolas

OpenMPI: ORTE was unable to reliably start one or more daemons

I've been at it for days but could not solve my problem.
I am running:
mpiexec -hostfile ~/machines -nolocal -pernode mkdir -p $dstpath where $dstpath points to current directory and "machines" is a file containing:
node01
node02
node03
node04
This is the error output:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:
* not finding the required libraries and/or binaries on
one or more nodes. Please check your PATH and LD_LIBRARY_PATH
settings, or configure OMPI with --enable-orterun-prefix-by-default
* lack of authority to execute on one or more specified nodes.
Please verify your allocation and authorities.
* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
Please check with your sys admin to determine the correct location to use.
* compilation of the orted with dynamic libraries when static are required
(e.g., on Cray). Please check your configure cmd line and consider using
one of the contrib/platform definitions for your system type.
* an inability to create a connection back to mpirun due to a
lack of common network interfaces and/or no route found between
them. Please check network connectivity (including firewalls
and network routing requirements).
--------------------------------------------------------------------------
[node01:06177] 1 more process has sent help message help-errmgr-base.txt / failed-daemon-launch
[node01:06177] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06181] [[6417,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
I have 4 machines, node01 to node04. In order to log into these 4 nodes, I have to first log in to node00. I am trying to run some distributed graph functions. The graph software is installed in node01 and is supposed to be synchronised to the other nodes using mpiexec.
What I've done:
Made sure all passwordless login are setup, every machine can ssh to any other machine with no issues.
Have a hostfile in the home directory.
echo $PATH gives /home/myhome/bin:/home/myhome/.local/bin:/usr/include/openmpi:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
echo $LD_LIBRARY_PATH gives
/usr/lib/openmpi/lib
This has previously worked before, but it just suddenly started giving these errors. I got my administrator to install fresh machines but it still gave such errors. I've tried doing it one node at a time but it gave the same errors. I'm not entirely familiar with command line at all so please give me some suggestions. I've tried reinstalling OpenMPI from source and from sudo apt-get install openmpi-bin. I'm on Ubuntu 16.04 LTS.
You should focus on fixing:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891

Starting Cassandra on the foreground

If I start the Cassandra service everything is ok, but when I try to start Cassandra on the foreground using "cassandra -f" I get the following error:
Error: Could not find or load main class
Files\DataStax-DDC\apache-cassandra.logs.gc.log
Do I need to configure anything in particular to run Cassandra in the foreground?
Looks like the space in your "Program Files" directory is not escaped in your CASSANDRA_HOME environment variable. It gets set in your cassandra-env.ps1 (in conf/) config file, you could manually set it.

DataStax Opscenter Agent doesn't connect: "Unable to locate cassandra.yaml configuration file"

I have installed Datastax Community Edition v2.0.3 on a Windows Server 2012 R2 Datacenter machine.
After a vanilla install, I can't connect to the database with OpsCenter because the agent failed to start: it says "0 of 1 agents connected" in the top of the page. I can connect to the database itself using CQLSH. This problem does not occur with v2.0.2.
My datastax_opscenter_agent-stderr.log file contains the following:
2013-12-05 16:16:54 Commons Daemon procrun stderr initialized
Exception in thread "Initialization" clojure.lang.ExceptionInfo: throw+: {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\" \"/etc/cassandra/conf/cassandra.yaml\" \"/etc/cassandra/cassandra.yaml\" \"C:\\\\Program Files\\\\DataStax Community\\\\apache-cassandra\\\\bin\\\\conf\\\\cassandra.yaml\"]"} {:object {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\" \"/etc/cassandra/conf/cassandra.yaml\" \"/etc/cassandra/cassandra.yaml\" \"C:\\\\Program Files\\\\DataStax Community\\\\apache-cassandra\\\\bin\\\\conf\\\\cassandra.yaml\"]"}, :environment {tar-location "C:\\Program Files\\DataStax Community\\apache-cassandra\\bin\\conf\\cassandra.yaml", conf nil, checked-files ["/etc/dse/cassandra/cassandra.yaml" "/etc/cassandra/conf/cassandra.yaml" "/etc/cassandra/cassandra.yaml" "C:\\Program Files\\DataStax Community\\apache-cassandra\\bin\\conf\\cassandra.yaml"]}}
at opsagent.util.cassandra_util$cassandra_conf_location.invoke(cassandra_util.clj:118)
at opsagent.util.cassandra_util$get_cassandra_conf.invoke(cassandra_util.clj:130)
at opsagent.opsagent$create_thrift_conf_vars.invoke(opsagent.clj:52)
at opsagent.opsagent$setup_thrift.invoke(opsagent.clj:138)
at opsagent.jmx$determine_ip.invoke(jmx.clj:333)
at opsagent.jmx$setup_jmx$fn__1309.invoke(jmx.clj:350)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
There is in fact a cassandra.yamlfile in the C:\Program Files\DataStax Community\apache-cassandra\conf directory, where it is supposed to be. The message above states that it looked in C:\\Program Files\\DataStax Community\\apache-cassandra\\bin\\conf\\cassandra.yaml, though (notice the \bin).
Setting the conf_location option in the opscenter\conf\clusters\local.conf, as per the log's suggestion, results in the same error.
The Windows Firewall has been turned off through all this.
Edit
I've been tinkering a little bit, and now I get the same exception in the log, except the path where it's been looking for cassanda.yaml has changed from
'C:\Program Files\DataStax Community\apache-cassandra\bin\conf\cassandra.yaml'
to
'C:\Program Files\DataStax Community\utils\conf\cassandra.yaml'
I have no idea what I did to change this.
I have tried copying cassandra.yaml to the directory above. This gets rid of the exception in datastax_opscenter_agent-stderr.log, but OpsCenter still says "0 of 1 agents connected". However, the Dashboard is now showing data in the graphs. I am also able to add a keyspace to the schema, but it won't show up after I created it. It is accessible using CLQSH, though. Obviously having cassandra.yaml in two different places is not ideal, either.
I fixed this problem by setting the conf_location property as below:
conf_location = C:\Program Files\DataStax Community\apache-cassandra\conf\cassandra.yaml
Initially I was using quotes around the path, which didn't work. Removing the quotes did the trick.
I had the same problem in Windows 7 x64 while installing DataStax Community Edition v1.2.15. I got it working and then after backup and rollback I used Beyond Compare to identify exactly what change actually fixed this. There is some ambiguity in khaledh's otherwise correct answer and in the equivalent SF question, thus I am providing a more complete answer.
In C:\Program Files\DataStax Community\opscenter\conf\clusters\local.conf, the default contents of ..
[cassandra]
seed_hosts = 127.0.0.1
.. should instead be ..
[cassandra]
seed_hosts = 127.0.0.1
conf_location = C:\\Program Files\\DataStax Community\\apache-cassandra\\conf\\cassandra.yaml
Restart the services and then in OpsCenter if it still says "0 of 1 agents" click on the "Fix now" link. In my case it was not until I clicked on this link that it self-updated and showed "1 of 1" without any prompting or alerts.
I fixed this issue by adding entry in the agent configuration file address.yaml,
Steps,
Login to Agent machine
Stop agent
Add the JAVA_HOME if not already added
Edit ( create if not exists ) address.yaml
( located at /etc/datastax-agent/ if you have installed OpsCenter Agents using package manager )
Add entry as
cassandra_conf:
Start the agent
Restart the opscenter - the new agent wonts show up until you restart opscenter
cassandra_conf: /usr/lib/cassandra-2.0.5/conf/cassandra.yaml
Create the if not exists
( For DataStax version 5.0.1 for both Opscenter and Agent )

Resources