Error while trying to cleanup Cassandra node - cassandra

I am facing issue while launching cleanup command with nodetool.
Cleanup did work fine until now. I did'nt find any modification on my configuration. I have no clue on what could have change.
nodetool > cleanup
error: Expecting URI in variable: [cassandra.config]. Found[cassandra.yaml]. Please prefix the file with [file:///] for local files and [file://<server>/] for remote files. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Found[cassandra.yaml]. Please prefix the file with [file:///] for local files and [file://<server>/] for remote files. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:80)
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:100)
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:262)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:180)
at org.apache.cassandra.config.DatabaseDescriptor.toolInitialization(DatabaseDescriptor.java:151)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:281)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:288)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:255)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:169)
Any idea ?
Regards,
Nicolas

Nodetool uses cassandra.yaml to to find the number of concurrent compactors. Since you have cassandra.config set its using that cassandra.yaml, but the cassandra.config has an invalid value so the nodetool is choking on it.

I did find a solution that works for me (but i think it's specific to my prod environment).
Somewhere, something should have used /etc/alternative/cassandra symlink, which was heading to my empty default cassandra configuration.
After i correct it :
[root#myhost ~]# ll /etc/alternatives/
total 116
lrwxrwxrwx 1 root root 30 Oct 2 12:40 cassandra -> /etc/cassandra/my-cassandra-instance
That done, i was able to cleanup my cassandra clusters.
Didn't find time to work this case much more.
Anyway thanks for the clues, it was a context issue.
Nicolas

Related

OpenMPI: ORTE was unable to reliably start one or more daemons

I've been at it for days but could not solve my problem.
I am running:
mpiexec -hostfile ~/machines -nolocal -pernode mkdir -p $dstpath where $dstpath points to current directory and "machines" is a file containing:
node01
node02
node03
node04
This is the error output:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:
* not finding the required libraries and/or binaries on
one or more nodes. Please check your PATH and LD_LIBRARY_PATH
settings, or configure OMPI with --enable-orterun-prefix-by-default
* lack of authority to execute on one or more specified nodes.
Please verify your allocation and authorities.
* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
Please check with your sys admin to determine the correct location to use.
* compilation of the orted with dynamic libraries when static are required
(e.g., on Cray). Please check your configure cmd line and consider using
one of the contrib/platform definitions for your system type.
* an inability to create a connection back to mpirun due to a
lack of common network interfaces and/or no route found between
them. Please check network connectivity (including firewalls
and network routing requirements).
--------------------------------------------------------------------------
[node01:06177] 1 more process has sent help message help-errmgr-base.txt / failed-daemon-launch
[node01:06177] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06181] [[6417,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891
I have 4 machines, node01 to node04. In order to log into these 4 nodes, I have to first log in to node00. I am trying to run some distributed graph functions. The graph software is installed in node01 and is supposed to be synchronised to the other nodes using mpiexec.
What I've done:
Made sure all passwordless login are setup, every machine can ssh to any other machine with no issues.
Have a hostfile in the home directory.
echo $PATH gives /home/myhome/bin:/home/myhome/.local/bin:/usr/include/openmpi:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
echo $LD_LIBRARY_PATH gives
/usr/lib/openmpi/lib
This has previously worked before, but it just suddenly started giving these errors. I got my administrator to install fresh machines but it still gave such errors. I've tried doing it one node at a time but it gave the same errors. I'm not entirely familiar with command line at all so please give me some suggestions. I've tried reinstalling OpenMPI from source and from sudo apt-get install openmpi-bin. I'm on Ubuntu 16.04 LTS.
You should focus on fixing:
Failed to parse XML input with the minimalistic parser. If it was not
generated by hwloc, try enabling full XML support with libxml2.
[node01:06177] [[6421,0],0] ORTE_ERROR_LOG: Error in file base/plm_base_launch_support.c at line 891

Starting Cassandra on the foreground

If I start the Cassandra service everything is ok, but when I try to start Cassandra on the foreground using "cassandra -f" I get the following error:
Error: Could not find or load main class
Files\DataStax-DDC\apache-cassandra.logs.gc.log
Do I need to configure anything in particular to run Cassandra in the foreground?
Looks like the space in your "Program Files" directory is not escaped in your CASSANDRA_HOME environment variable. It gets set in your cassandra-env.ps1 (in conf/) config file, you could manually set it.

Changing data file directories Cassandra

I'm trying to change the Cassandra data, commit log and saved caches directories by defining a custom shell script for CASANDRA_INCLUDE. I'm modifying the properties in the script as follows :
***
data_file_directories = "/usr/pic1/kearanky/cassandra/data"
commitlog_directory = "/usr/pic1/kearanky/cassandra/commitlog"
saved_caches_directory: "/usr/pic1/kearanky/cassandra/saved_caches"
***
When I run cassandra I get the error "data_file_directories: command not found". How can I modify the directories correctly?
PS: I don't have write access to cassandra.yaml and can't create the default directories it uses.
referrer to this answer Make your own cassandra.yaml with your custom directories and then run cassandra with with -d flag and cassandra.config=directory
or set $CASSANDRA_HOME variable in your .bashrc and then run cassandra

Cassandra dead but pid file exists

I have novice to cassandra and tried my hands to install cassandra-2.1.2 on centos 7.0.
After complete installation execute cqlsh command and created few keyspace(s) and column family.
Which seems to me in first glance its working perfectly.
But later onwards i realized below issues:
1- when i execute "service cassandra status" command, i got below error:
Output:Cassandra dead but pid file exists.
I googled the above issue and found some links
http://www.datastax.com/support-forums/topic/dse-dead-but-pid-file-exists
https://baioradba.wordpress.com/2014/06/13/how-to-install-cassandra-on-centos-6-5/
and found that I had same configuration mentioned in above links but the same error still persists.
Please tell me the root cause and how to resolve it.
2- Second issue is in the cassandra.log file.
When I analysed the cassandra.log file there was an expection as :
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Below is the complete log:
12:01:40.816 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:158) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:133) ~[apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:110) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465) [apache-cassandra-2.1.3.jar:2.1.3]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554) [apache-cassandra-2.1.3.jar:2.1.3]
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file://<server>/ for remote files. Aborting.
Fatal configuration error; unable to start. See log for stacktrace.
I again searched the same issue in google and but the links were not that useful as they contained the java class code for cassandra.config .
Again please tell the root cause and how to resolve it?
Thanks in advance.
rm /var/run/cassandra.pid
Run ps -ef | grep cassandra
Kill the pid of the cassandra process.
Start cassandra
fix this issue, Edit the cassandra-env.sh:
sudo vi /etc/cassandra/conf/cassandra-env.sh
increase heap size for cassandra .. this should resolve your issue
Check if you have enough memory to start cassandra service with this command:
cat /proc/meminfo
I was running Hortonworks VM with Virtualbox, and I had a lot of Hadoop components started which needed a lot of memory, so for me the solution was to stop unnecessary Hadoop components and add some extra memory to the virtual machine.
From https://github.com/apache/cassandra/blob/cassandra-2.1/examples/client_only/README.txt#L43-L49 :
cassandra.yaml can be on the classpath as is done here, can be
specified (by modifying the script) in a location within the classpath
like this: java -Xmx1G
-Dcassandra.config=/path/in/classpath/to/cassandra.yaml ... or can be retrieved from a location outside the classpath like this: ...
-Dcassandra.config=file:///path/to/cassandra.yaml ... or ... -Dcassandra.config=http://awesomesauce.com/cassandra.yaml ...
So you probably had a misconfigured startup option.
Remove the pid file. Try
rm /var/run/cassandra.pid

Cannot create KeySpace using Cassandra-CLI

I have installed and started up Cassandra server(1.0.8). I can connect to the server using the CLI application. But as soon as I try to create a keyspace "twissandra" following the step from CassandraCLI
I end up getting the following error
I can see the cassandra.yaml file in "config" directory of the installation.
EDIT - THE ANSWER
OK So after a few days of short chitty chats with libjack. The error was tracked down.
REM set CLASSPATH="%CASSANDRA_HOME%\conf"
The line above is a remark (comment if you may) I had to go through the whole BAT file line by line before finally removing the REM clause.
The cassandra.yaml is expected to be found in the classpath. By default, the cassandra.bat (my version from 1.07 zip) adds the $CASSANDRA_HOME\conf directory to the classpath (as well as other necessary Jars)
If CASSANDRA_HOME is not set, it uses the directory above the script location.
To test, perhaps modify the cassandra.bat to echo out all the commands and see where things get messed up.

Resources