As sstables are immutable and sstable split has to be performed offline ie. with node shutdown. Wouldn't it also be possible to split copies of extreme large sstables offline/in a sideline dir, while keeping a node online then following swap the extreme sstables with a set of splitted sstable files during a short restart of a node to minimize node downtime?
Or would it be better to decommission a node, spreading data over rest of cluster and then rejoin as a new empty node
Eg. having some large sstables which ain't getting into a compaction view any time soon. I'll like to split such offline say in another directory/FS/on another box, just where ever out of scope from running node while still having the node servicing redundancy from original sstable path. Only it seems sstablesplit want to find the configuration or can it be tricked to otherwise do a split out-reach from the running node?
Tried on a copy of a sstable file to split it, but:
on-a-offlinebox$ sstablesplit --debug -s SOME-VALUE-IN-MB mykeyspc-mycf-*-Data.db 16:58:13.197 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal
configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI
in variable: [cassandra.config]. Please prefix the file with file:///
for local files or file:/// for remote files. Aborting. If you
are executing this from an external tool, it needs to set
Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73)
~[apache-cassandra-2.1.15.jar:2.1.15]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84)
~[apache-cassandra-2.1.15.jar:2.1.15]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161)
~[apache-cassandra-2.1.15.jar:2.1.15]
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:136)
~[apache-cassandra-2.1.15.jar:2.1.15]
at org.apache.cassandra.tools.StandaloneSplitter.main(StandaloneSplitter.java:56)
[apache-cassandra-2.1.15.jar:2.1.15] Expecting URI in variable:
[cassandra.config]. Please prefix the file with file:/// for local
files or file:/// for remote files. Aborting. If you are
executing this from an external tool, it needs to set
Config.setClientMode(true) to avoid loading configuration. Fatal
configuration error; unable to start. See log for stacktrace.
If you can afford downtime for the node, just do it (split the tables). Anyway, if you will do this split on another machine/another dir you will need to run repair on the node (due to "offline" time of the rebuilded tables) after reloading sstables.
You can also try to drop this tables data files from your node and running repair, it will be probably minimal downtime for the node:
Stop the node -> Delete big sstables -> Start the node -> Repair.
EDIT: Since Cassandra 3.4, you can run compact command on specific sstables/files. On any earlier version, you can use forceUserDefinedCompaction jmx call. You can use one of these, or make a jmx call by yourself:
http://wiki.cyclopsgroup.org/jmxterm/manual.html
https://github.com/hancockks/cassandra-compact-cf
https://gist.github.com/jeromatron/e238e5795b3e79866b83
Example code with jmxterm:
sudo java -jar jmxterm-1.0-alpha-4-uber.jar -l localhost:7199
bean org.apache.cassandra.db:type=CompactionManager
run forceUserDefinedCompaction YourKeySpaceName_YourFileName.db
Also, if "big tables" problem occurs all the time, consider moving to LCS.
Related
Our agent stoped working and the only way to get it back is by restarting the agent in every node, the error we get:
ERROR [clojure-agent-send-off-pool-11618] df failed on execute;
returning default value - Cannot run program "df": error=24, Too many open files
Why it's need to have so much files open? How we can reset it without the need to restart the agent and make it automatically.
As per the DSE documentation on production settings, make sure that you have made the following adjustments to your /etc/security/limits.d/cassandra.conf file:
cassandra - memlock unlimited
cassandra - nofile 100000
cassandra - nproc 32768
cassandra - as unlimited
Note: This assumes that you are running Cassandra as the cassandra user. If that is not the case, adjust accordingly.
Why it's need to have so much files open?
Cassandra ultimately writes its data to SSTable files on the underlying filesystem. It also has to serve reads from any of those aforementioned files. Additionally, if you have rows that have been updated a lot and thus contain obsoleted data over time (and compaction has not run), it is entirely possible that a single row might need to be read from multiple files.
In short, databases need to work with many files simultaneously, and Cassandra is no exception.
I'm encountering the same problem as Cassandra system.hints table is empty even when the one of the node is down:
I am learning Cassandra from academy.datastax.com. I am trying the Replication and Consistency demo on local machine. RF = 3 and Consistency = 1.
When my Node3 is down and I am updating my table using update command, the SYSTEM.HINTS table is expected to store hint for node3 but it is always empty.
#amalober pointed out that this was due to a difference the Cassandra version being used. From the Cassandra docs at DataStax:
In Cassandra 3.0 and later, the hint is stored in a local hints directory on each node for improved replay.
This same question was asked 3 years ago, How to access the local data of a Cassandra node, but the accepted solution was to
...Hack something together using the Cassandra source that reads SSTables and have that feed the local client you're hoping to build. A great starting point would be looking at the source of org.apache.cassandra.tools.SSTableExport which is used in the sstable2json tool.
Is there an easier way to access the local hints directory of a Cassandra node?
Is there an easier way to access the local hints directory of a Cassandra node?
The hint directory is defined in $CASSANDRA_HOME/conf/cassandra.yaml file (sometimes it is located under /etc/cassandra also, depending on how you install Cassandra)
Look for the property hints_directory
I guess you are using ccm. So, the hint file should be in $CASSANDRA_HOME/.ccm/yourcluster/yournode/hints directory
I haven't been able to reproduce your issue with not getting a hints file. Every attempt I had resulted in the hints file as expected. There is a way to view the hints easier now.
We added a dump for hints in sstable-tools that you can use to view the mutations in the HH files. We may in the future add ability to use the HH files like sstables in the shell (use mutations to build memtable and include in queries) but for now its pretty raw.
Its pretty simple (sans metadata setup) if you wanna do analysis of data yourself. You can see what we did here and change to your needs: https://github.com/tolbertam/sstable-tools/blob/master/src/main/java/org/apache/cassandra/hints/HintsTool.java#L39
I have two different independent machines running Cassandra and I want to migrate the data from one machine to the other.
Thus, I first took a snapshot of my Cassandra Cluster on machine 1 according to the datastax documentation.
Then I moved the data to machine 2, where I'm trying to import it with sstableloader.
As a note: The keypsace (open_weather) and tablename (raw_weather_data) on the machine 2 have been created and are the same as on machine 1.
The command I'm using looks as follows:
bin/sstableloader -d localhost "path_to_snapshot"/open_weather/raw_weather_data
And then get the following error:
Established connection to initial hosts
Opening sstables and calculating sections to stream
For input string: "CompressionInfo.db"
java.lang.NumberFormatException: For input string: "CompressionInfo.db"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:276)
at org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:235)
at org.apache.cassandra.io.sstable.Component.fromFilename(Component.java:120)
at org.apache.cassandra.io.sstable.SSTable.tryComponentFromFilename(SSTable.java:160)
at org.apache.cassandra.io.sstable.SSTableLoader$1.accept(SSTableLoader.java:84)
at java.io.File.list(File.java:1161)
at org.apache.cassandra.io.sstable.SSTableLoader.openSSTables(SSTableLoader.java:78)
at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:162)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:106)
Unfortunately I have no idea why?
I'm not sure if it is related to the issue, but somehow on machine 1 my *.db files are name rather "strange" as compared to the *.db files I already have on machine 2.
*.db files from machine 1:
la-53-big-CompressionInfo.db
la-53-big-Data.db
...
la-54-big-CompressionInfo.db
...
*.db files from machine 2:
open_weather-raw_weather_data-ka-5-CompressionInfo.db
open_weather-raw_weather_data-ka-5-Data.db
What am I missing? Any help would be highly appreciated. I'm also open to any other suggestions. The COPY command will most probably not work since it is Limited to 99999999 rows as far as I know.
P.s. I didn't want to create a overly huge post, but if you need any further information to help me out, just let me know.
EDIT:
Note that I'm using Cassandra in the stand-alone mode.
EDIT2:
After installing the same version 2.1.4 on my destination machine (machine 2), I still get all the same error. With SSTableLoader I still get the above mentioned error and with copying the files manually (as described by LHWizard), I still get empty tables after starting Cassandra again and performing a SELECT command.
Regarding the initial tokens, I get a huge list of tokens if I perform node ring on machine 1. I'm not sure what to do with those?
your data is already in the form of a snapshot (or backup). What I have done in the past is the following:
install the same version of cassandra on the restore node
edit cassandra.yaml on the restore node - make sure that cluster_name and snitch are the same.
edit seeds: list and any other properties that were altered in the original node.
get the schema from the original node using cqlsh DESC KEYSPACE.
start cassandra on the restore node and import the schema.
(steps 6 & 7 may not be completely necessary, but this is what I do.)
stop cassandra, delete the contents of /var/lib/cassandra/data/, commitlog/, and saved_caches/* folders.
restart cassandra on the restore node to recreate the correct folders, then stop it
copy the contents of the snapshots folder to each corresponding table folder in the restore node, then start cassandra. You probably want to run nodetool repair.
You don't really need to bulk import the data, it's already in the correct format if you are using the same version of cassandra, although you didn't specify that in your original question.
I've got a mostly idle (till now) 4 node cassandra cluster that's hooked up with opscenter. There's a table that's had very little writes in the last week or so (test cluster). It's running 2.1.0. Happened to ssh in, and out of curiosity, ran du -sh * on the data directory. Here's what I get:
4.2G commitlog
851M data
188K saved_caches
There's 136 files in the commit log directory. I flushed, and then drained cassandra, stopped and started the service. Those files are still there. What's the best way to get rid of these? Most of the stuff is opscenter related, and I'm inclined to just blow them away as I don't need the test data. Wondering what to do in case this pops up again. Appreciate any tips.
The files in the commit log directory have a fixed size determined by your settings in the cassandra.yaml. All files have a pre-allocated size, so you cannot change it by making flush, drain or other operations on the cluster.
You have to change the configuration if you want to reduce their size.
Look at the configuration settings "commitlog_total_space_in_mb" and "commitlog_segment_size_in_mb" to configure the size of each file and the total space occupied by all of them.
When I copy a node's snapshot to its /var/lib/cassandra/data///, and run 'nodetool refresh', what will happen to this newly replaced sstables and original sstables, because the original sstables is still there, and some new writes still in commitlog and memtables.
Does backup one node's snapshot make sense? Because other nodes may have the the data with more recent timestamp.
Does backup one node's data make sense?
Not in a multi node environment. If you're removing a node from the cluster or running a rolling upgrade maybe, but not for backup purposes on a live cluster.
This is a problem that tends to get addressed by using a parallel ssh tool. The example given by DataStax is pssh. This will create all the snapshots at the same time on each node giving you consistent data assuming you're not dealing with heavy writes (since C* is eventually consistent).