I am trying to add another node to a Production cassandra cluster as the disc space utilization across nodes is reaching over 90%. However, the node is in joining state for over 2 days. I also noticed that one of the node went down(DN) as it is at 100% disc space utilization. Cassandra server is unable to run on this instance!!
Will this affect bootstrapping completion of the new node?
Any immediate solutions for restoring space on the node that went down?
If I remove this out of the ring, this may add more stress of data load and increase disc space on the other nodes.
Can I remove any SSTable(like the list of files) temporarily out of the instance, bring up the server, perform clean-up and then add back these files?
-rw-r--r--. 1 polkitd input 5551459 Sep 17 2020 mc-572-big-CompressionInfo.db
-rw-r--r--. 1 polkitd input 15859691072 Sep 17 2020 mc-572-big-Data.db
-rw-r--r--. 1 polkitd input 8 Sep 17 2020 mc-572-big-Digest.crc32
-rw-r--r--. 1 polkitd input 22608920 Sep 17 2020 mc-572-big-Filter.db
-rw-r--r--. 1 polkitd input 5634549206 Sep 17 2020 mc-572-big-Index.db
-rw-r--r--. 1 polkitd input 12538 Sep 17 2020 mc-572-big-Statistics.db
-rw-r--r--. 1 polkitd input 44510338 Sep 17 2020 mc-572-big-Summary.db
-rw-r--r--. 1 polkitd input 92 Sep 17 2020 mc-572-big-TOC.txt
If you are using vnodes then downed node will surelyimpact bootstrapping. For immediate relife, identify tables which are not used in traffic and move sstables to backup from that table.
I resolved this by temporarily increasing the EBS volume(disc space)on that node, brought up the server, then removed the node out of the cluster, cleared out cassandra data folders, decreased the EBS Volume and then added back the node to the cluster.
One thing that I noticed was removing the node out of the cluster, increased disc space on the other nodes. So I added additional nodes to distribute the load, then ran clean up on all other nodes before moving on to removing the node out of the cluster.
Related
under folder - /var/hadoop/zookeeper/version-2/
we can see that Zookeeper transaction logs and snapshot files are created very frequently (multiple files in every minute) and that fills up the Filesystem in a very short time.
ROOT CAUSE
One or more application are creating or modifying the znodes too frequently, causing too many transactions in a short duration. This leads to the creation of too many transactional log files and snapshot files since they get rolled over after 100,000 entries by default (as defined by zookeeper property 'snapCount')
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:24 log.570021fa92
-rw-r--r-- 1 zookeeper hadoop 490656299 Jul 28 17:24 snapshot.5700232ffa
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:29 log.5700232ffc
-rw-r--r-- 1 zookeeper hadoop 490656389 Jul 28 17:29 snapshot.5700249d7f
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:33 log.5700249d78
-rw-r--r-- 1 zookeeper hadoop 490656275 Jul 28 17:33 snapshot.570025fdaf
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:36 log.570025fdae
-rw-r--r-- 1 zookeeper hadoop 490656275 Jul 28 17:36 snapshot.570026c447
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:40 log.570026c449
-rw-r--r-- 1 zookeeper hadoop 490658969 Jul 28 17:40 snapshot.570027caed
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:43 log.570027caef
-rw-r--r-- 1 zookeeper hadoop 490658981 Jul 28 17:43 snapshot.570028a0d0
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:48 log.570028a0d2
-rw-r--r-- 1 zookeeper hadoop 165081088 Jul 28 17:48 snapshot.57002a0268
-rw-r--r-- 1 zookeeper hadoop 67108880 Jul 28 17:48 log.57002a026b
.
.
.
.
when we opened one of the log as - log.57002a026b we saw encrypted log
any suggestion how to unencrypted the logs above ?
or how to know which is the application thatcreating or modifying the znodes too frequently ?
PROBLEM
Zookeeper transaction logs and snapshot files are created very frequently (multiple files in every minute) and that fills up the FileSystem in a very short time.
ROOT CAUSE
One or more application are creating or modifying the znodes too frequently, causing too many transactions in a short duration. This leads to the creation of too many transactional log files and snapshot files since they get rolled over after 100,000 entries by default (as defined by zookeeper property 'snapCount')
RESOLUTION
The resolution for such cases involves reviewing the zookeeper transaction logs to find the znodes that are updated/created most frequently using the following command on one of the zookeeper servers:
# cd /usr/hdp/current/zookeeper-server
# java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/logxxx
(where 'dataDir' is set to '/hadoop/zookeeper' within zookeeper configuration)
Once the frequently updating znodes are identified using the above command, one should continue with fixing the related application that is creating such a large number of updates on zookeeper.
An example of such an application that can cause this problem is Hbase, when there are very large number of regions stuck in transition and they repeatedly fail to become online.
I'm using the current version of mbpfan (mbpfan-git-2.2.1.r4.g52d8973-1-x86_64) from Arch User Repository, running Arch linux on a MacBookPro 6,1 mid-2010 (on external usb, actually). Mbpfan is a small daemon designed to control mac fan speed and solve heat problems.
After a recent upgrade to 5.8.1 kernal, from 5.7.12, mbpfan does not work,
with error
Aug 20 17:19:56 ehost kernel: applesmc: send_byte(0x03, 0x0300) fail:
0x00 Aug 20 17:19:56 ehost kernel: applesmc: FS! : write data fail
and
ehost mbpfan[380]: Could not set fan speed: Input/output error Aug 20
17:21:56 ehost kernel: applesmc: send_byte(0x40, 0x0300) fail: 0x00
Aug 20 17:21:56 ehost kernel: applesmc: F0Tg: write data fail
FS! fail is saying, can't change fan to "manual," and F0Tg is the first fan where it
tries to write a new speed.
From looking in /sys/devices/platform/applesmc.768 one can see that even root can only write to fan1_manual, fan1_min, and
fan1_output, for example.
-r--r--r-- 1 root root 4096 Aug 18 21:00 fan1_input
-r--r--r-- 1 root root 4096 Aug 18 21:01 fan1_label
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_manual
-r--r--r-- 1 root root 4096 Aug 18 21:00 fan1_max
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_min
-rw-r--r-- 1 root root 4096 Aug 18 21:00 fan1_output
The above permissions have not changed from earlier kernels when mbpfan worked.
They are exactly the same on the installations where mbpfan works (different usb stick, same machine).
Permissions should not be an issue, as I understand that one writes to "fan1_output," which is rw, to set the desired fan speed (counterintuitively).
As a test, I upgraded another installation with a working mbpfan (kernel 5.7.9) to 5.8.1, and the same no write problem appears.
--It seems that something in the kernel that affects mbpfan has changed between
5.7.12 and 5.8.1.
Naturally, I have reported this at the developer's git hub area. I've been digging around in the code a bit, but it's tough going so far as I really know nothing about drivers. --I'd really appreciate any suggestions on what I might try to get this working...
Using HDP 2.5.3 and I've been trying to debug some YARN container classpath issues.
Since HDP includes both Spark 1.6 and 2.0.0, there have been some conflicting versions
Users I support are successfully able to use Spark2 with Hive queries in YARN client mode, but not from cluster mode they get errors about tables not found, or something like that because the Metastore connection isn't established.
I am guessing that setting either --driver-class-path /etc/spark2/conf:/etc/hive/conf or passing --files /etc/spark2/conf/hive-site.xml after spark-submit would work, but why isn't hive-site.xml loaded already from the conf folder?
Accoringing to Hortonworks docs, says hive-site should be placed in $SPARK_HOME/conf, and it is...
I see hdfs-site.xml and core-site.xml, and other files that are part of HADOOP_CONF_DIR, for example, and this is the from the YARN UI container info.
2232355 4 drwx------ 2 yarn hadoop 4096 Aug 2 21:59 ./__spark_conf__
2232379 4 -r-x------ 1 yarn hadoop 2358 Aug 2 21:59 ./__spark_conf__/topology_script.py
2232381 8 -r-x------ 1 yarn hadoop 4676 Aug 2 21:59 ./__spark_conf__/yarn-env.sh
2232392 4 -r-x------ 1 yarn hadoop 569 Aug 2 21:59 ./__spark_conf__/topology_mappings.data
2232398 4 -r-x------ 1 yarn hadoop 945 Aug 2 21:59 ./__spark_conf__/taskcontroller.cfg
2232356 4 -r-x------ 1 yarn hadoop 620 Aug 2 21:59 ./__spark_conf__/log4j.properties
2232382 12 -r-x------ 1 yarn hadoop 8960 Aug 2 21:59 ./__spark_conf__/hdfs-site.xml
2232371 4 -r-x------ 1 yarn hadoop 2090 Aug 2 21:59 ./__spark_conf__/hadoop-metrics2.properties
2232387 4 -r-x------ 1 yarn hadoop 662 Aug 2 21:59 ./__spark_conf__/mapred-env.sh
2232390 4 -r-x------ 1 yarn hadoop 1308 Aug 2 21:59 ./__spark_conf__/hadoop-policy.xml
2232399 4 -r-x------ 1 yarn hadoop 1480 Aug 2 21:59 ./__spark_conf__/__spark_conf__.properties
2232389 4 -r-x------ 1 yarn hadoop 1602 Aug 2 21:59 ./__spark_conf__/health_check
2232385 4 -r-x------ 1 yarn hadoop 913 Aug 2 21:59 ./__spark_conf__/rack_topology.data
2232377 4 -r-x------ 1 yarn hadoop 1484 Aug 2 21:59 ./__spark_conf__/ranger-hdfs-audit.xml
2232383 4 -r-x------ 1 yarn hadoop 1020 Aug 2 21:59 ./__spark_conf__/commons-logging.properties
2232357 8 -r-x------ 1 yarn hadoop 5721 Aug 2 21:59 ./__spark_conf__/hadoop-env.sh
2232391 4 -r-x------ 1 yarn hadoop 281 Aug 2 21:59 ./__spark_conf__/slaves
2232373 8 -r-x------ 1 yarn hadoop 6407 Aug 2 21:59 ./__spark_conf__/core-site.xml
2232393 4 -r-x------ 1 yarn hadoop 812 Aug 2 21:59 ./__spark_conf__/rack-topology.sh
2232394 4 -r-x------ 1 yarn hadoop 1044 Aug 2 21:59 ./__spark_conf__/ranger-hdfs-security.xml
2232395 8 -r-x------ 1 yarn hadoop 4956 Aug 2 21:59 ./__spark_conf__/metrics.properties
2232386 8 -r-x------ 1 yarn hadoop 4221 Aug 2 21:59 ./__spark_conf__/task-log4j.properties
2232380 4 -r-x------ 1 yarn hadoop 64 Aug 2 21:59 ./__spark_conf__/ranger-security.xml
2232372 20 -r-x------ 1 yarn hadoop 19975 Aug 2 21:59 ./__spark_conf__/yarn-site.xml
2232397 4 -r-x------ 1 yarn hadoop 1006 Aug 2 21:59 ./__spark_conf__/ranger-policymgr-ssl.xml
2232374 4 -r-x------ 1 yarn hadoop 29 Aug 2 21:59 ./__spark_conf__/yarn.exclude
2232384 4 -r-x------ 1 yarn hadoop 1606 Aug 2 21:59 ./__spark_conf__/container-executor.cfg
2232396 4 -r-x------ 1 yarn hadoop 1000 Aug 2 21:59 ./__spark_conf__/ssl-server.xml
2232375 4 -r-x------ 1 yarn hadoop 1 Aug 2 21:59 ./__spark_conf__/dfs.exclude
2232359 8 -r-x------ 1 yarn hadoop 7660 Aug 2 21:59 ./__spark_conf__/mapred-site.xml
2232378 16 -r-x------ 1 yarn hadoop 14474 Aug 2 21:59 ./__spark_conf__/capacity-scheduler.xml
2232376 4 -r-x------ 1 yarn hadoop 884 Aug 2 21:59 ./__spark_conf__/ssl-client.xml
As you might see, hive-site is not there, even though I definitely have conf/hive-site.xml for spark-submit to take
[spark#asthad006 conf]$ pwd && ls -l
/usr/hdp/2.5.3.0-37/spark2/conf
total 32
-rw-r--r-- 1 spark spark 742 Mar 6 15:20 hive-site.xml
-rw-r--r-- 1 spark spark 620 Mar 6 15:20 log4j.properties
-rw-r--r-- 1 spark spark 4956 Mar 6 15:20 metrics.properties
-rw-r--r-- 1 spark spark 824 Aug 2 22:24 spark-defaults.conf
-rw-r--r-- 1 spark spark 1820 Aug 2 22:24 spark-env.sh
-rwxr-xr-x 1 spark spark 244 Mar 6 15:20 spark-thrift-fairscheduler.xml
-rw-r--r-- 1 hive hadoop 918 Aug 2 22:24 spark-thrift-sparkconf.conf
So, I don't think I am supposed to place hive-site in HADOOP_CONF_DIR as HIVE_CONF_DIR is separated, but my question is that how do we get Spark2 to pick up the hive-site.xml without needing to manually pass it as a parameter at runtime?
EDIT Naturally, since I'm on HDP I am using Ambari. The previous cluster admin has installed Spark2 clients on all of the machines, so all of the YARN NodeManagers that could be potential Spark drivers should have the same config files
You can use spark property - spark.yarn.dist.files and specify path to hive-site.xml there.
The way I understand it, in local or yarn-client modes...
the Launcher checks whether it needs Kerberos tokens for HDFS, YARN, Hive, HBase
> hive-site.xml is searched in the CLASSPATH by the Hive/Hadoop client libs (including in driver.extraClassPath because the Driver runs inside the Launcher and the merged CLASSPATH is already built at this point)
the Driver checks which kind of metastore to use for internal purposes: a standalone metastore backed by a volatile Derby instance, or a regular Hive metastore
> that's $SPARK_CONF_DIR/hive-site.xml
when using the Hive interface, a Metastore connection is used to read/write Hive metadata in the Driver
> hive-site.xml is searched in the CLASSPATH by the Hive/Hadoop client libs (and the Kerberos token is used, if any)
So you can have one hive-site.xml stating that Spark should use an embedded, in-memory Derby instance to use as a sandbox (in-memory implying "stop leaving all these temp files behind you") while another hive-site.xml gives the actual Hive Metastore URI. And all is well.
Now, in yarn-cluster mode, all that mechanism pretty much explodes in a nasty, undocumented mess.
The Launcher needs its own CLASSPATH settings to create the Kerberos tokens, otherwise it fails silently. Better go to the source code to find out which undocumented Env variable you shoud use.
It may also need an override in some properties because the hard-coded defaults suddenly are not the defaults any more (silently).
The Driver cannot tap the original $SPARK_CONF_DIR, it has to rely on what the Launcher has made available for upload. Does that include a copy of $SPARK_CONF_DIR/hive-site.xml? Looks like it's not the case.
So you are probably using a Derby thing as a stub.
And the Driver has to to do with whatever YARN has forced on the container CLASSPATH, in whatever order.
Besides, the driver.extraClassPath additions do NOT take precedence by default; for that you have to force spark.yarn.user.classpath.first=true (which is translated to the standard Hadoop property whose exact name I can't remember right now, especially since there are multiple props with similar names that may be deprecated and/or not working in Hadoop 2.x)
Think that's bad? Try out connecting to a Kerberized HBase in yarn-cluster mode. The connection is done in the Executors, that's another layer of nastyness. But I disgress.
Bottom line: start your diagnostic again.
A. Are you really, really sure that the mysterious "Metastore connection errors" are caused by missing properties, and specifically the Metastore URI?
B. By the way, are your users explicitly using a HiveContext???
C. What is exactly the CLASSPATH that YARN presents to the Driver JVM, and what is exactly the CLASSPATH that the Driver presents to the Hadoop libs when opening the Metastore connection?
D. If the CLASSPATH built by YARN is messed up for some reason, what would be the minimal fix -- change in precedence rules? addition? both?
In the cluster mode configuration is read from the conf directory of the machine, which runs the driver container, not the one use for spark-submit.
Found an issue with this
You create a org.apache.spark.sql.SQLContext before creating hive context the hive-site.xml is not picked properly when you create hive context.
Solution : Create the hive context before creating another SQL context.
Need: create keyspace on alternate device
Problem: service aborts on startup with dir-create failure messages below.
INFO [main] 2017-01-06 00:45:03,300 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
ERROR [main] 2017-01-06 00:45:03,393 Directories.java:239 - Failed to create /var/lib/cassandra/data/opus/aa-15be7240d3db11e6ad0eed0a1d791016 directory
ERROR [main] 2017-01-06 00:45:03,397 DefaultFSErrorHandler.java:92 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
Context: Cassandra 3.9 single-node ubuntu 16.04; directory perms are below.
01:52 opus/ cd /var/lib/cassandra/data
01:52 opus/ ls -l
total 24
drwxr-xr-x 3 cassandra cassandra 4096 Jan 6 00:41 opus
drwxr-xr-x 24 cassandra cassandra 4096 Jan 5 23:49 system
drwxr-xr-x 6 cassandra cassandra 4096 Jan 5 23:50 system_auth
drwxr-xr-x 5 cassandra cassandra 4096 Jan 5 23:50 system_distributed
drwxr-xr-x 12 cassandra cassandra 4096 Jan 5 23:50 system_schema
drwxr-xr-x 4 cassandra cassandra 4096 Jan 5 23:50 system_traces
01:52 opus/ cd opus
01:52 opus/ ls -l
total 4
drwxr-xr-x 3 cassandra cassandra 4096 Jan 6 00:41 aa-15be7240d3db11e6ad0eed0a1d791016
when the link is installed
01:57 data/ ls -l
total 20
lrwxrwxrwx 1 root root 35 Jan 6 01:57 opus -> /media/opus/quantdrive/opus
Steps:
Vanilla install of cassandra 3.9;
Create keyspace in cqlsh create keyspace opus with replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
Create table use opus; create table aa(aa int, primary key(aa));
Stop cassandra
Move keyspace dir mv /var/lib/cassandra/data/opus /media/opus/quantdrive
Create symbolic link ln -s /media/opus/quantdrive/opus /var/lib/cassandra/opus
Start cassandra [FAILS AS ABOVE] with create directory, when directory already present
No change in perms on opus keyspace directory, I just moved it. When I move it back, cassandra starts fine.
I would be grateful for any help with this and I apologize in advance if I the solution to my problem is described elsewhere or if I'm missing the obvious.
Move the mount point for the target drive from a user-owned directory to a root-owned one. I moved the mount-point in my case from /media/opus/quantdrive which is owned by user opus to /mnt/quantdrive which is owned by root and everything worked fine.
I'm trying to upgrade from Cassandra to the latest Datastax Enterprise and everything went fine except the fact I can't get my data back.
Basically, I had a clean cassandra after the upgrade, then I recreated the schema and trying to somehow link the files that are left from old db to the new db.
That's what I have right now in /var/lib/cassandra/data/wowch directory for example:
drwxr-x--- 4 cassandra cassandra 4.0K Feb 27 13:05 users-247834809d2011e58d82b7a748b1d9c2/
drwxr-xr-x 2 cassandra cassandra 4.0K Feb 27 18:53 users-f41a5300dd5611e58bc7b7a748b1d9c2/
As I get, the older directory is what was in the db before the upgrade. It contains some db files:
total 144K
drwxr-x--- 4 cassandra cassandra 4.0K Feb 27 13:05 ./
drwxr-x--- 60 cassandra cassandra 20K Feb 27 14:35 ../
drwxr-x--- 2 cassandra cassandra 4.0K Dec 7 21:21 backups/
-rwxr-x--- 2 cassandra cassandra 51 Jan 20 00:05 ma-46-big-CompressionInfo.db*
-rwxr-x--- 2 cassandra cassandra 828 Jan 20 00:05 ma-46-big-Data.db*
-rwxr-x--- 2 cassandra cassandra 10 Jan 20 00:05 ma-46-big-Digest.crc32*
-rwxr-x--- 2 cassandra cassandra 16 Jan 20 00:05 ma-46-big-Filter.db*
-rwxr-x--- 2 cassandra cassandra 83 Jan 20 00:05 ma-46-big-Index.db*
-rwxr-x--- 2 cassandra cassandra 4.9K Jan 20 00:05 ma-46-big-Statistics.db*
-rwxr-x--- 2 cassandra cassandra 92 Jan 20 00:05 ma-46-big-Summary.db*
-rwxr-x--- 2 cassandra cassandra 92 Jan 20 00:05 ma-46-big-TOC.txt*
-rwxr-x--- 2 cassandra cassandra 43 Feb 12 15:05 ma-47-big-CompressionInfo.db*
-rwxr-x--- 2 cassandra cassandra 41 Feb 12 15:05 ma-47-big-Data.db*
-rwxr-x--- 2 cassandra cassandra 10 Feb 12 15:05 ma-47-big-Digest.crc32*
-rwxr-x--- 2 cassandra cassandra 16 Feb 12 15:05 ma-47-big-Filter.db*
-rwxr-x--- 2 cassandra cassandra 20 Feb 12 15:05 ma-47-big-Index.db*
-rwxr-x--- 2 cassandra cassandra 4.5K Feb 12 15:05 ma-47-big-Statistics.db*
-rwxr-x--- 2 cassandra cassandra 92 Feb 12 15:05 ma-47-big-Summary.db*
-rwxr-x--- 2 cassandra cassandra 92 Feb 12 15:05 ma-47-big-TOC.txt*
-rwxr-x--- 2 cassandra cassandra 43 Feb 12 16:05 ma-48-big-CompressionInfo.db*
-rwxr-x--- 2 cassandra cassandra 169 Feb 12 16:05 ma-48-big-Data.db*
-rwxr-x--- 2 cassandra cassandra 10 Feb 12 16:05 ma-48-big-Digest.crc32*
-rwxr-x--- 2 cassandra cassandra 16 Feb 12 16:05 ma-48-big-Filter.db*
-rwxr-x--- 2 cassandra cassandra 20 Feb 12 16:05 ma-48-big-Index.db*
-rwxr-x--- 2 cassandra cassandra 4.9K Feb 12 16:05 ma-48-big-Statistics.db*
-rwxr-x--- 2 cassandra cassandra 92 Feb 12 16:05 ma-48-big-Summary.db*
-rwxr-x--- 2 cassandra cassandra 92 Feb 12 16:05 ma-48-big-TOC.txt*
-rwxr-x--- 1 cassandra cassandra 31 Dec 7 21:26 manifest.json*
drwxr-x--- 3 cassandra cassandra 4.0K Feb 27 13:05 snapshots/
I tried to copy all the stuff from here to the users-f41a5300dd5611e58bc7b7a748b1d9c2/ directory and run nodetool repair or nodetool refresh -- wowch users but had no success — the data is still not loaded.
Did I forget something? What is the right way of doing it and how to get the data back?
Depending on the version of Cassandra/DSE you were previously running you may need to run a nodetool upgradesstables. You can see the documentation here.
https://docs.datastax.com/en/cassandra/2.0/cassandra/tools/toolsUpgradeSstables.html
It's possible that you've run into this issue but without more info I can't say for sure.
You also haven't provided info on which version you started with and ended. A little more info would be very helpful. Can you also clarify - are you upgrading from community Cassandra to DSE? I couldn't tell from the way your question was worded.
Stuff to check: Do you have the token assignments from the old version? I didn't use vnodes and I found that I had to manually set initial_token in cassandra.yaml after a backup/restore of my cluster. Make sure that cassandra owns all of the dirs and files. After you import the schema, stop DSE and then empty the contents of the commitlog directory. move your data if necessary into the new folders and then restart DSE. Hope this helps.