inserting in table emp1234 but got an error in cassandra - cassandra

This is what I tried:
emp1234(user_id,age,city,country,height,mobile_no,name,sex,state,weight) FROM '/tmp/user.csv' WITH HEADER = FALSE;
But it returns the following error:
error -
Failed to import 17 rows: WriteTimeout - Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}, will retry later, attempt 1 of 5


The most probable reason for it is that you have different DC name in the replication factor settings of keyspace, than in your cluster.
Get DC name(s) via nodetool status, check the names in the settings of the keyspace (via DESCRIBE keyspace_name;), they should be the same, including character case. If they are different, change replication factor via
alter KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC_name': X};

Related

OverloadException in single node cassandra

I read through this page : https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html and identified that OverloadedException in Cassandra is due to the
"The coordinator tracks how many hints it is currently writing, and if the number increases too much, the coordinator refuses writes and throws the OverloadedException exception."
But I use a single node and able to get Overload exception frequently so what could be the reason for Overload exception in singlenode with Consistency as 1 and ReplicationFactor as 1?
EDITED :
Total hints In progress JMX
I checked in the code :
private static void checkHintOverload(InetAddressAndPort destination)
{
// avoid OOMing due to excess hints. we need to do this check even for "live" nodes, since we can
// still generate hints for those if it's overloaded or simply dead but not yet known-to-be-dead.
// The idea is that if we have over maxHintsInProgress hints in flight, this is probably due to
// a small number of nodes causing problems, so we should avoid shutting down writes completely to
// healthy nodes. Any node with no hintsInProgress is considered healthy.
if (StorageMetrics.totalHintsInProgress.getCount() > maxHintsInProgress
&& (getHintsInProgressFor(destination).get() > 0 && shouldHint(destination)))
{
throw new OverloadedException("Too many in flight hints: " + StorageMetrics.totalHintsInProgress.getCount() +
" destination: " + destination +
" destination hints: " + getHintsInProgressFor(destination).get());
}
}
1) Is this the only way to get the Overloaded exception ?
2) Why am I getting the Overload exception in single node ?
3) When this checkHintOverload method is called in Single node?
NOTE :
1) My Keyspace has been configured with NetworkTopologyStrategy, Will that be a reason? : CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter_1': '1'} AND durable_writes = true;
2) hinted_handoff = enabled in cassandra.yaml, Still I wonder will it trigger the hints in Single node, If so why?
3) consistency level is Quorum in this single node
Could any of the three parameter might be a reason for this?

Cassandra issuing an error while selecting the data in table "NoHostAvailable:"

I have created the keyspace and also created a table using Cassandra 3.0 server. I am using the 3 nodes architecture. And three of the servers are working and able to connect the 3 nodes. However when i insert or selecting the data using the CQL, Its showing the error saying that "NoHostAvailable:". Please could anyone provide me the reason and solution for this issue.
Topology
nodetool status output
UN 172.30.1.7 230.22 KB 256 ? 2103dcd3-f09b-47da-a187-bf28b42b918e rack1
DN 172.30.1.20 ? 256 ? 683db65d-0836-40e4-ab5b-fa0db20bae30 rack1
DN 172.30.1.2 ? 256 ? 2b1f15d1-2f92-41ef-a03e-0e5f5f578cf4 rack1
Schema
Keyspace
CREATE KEYSPACE test WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2};
Table
CREATE TABLE testrep(id INT PRIMARY KEY);

Note that from nodetool status, 2 out of your 3 node cluster is down(DN).
You might be inserting with a Consistency Level that cannot be satisfied.
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 237.31 MiB 256 ? 3c8a8d8d-992c-4b7c-a220-6951e37870c6 rack1
cassandra#cqlsh> create KEYSPACE qqq WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2};
cassandra#cqlsh> use qqq;
cassandra#cqlsh:qqq> CREATE TABLE testrep(id INT PRIMARY KEY);
cassandra#cqlsh:qqq> insert into testrep (id) VALUES ( 1);
cassandra#cqlsh:qqq> CONSISTENCY
Current consistency level is ONE.
cassandra#cqlsh:qqq> CONSISTENCY TWO ;
Consistency level set to TWO.
cassandra#cqlsh:qqq> insert into testrep (id) VALUES (2);
NoHostAvailable:
cassandra#cqlsh:qqq> exit

Cassandra WriteTimeout when "IS NOT NULL" condition of materialized view not met

I have this table and materialized view:
CREATE TABLE bubu (
a text,
b text,
c text,
d text,
PRIMARY KEY (a, b));
CREATE MATERIALIZED VIEW bubu_mv AS
SELECT *
FROM bubu
WHERE a IS NOT NULL AND b IS NOT NULL AND c IS NOT NULL
PRIMARY KEY (a, c, b)
When I do this insert and then this update, the insert succeeds but the update returns write timeout:
insert into bubu (a,b,c,d) values ('1','2',null,'3');
update bubu set d = '5' where a = '1' and b = '2';
WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
this happens on cassandra 3.7 and does not happen on cassandra 3.2.1 (just 2 versions I tried).
also, if the insert is like this, it does not happen:
insert into bubu (a,b) values ('1','2');
If I increase write_request_timeout_in_ms in conf/cassandra.yaml to 20000, I get timeout after ~10 seconds:
update bubu set d = '5' where a = '1' and b = '2';
OperationTimedOut: errors={}, last_host=127.0.0.1
Is this a bug in cassandra or am I doing something wrong?

Can you increase timeout value in the config file conf/cassandra.yaml and check. Restart your server if it requires.
write_request_timeout_in_ms: 20000

Cassandra 3.3 - cluster node - irregular behavior

I have a Cassandra Cluster conformed with the nodes
{ 192.168.120.57, 192.168.120.58, 192.168.120.59 }, with replication factor of 2
I detected the node 192.168.120.59 was behaving irregularly
-> The service was running (sudo service cassandra status)
-> The nodetool also told me that the node was running
XXXX#cassandra-prod03:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.120.57 4.05 GB 256 ? 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb rack1
UN 192.168.120.58 4.4 GB 256 ? 53bc77ff-50b3-4bf0-bd55-17e2bb7c8208 rack1
UN 192.168.120.59 2.77 GB 256 ? 03d8d7fd-1d86-4034-a537-9915adb0d4b3 rack1
BUT -> I couldn't connect by DevCenter (the last time I tried I could and the cassandra.yml hasn't changed)
I looked into the log of this node (192.168.120.59), and showed:
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,827 HintsDispatchExecutor.java:224 - Failed to dispatch hints file 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb-1465184315371-1.hints: file is corrupted ({})
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,830 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:1,1,main]
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,834 StorageService.java:470 - Stopping gossiper
ERROR [HintsDispatcher:1] 2016-06-23 16:57:03,842 StorageService.java:480 - Stopping native transport
In the other nodes, the log showed many many times:
DEBUG [GossipTasks:1] 2016-06-24 07:56:21,551 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:22,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:23,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:24,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:25,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:26,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:27,554 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
These are the things I have done and which didn't work:
1) reboot the machine.
2) nodetool repair.
XXXXs#cassandra-prod03:~$ nodetool repair
[2016-06-23 17:07:37,582] Starting repair command #1, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
Jun 23, 2016 5:53:57 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException
error: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
3) nodetool scrub.
After this I followed this page https://docs.datastax.com/en/cassandra/1.2/cassandra/operations/ops_backup_noderestart_t.html
On the node that presented the irregular behaviour:
1) Deleted all tables
2) Copy and Paste the tables that were in the snapshot sub-directory
3) Restarted cassandra service (sudo service cassandra start)
Now the Dev-Center can connect
4) I Ran nodetool repair again, and it seems that is not finishing. It has been running for more than 3 hours and is still in 0% (but for now didn't throw the other error), but I think it is not working.
XXXX#cassandra-prod03:/var/lib/cassandra/data/recommendersjobs$ nodetool repair
[2016-06-24 12:24:13,901] Starting repair command #2, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
[2016-06-24 12:24:17,235] Repair session 7990e900-3a30-11e6-9861-e998758604bf for range [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]] failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser, [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]]] Validation failed in /192.168.120.58 (progress: 0%)
In this log appears:
1) failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser (near the middle)
2) Validation failed in /192.168.120.58 (progress: 0%) (in the last part)
While this repair is running (I think it is running), in the log of this node and in the others nodes of the cluster, this appears many times:
DEBUG [ReadRepairStage:4] 2016-06-24 14:08:18,318 ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-3297346077674417080, 42555f31313131303131333538) (d41d8cd98f00b204e9800998ecf8427e vs 9e7f1588ffcd53d13ab3f8bb2be0f05e)
at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[apache-cassandra-3.3.jar:3.3]
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) ~[apache-cassandra-3.3.jar:3.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
NOTE 1: The cassandra was and is working because of the replication factor of 2.
NOTE 2: The cassandra Cluster is being used in production, and applications are inserting and reading rows.
What can I do to correct this irregular behaviour?
I thought of creating a new node and swap it with the irregular node, what are the correct steps for doing this?

When a node does not respond / is malfunctioning
If you can create a new VM that will repace the malfunctioning node.
See: https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html
If you do not want to create a new VM and use the existing malfunctioning node.
IMPORTANT: NEED TO HAVE A REPLICATION FACTOR THAT ALLOWS YOU TO DELETE ALL THE DATA OF THE AFFECTED NODE.
nodetool decommission (On that node)
See: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsDecommission.html
sudo rm -rf /var/lib/cassandra/*
clear/delete all data in the node
See: https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceClearCpkgData.html
sudo service cassandra start
After a time,
nodetool repair
If the nodetool repair FAILS:
For tables in keyspace:
nodetool scrub <keyspace> <table>
in EACH of the nodes→ see log of the node and wait to complete
run nodetool repair <keyspace> <table>
only in one node (anyone)
NOTE: it is not necessary to run for each table, yo can run:
nodetool scrub <keyspace>
in EACH of the nodes→ see log of the node and wait to complete
nodetool repair <keyspace>
I first suggest trying for 1 table for seeing the log of only that table and comprehend better what is going on
NOTE:
set the loggin levels to INFO to see the important things
nodetool setlogginglevel org.apache.cassandra INFO

Cassandra - Delete not working

Sometimes; when I perform a DELETE; it doesn't work.
My config : [cqlsh 5.0.1 | Cassandra 3.0.3 | CQL spec 3.4.0 | Native protocol v4]
cqlsh:my_db> SELECT * FROM conversations WHERE user_id=120 AND conversation_id=2 AND peer_type=1;
user_id | conversation_id | peer_type | message_map
---------+-----------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
120 | 2 | 1 | {0: {real_id: 68438, date: 1455453523, sent: True}, 1: {real_id: 68437, date: 1455453520, sent: True}, 2: {real_id: 68436, date: 1455453517, sent: True}, 3: {real_id: 68435, date: 1455453501, sent: True}, 4: {real_id: 68434, date: 1455453500, sent: True}, 5: {real_id: 68433, date: 1455453499, sent: True}, 6: {real_id: 68432, date: 1455453498, sent: True}, 7: {real_id: 68431, date: 1455453494, sent: True}, 8: {real_id: 68430, date: 1455453480, sent: True}}
(1 rows)
cqlsh:my_db> DELETE message_map FROM conversations WHERE user_id=120 AND conversation_id=2 AND peer_type=1;
cqlsh:my_db> SELECT * FROM conversations WHERE user_id=120 AND conversation_id=2 AND peer_type=1;
user_id | conversation_id | peer_type | message_map
---------+-----------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
120 | 2 | 1 | {0: {real_id: 68438, date: 1455453523, sent: True}, 1: {real_id: 68437, date: 1455453520, sent: True}, 2: {real_id: 68436, date: 1455453517, sent: True}, 3: {real_id: 68435, date: 1455453501, sent: True}, 4: {real_id: 68434, date: 1455453500, sent: True}, 5: {real_id: 68433, date: 1455453499, sent: True}, 6: {real_id: 68432, date: 1455453498, sent: True}, 7: {real_id: 68431, date: 1455453494, sent: True}, 8: {real_id: 68430, date: 1455453480, sent: True}}
(1 rows)
CQLSH doesn't return me any error on the DELETE instruction, but it's like if it wasn't taken in account.
Do you know why ?
NB : This is my table definition :
CREATE TABLE be_telegram.conversations (
user_id bigint,
conversation_id int,
peer_type int,
message_map map<int, frozen<message>>,
PRIMARY KEY (user_id, conversation_id, peer_type)
) WITH CLUSTERING ORDER BY (conversation_id ASC, peer_type ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

A DELETE statement removes one or more columns from one or more rows in a table, or it removes the entire row if no columns are specified. Cassandra applies selections within the same partition key atomically and in isolation.
When a column is deleted, it is not removed from disk immediately. The deleted column is marked with a tombstone and then removed after the configured grace period has expired. The optional timestamp defines the new tombstone record.
About deletes in Cassandra
The way Cassandra deletes data differs from the way a relational database deletes data. A relational database might spend time scanning through data looking for expired data and throwing it away or an administrator might have to partition expired data by month, for example, to clear it out faster. Data in a Cassandra column can have an optional expiration date called TTL (time to live).
Facts about deleted data to keep in mind are:
Cassandra does not immediately remove data marked for deletion from
disk. The deletion occurs during compaction.
If you use the sized-tiered or date-tiered compaction strategy, you
can drop data immediately by manually starting the compaction
process. Before doing so, understand the documented disadvantages of
the process.
A deleted column can reappear if you do not run node repair
routinely.
Why deleted data can reappear
Marking data with a tombstone signals Cassandra to retry sending a
delete request to a replica that was down at the time of delete. If
the replica comes back up within the grace period of time, it
eventually receives the delete request. However, if a node is down
longer than the grace period, the node can miss the delete because the
tombstone disappears after gc_grace_seconds. Cassandra always attempts
to replay missed updates when the node comes back up again. After a
failure, it is a best practice to run node repair to repair
inconsistencies across all of the replicas when bringing a node back
into the cluster. If the node doesn't come back within
gc_grace,_seconds, remove the node, wipe it, and bootstrap it again.
In your case, compaction is sized-tiered. So please try compaction process.
Compaction
Periodic compaction is essential to a healthy Cassandra database
because Cassandra does not insert/update in place. As inserts/updates
occur, instead of overwriting the rows, Cassandra writes a new
timestamped version of the inserted or updated data in another
SSTable. Cassandra manages the accumulation of SSTables on disk using
compaction.
Cassandra also does not delete in place because the SSTable is
immutable. Instead, Cassandra marks data to be deleted using a
tombstone. Tombstones exist for a configured time period defined by
the gc_grace_seconds value set on the table. During compaction, there
is a temporary spike in disk space usage and disk I/O because the old
and new SSTables co-exist. This diagram depicts the compaction
process:
Compaction merges the data in each SSTable data by partition key,
selecting the latest data for storage based on its timestamp.
Cassandra can merge the data performantly, without random IO, because
rows are sorted by partition key within each SSTable. After evicting
tombstones and removing deleted data, columns, and rows, the
compaction process consolidates SSTables into a single file. The old
SSTable files are deleted as soon as any pending reads finish using
the files. Disk space occupied by old SSTables becomes available for
reuse.
Data input to SSTables is sorted to prevent random I/O during SSTable
consolidation. After compaction, Cassandra uses the new consolidated
SSTable instead of multiple old SSTables, fulfilling read requests
more efficiently than before compaction. The old SSTable files are
deleted as soon as any pending reads finish using the files. Disk
space occupied by old SSTables becomes available for reuse.
so try this
nodetool <options> repair
options are:
( -h | --host ) <host name> | <ip address>
( -p | --port ) <port number>
( -pw | --password ) <password >
( -u | --username ) <user name>
-- Separates an option and argument that could be mistaken for a option.
keyspace is the name of a keyspace.
table is one or more table names, separated by a space.
This command starts the compaction process on tables that use the SizeTieredCompactionStrategy or DateTieredCompactionStrategy. You can specify a keyspace for compaction. If you do not specify a keyspace, the nodetool command uses the current keyspace. You can specify one or more tables for compaction. If you do not specify a table(s), compaction of all tables in the keyspace occurs. This is called a major compaction. If you do specify a table(s), compaction of the specified table(s) occurs. This is called a minor compaction. A major compaction consolidates all existing SSTables into a single SSTable. During compaction, there is a temporary spike in disk space usage and disk I/O because the old and new SSTables co-exist. A major compaction can cause considerable disk I/O.

Resources