In this [1] document, when it describes cfstats output it says, Read count is the Number of pending read requests. Is that correct? I was thinking that is all read requests received since last server restart.
can someone please clarify this?
[1] http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/operations/ops_monitoring_c.html
Thanks,
Bhathiya
Yes, you're right, the docs are wrong. cfstats read count is the number of local read requests for the table since start up.
Related
I am PhD student in Seoul National University. My name is Seokwon Choi. I impressed research paper(Analysis for network partition fault). I hope to present this paper with my lab member at lab seminar time.
However, I read your research paper and your presentation slide. I have one question.
Why the read operation read Y value in VoltDB? Actually replication is fail, so write is fail. Why it update Y value in local storage?
and read operation read value Y that updated locally?
I think read operation should read commit value(written successfully: in this case-> value X).
I try to find VoltDB Document. It can allow dirty read in VoltDB. Why allow dirty read when it happens network partition in VoltDB?
Is there any reason to work like this?
I attached picture of dirty read when network partition
Thank you
Best Regards
From Seokwon Choienter image description here
VoltDB does not allow dirty reads. In your picture, you show a 3-node cluster where 1 node gets partitioned from the other 2 and the single node is a partition master.
Event1: Network partition
Event2: Write to minority (and you show that the write fails, which is correct)
Event3: Read from minority (and you show a dirty read, which is incorrect).
Event 3 is not possible. The single node that gets partitioned from the other two will shut down its client interface and then crash, never allowing event 3 to happen.
We ran Jepsen tests several years ago and fixed a defect in V6.4 that in some circumstances would allow that dirty read from event#3. See https://aphyr.com/posts/331-jepsen-voltdb-6-3 and https://www.voltdb.com/blog/2016/07/12/voltdb-6-4-passes-official-jepsen-testing/ for the full details on the guarantees from VoltDB, the Jepsen testing we did, and the defects that were fixed in order to pass the test.
Disclosure: I work for VoltDB.
How to find out when a Cassandra table becomes "eventually consistent"? Is there a definitive way to determine this at a given point in time? Preferably programatically through the Datastax driver API? I checked out the responses to the following related questions but there does not seem to be anything more concrete than "check the nodetool netstats output"
Methods to Verify Cassandra Node Sync
how do i know if nodetool repair is finished
If your system is always online doing operations then it may never become full consistent at single point of time untill you are on Consistency level "ALL".
Repairs process logs error in log file if it does not get reply from other replica nodes cause they were down/timeout etc.
you can check the logs if no error WRT AntiEntropy/stream it means your system is almost consistence.
I was unable to find a good documentation/explanation as to what severity indicates in nodetool gossipinfo. was looking for a detailed explanation but could not find a suitable one.
The severity is a value added to the latency in the dynamic snitch to determine which replica a coordinator will send the read's DATA and DIGEST requests to.
Its value would depend on the IO used in compaction and also it would try to read /proc/stat (same as the iostat utility) to get actual disk statistics as its weight. In post 3.10 versions of cassandra this is removed in https://issues.apache.org/jira/browse/CASSANDRA-11738. In pervious versions you can disable it by setting -Dcassandra.ignore_dynamic_snitch_severity in jvm options. The issue is that it weighting the io use the same as the latency. So if a node is GC thrashing and not doing much IO because of it, it could end up being treated as the target of most reads even though its the worst possible node to send requests to.
Now you can still use JMX to set the value still (to 1) if you want to exclude it from being used for reads. A example use case is using nodetool disablebinary so application wont query it directly, then setting the severity to 1. That node would then only be queried by cluster if theres a CL.ALL request or a read repair. Its a way to take a node "offline" for maintenance from a read perspective but still allow it to get mutations so it doesn't fall behind.
Severity reports activity that happens on the particular node (compaction, etc.), and this information then is used to make a decision on what node could better handle the request. There is discussion in original JIRA about this functionality & how this information is used.
P.S. Please see Chris's answer about changes in post 3.10 versions - I wasn't aware about these changes...
Sorry if this is an existing question, but any of the existing ones resolved my problem..
I've installed Cassandra single noded. I don't have a large application right now, but I think this can be the case soon, and I will need more and more nodes..
Well, I'm saving data from a stream to Cassandra, and this were going well, but suddently, when I tried to read data, I've started to receive this error:
"Not enough replica available for query at consistency ONE (1 required but only 0 alive)"
My keyspace was built using simplestrategy with replication_factor = 1. Im saving data separated by a field called "catchId", so most of my queries are like: "select * from data where catchId='xxx'". catchId is a partition key.
I'm using the cassandra-driver-core version 3.0.0-rc1.
The thing is that I don't have that much of data rigth now, and I'm thinking if it will be better to use a RDBMS for now, and migrate to Cassandra only when I have a better infrastructure.
Thanks :)
It seems that your node is unable to respond when you try to make your read (in general this error appears for more than one node). If you do not have lots of data, it's very strange, so this is probably a bad design choice. This can emanate from several things, so you have to make a few investigations.
study your logs ! In particular the system.log
you can change your read_request_timeout_in_ms parameter in cassandra.yaml. Although it's not agood idea in production, it will say you if it's just temporary problem (your request succeed after a little time) or a bigger problem
study your CPU and memory behavior when you are doing requests
if you are very motivated, you can install opscenter which will you give more valuable informations
How and how many write requests are you doing ? It can overwhelm cassandra (even if it's designed for). I recommend to make async requests to avoid problems.
Can anyone please let me know what to look/monitor in nodetool netstats command and its thresholds.
when we say we monitor the number of active, pending, and completed commands and responses, what should the threshold be? I was reading a blog and they told the thresholds are 5 and 10. I am having trouble understanding is it 5 pending commands or 5% of pending commands or a ratio between pending and active commands.
Sorry if this is silly. I am new to Cassandra.
The numbers you see in the netstats output are actual counts. See this doc link:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsNetstats.html
Could you elaborate on what you mean by thresholds, where are you seeing this, or what blog are you referring to?