I have a requirement where I need to read the documents immediately after writing to the Cloudant DB. Sometimes the DB returns inconsistent results which is expected. But can we avoid eventual consistency by setting the quorum parameter to a higher value (> 2) in the query?
{
"selector": {
"year": {
"$gt": 2010
}
},
"r": 5
}
No.
There is no way to "remove" eventual consistency in Cloudant (safely). None. My advice would be to not mess with quorum and replicas other than the default values which are set as they are for very good reasons.
If you find yourself needing to work around eventual consistency, either your workload or your data model is unsuited for Cloudant.
Here's a good intro (disclaimer: I'm the author) https://blog.cloudant.com/2019/11/21/Best-and-Worst-Practices.html
Related
I'm creating a logging system to monitor our (200 ish) main application installations, and Cosmos db seems like a good fit due to the amount of data we'll collect, and to allow a varying schema for the log data (particularly the Tags array - see document schema below).
But, never having used CosmosDb before I'm slightly unsure of what to use for my partition key.
If I partitioned by CustomerId, there would likely be several Gb of data in each of the 200 partitions, and the data will usually be queried by CustomerId, so this was my first choice for the partition key.
However I was planning to have a 'log stream' view in the logging system, showing logs coming in for all customers.
Would this lead to running a horribly slow / expensive cross partition query?
If so, is there an obvious way to avoid / limit the cost & speed implications of this cross partition querying? (Other than just taking out the log stream view for all customers!)
{
"CustomerId": "be806507-7cc4-4db4-881b",
"CustomerName": "Our Customer",
"SystemArea": 1,
"SystemAreaName": "ExchangeSync",
"Message": "Updated OK",
"Details": "",
"LogLevel": 2,
"Timestamp": "2018-11-23T10:59:29.7548888+00:00",
"Tags": {
"appointmentId": "109654",
"appointmentGroupId": "86675",
"exchangeId": "AAMkA",
"exchangeAlias": "customer.name#customer.com"
}
}
(Note - There isn't a defined list of SystemArea types we'll use yet, but it would be a lot fewer than the 200 customers)
Cross partition queries should be avoided as much as possible. If your querying is likely to happen with customer id then the customerid is a good logical partition key. However you have to keep in mind that there is a limit of 10GB per logical partition data.
A cross partition query across the whole database will lead to a very slow and very expensive operation but if it's not functionality critical and it's just used for infrequent reporting, it's not too much of a problem.
Why the documents are fetched from primary shard and replica shard when I run the same query again and again. Because of this I am getting different search results.
Example response - 1 - Replica
"_shard": 0,
"_node": "node_1",
"_index": "sample_ind",
"_type": "my_type",
"_id": "E1",
"_score": 2.9560382,
Response-2 Primary shard
"_shard": 0,
"_node": "node_2",
"_index": "sample_ind",
"_type": "my_type",
"_id": "E2",
"_score": 2.956294,
node-1 has the replica shard and node-2 has the primary shard. How the query fetch works and why the response comes from primary shard and replica shard when i run the same query multiple time ?
Difficult to say, may you give me more detail about your results ?
Elastic's site contains a good article to understand how to query fetch results from primary/replica shards: https://www.elastic.co/guide/en/elasticsearch/guide/current/_query_phase.html
Hth,
This is basic Elasticsearch information and I strongly suggest to go over the documentation to at least grasp the elementary knowledge about Elasticsearch.
In short, when a query comes to the cluster, the shards that need to be queried can be either primaries or replicas. It does not matter, they have the same data in them and can perform the query equally. I don't recommend running your queries against only primaries or only replicas, as it will create hot-spots in your cluster and can destabilize the cluster.
Also, the scoring on primaries and replicas should be almost the same. Part of the algorithm to calculate the score involves how many documents exist in the shard and the frequency of terms in these documents. The tricky part is that when you update or delete a document, that document is not immediately removed from disk, it is only marked for deletion. In the background Elasticsearch merges shard files and takes smaller, similar in size segments and create a bigger segment and deletes the smaller ones. At merging time the marked-as-deleted documents actually are removed from the index.
Until then, these documents are not returned in searches, but they are considered when calculating the scores as mentioned above.
Currently I'm doing filtered replication by monitoring the below resource:
_changes?filter=_selector&include_docs=true&attachments=true&limit=20
As you can see, I'm using a selector defined by
"selector": {
"type": "Property"
}
and everything is working great. Now I need to add another criteria which is geospatial index. I want to replicate documents with locations in a radius. e.g.
lat=-11.05987446&lon=12.28339928&radius=100
How can I replicate using the above filtered replication technique and replicate documents within a radius?
Thanks
The selector filter for _changes is not backed by an index - it just uses the same syntax as Query selectors, which currently does not support geospatial operations.
I think you have 3 options:
1. Use a bounding box
Your selector would then look something like:
"selector": {
"type": "Property",
"lat": {
"$gt": -11
},
"lat": {
"$lt": 11
},
"lon": {
"$gt": 12
},
"lon": {
"$lt": 14
}
}
Perhaps you could then further restrict the results on the client if you need exactly a radial search.
2. Implement a radius search in a JavaScript filter
This means dropping the use of the `selector, would be relatively slow (anything that involves JavaScript in Couch/Cloudant will be) but gives you exactly the result you want.
3. Run a query and replicate the resulting ids
Use a search or geospatial query to get the set of _ids you need and use a doc_id based replication to fetch them.
Lastly, it's worth considering whether you really want replication (which implies the ability for documents to sync both ways) or if you're just caching / copying data to the client. Replication carries some overhead beyond just copying the data (it needs to figure out the delta between the client and server, retrieve the rev history for each doc, etc) so, if you don't need to write the docs back to the server, maybe you don't need it.
If you do go down the replication route, you may need to handle cases where documents that you previously replicated no longer match the query so updates do not propagate in subsequent replications.
If not, you may be better off just running a query with include_docs=true and manually inserting the documents to a local database.
The _selector filter for the changes feed isn't backed by an index, it's basically a handy shortcut to achieve the same thing as a javascript filter, but much faster as it's executed directly in the Erlang.
As it's not index-backed, you can't tap into a geo-index that way.
You'd be better off to run either a bounding box or radius query to get the ids and then fetch those documents with a bulk_get or with a post to all_docs with the ids in the body.
https://cloudant.com/wp-content/uploads/Cloudant-Geospatial-technical-overview.pdf
I have an application that perform only inserts/deletes in cassandra. All write operations (inserts/deletes) application perform using consistency level QUORUM, read operation currently is executed using QUORUM as well, but i`m wondering if in this case (when there is no updates to data) consistency level ONE would give same results as QUORUM.
Not necessarily. It could be that your read request goes to the one node which has not (yet) received/applied the updates. The QUORUM consistency level does allow for some nodes to not have the updated data just yet; by using a consistency level of ONE for your read, you might read stale data. Perhaps your application can deal with this situation -- that's for you to decide. But you should know that a consistency level of ONE for reads may not return the data you expect in your situation.
Hope this helps!
This question is about NoSQL (for instance take cassandra).
Is it true that when you use a NoSQL database without data replication that you have no consistency concerns? Also not in the case of access concurrency?
What happens in case of a partition where the same row has been written in both partitions, possible multiple times? When the partition is gone, which written value is used?
Let's say you use N=5 W=3 R=3. This means you have guaranteed consistency right? How good is it to use this quorum? Having 3 nodes returning the data isn't that a big overhead?
Can you specify on a per query basis in cassandra whether you want the query to have guaranteed consistency? For instance you do an insert query and you want to enforce that all replica's complete the insert before the value is returned by a read operation?
If you have: employees{PK:employeeID, departmentId, employeeName, birthday} and department{PK:departmentID, departmentName} and you want to get the birthday of all employees with a specific department name. Two problems:
you can't ask for all the employees with a given birthday (because you can only query on the primary key)
You can't join the employee and the department column families because joins are impossible.
So what you can do is create a column family:
departmentBirthdays{PK:(departmentName, birthday), [employees-whos-birthday-it-is]}
In that case whenever an employee is fired/hired it has to be removed/added in the departmentBirthdays column family. Is this process something you have to do manually? So you have to manually create queries to update all redundant/denormalized data?
I'll answer this from the perspective of cassandra, coz that's what you seem to be looking at (hardly any two nosql stores are the same!).
For a single node, all operations are in sequence. Concurrency issues can be orthogonal though...your web client may have made a request, and then another, but due to network load, cassandra got the second one first. That may or may not be an issue. There are approaches around such problems, like immutable data. You can also leverage "lightweight transactions".
Cassandra uses last write wins to resolve conflicts. Based on you replication factor and consistency level for your query, this can work well.
Quurom for reads AND writes will give you consistency. There is an edge case..if the coordinator doesn't know a quorum node is down, it sends the write requests, then the write would complete when quorum is re-established. The client in this case would get a timeout and not a failure. The subsequent query may get the stale data, but any query after that will get latest data. This is an extreme edge case, and typically N=5, R=3, W3= will give you full consistency. Reading from three nodes isn't actually that much of an overhead. For a query with R=3, the client would make that request to the node it's connected to (the coordinator). The coordinator will query replicas in parallel (not sequenctially). It willmerge up the results with LWW to get the result (and issue read repairs etc. if needed). As the queries happen in parallel, the overhead is greatly reduced.
Yes.
This is a matter of data modelling. You describe one approach (though partitioning on birthday rather than dept might be better and result in more even distribution of partitions). Do you need the employee and department tables...are they needed for other queries? If not, maybe you just need one. If you denormalize, you'll need to maintain the data manually. In Cassandra 3.0, global indexes will allow you to query on an index without being inefficient (which is the case when using a secondary index without specifying the partition key today). Yes another option is to partition employeed by birthday and do two queries, and do the join in memory in the client. Cassandra queries hitting a partition are very fast, so doing two won't really be that expensive.