I am working on Bulk insert stored procedure in Cosmos database using document client but the challenge that I am facing is, I need to insert documents in bulk that may have different partition keys.
Is there any way to achieve it?
I am currently using the below code:
Uri uri = UriFactory.CreateStoredProcedureUri("test-db", "test-collection", "sp_bulk_insert");
RequestOptions options = new RequestOptions { PartitionKey = new PartitionKey("patient")};
var result = await _client.ExecuteStoredProcedureAsync<string>(uri, options , patientInfo, pageInfo);
return result;
But I also have pageInfo object having partition key: "page" but given PartitionKey in RequestOptions is "patient" that is the partition key of patientInfo object
When I am trying to execute the SP it is giving following error:
Requests originating from scripts cannot reference partition keys other than the one for which client request was submitted
Stored procedures are scoped to a single partition key so this not possible. Also there is no reason to use stored procedures for bulk operations. You are better off using the .NET SDK v3 and leveraging the bulk support in there. https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/BulkSupport
Thanks.
Related
I am using GraphQL for Cassandra database operation. The following search query work perfectly when filtering the column with partition key:
query oneUsers{
users(value: { username:"username" }) {
values {
id
name
username
}
}
}
But getting following errors when trying to search other columns:
graphql: Exception while fetching data (/users) : org.apache.cassandra.stargate.exceptions.InvalidRequestException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING
What is the best way to search columns in Cassandra using GraphQL?
You can only retrieve records from Cassandra by specifying (a) the partition key, or (b) the primary key column(s).
If you would like to filter by non-primary key columns, you will need to create an index on the column. For example if you want to filter by id, create an index with:
mutation createIndexes {
users: createIndex(
keyspaceName:"myks",
tableName:"users",
columnName:"id",
indexName:"id_idx"
)
}
You should then be able to query by id.
For details and more examples, see Developing with the Astra DB GraphQL API. Cheers!
How can I delete a document from a collection.
AsyncDocumentClient client = getDBClient();
RequestOptions options = new RequestOptions();
options.setPartitionKey(new PartitionKey("143003"));
client.deleteDocument(String.format("dbs/test-lin/colls/application/docs/%s", document.id()), options);
I am trying to delete a set of documents from collection based on some condition. I have set the partition key. The read-write keys are being used (So no permission issue).
There are no errors when executing this code. The document is not getting deleted from the collection.
How to fix the issue?
#Suj Patil
You should call subscribe(). The publisher does not do anything until some one subscribes.
client.deleteDocument(String.format("dbs/test-lin/colls/application/docs/%s", document.id()), options).subscribe()
I have a collection with the partition key "flightConversationId".
I am doing a very simple query, BY THE PARTITON KEY FIELD
SELECT * from root WHERE root.flightConversationId="b36d13c0-cbec-11e7-a4ad-8fcedf370f98"
When doing this query via the nodeJS SDK, it will work one second, and fail the next with the error:
Cross partition query is required but disabled. Please set x-ms-documentdb-query-enablecrosspartition to true, specify x-ms-documentdb-partitionkey, or revise your query to avoid this exception.
I realize I could enable cross-partition querying, but I do not need cross partition queries. What is going on???
This situation seemed to resolve itself over time.
My theory is that when we deleted a collection and recreated it with a new partition key, it took a long time for all remnants of the original collection to really be deleted from the cloud, and that some requests were going to the "old" collection that had the same name as the "new".
You have to explicitly scope the query to a partition by providing an FeedOptions or RequestOptions class with a partitionKey property. Using the PartitionKey in your where clause isn't enough without that explicit scope. This is for C# but should be same object model:
https://learn.microsoft.com/en-us/azure/cosmos-db/documentdb-partition-data
Document result = await client.ReadDocumentAsync(
UriFactory.CreateDocumentUri("db", "coll", "XMS-001-FE24C"),
new RequestOptions { PartitionKey = new PartitionKey("XMS-0001") });
jsDoc:
http://azure.github.io/azure-documentdb-node/global.html#RequestOptions
http://azure.github.io/azure-documentdb-node/global.html#FeedOptions
Is it possible to insert a new document into a Couchbase bucket without specifying the document's ID? I would like use Couchbase's Java SDK create a document and have Couchbase determine the document's UUID with Groovy code similar to the following:
import com.couchbase.client.java.CouchbaseCluster
import com.couchbase.client.java.Cluster
import com.couchbase.client.java.Bucket
import com.couchbase.client.java.document.JsonDocument
// Connect to localhost
CouchbaseCluster myCluster = CouchbaseCluster.create()
// Connect to a specific bucket
Bucket myBucket = myCluster.openBucket("default")
// Build the document
JsonObject person = JsonObject.empty()
.put("firstname", "Stephen")
.put("lastname", "Curry")
.put("twitterHandle", "#StephenCurry30")
.put("title", "First Unanimous NBA MVP)
// Create the document
JsonDocument stored = myBucket.upsert(JsonDocument.create(person));
No, Couchbase documents have to have a key, that's the whole point of a key-value store, after all. However, if you don't care what the key is, for example, because you retrieve documents through queries rather than by key, you can just use a uuid or any other unique value when creating the document.
It seems there is no way to have Couchbase generate the document IDs for me. At the suggestion of another developer, I am using UUID.randomUUID() to generate the document IDs in my application. The approach is working well for me so far.
Reference: https://forums.couchbase.com/t/create-a-couchbase-document-without-specifying-an-id/8243/4
As you already found out, generating a UUID is one approach.
If you want to generate a more meaningful ID, for instance a "foo" prefix followed by a sequence number, you can make use of atomic counters in Couchbase.
The atomic counter is a document that contains a long, on which the SDK relies to guarantee a unique, incremented value each time you call bucket.counter("counterKey", 1, 2). This code would take the value of the counter document "counterKey", increment it by 1 atomically and return the incremented value. If the counter doesn't exist, it is created with the initial value 2, which is the value returned.
This is not automatic, but a Couchbase way of creating sequences / IDs.
I have a use case where I have to remove a subset of entities stored in couchbase, e.g. removing all entities with keys starting with "pii_".
I am using NodeJS SDK but there is only one remove method which takes one key at a time: http://docs.couchbase.com/sdk-api/couchbase-node-client-2.0.0/Bucket.html#remove
In some cases thousands of entities need to be deleted and it takes very long time if I delete them one by one especially because I don't keep list of keys in my application.
I agree with the #ThinkFloyd when he saying: Delete on server should be delete on server, rather than requiring three steps like get data from server, iterate over it on client side and finally for each record fire delete on the server again.
In this regards, I think old fashioned RDBMS were better all you need to do is 'DELETE * from database where something=something'.
Fortunately, there is something similar to SQL is available in CouchBase called N1QL (pronounced nickle). I am not aware about JavaScript (and other language syntax) but this is how I did it in python.
Query to be used: DELETE from <bucketname> b where META(b).id LIKE "%"
layer_name_prefix = cb_layer_key + "|" + "%"
query = ""
try:
query = N1QLQuery('DELETE from `test-feature` b where META(b).id LIKE $1', layer_name_prefix)
cb.n1ql_query(query).execute()
except CouchbaseError, e:
logger.exception(e)
To achieve the same thing: alternate query could be as below if you are storing 'type' and/or other meta data like 'parent_id'.
DELETE from <bucket_name> where type='Feature' and parent_id=8;
But I prefer to use first version of the query as it operates on key, and I believe Couchbase must have some internal indexes to operate/query faster on key (and other metadata).
The best way to accomplish this is to create a Couchbase view by key and then range query over that view via your NodeJS code, making deletes on the results.
http://docs.couchbase.com/admin/admin/Views/views-querySample.html
http://docs.couchbase.com/couchbase-manual-2.0/#couchbase-views-writing-querying-selection-partial
http://docs.couchbase.com/sdk-api/couchbase-node-client-2.0.8/ViewQuery.html
For example, your Couchbase view could look like the following:
function(doc, meta) {
emit(meta.id, null);
}
Then in your NodeJS code, you could have something that looks like this:
var couchbase = require('couchbase');
var ViewQuery = couchbase.ViewQuery;
var query = ViewQuery.from('designdoc', 'by_id');
query.range("pii_", "pii_" + "\u0000", false);
var myBucket = myCluster.openBucket();
myBucket.query(query, function(err, results) {
for(i in results) {
// Delete code in here
}
});
Of course your Couchbase design document and view will be named differently than the example that I gave, but the important part is the ViewQuery.range function that was used.
All document ids prefixed with pii_ would be returned, in which case you can loop over them and start deleting.
Best,