I had Fetch Multiple Documents. I had cast all the Documents to there respective data Models. After Updating properties of each Document, I need to save all the documents from a stored procedure.
I have Read the Documentation of stored procedure Collection in this the Method replaceDocument(documentLink, document, optionsopt, callbackopt) Required documentLink which I can not find after Casting.
I have tried this function but it did not work
function bulkReplace(docs) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var count = 0;
if (!docs) throw new Error("The array is undefined or null.");
var docsLength = docs.length;
if (docsLength == 0) {
getContext().getResponse().setBody(0);
return;
}
tryCreate(docs[count], callback);
function tryCreate(doc, callback) {
var isAccepted = collection.replaceDocument(doc._self, doc, callback);
if (!isAccepted) getContext().getResponse().setBody(count);
}
function callback(err, doc, options) {
if (err) throw err;
count++;
if (count >= docsLength) {
getContext().getResponse().setBody(count);
} else {
tryCreate(docs[count], callback);
}
}
}
Exception details:
"Unable to Process Request","errorDetails":"The server encountered a
problem while processing your request, please try again
"technicalReason":"Type:Microsoft.Azure.Documents.DocumentClientException
Source:Microsoft.Azure.Documents.Client Message:Message:
{\"Errors\":[\"Encountered exception while executing Javascript.
Exception = Error: Invalid document link: \"undefined\".\r\nStack
trace: Error: Invalid document link: \"undefined\".\n at
validateDocumentLink
(bulkReplace.js:349:21)\natreplaceDocument(bulkReplace.js:780:17)\n
at tryCreate (bulkReplace.js:45:9)
I tested your code on my side and it works.
As we know, documents in azure document db has a few auto-generate fields including "_self". You do not need to make another query in stored procedure.You just need to ensure the documents in docs param you import contain the right "_self" field, otherwise the invalid document link exception occurs.
For example :
var doc = { "id": "2",
"name" : "ccc",
"_self": "dbs/duUuAA==/colls/duUuAPfBbAA=/docs/duUuAPfBbAAzAQAAAAAAAA==/",
}
I suggest you use console.log() to print the doc._self in your code to check the value of it.
Hope it helps you.
Related
In a post function, I am trying to retrieve the nth activity of a user (since I have a dropdown that return the index number of the activity). When I run the query
collection.find({'local.email':req.user.local.email},
{'local.activities':{$slice : [currActivity,1]}});
I receive the correct activity object in Robo3T.
But, when I call the same query in Node inside a post function, it returns an undefined.
app.post('/addlog',function(req,res){
var currActivity = req.body.curAct;
var score = req.body.score;
var comment = req.body.reason;
mongoose.connect('mongodb://****:****#ds044907.mlab.com:44907/intraspect',function (err, database) {
if (err)
throw err
else
{
db = database;
var collection = db.collection('users');
var retrievedAct = collection.find({'local.email':req.user.local.email},
{'local.activities':{$slice : [currActivity,1]}}).toArray().then(console.log(retrievedAct));
if (retrievedAct.length > 0) { printjson (retrievedAct[0]); }
console.log(currActivity);
console.log(retrievedAct[0]);
// console.log(req.body.newAct);
collection.update({'local.activities.name':retrievedAct[0]},
{$push: {'local.activities.log' : {
comments: comment,
score: score,
log_time: Date.now()
}}})
.then(function(){
res.redirect('/homepage');
})
.catch(function() {
console.log('Error');
});
}
});
});
I checked that the currActivity variable does infact contain the integer value for the nth activity.
If you want the result of collection.find().toArray(), as specified in the docs, you have two options:
Passing a callback to .toArray() like you did with mongoose.connect()
Using the Promise that it returns if you don't pass a callback
Now you are doing neither of them.
Also, you are mixing callback style and Promises in your code. I recommend you unificate your code. If you are using a Node.js version bigger than 8, using async/await could be nice, it makes it simpler.
I am trying to access a key with unknown parent as :
var ref = admin.database().ref("user_contacts/{pushId}").child(event.data.ref.parent.key);
ref.once("value", function(querySnapshot) {
var qVal = querySnapshot.val();
console.log("Query Value:", qVal);
});
Snapshot always return null, data exists for this key under multiple.
{pushId} //push ids are unknown here.
I also tried this:
var ref = admin.database().ref("user_contacts/{pushId}").orderByChild(snapshot.ref.parent.key);
ref.once("value", function(querySnapshot) {
var qVal = querySnapshot.val();
console.log("Query Value:", qVal);
});
but same response.
Is there any else way to achieve this? Or any body here can suggest an improvement in my method? I am just trying to search a key in the database.
Hi I am using 16 collections to insert around 3-4 million json objects ranging from 5-10k per object.I am using stored procedure to insert these documents.I have 22 Capacity Unit.
function bulkImport(docs) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
// The count of imported docs, also used as current doc index.
var count = 0;
// Validate input.
if (!docs) throw new Error("The array is undefined or null.");
var docsLength = docs.length;
if (docsLength == 0) {
getContext().getResponse().setBody(0);
}
// Call the CRUD API to create a document.
tryCreateOrUpdate(docs[count], callback);
// Note that there are 2 exit conditions:
// 1) The createDocument request was not accepted.
// In this case the callback will not be called, we just call setBody and we are done.
// 2) The callback was called docs.length times.
// In this case all documents were created and we don't need to call tryCreate anymore. Just call setBody and we are done.
function tryCreateOrUpdate(doc, callback) {
var isAccepted = true;
var isFound = collection.queryDocuments(collectionLink, 'SELECT * FROM root r WHERE r.id = "' + doc.id + '"', function (err, feed, options) {
if (err) throw err;
if (!feed || !feed.length) {
isAccepted = collection.createDocument(collectionLink, doc, callback);
}
else {
// The metadata document.
var existingDoc = feed[0];
isAccepted = collection.replaceDocument(existingDoc._self, doc, callback);
}
});
// If the request was accepted, callback will be called.
// Otherwise report current count back to the client,
// which will call the script again with remaining set of docs.
// This condition will happen when this stored procedure has been running too long
// and is about to get cancelled by the server. This will allow the calling client
// to resume this batch from the point we got to before isAccepted was set to false
if (!isFound && !isAccepted) getContext().getResponse().setBody(count);
}
// This is called when collection.createDocument is done and the document has been persisted.
function callback(err, doc, options) {
if (err) throw err;
// One more document has been inserted, increment the count.
count++;
if (count >= docsLength) {
// If we have created all documents, we are done. Just set the response.
getContext().getResponse().setBody(count);
} else {
// Create next document.
tryCreateOrUpdate(docs[count], callback);
}
}
my C# codes looks like this
public async Task<int> Add(List<JobDTO> entities)
{
int currentCount = 0;
int documentCount = entities.Count;
while(currentCount < documentCount)
{
string argsJson = JsonConvert.SerializeObject(entities.Skip(currentCount).ToArray());
var args = new dynamic[] { JsonConvert.DeserializeObject<dynamic[]>(argsJson) };
// 6. execute the batch.
StoredProcedureResponse<int> scriptResult = await DocumentDBRepository.Client.ExecuteStoredProcedureAsync<int>(sproc.SelfLink, args);
// 7. Prepare for next batch.
int currentlyInserted = scriptResult.Response;
currentCount += currentlyInserted;
}
return currentCount;
}
The problem I am facing is out of 400k documents that I try to insert at times documents get missed with out giving any error.
The application is worker role deployed on cloud.
If I increase the number of threads or instances inserting in documentDB the number of documents missed are much higher.
how to figure out what is the problem.Thanks in Advance.
I found that when trying this code I would get an error at docs.length which stated that length was undefined.
function bulkImport(docs) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
// The count of imported docs, also used as current doc index.
var count = 0;
// Validate input.
if (!docs) throw new Error("The array is undefined or null.");
var docsLength = docs.length; // length is undefined
}
After many tests (could not find anything in Azure documentation) I realized that I could not pass an array as was suggested. The parameter had to be an object. I had to modify the batch code like this in order for it to run.
I also found I could not simply try and pass an array of documents in the DocumentDB script explorer (Input box) either. Even though the placeholder help text says you can.
This code worked for me:
// psuedo object for reference only
docObject = {
"items": [{doc}, {doc}, {doc}]
}
function bulkImport(docObject) {
var context = getContext();
var collection = context.getCollection();
var collectionLink = collection.getSelfLink();
var count = 0;
// Check input
if (!docObject.items || !docObject.items.length) throw new Error("invalid document input parameter or undefined.");
var docs = docObject.items;
var docsLength = docs.length;
if (docsLength == 0) {
context.getResponse().setBody(0);
}
// Call the funct to create a document.
tryCreateOrUpdate(docs[count], callback);
// Obviously I have truncated this function. The above code should help you understand what has to change.
}
Hopefully Azure documentation will catch up or become easier to find if I missed it.
I'll also be placing a bug report for the Script Explorer in hopes that the Azurites will update.
It’s important to note that stored procedures have bounded execution, in which all operations must complete within the server specified request timeout duration. If an operation does not complete with that time limit, the transaction is automatically rolled back. In order to simplify development to handle time limits, all CRUD (Create, Read, Update, and Delete) operations return a Boolean value that represents whether that operation will complete. This Boolean value can be used a signal to wrap up execution and for implementing a continuation based model to resume execution (this is illustrated in our code samples below).
The bulk-insert stored procedure provided above implements the continuation model by returning the number of documents successfully created. This is noted in the stored procedure's comments:
// If the request was accepted, callback will be called.
// Otherwise report current count back to the client,
// which will call the script again with remaining set of docs.
// This condition will happen when this stored procedure has been running too long
// and is about to get cancelled by the server. This will allow the calling client
// to resume this batch from the point we got to before isAccepted was set to false
if (!isFound && !isAccepted) getContext().getResponse().setBody(count);
If the output document count is less than the input document count, you will need to re-run the stored procedure with the remaining set of documents.
Since May 2018 there is a new Batch SDK for Cosmos DB. There is a GitHub repo to get you started.
I have been able to import 100.000 records in 9 seconds. And using Azure Batch to fan out the inserts, I have done 19 mln records in 1m15s. This was on a 1.66mln RU/s collection, which you obviously can scale down after import.
I am using node.js with mongoose. The problem i am facing is i am getting newModifier1 printed but outside that function the value is null.
Here is my code:
// Find userSchema
newModifier1 = "";
exports.findModifier = function(modifierName){
modifierModel.find({'name' : modifierName},function(err,result){
if(err){
console.log("Error : "+err);
throw err;
}
else{
newModifier1 = result;
// console.log("Modifier is searched successfully : "+newModifier1);
}
console.log("Modifier is searched successfully1 : "+newModifier1);
});
// newModifier1=temp;
return newModifier1; // it takes newModifier1 = "" value here
}
Any ideas what the problem could be?
This is what is happening:
// this is "global" an would be weirdly overwritten
// if function is called multiple times before finishing
newModifier1 = "";
exports.findModifier = function(modifierName){
// TIMESTAMP: 0
modifierModel.find({'name' : modifierName},function(err,result){
// TIMESTAMP: 2
if(err){
console.log("Error : "+err);
throw err;
}
else{
newModifier1 = result;
// console.log("Modifier is searched successfully : "+newModifier1);
}
console.log("Modifier is searched successfully1 : "+newModifier1);
});
// TIMESTAMP: 1
return newModifier1; // it takes newModifier1 = "" value here
}
I added some notes, when what is happening. As you can see and because of the async nature of node.js you return the value before you get a result back from the database.
You need familiarize yourself with the async flow and callback function.
Pass a callback function to findModifier and wait for the database to return a result.
modifierModel.find runs asynchronously and probably findModifier method is returning before the callback of find method executes. Although you see it being printed out what is returned from the method is en empty string anyway. You can use a library like async.
I am trying to link up a redis database with a Node.js application I am building to be able to store comments about items. I'm using the node_redis library to handle the connection. When I attempt to retrieve the comments out of the database however only "[true]" is returned. For testing purposes I have stuffed everything into one method and I have hardcoded the values in, but I still receive "[true]".
exports.getComment = function (id){
var comments = new Array();
rc.hmset("hosts", "mjr", "1", "another", "23", "home", "1234");
comments.push(rc.hgetall("hosts", function (err, obj) {
var comment = new Array();
if(err){
comment.push("Error");
} else {
comment.push(obj);
}
return comment;
}));
return comments;
}
Updated the code according to the tutorial and here is the result:
Retrieving the comment:
exports.getComment = function (id, callback){
rc.hgetall(id, callback);
}
Adding the comment:
exports.addComment = function (id, area, content, author){
//add comment into the database
rc.hmset("comment",
"id", id,
"area", area,
"content", content,
"author" , author,
function(error, result) {
if (error) res.send('Error: ' + error);
});
//returns nothing
};
Code to render:
var a = [];
require('../lib/annotations').addComment("comment");
require('../lib/annotations').getComment("comment", function(comment){
a.push(comment)
});
res.json(a);
Node.js is asynchronous. Which means it asynchronously does the redis stuff, and then gets the result back in the callback function.
I suggest you read this tutorial and fully understand it before getting further: http://howtonode.org/node-redis-fun
Basically, this way won't work:
function getComments( id ) {
var comments = redis.some( action );
return comments;
}
But it has to be this way:
function getComments( id, callback ) {
redis.some( action, callback );
}
This way, you use the API like this:
getComments( '1', function( results ) {
// results are available!
} );
The problem lays within the actual Redis-Node library when the call to addComment is made as it is below.
require('../lib/annotations').getComment("comment", function(comment){
a.push(comment)
});
This call is missing an argument in the callback function. The first argument is the error report which should return null if everything is ok, the second is the actual data. So it should be structured like the call below.
require('../lib/annotations').getComment("comment", function(comment){
a.push(err, comment)
});