I need to store user's info in DynamoDB and send a mail to the same user if it doesn't already exist in DynamoDB table. I am doing this in for loop. The list contains only 2 records. The issue is only the second record gets inserted in table and the mail is sent twice to the same user. Here is the code:
module.exports.AddUser = function(req, res, usersList, departmentId) {
var _emailId = "";
var _userName = "";
var _departmentId = departmentId;
for (var i = 0; i < usersList.length; i++) {
_emailId = usersList[i].emailId;
_userName = usersList[i].userName;
var params = {
TableName: "UsersTable",
Key: {
"emailId": _emailId,
"departmentId": _departmentId
}
};
docClient.get(params, function(err, data) {
if (!err) {
if (!data.items)
AddUserAndSendEmail("UsersTable", _emailId, _userName);
//The above function is being called twice but for the same user.
//It has a check so not inserting the same record twice but
//sending two mails to the same user.
}
});
}
res.end("success");
}
function AddUserAndSendEmail(tableName, emailId, _userName) {
var params = {
TableName: tableName,
Item: {
"emailId": emailId,
"departmentId": 101//Default Department
}
};
docClient.put(params, function(err, data) {
if (!err) {
//Send Email Code Here
} else {
console.log("error");
}
});
}
What could be the reason for this strange behavior? Really frustrated, I am about to give up on this.
1) Please note that DynamoDB is eventually consistent. If you insert the item and check whether the item exists immediately, it may not always find the item in the database.
This means the second iteration of the loop may not always find the first item inserted into the table.
2) If the item already exists in the table, the Put api will update the item and give successful response.
This means the Put will be successful for the same email id and department id in the second iteration because it updates the record if it is already present.
GetItem – The GetItem operation returns a set of Attributes for an
item that matches the primary key. The GetItem operation provides an
eventually consistent read by default. If eventually consistent reads
are not acceptable for your application, use ConsistentRead.
PutItem – Creates a new item, or replaces an old item with a new item
(including all the attributes). If an item already exists in the
specified table with the same primary key, the new item completely
replaces the existing item. You can also use conditional operators to
replace an item only if its attribute values match certain conditions,
or to insert a new item only if that item doesn’t already exist.
Based on the above points, there is a possibility to get two emails if you have same email id and department id in the array.
Related
I use firebase on node.js .
My given structure should look like this:
{
...
batch-1:
id-1(suppose):
name:...
phone:...
id-2:
...
id-3:
...
batch-2:
...
batch-3:
...
...
batch-n:
...
}
How can I get an id-1 object by its identifier in such an architecture?
Does the database have to go around all the batches?
Is there a better solution?
The main task: Create a batch with many objects that will have SHORT and a UNIQUE identifier and optimally receive data by this identifier
To search for a particular ID that is a child of a list of unknown IDs, you need to use orderByChild(). In your use case, you are looking for a particular ID in a list of batch IDs. If you used orderByChild() on this list, you would get back results for each and every batch ID, even if it didn't have the ID you wanted. This is because even null (non-existant) values are included (and sorted at the start) in the results. To get the data of the desired ID, you would get the data for the last result of the query, which if it existed, would be sorted to the end of the list. Note that if the desired ID doesn't exist, the last result (if there are any results) would have a null value. To return only the last result of the query, you would use limitToLast(1).
Putting this all together, gives the following code:
let idToFind = "unique-id-1";
let batchesRef = firebase.database().ref(); // parent key of "batch-1", "batch-2", etc.
// assumed to be the database root here
batchesRef.orderByChild(idToFind).limitToLast(1).once('value')
.then((querySnapshot) => {
if (!querySnapshot.numChildren()) { // handle rare no-results case
throw new Error('expected at least one result');
}
let dataSnapshot;
querySnapshot.forEach((snap) => dataSnapshot = snap); // get the snapshot we want out of the query's results list
if (!dataSnapshot.exists()) { // value may be null, meaning idToFind doesn't exist
throw new Error(`Entry ${idToFind} not found.`);
}
// do what you want with dataSnapshot
console.log(`Entry ${idToFind}'s data is:`, dataSnapshot.val());
})
.catch((error) => {
console.log("Unexpected error:", error);
})
For small data sets, the above code will work just fine. But if the list of batches starts growing quite large, you may wish to build an index that maps a particular ID to the batch ID that contains it.
Here is my method which allows you to search by id or to search by key value such as email uniqueemail
// gets primary key
const getSnapshotValKey = snapshot => (Object.keys(snapshot).length > 0 ? Object.keys(snapshot)[0] : null)
const getUser = async ({ id, key, value }) => {
let user = null
const ref = id ? '/users/' + id : 'users'
const userRef = admin.database().ref(ref)
const valueRef = id ? userRef : await userRef.orderByChild(key).equalTo(value)
const snapshot = await valueRef.once('value')
const val = snapshot.val()
if (val) {
const key = id || getSnapshotValKey(val)
user = {
id: key,
...(id ? val : val[key]),
}
}
return user
}
I have to delete some documents from azure cosmos DB through azure portal.
I wrote a stored procedure in container which will delete the data which has to be deleted.
But at the time of execution of stored procedure it will ask for partition key value. I have to delete documents which are having different partition keys.
Below given is the Stored Procedure Used.
function bulkDeleteProcedure(deletedate) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var response = getContext().getResponse();
var query = "SELECT * FROM c where c._ts = " + deletedate;
var responseBody = {
docs_deleted: 0,
continuation: true
};
// Validate input.
if (!query) throw new Error("The query is undefined or null.");
fetchData();
function fetchData(continuation) {
var requestOptions = {continuation: continuation};
var isAccepted = collection.queryDocuments(collectionLink, query, requestOptions, function (err, items, responseOptions) {
if (err) throw err;
// response.setBody(items);
if (items.length > 0) {
// Begin deleting documents as soon as documents are returned form the query results.
// tryDelete() resumes querying after deleting; no need to page through continuation tokens.
// - this is to prioritize writes over reads given timeout constraints.
tryDelete(items);
// response.setBody(items.length + " : inside retrvd docs");
} else if (responseOptions.continuation) {
// Else if the query came back empty, but with a continuation token; repeat the query w/ the token.
fetchData(responseOptions.continuation);
} else {
responseBody.continuation = false;
response.setBody(responseBody);
}
});
// If we hit execution bounds - return continuation: true.
if (!isAccepted) {
response.setBody(responseBody);
}
}
// Recursively deletes documents passed in as an array argument.
// Attempts to query for more on empty array.
function tryDelete(documents) {
if (documents.length > 0) {
// Delete the first document in the array.
var isAccepted = collection.deleteDocument(documents[0]._self, {}, function (err, responseOptions) {
if (err) throw err;
responseBody.deleted++;
documents.shift();
// Delete the next document in the array.
tryDelete(documents);
});
// If we hit execution bounds - return continuation: true.
if (!isAccepted) {
response.setBody(responseBody);
}
} else {
// If the document array is empty, query for more documents.
fetchData();
}
}
}
Suppose my partition key is vehicle type and I am having vehicle type values as 01 to 10. The sql query as per my requirement will return documents with 10 different partition key values.
Current scenario is like i have to run the stored procedure 10 times by providing each partition key value each time.
Is it possible to run this stored procedure for different partition key values in a single go?
Current scenario is like i have to run the stored procedure 10 times
by providing each partition key value each time. Is it possible to run
this stored procedure for different partition key values in a single
go?
Unfortunately no. You can't provide multiple partition keys to a stored procedure. You will need to execute the stored procedure 10 times.
One thing you could do is use any available SDK and write code to execute stored procedure in a loop. You could create an array of partition keys and loop through that and execute stored procedure for each partition key in that array.
I am new to firebase. I have data in Firebase realtime database with following structure. My requirement is to add a new field to multiple (selected) records present under “user”.
Initial Data
db—>user
—pushId_1 (Auto Generated via push command)
name: user1
pushId_2
name: user2
Required final Data with one more field added (“newField”)
db—>user
—pushId_1
name: user1
newField: sample data
pushId_2
name: user2
newField: sample data
I have written following code to do this.
exports.sampleFunction = functions.database.ref('/db/user/{pushId}/')
.onCreate((snap, context) => {
console.log('onCreate called with snapshot id = ' + snap.key);
admin.database().ref('/db/user/').once('value').then(snapshot => {
if (snapshot.numChildren() > 1) {
var updates = {};
var count = 0;
snapshot.forEach((child) => {
if (!child.hasChild('newField') && count < 2) {
updates[child.key] = { newField: 'sample data' };
count++;
}
});
//return snapshot.ref.set(updates); //set is also giving same result.
return snapshot.ref.update(updates);
}
return null;
}).catch(error => {
console.error("error in fetching data from db." + error);
});
return null;
});
Problem is, this code is deleting existing field "name" and replacing it with “newField”.
Can you please help me in appending this new field to data without deleting existing fields.
Thanks
That's because you are using set instead of updating the node.
Set replaces any content while update adds fields if missing and only replace fields if they are present in new data.
return snapshot.ref.update(updates);
Also you are setting the data wrong to update. In key you need to have path relative to the ref you are calling the update instead of nested objects.
Instead of updates[child.key] = { newField: 'sample data' };, it should be
updates[`${child.key}/newField`] = 'sample data';
now when you call update with the child parent i.e. snapshot, it knows exactly which fields to update.
See the docs for more details: https://firebase.google.com/docs/database/admin/save-data#section-update
I have a list of documents that belong to a partitioned collection. Instead of querying for every document from the .NET client and either do update or insert, I thought I could use a Stored Procedure to accomplish this.
What I did not initially realize is that Stored Procedures are executed in the transaction scope of a single partition key. So I am getting PartitionKey value must be supplied for this operation.
The thing is that the documents (that I am trying to upsert) may belong to different partitions. How can I accomplish this in the Stored Procedure? In my case, the SP is useless unless it can operate on multiple partitions.
This is how I constructed my SP:
function upsertEcertAssignments(ecerts) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var response = getContext().getResponse();
// Validate input
if (!ecerts) throw new Error("The ecerts is null or undefined");
if (ecerts.length == 0) throw new Error("The ecerts list size is 0");
// Recursively call the 'process' function
processEcerts(ecerts, 0);
function processEcerts(ecerts, index) {
if (index >= ecerts.length) {
response.setBody(index);
return;
}
var query = {query: "SELECT * FROM DigitalEcerts c WHERE c.code = #code AND c.collectionType = #type", parameters: [{name: "#code", value: ecerts[index].code}, {name: "#type", value: 0}]};
var isQueryAccepted = collection.queryDocuments(collectionLink, query, {partitionKey: ecerts[index].code}, function(err, foundDocuments, foundOptions) {
if (err) throw err;
if (foundDocuments.length > 0) {
var existingEcert = foundDocuments[0];
ecerts[index].id = existingEcert.id;
var isAccepted = __.replaceDocument(existingEcert._self, ecerts[index], function(err, updatedEcert, replacedOptions) {
if (err) throw err;
processEcerts(ecerts, index + 1);
});
if (!isAccepted) {
response.setBody(index);
}
} else {
var isAccepted = __.createDocument(__.getSelfLink(), ecerts[index], function(err, insertedEcert, insertedOptions) {
if (err) throw err;
processEcerts(ecerts, index + 1);
});
if (!isAccepted) {
response.setBody(index);
}
}
});
if (!isQueryAccepted)
response.setBody(index);
}
}
From .NET, if I call it like this, I get the partitionKey value problem:
var continuationIndex = await _docDbClient.ExecuteStoredProcedureAsync<int>(UriFactory.CreateStoredProcedureUri(_docDbDatabaseName, _docDbDigitalEcertsCollectionName, "UpsertDigitalMembershipEcertAssignments"), digitalEcerts);
If I call it with a partition key, it works...but it is useless:
var continuationIndex = await _docDbClient.ExecuteStoredProcedureAsync<int>(UriFactory.CreateStoredProcedureUri(_docDbDatabaseName, _docDbDigitalEcertsCollectionName, "UpsertDigitalMembershipEcertAssignments"), new RequestOptions { PartitionKey = new PartitionKey(digitalEcerts[0].Code) }, digitalEcerts.Take(1).ToList());
I appreciate any pointer.
Thanks.
By the sound of it, your unique id is a combination of code and type. I would recommend making your id property to be the combination of two.
This guarantees that your id is unique but also eliminates the need to query for it.
If the collection the stored procedure is registered against is a
single-partition collection, then the transaction is scoped to all the
documents within the collection. If the collection is partitioned,
then stored procedures are executed in the transaction scope of a
single partition key. Each stored procedure execution must then
include a partition key value corresponding to the scope the
transaction must run under.
You could refer to the description above which mentioned here. We can query documents cross partitions via setting EnableCrossPartitionQuery to true in FeedOptions parameter. However, the RequestOptions doesn't have such properties against executing stored procedure.
So, It seems you have to provide partition key when you execute sp. Of course, it can be replaced by upsert function. It is useless from the perspective of the business logic, but if bulk operations, the SP can release some of the performance pressure because the SP is running on the server side.
Hope it helps you.
I'm using nodejs with the module cradle to interact with the couchdb server, the question is to let me understanding the reduce process to improve the view query...
For example, I should get the user data from his ID with a view like this:
map: function (doc) { emit(null, doc); }
And in node.js (with cradle):
db.view('users/getUserByID', function (err, resp) {
var found = false;
resp.forEach(function (key, row, id) {
if (id == userID) {
found = true;
userData = row;
}
});
if (found) {
//good, works
}
});
As you can see, this is really bad for large amount of documents (users in the database), so I need to improve this view with a reduce but I don't know how because I don't understand of reduce works.. thank you
First of all, you're doing views wrong. View are indexes at first place and you shouldn't use them for full-scan operations - that's ineffective and wrong. Use power of Btree index with key, startkey and endkey query parameters and emit field you like to search for as key value.
In second, your example could be easily transformed to:
db.get(userID, function(err, body) {
if (!err) {
// found!
}
});
Since in your loop you're checking row's document id with your userID value. There is no need for that loop - you may request document by his ID directly.
In third, if your userID value isn't matches document's ID, your view should be:
function (doc) { emit(doc.userID, null); }
and your code will be looks like:
db.view('users/getUserByID', {key: userID}, function (err, resp) {
if (!err) {
// found!
}
});
Simple. Effective. Fast. If you need matched doc, use include_docs: true query parameter to fetch it.