NodeJS: Handling transactions with NoSQL databases? - node.js

Consider a promise-chained chunk of code for example:
return Promise.resolve()
.then(function () {
return createSomeData(...);
})
.then(function () {
return updateSomeData(...);
})
.then(function () {
return deleteSomeData(...);
})
.catch(function (error) {
return ohFishPerformRollbacks();
})
.then(function () {
return Promise.reject('something failed somewhere');
})
In the above code, let's say something went wrong in the function updateSomeData(...). Then one would have to revert the create operation that was executed before this.
In another case, if something went wrong in the function deleteSomeData(...), then one would want to revert the operations executed in createSomeData(...) and updateSomeData(...).
This would continue as long as all the blocks have some revert operations defined for themselves in case anything goes wrong.
Only if there was a way in either NodeJs or the database or somewhere in the middle, that would revert all the transactions happening under the same block of code.
One way I can think of this to happen is by flagging all the rows in database with a transactionId (ObjectID) and a wasTransactionSuccessful(boolean), so that CRUD operations could be clubbed together with their transactionIds, and in case something goes wrong, those transactions could be simply deleted from the database in the ending catch block.
I read about rolling back transactions in https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/. But I want to see if it can be done in a more simpler fashion and in a generic manner for NoSQL databases to adapt.

I am not sure if this would satisfy your use case, but I hope it would.
let indexArray = [1, 2, 3];
let promiseArray = [];
let sampleFunction = (index) => {
return new Promise((resolve, reject) => {
setTimeout(resolve, 100, index);
});
}
indexArray.map((element) => {
promiseArray.push(sampleFunction(element));
});
Promise.all(promiseArray).then((data) => {
// do whatever you want with the results
}).catch((err) => {
//Perform your entire rollback here
});
async.waterfall([
firstFunc,
secondFunc
], function (err, result) {
if (err) {
// delete the entire thing
}
});
Using the async library would give you a much elegant solution than going with chaining.

Related

What's the proper way to return a promise in this triggered Cloud Function

I understand that for a triggered function, I must always return a promise. Look at the following example:
//Example
exports.onAuthUserDelete = functions.auth.user().onDelete(async (user) => {
let userId = user.uid;
try {
await firestore.collection('Users').doc(userId).delete();
return Promise.resolve();
} catch (error) {
logger.error(error);
return Promise.reject(error);
}
});
My questions are:
Is return Promise.resolve() required or can I just do return firestore.collection('Users').doc(userId).delete()? If I opt to go with the latter, what would happen if the command failed? Will it still trigger catch()?
Is it better to just start every function with the following template to make sure a promise is always returned?
//Is it better to start with this boilerplate
exports.onAuthUserDelete = functions.auth.user().onDelete(async (user) => {
return new Promise((resolve, reject) => {
//My code goes here...
});
}
Firestore's delete operation already returns a promise, so there's no need to create your own. As far as I can see that first example is example the same as:
exports.onAuthUserDelete = functions.auth.user().onDelete((user) => {
return firestore.collection('Users').doc(user.uid).delete();
});
Given that, I highly recommend using the above shorter version.

Node express multiple queries in one route that depend on each other

I am just about getting familiar with node and express and programming in general, but this is a more complex issue I am trying to solve. Please if you can provide with some best practice in this kind of scenario.
I am trying to run two queries to my database where the first one is dependent on the result of the first. Q1. Return a list of ids. Q2. Return id and coord for each of the ids. I want to respond with a json object that look something like this
[
{ id: 451, coords: 'POINT(12.5574 43.8351)' },
{ id: 56, coords: 'POINT(13.5574 44.8351)' }
]
Currently I cannot get it to work, I know there is probably several issues with my example code, but I have pretty much got stuck. Maybe I am overthinking this and make it harder than it is, or bad practice in general.
How can I run multiple queries where the second use the output from the first one and then build the correct object to respond with. Any pointers would be much appreciated.
router.get('/asset/:id', (req, res) => {
let latLngOfAssets = []
// get associated assets
function getConnectionsById() {
queries.getConnectionsById(req.params.id) // return list of objects
.then(data => {
if (data) {
data.forEach(function(element) {
getLatLngofAsset(element.til_poi_id) // for each id in list call function to get coordinate
});
} else {
throw new Error('No data returned');
}
console.log(latLngOfAssets) // What I want to respond with res.json(latlngofassets)
})
}
function getLatLngofAsset(id) {
queries.getPoilatlng(id) // Return geometry as text for asset with id
.then(data =>{
let newassetobj = {}
if (data) {
newassetobj["id"] = data.rows[0].id
newassetobj["coords"] = data.rows[0].st_astext
//console.log(newassetobj) // structure of each object { id: 451, coords: 'POINT(12.5574 43.8351)' }
latLngOfAssets.push(newassetobj) // making list of objects i want to respond with
} else {
throw new Error('Did not get any poi');
}
})
}
getConnectionsById()
.catch(err => { // {message: "Cannot read property 'then' of undefined", error: {…}}
console.error('Something went wrong', err);
});
});
You've done a good job separating out the two distinct sections of your code into separate functions - what you're missing is the ability to tie them together. This portion of your code is not doing what I think you are trying to accomplish:
data.forEach(function(element) {
getLatLngofAsset(element.til_poi_id)
});
Because getLatLngofAsset() is Promise-based* you need to use it like a Promise. You first need to make getLatLngofAsset return the Promise chain it creates. Then, it can be await-ed inside getConnectionsById using an async function:
function getConnectionsById() {
queries.getConnectionsById(req.params.id)
.then(data => {
if (data) {
data.forEach(async function(element) { // <-- note the async keyword here
await getLatLngofAsset(element.til_poi_id)
});
} else {
throw new Error('No data returned');
}
console.log(latLngOfAssets) // What I want to respond with res.json(latlngofassets)
})
}
This is a start - there are a couple more things we can tackle once you understand the relationship between the functions you have declared and the Promises they create & return.

Async/await in Express with multiple MongoDB queries

I have a fairly straightforward CRUD app which renders the results of two queries onto one page. The problem that arose once I got this to "work" was that the page required a refresh in order to display the results. On first load, no results were displayed.
I came to figure out that this is a problem/symptom of Node's asynchronous nature. I've been trying to approach this problem by using async/await, and from hours of messing with things, I feel like I'm quite close to the solution, but it's just not working out - I still need a manual refresh to display/render the results on the .ejs page.
The code:
var entries = [];
var frontPageGoals = [];
app.get('/entries', async (req,res) => {
if (req.session.password) {
const entriesColl = await
db.collection('entries')
.find()
.sort({date: -1})
.toArray((err, result) => {
if (err) { console.log(err) }
else {
for (i=0; i<result.length; i++) {
entries[i] = result[i];
}
}
});
const goalsColl = await
db.collection('goals')
.find()
.toArray((err, result) => {
if (err) {console.log(err)}
else {
for (i=0; i<result.length; i++) {
frontPageGoals[i] = result[i];
}
}
});
res.render('index.ejs', {entries: entries, frontPageGoals: frontPageGoals});
}
else {
res.redirect('/');
}
});
Now, I can conceive of a few problems here, but honestly I'm just at my wits end trying to figure this out. For example, I'm sure it's problematic that the empty lists which will contain the results to be passed when the page renders are outside the actual async function. But after trying to move them a dozen different places within the async area... still no dice.
Any help would be hugely appreciated! This is basically the last big "thing" I need done for this app.
I'm not 100% sure about your database driver, but assuming that the toArray() returns a promise (which it does in the default mongodb driver), the await will actually return the value you expect in your callback, result in your case, or in case there was an error, which you expected it as err in your callback, it will be thrown, thus forcing you to use try-catch blocks, in your case, you would just use console.log(err) in the catch block, since you aren't doing any handling
Here's your code after updating :
app.get("/entries", async (req, res) => {
if (req.session.password) {
try {
const entries = await db
.collection("entries")
.find()
.sort({ date: -1 })
.toArray();
const frontPageGoals = await db
.collection("goals")
.find()
.toArray();
res.render("index.ejs", {
entries: entries,
frontPageGoals: frontPageGoals
});
} catch (err) {
console.log(err);
}
} else {
res.redirect("/");
}
});
EDIT
However, if you don't know about promises -which async/await are basically promises-, and wanna just do it using callbacks -not advised-, you would have to just send your response in the callback, and nest the 2nd query in the first query's callback, here is the code,, with some comments to hopefully help you out:
app.get("/entries", (req, res) => {
if (req.session.password) {
// First query
db.collection("entries")
.find()
.sort({ date: -1 })
.toArray((err, entryResult) => {
if (err) {
console.log(err);
} else {
// In the callback of the first query, so it will
// execute 2nd query, only when the first one is done
db.collection("goals")
.find()
.toArray((err, frontPageResult) => {
if (err) {
console.log(err);
} else {
// In the callback of the 2nd query, send the response
// here since both data are at hand
res.render("index.ejs", {
entries: entryResult,
frontPageGoals: frontPageResult
});
}
});
}
});
} else {
res.redirect("/");
}
});
I have removed the async keyword since you no longer need it
I renamed the callback arguments, instead of just result, because both callbacks would have the same argument name, and you would have had to store it in a temp variable

How to handle chained promises in a loop in nodejs with bluebird

The gist of the problem is:
for (let i=0;i<list.length;i++) {
AsyncCall_1({'someProperty': list[i] })
.then((resp_1) => {
resp_1.doSomething();
resp_1.AsyncCall_2()
.then((resp_2) => {
resp_2.doSomething();
})
})
}
after last resp.AsyncCall_2.then(()=> {
//do something
})
I need to sequentially chain all the promises so that, the loop waits for the "resp.AsyncCall_2" function to be resolved for its next iteration. After last "resp.AsyncCall_2" call do something. (since all the promises will be resolved the)
Actual Problem:
for (var i=0;i<todo.assignTo.length;i++) {
Users.findOne({'username': todo.assignTo[i] })
.then((user) => {
user.assigned.push(todo.title);
user.notificationCount.assignedTodosCount++;
user.save()
.then((user) => {
console.log("todo is assigned to the user: " + user.username)
})
})
}
//to something at last call resloved (I know this is wrong way of doing this)
Users.find({})
.then((users)=> {
var promises = [];
for (var i=0;i<users.length;i++) {
users[i].notificationCount.totalTodosCount++;
promises.push(users[i].save());
}
Promise.all(promises)
.then(()=> {
res.statusCode = 200;
res.setHeader('Content-Type', 'application/json');
console.log("todo is successfully posted");
res.json({success : true, todo});
},(err) => next(err))
.catch((err) => next(err));
})
Thank You in Advance..
In modern versions of node.js, you can just use async/await and don't need to use Bluebird iteration functions:
async function someMiddlewareFunc(req, res, next) {
try {
for (let item of list) {
let resp_1 = await AsyncCall_1({'someProperty': item });
resp_1.doSomething();
let resp_2 = await resp_1.AsyncCall_2();
resp_2.doSomething();
}
// then do something here after the last iteration
// of the loop and its async operations are done
res.json(...);
} catch(err) {
next(err);
}
}
This will serialize the operations (which is what you asked for) so the 2nd iteration of the loop doesn't start until the async operations in the first iteration is done.
But, it doesn't appear in your real code that you actually need to serialize the individual operations and serializing things that don't have to be serialized usually makes the end-to-end time to complete them be longer. So, you could run all the items in your loop in parallel, collect all the results at the end and then send your response and Bluebird's Promise.map() would be quite useful for that because it combines a .map() and a Promise.all() into one function call:
function someMiddlewareFunc(req, res, next) {
Promise.map(list, (item) => {
return AsyncCall_1({'someProperty': item}).then(resp_1 => {
resp_1.doSomething();
return resp_1.AsyncCall_2();
}).then(resp_2 => {
return resp_2.doSomething();
});
}).then(results => {
// all done here
res.json(...);
}).catch(err => {
next(err);
});
}
FYI, when using res.json(...), you don't need to set these res.statusCode = 200; or res.setHeader('Content-Type', 'application/json'); as they will be done for you automatically.
Further notes about Bluebird's Promise.map(). It accepts a {concurrency: n} option that tells Bluebird how many operations are allowed to be "in flight" at the same time. By default, it runs them all in parallel at the same time, but you can pass any number you want as the concurrency option. If you pass 1, it will serialize things. Using this option can be particularly useful when parallel operation is permitted, but the array is very large and iterating all of them in parallel runs into either memory usage problems or overwhelms the target server. In that case, you can set the concurrency value to some intermediate value that still gives you some measure of parallel execution, but doesn't overwhelm the target (some number between 5 and 20 is often appropriate - it depends upon the target service). Sometimes, commercial services (like Google) also have limits about how many requests they will handle at the same time from the same IP address (to protect them from one account using too much of the service at once) and the concurrency value can be useful for that reason too.
Have you tried Promise.each?
const users = todo.assignTo.map(function(user) {
return Users.findOne({'username': assigned_to });
}
Promise.each(users, function(user) {
user.assigned.push(todo.title);
user.notificationCount.assignedTodosCount++;
user.save()
.then((user) => {
console.log("todo is assigned to the user: " + user.username)
})
}

How can promise resolve all data with multiple layer of callbacks in NodeJS

Suppose I have the following source code in NodeJS using promise (connecting to mongodb with 3 collections, well, this is not important)
function getData(){
return new promise((resolve, reject) => {
var dataArray = [];
getCollection1().then((rows1) => {
getCollection2().then((rows2) => {
var val = someDataProcessingFunction(rows1, rows2);
dataArray.push(val);
resolve(dataArray);
}, (err) => {
reject(err);
}
}, (err) => {
reject(err);
});
getCollection3().then((rows3) => {
rows3.forEach((row3) => {
dataArray.push(row3);
});
resolve(dataArray);
}, (err) => {
reject(err);
});
});
}
The problem is, there is multiple places where dataArray is pushed, but I don't know when all the pushing is completed. so that I don't know when to resolve, to return the data to user, for example, by response.send in express. As to the code above while I am using resolves several times, I will only get part of the whole data returned.
I can think out two workarounds, neither is good. one is to surround all resolves in setTimeout, which means I must delay the response to user.
another is to count all the push times at the beginning, every time subtract 1 when one item is pushed. when the times goes 0, do resolve. but it is often difficult to calculate the number before doing the real loops.
Is there better solutions? Thank you and love you :)
First of all, avoid the Promise constructor antipattern and use return values instead! For multiple layers of asynchronous functions, just use multiple layers of promises. To await multiple promises at once, use Promise.all:
function getData(){
return Promise.all([
getCollection1(),
getCollection2(),
getCollection3()
]).then(([rows1, rows2, rows3]) => {
return [someDataProcessingFunction(rows1, rows2)].concat(rows3);
});
}

Resources