I have a route that uses the below shown method, in a node app that uses express.
I create a transaction but don't use it in the update method. Sequelize is configured to not use managed transactions and auto commit is set to false.
When this route is called multiple times/under load around 7 - 10 times per second (the number of calls differ), I end up with 5 or so dangling transactions even though commit is called for the transaction at the end of the method call.
(Because of these dangling transactions, subsequent calls and my node app is not able to make anymore db calls)
But if I pass the transaction in the params object this behaviour doesnt occur. And I don't get dangling transactions.
What can be the reason this is happening?
updateItem = (obj) => {
this.logDebug(`Updating item - `, obj);
return new Promise(async (resolve, reject) => {
let transaction;
try {
transaction = await this.getTransaction();
} catch(error) { return reject(error);
const params = {
where: {
id: obj.id
},
returning: true,
plain: true
};
return models[modelName].
update(obj, params).then(result => {
if (!result) { return result; }
result = JSON.parse(JSON.stringify(result[1]));
return result;
}).
then(async (result) => {
await transaction.commit();
return resolve(result);
}).
catch(async error => {
this.logError(`Failed to update - `, error);
await transaction.rollback();
return reject(error);
});
});
};
Causes dangling transaction.
const params = {
where: {
id: obj.id
},
returning: true,
plain: true
};
No dangling transactions occur.
const params = {
where: {
id: obj.id
},
returning: true,
plain: true,
transaction
};
Using Nodejs 12.4, Sequelize 5.21.9, postgres 9.x
Just using the created transaction in the params object somehow doesn't cause dangling transaction.
While NOT using the transaction in the params causes this issue of dangling transactions.
Wanted to know the cause of this behaviour? Is it a bug with my code? Or bug with Sequelize?
There are a few points here;
1- there is a missing curly bracket at the end of the first line (a simple type i guess)
2- you dont't have to use the returns other than the first one
3- it is not required to use double "then" fucntion at the end
4- if json parse fails your transaction probably leak
With all these the code may be updated like this;
updateItem = (obj) => {
this.logDebug(`Updating item - `, obj);
return new Promise(async (resolve, reject) => {
let transaction;
try {
transaction = await this.getTransaction();
} catch (error) { return reject(error); }
const params = {
where: {
id: obj.id
},
returning: true,
plain: true
};
return models[modelName].
update(obj, params).then(async (result) => {
try {
resolve(JSON.parse(JSON.stringify(result[1])));
await transaction.commit();
} catch (err) {
reject(err);
}
}).catch(async error => {
this.logError(`Failed to update - `, error);
await transaction.rollback(); // an extra try catch wont hurt here...
reject(error);
});
});
};
But things may get complicated if you want to make multiple updates in the future. I may suggest you to check Multiple Transaction Manager's Sequelize context for a clean way to arrange your transactions.
Related
I have a set of functions in Node.js that I would like to load in a certain order. I will provide some mockup code abstracted and simplified:
function updateMyApp() {
loadDataToServer()
.then(() => useData())
.then(() => saveData())
.then(() => { console.log("updateMyApp done") })
}
function loadDataToServer() {
return new Promise( (resolve, reject) {
...preparing data and save file to cloud...
resolve()})
}
function handleDataItem(item) {
// Function that fetches data item from database and updates each data item
console.log("Name", item.name)
}
function saveData() {
// Saves the altered data to some place
}
useData is a bit more complex. In it I would like to, in order:
console.log('Starting alterData()')
Load data, as json, from the cloud data source
Iterate through every item in the json file and do handleDataItem(item) on it.
When #2 is done -> console.log('alterData() done')
Return a resolved promise back to updateMyApp
Go on with saveData() with all data altered.
I want the logs to show:
Starting useData()
Name: Adam
Name: Ben
Name: Casey
useData() done
my take on this is the following:
function useData() {
console.log('Starting useData()')
return new Promise( function(resolve, reject) {
readFromCloudFileserver()
.then(jsonListFromCloud) => {
jsonListFromCloud.forEach((item) => {
handleDataItem(item)
}
})
.then(() => {
resolve() // I put resolve here because it is not until everything is finished above that this function is finished
console.log('useData() done')
}).catch((error) => { console.error(error.message) })
})
}
which seems to work but, as far as I understand this is not how one is supposed to do it. Also, this seems to do the handleDataItem outside of this chain so the logs look like this:
Starting useData()
useData() done
Name: Adam
Name: Ben
Name: Casey
In other words. It doesn't seem like the handleDataItem() calls are finished when the chain has moved on to the next step (.then()). In other words, I can not be sure all items have been updated when it goes on to the saveData() function?
If this is not a good way to handle it, then how should these functions be written? How do I chain the functions properly to make sure everything is done in the right order (as well as making the log events appear in order)?
Edit: As per request, this is handleDataItem less abstracted.
function handleDataItem(data) {
return new Promise( async function (resolve) {
data['member'] = true
if (data['twitter']) {
const cleanedUsername = twitterApi.cleanUsername(data['twitter']).toLowerCase()
if (!data['twitter_numeric']) {
var twitterId = await twitterApi.getTwitterIdFromUsername(cleanedUsername)
if (twitterId) {
data['twitter_numeric'] = twitterId
}
}
if (data['twitter_numeric']) {
if (data['twitter_protected'] != undefined) {
var twitterInfo = await twitterApi.getTwitterGeneralInfoToDb(data['twitter_numeric'])
data['twitter_description'] = twitterInfo.description
data['twitter_protected'] = twitterInfo.protected
data['twitter_profile_pic'] = twitterInfo.profile_image_url.replace("_normal", '_bigger')
data['twitter_status'] = 2
console.log("Tweeter: ", data)
}
} else {
data['twitter_status'] = 1
}
}
resolve(data)
}).then( (data) => {
db.collection('people').doc(data.marker).set(data)
db.collection('people').doc(data.marker).collection('positions').doc(data['report_at']).set(
{
"lat":data['lat'],
"lon":data['lon'],
}
)
}).catch( (error) => { console.log(error) })
}
The twitterAPI functions called:
cleanUsername: function (givenUsername) {
return givenUsername.split('/').pop().replace('#', '').replace('#', '').split(" ").join("").split("?")[0].trim().toLowerCase()
},
getTwitterGeneralInfoToDb: async function (twitter_id) {
var endpointURL = "https://api.twitter.com/2/users/" + twitter_id
var params = {
"user.fields": "name,description,profile_image_url,protected"
}
// this is the HTTP header that adds bearer token authentication
return new Promise( (resolve,reject) => {
needle('get', endpointURL, params, {
headers: {
"User-Agent": "v2UserLookupJS",
"authorization": `Bearer ${TWITTER_TOKEN}`
}
}).then( (res) => {
console.log("result.body", res.body);
if (res.body['errors']) {
if (res.body['errors'][0]['title'] == undefined) {
reject("Twitter API returns undefined error for :'", cleanUsername, "'")
} else {
reject("Twitter API returns error:", res.body['errors'][0]['title'], res.body['errors'][0]['detail'])
}
} else {
resolve(res.body.data)
}
}).catch( (error) => { console.error(error.message) })
})
},
// Get unique id from Twitter user
// Twitter API
getTwitterIdFromUsername: async function (cleanUsername) {
const endpointURL = "https://api.twitter.com/2/users/by?usernames="
const params = {
usernames: cleanUsername, // Edit usernames to look up
}
// this is the HTTP header that adds bearer token authentication
const res = await needle('get', endpointURL, params, {
headers: {
"User-Agent": "v2UserLookupJS",
"authorization": `Bearer ${TWITTER_TOKEN}`
}
})
if (res.body['errors']) {
if (res.body['errors'][0]) {
if (res.body['errors'][0]['title'] == undefined) {
console.error("Twitter API returns undefined error for :'", cleanUsername, "'")
} else {
console.error("Twitter API returns error:", res.body['errors'][0]['title'], res.body['errors'][0]['detail'])
}
} else {
console.error("Twitter API special error:", res.body)
}
} else {
if (res.body['data']) {
return res.body['data'][0].id
} else {
//console.log("??? Could not return ID, despite no error. See: ", res.body)
}
}
},
You have 3 options to deal with your main issue of async methods in a loop.
Instead of forEach, use map and return promises. Then use Promise.all on the returned promises to wait for them to all complete.
Use a for/of loop in combination with async/await.
Use a for await loop.
It sounds like there's a problem in the implementation of handleDataItem() and the promise that it returns. To help you with that, we need to see the code for that function.
You also need to clean up useData() so that it properly returns a promise that propagates both completion and errors.
And, if handleDataItem() returns a promise that is accurate, then you need to change how you do that in a loop here also.
Change from this:
function useData() {
console.log('Starting useData()')
return new Promise( function(resolve, reject) {
readFromCloudFileserver()
.then(jsonListFromCloud) => {
jsonListFromCloud.forEach((item) => {
handleDataItem(item)
}
})
.then(() => {
resolve() // I put resolve here because it is not until everything is finished above that this function is finished
console.log('useData() done')
}).catch((error) => { console.error(error.message) })
})
}
to this:
async function useData() {
try {
console.log('Starting useData()')
const jsonListFromCloud = await readFromCloudFileserver();
for (let item of jsonListFromCloud) {
await handleDataItem(item);
}
console.log('useData() done');
} catch (error) {
// log error and rethrow so caller gets the error
console.error(error.message)
throw error;
}
}
The structural changes here are:
Switch to use async/await to more easily handle the asynchronous items in a loop
Remove the promise anti-pattern that wraps new Promise() around an existing promise - no need for that AND you weren't capturing or propagating rejections from readFromCloudFileServer() which is a common mistake when using that anti-pattern.
rethrow the error inside your catch after logging the error so the error gets propagated back to the caller
I tried querying Firestore using .get():
//Cloud function to perform leaderboard calculation
exports.scheduledLeaderboardFunction = functions.pubsub.schedule('00 21 * * *')
.timeZone('America/Los_Angeles')
.onRun(async (context) => {
var globalPostsArray = [];
try {
await admin.firestore()
.collection('globalPosts')
.get()
.then((querySnapshot) => {
if(querySnapshot.exists) {
querySnapshot.forEach((res) => {
const {
//Fields
//Removed
} = res.data();
globalPostsArray.push({
//Fields
//Removed
});
});
}
else {
throw new Error("Data doesn't exist") <-------- This error is thrown
}
return null
})
.then(() => {
if (globalPostsArray.length > 0) {
console.log(globalPostsArray)
}
else {
throw new Error("length not greater than 0")
}
return null;
})
}
catch(error) {
console.log(error);
}
return null;
});
but in the firebase cloud log, I get the following error printed:
Error: Data doesn't exist
Which means querySnapshot doesn't exist when I use .get() (the error is thrown).
globalPosts, the collection I am querying, is NOT empty
if I can make .get() to work, that would work for my issue, since I am not waiting for updates which is what .onSnapshot() is good for.
summary: onSnapshot() worked for fetching the data from Firestore, but I can't use .then() to wait for the data so I can finish up the work. get() isn't working, but I can use .then() to wait for the collection if it does work.
How can I fix my issue?
EDIT: changed the function to this, but its still not working
//Cloud function to perform leaderboard calculation
exports.scheduledLeaderboardFunction = functions.pubsub.schedule('00 21 * * *')
.timeZone('America/Los_Angeles')
.onRun(async (context) => {
try {
await admin.firestore().collection('globalPosts').orderBy("date_created", "desc")
.get()
.then(function(querySnapshot) {
if(querySnapshot) {
querySnapshot.forEach(function(doc) {
// doc.data() is never undefined for query doc snapshots
console.log(doc.id, " => ", doc.data());
});
return null
}
else {
throw new Error("Data doesn't exist")
}
})
}
catch(error) {
console.log(error);
}
return null;
});
Proof globalPosts isn't empty:
Same error:
Error: Data doesn't exist
I think you're on the right track with .get(), but don't mix async syntax.
const shortSnapshot = await firebase.firestore()
.collection("stuff")
.where("stuff", "==", "new")
.limit(1)
.get();
if (shortSnapshot.empty) {
console.error("No stuff");
return response.sendStatus(404);
}
console.log(shortSnapshot.docs[0].data());
There is no .then for .onSnapShot - it attaches a Listener Function, and returns synchronously, with no data - it does NOT return a promise.
the anonymous function you created and passed - it begins (querySnapshot) => { is called with a querySnapshot as it's argument when the Listener is triggered. Judging by the other parts of the code, you're attempting a cloud function? As a general rule, listeners are NOT the correct approach for a Cloud Function, as these functions are intended to be short-lived.
The message you are shown is precisely what would be expected if your collection 'globalPosts' were empty - and you show nothing here to indicate that this is an error.
Removing the try, catch fixed my issue!
await admin.firestore().collection('globalPosts').orderBy("date_created", "desc")
.get()
.then(function(querySnapshot) {
if(querySnapshot) {
querySnapshot.forEach(function(doc) {
// doc.data() is never undefined for query doc snapshots
console.log(doc.id, " => ", doc.data());
});
return null
}
else {
throw new Error("Data doesn't exist")
}
})
The error of "data doesn't exist" is no longer an issue
I'm struggling trying to chain together three requests that require synchrony in node.js. Here is my attempt at using promises, but i am getting an error saying that db.run isn't a function. The first action should insert into my sqlite db. The most important thing i need is the
this.lastID variable, which lists the id of the last enetered item. Before attempting to use promises, I was having trouble with scoping. This is important because i need to take this value and use use it in my JSON object under the callback key. Lastly, Im using the requests npm package to send the request.
I am using the bluebird promises library, sqlite3 npm package, nodejs, express.
Any help with this would be awesome because I'm lost.
function db() {
return new Promise(function(resolve, reject) {
db.run(`INSERT INTO scan_requests(name, date) VALUES(?,?);`, [name,date], function(err) {
if (err) {
console.log(err)
}
let q = this.lastID
resolve(q)
})
})
}
db()
.then(function(q) {
let options = {
url: 'API',
body: {
name: req.name,
scan_callback: `http://localhost:80/${q}`
},
json: true
}
resolve(options)
}).then(function(options) {
console.log(options)
})
1st rule of "promises" ... always return your "promises". Except when you create a new one.
Try this ...
app.post('/route', function (req,res) {
new Promise(function(resolve, reject) {
db.run(`INSERT INTO scan_requests(req.name,req.date) VALUES(?,?);`, [name,date]).then(function(result) {
let options = {
url: 'http://API',
body: {
name: req.name,
date: req.date
callback: `http://localhost:80/${this.lastID}`,
},
json: true
}
// this resolves this promise ... it is now passed on
resolve(options);
}).then(function(options) {
// options is now the result from the promise
console.log(options)
request
.post(options)
.on('error', function(err) {
console.log(err)
})
.pipe(res)
});
});
});
UPDATE (question modified)
You're using resolve(options) but resolve is not in scope there (it doesn't exist). Remember the first rule of promises ...
db()
.then(function(q) {
let options = {
url: 'API',
body: {
name: req.name,
scan_callback: `http://localhost:80/${q}`
},
json: true
}
// *** change the following line ***
// --- you must return your data ---
return options;
}).then(function(options) {
console.log(options)
// -------------------------
// --- contrived example ---
// -------------------------
return { success: true };
}).then(status => {
console.log(`Success ${status.success}`);
});
The example includes a rather useless but illustrative example of how to continue passing data down the "promise chain".
I am relatively new to node.js, postgresql, promises and infact stackoverflow so apologies in advance if things sound a little disjointed!
I am currently trying to run multiple queries within chained promises spread across various controllers. I want to run all the queries within the same transaction or task to eliminate multiple connects and disconnects to the database.
I have tried the following where I am adding a student and assigning two mentors for that student. The HTTP request is routed to the student controller which adds a student via the student repository. The student repository is where the task starts and is returned to the controller which forwards it to the mentor controller and along the chain it goes...
# HttpPost("/api/students/create")
addStudent( # Req()request) {
var studentid;
var mentorids= [];
//Add the student
return this.studentRepository.addStudent(request.body.student)
.then(newStudentId => {
studentid = newStudentId;
//Add the mentors, passing the transaction object received back from
studentRepository
return this.mentorController.addMentor(request.body.schoolid, request.body.mentors, newStudentId.transaction)
.then(result => {
var data = [];
console.log(result);
for (var role in result) {
data.push({
mentorid: result[role].roleid,
studentid: studentid
});
}
//Assigns the student to mentors in the link table
return this.studentRepository.assignMentor(data)
.then(result => {
return result;
})
})
});
}
Student repository
addStudent(student): any {
return this.collection.task(t => {
return this.collection.one(this.sql.addStudent, student)
.then(studentid => {
return {
studentid: studentid.studentid,
transaction: t
}
});
})
}
Mentor controller
addMentor(institutionid: number, mentors, t): any {
var promises = [];
var mentorIds = [];
for (var role in mentors) {
promises.push(this.roleController.registerRole(institutionid,mentors[role].role,t));
}
return t.batch(promises)
.then(result => {
return Promise.resolve(result);
})
}
Role controller
# HttpPost("/api/roles/register")
registerRole(institutionid, # Req()request, t ? ) : any {
console.log(request);
return this.roleRepository.checkRoleEnrollment(institutionid, request.email, request.roletype, t)
.then(result => {
return this.roleRepository.addRoleEnrollment(institutionid, request, t)
.then(data => {
return this.roleRepository.updateRoleEnrollment(data.roleenrollmentid, data.roleid)
.then(d => {
return data;
})
})
})
.catch (error => {
return Promise.reject(error);
});
}
I am getting the following error when I call checkEnrollment in the Role Controller:
"name": "Error",
"message": "Unexpected call outside of task.",
"stack": "Error: Unexpected call outside of task. at Task.query
(\api\node_modules\pg-promise\lib\task.js:118:19)
at Task.obj.oneOrNone (\api\node_modules\pg-promise\lib\database.js:491:31)
at RoleRepository.checkRoleEnrollment....
Any help would be much appreciated. Thanking you in advance.
As per my earlier comment:
That error means you are trying to access connection t allocated by a task somewhere outside of the task's callback function, i.e. the task's callback has returned, the connection was released, and then you are using the connection object allocated by the task from somewhere else, which is, of course, invalid.
b.t.w. I'm the author of pg-promise ;)
Below is what your code effectively doing, in a simplified form:
var cnReference;
db.task(t => {
cnReference = t;
// can only use `t` connection while executing the callback
})
.then(data => {
// we are now outside of the task;
// the task's connection has been closed already,
// and we can do nothing with it anymore!
return cnReference.query('SELECT...');
})
.catch(error => {
// ERROR: Unexpected call outside of task.
// We cannot use a task connection object outside of the task's callback!
});
You need to correct the implementation to make sure this doesn't happen.
I'm running into an issue which I don't fully understand. I feel like there are likely concepts which I haven't grasped, code that could be optimized, and possibly a bug thrown in for good measure.
To greatly simplify the overall flow:
A request is made to an external API
The returned JSON object is parsed and scanned for link references
If any link references are found, additional requests are made to populate/replace link references with real JSON data
Once all link references have been replaced, the original request is returned and used to build content
Here, is the original request (#1):
await Store.get(Constants.Contentful.ENTRY, Contentful[page.file])
Store.get is represented by:
async get(type, id) {
return await this._get(type, id);
}
Which calls:
_get(type, id) {
return new Promise(async (resolve, reject) => {
var data = _json[id] = _json[id] || await this._api(type, id);
console.log(data)
if(isAsset(data)) {
resolve(data);
} else if(isEntry(data)) {
await this._scan(data);
resolve(data);
} else {
const error = 'Response is not entry/asset.';
console.log(error);
reject(error);
}
});
}
The API call is:
_api(type, id) {
return new Promise((resolve, reject) => {
Request('http://cdn.contentful.com/spaces/' + Constants.Contentful.SPACE + '/' + (!type || type === Constants.Contentful.ENTRY ? 'entries' : 'assets') + '/' + id + '?access_token=' + Constants.Contentful.PRODUCTION_TOKEN, (error, response, data) => {
if(error) {
console.log(error);
reject(error);
} else {
data = JSON.parse(data);
if(data.sys.type === Constants.Contentful.ERROR) {
console.log(data);
reject(data);
} else {
resolve(data);
}
}
});
});
}
When an entry is returned, it is scanned:
_scan(data) {
return new Promise((resolve, reject) => {
if(data && data.fields) {
const keys = Object.keys(data.fields);
keys.forEach(async (key, i) => {
var val = data.fields[key];
if(isLink(val)) {
var child = await this._get(val.sys.linkType.toUpperCase(), val.sys.id);
this._inject(data.fields, key, undefined, child);
} else if(isLinkArray(val)) {
var children = await* val.map(async (link) => await this._get(link.sys.linkType.toUpperCase(), link.sys.id));
children.forEach((child, index) => {
this._inject(data.fields, key, index, child);
});
} else {
await new Promise((resolve) => setTimeout(resolve, 0));
}
if(i === keys.length - 1) {
resolve();
}
});
} else {
const error = 'Required data is unavailable.';
console.log(error);
reject(error);
}
});
}
If link references are found, additional requests are made and then the resulting JSON is injected into the original JSON in place of the reference:
_inject(fields, key, index, data) {
if(isNaN(index)) {
fields[key] = data;
} else {
fields[key][index] = data;
}
}
Notice, I'm using async, await, and Promise's I believe in their intended manor. What ends up happening: The calls for referenced data (gets resulting of _scan) end up occurring after the original request is returned. This ends up providing incomplete data to the content template.
Additional information concerning my build setup:
npm#2.14.2
node#4.0.0
webpack#1.12.2
babel#5.8.34
babel-loader#5.4.0
I believe the issue is in your forEach call in _scan. For reference, see this passage in Taming the asynchronous beast with ES7:
However, if you try to use an async function, then you will get a more subtle bug:
let docs = [{}, {}, {}];
// WARNING: this won't work
docs.forEach(async function (doc, i) {
await db.post(doc);
console.log(i);
});
console.log('main loop done');
This will compile, but the problem is that this will print out:
main loop done
0
1
2
What's happening is that the main function is exiting early, because the await is actually in the sub-function. Furthermore, this will execute each promise concurrently, which is not what we intended.
The lesson is: be careful when you have any function inside your async function. The await will only pause its parent function, so check that it's doing what you actually think it's doing.
So each iteration of the forEach call is running concurrently; they're not executing one at a time. As soon as the one that matches the criteria i === keys.length - 1 finishes, the promise is resolved and _scan returns, even though other async functions called via forEach are still executing.
You would need to either change the forEach to a map to return an array of promises, which you can then await* from _scan (if you want to execute them all concurrently and then call something when they're all done), or execute them one-at-a-time if you want them to execute in sequence.
As a side note, if I'm reading them right, some of your async functions can be simplified a bit; remember that, while awaiting an async function call returns a value, simply calling it returns another promise, and returning a value from an async function is the same as returning a promise that resolves to that value in a non-async function. So, for example, _get can be:
async _get(type, id) {
var data = _json[id] = _json[id] || await this._api(type, id);
console.log(data)
if (isAsset(data)) {
return data;
} else if (isEntry(data)) {
await this._scan(data);
return data;
} else {
const error = 'Response is not entry/asset.';
console.log(error);
throw error;
}
}
Similarly, _scan could be (assuming you want the forEach bodies to execute concurrently):
async _scan(data) {
if (data && data.fields) {
const keys = Object.keys(data.fields);
const promises = keys.map(async (key, i) => {
var val = data.fields[key];
if (isLink(val)) {
var child = await this._get(val.sys.linkType.toUpperCase(), val.sys.id);
this._inject(data.fields, key, undefined, child);
} else if (isLinkArray(val)) {
var children = await* val.map(async (link) => await this._get(link.sys.linkType.toUpperCase(), link.sys.id));
children.forEach((child, index) => {
this._inject(data.fields, key, index, child);
});
} else {
await new Promise((resolve) => setTimeout(resolve, 0));
}
});
await* promises;
} else {
const error = 'Required data is unavailable.';
console.log(error);
throw error;
}
}