MongooseError: Query was already executed - node.js

I updated Mongoose to the latest version (6.0.1) and now I'm getting this error whenever .findOne() is executed:
MongooseError: Query was already executed: Streams.findOne({ i: 6 })
at model.Query._wrappedThunk [as _findOne] (C:\Users\honza\ZiggerTestMaster\node_modules\mongoose\lib\helpers\query\wrapThunk.js:21:19)
at C:\Users\honza\ZiggerTestMaster\node_modules\kareem\index.js:370:33
at processTicksAndRejections (node:internal/process/task_queues:78:11)
at runNextTicks (node:internal/process/task_queues:65:3)
at listOnTimeout (node:internal/timers:526:9)
at processTimers (node:internal/timers:500:7) {
originalStack: 'Error\n' +
' at model.Query._wrappedThunk [as _findOne] (C:\\Users\\honza\\ZiggerTestMaster\\node_modules\\mongoose\\lib\\helpers\\query\\wrapThunk.js:25:28)\n' +
' at C:\\Users\\honza\\ZiggerTestMaster\\node_modules\\kareem\\index.js:370:33\n' +
' at processTicksAndRejections (node:internal/process/task_queues:78:11)\n' +
' at runNextTicks (node:internal/process/task_queues:65:3)\n' +
' at listOnTimeout (node:internal/timers:526:9)\n' +
' at processTimers (node:internal/timers:500:7)'
}
My code is as follows:
var visitorData = Visitor.find({});
app.get("/retrieve", function(req,res){
visitorData.exec(function (err,data) {
if (err) {
throw err;
}
res.render("retrieve", { title:"View Visitor Data", records: data});
});
});
It executes properly the first time I open the route but, whenever I refresh it, it throws the above error. Has been happening since Mongoose got the latest version.

Had the same issues, but the release notes on mongoose helped, I chained a .clone() method to the .find() method:
https://mongoosejs.com/docs/migrating_to_6.html#duplicate-query-execution
Mongoose no longer allows executing the same query object twice. If
you do, you'll get a Query was already executed error. Executing the
same query instance twice is typically indicative of mixing callbacks
and promises, but if you need to execute the same query twice, you can
call Query#clone() to clone the query and re-execute it.
So all you need to do is to add a .clone() method to the end of the mongoose method that needs to be called concurrently like below (I restructured your code a bit):
app.get("/retrieve", function (req, res) {
Visitor.find({}, function (err, data) {
if (!err) {
res.render("retrieve", { title: "View Visitor Data", records: data });
} else {
throw err;
}
}).clone().catch(function(err){ console.log(err)})
});

wrap your query within async-await.
const visitorData = async() => {
return await Visitor.find({});
}

Don't use the callback within the findOne, instead
use .then() , after the Query, like so
Visitor.find({})
.then(function (err, data) {
if (!err) {
res.render("retrieve", { title: "View Visitor Data", records: data });
} else {
throw err;
}
}
This should most likely work , otherwise use clone() if necessary , as stated by
Avis

The error predominantly arises because the same query is executed more than once, the most often cause being unknowingly using promise and callback at the same time, to avoid such mistakes this error is thrown.
To get rid of this error, stick to either promise or callback when executing this query method.
Use:
await Model.findOneAndUpdate(query,doc,options)
// (or) regular .then() .catch() syntax
Model.findOneAndUpdate(query)
.then(res=>...)
.catch(err=>...)
(OR)
Model.findOneAndUpdate(query, doc, options, callback)
The same principle applies to similar methods like findByIdAndUpdate.

Mongoose v6 does not allow duplicate queries.
Mongoose no longer allows executing the same query object twice. If you do, you'll get a Query was already executed error. Executing the same query instance twice is typically indicative of mixing callbacks and promises, but if you need to execute the same query twice, you can call Query#clone() to clone the query and re-execute it. See gh-7398
Duplicate Query Execution
In my case, I assumed my Apollo Server resolver will wait for the promise to resolve and return the result, but that wasn't the case. So I had to add async/await to the mongoose query.

I'm not sure about the details but mine and my co-workers problems got fixed by downgrading mongoose to 5.13.14

Mongoose no longer allows executing the same query object twice. If you do, you'll get a Query was already executed error. Executing the same query instance twice is typically indicative of mixing callbacks and promises, but if you need to execute the same query twice, you can call ".clone();" to clone the query and re-execute it.

// Mongoose 6.6.3
// Query.prototype.clone()
// Returns:
// «Query» copy
// Make a copy of this query so you can re-execute it.
// Example:`enter code here`
const q = Book.findOne({ title: 'Casino Royale' });
await q.exec();
await q.exec();
// Throws an error because you can't execute a query
twice
await q.clone().exec(); // Works

My mistake was using find() for finding a record like this:
foo.find(a=> a.id === id) //It is wrong!
It should be:
foo.findById(id)

The newer version of Mongoose (version 6+) doesn't support executing the same query object twice. So, just downgrade to an older version, like version 5.13.15.
Uninstall the current Mongoose package & then download the older version:
npm uninstall mongoose
npm i mongoose#5
Hope, this can resolve the issue.

Related

Saving multiple documents per seconds on MongoDB returns timeout

I am trying to map through an array [of nearly 1200 objects] with the Javascript's .map() method to individually save each object to a Mongodb database (enriched with extra data).
The first 270 or so would get saved without problems to Mongodb. But subsequent attempts to save would resolve in this error:
Request failed: Error: Timeout of 60000ms exceeded
at RequestBase._timeoutError (/home/i8mi/Desktop/work/projects/s/mailer/node_modules/superagent/lib/request-base.js:722:13)
at Timeout.<anonymous> (/home/i8mi/Desktop/work/projects/s/mailer/node_modules/superagent/lib/request-base.js:736:12)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
timeout: 60000,
code: 'ECONNABORTED',
errno: 'ETIME',
response: undefined
}
The whole operation ends in less than 2 minutes - the successful saves and the failures.
I understand this is a timeout error from nodejs(superagent) error. I can't seem to find any information on rate limits for mongoDB cloud.
At the end there would be erros like these:
(node:13061) UnhandledPromiseRejectionWarning: MongoNetworkTimeoutError: connection timed out
at connectionFailureError (/home/i8mi/Desktop/work/projects/s/mailer/node_modules/mongodb/lib/core/connection/connect.js:342:14)
at TLSSocket.<anonymous> (/home/i8mi/Desktop/work/projects/s/mailer/node_modules/mongodb/lib/core/connection/connect.js:310:16)
at Object.onceWrapper (events.js:420:28)
at TLSSocket.emit (events.js:314:20)
at TLSSocket.Socket._onTimeout (net.js:484:8)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7)
Any idea what could possibly be done to solve this? Do I need something like Redis, Kafka or RabbitMQ for this?
Stack versions:
- Node v12.19.1
- Mongoose: "^5.12.2",
- Database: MongoDB Cloud (free tier)
Update:
Actual code:
function GetRequestResourceIds(data){
console.log(data[0].tagList)
TagList.find({_id: data[0].tagList}, (err, result) => {
console.log(result)
return new Promise(async(resolve, reject) => {
if(!err && result){
// Now create mxLead for each person in this Tag List
// find the greeting info to use
const allRecipients = result[0].jsonList
// console.log(result)
await Greeting.find({_id: data.greeting}, (err, result) => {
if(!err && result){
// do this for each person in the list
const avartarGreeting = result.avartarGreeting
const initialTextGreeting = result.textGreeting
allRecipients.map(async(recipient) => {
//construct recipient (baseRecipient) data here
const baseRecipient = {
firstName: recipient.data[0],
lastName: recipient.data[1],
emailAddress: recipient.data[2],
avartarUrl: recipient.data[3],
}
// const textGreeting = initialTextGreeting+baseRecipient.firstName
const textGreeting = initialTextGreeting
const avData = {
avartarGreeting,
textGreeting,
avartarUrl: baseRecipient.avartarUrl,
}
return personalizedAvatar.personalizedAvatar(avData)
.then(newAvatar => {
const newAvatars = new Avatar({
recipientEmail: baseRecipient.emailAddress,
recipientFirstName: baseRecipient.firstName,
recipientLastName: baseRecipient.lastName,
newAvatar,
status: "done",
})
newAvatars.save()
// console.log("success")
resolve('success')
})
})
}
})
// return result
}
else {
console.log("failed")
resolve("failed")
}
})
})
}
module.exports.GetRequestResourceIds = GetRequestResourceIds
This is intended for a use case where game developers can generate avatars for players automatically as they achieve ranks, join a group or just for seasonal greetings. So they could be sending an array of one, hundreds or thousands users.
your error's stack trace does not mention MongoDB at any point. superagent is an HTTP client, which has nothing to do with MongoDB - so that timeout is coming from somewhere else.
The second error, which I suppose comes from the actual piece of code that does the writing, indicates a connection timeout - either when attempting to connect (are you establishing a new connection for each write?) or when writing the documents due to connection starvation. In either case, you are probably suffering from unbounded concurrency - a condition where arbitrarily many asynchronous operations are executing at the same time (read more: Promise.allpocalypse) - though I cannot be sure without seeing some code.
When writing large numbers of documents to MongoDB, you should probably be using collection.insertMany (see code: MongoDB usage examples), which enables you to pass whole arrays of documents to save.
You do not need a message queue for writing large numbers of documents to MongoDB, unless each of them requires special processing by workers.
EDIT: Based on the now-included code, here are some possible improvements:
Iterate over the recipient list first, compute everything you need and store in some variable
Save the new documents in bulk - here, we assume they're all new, so insertMany should do
Handle all promises in a way that doesn't lose rejections (.catch())
Avoid mixing new Promise(), .then() and async functions - settle on one style, preferably async/await if possible (this lets you avoid .catch() and just do try {} catch () instead)
Do not resolve with "failure" if an error occurs - that's what exceptions and throw are for
The way you are trying to accomplish this task doesn't sound healthy or db/network-friendly. You should rather research and implement mongoDb bulkWrite operations - or at the very least find a way of using insertMany(), updateMany(), or any of its peers.
You are probably making too many requests for execution, try to put code inside foreach and execute one request at a time.
jobs.forEach(a => {mongoose.set("strictQuery", false);
mongoose.connect('your string')
.then(()=> {
const job = new Job({
_id: new mongoose.Types.ObjectId(),
title: a.title,
location: a.location,
});
job
.save()
.catch(err => {
console.log(err);
});
});
});
The example above is my code, so see that just as the inspiration :)

Using async-await to execute a task AFTER dropping a collection from mongodb database

I am trying to use async await to execute an http request before executing some other code.
More precisely, I would like to drop a collection in my mongodb database, before executing some others tasks. Here's what I did:
app.component.ts:
async deleteRiskRatingData2() {
await this.saveInformationService
.deleteRiskRatingInformation()
.subscribe((data: string) => {
console.log('Deleting risk Rating');
console.log(this.riskRatingTable);
});
console.log('TASKS TO BE EXECUTED AFTER DROPIING COLLECTION');
}
save-information.service.ts
deleteRiskRatingInformation() {
console.log('INIDE deleteRiskRatingInformation INSIDE SAVE-INFORMATION.SERVICE');
return this.http.get(`${this.uri}/dropRiskRatingCollection`);
}
In the backend:
server.js
router.route('/dropRiskRatingCollection').get((req, res) => {
RiskRating.remove({},(err) => {
if (err)
console.log(err);
else
res.json("Risk Rating Collection has been dropped!");
});
});
And this is what happens:
I though my implementation of Async/Await should allow me to execute the:
console.log('TASKS TO BE EXECUTED AFTER DROPPING COLLECTION');
After the dropping of the collection request has been executed. But that didn't happen as you see. And I really don't understand why.
Any idea why is this happening? Is my logic flawed somewhere? And how can I achieve my goal?
Thank you!
async-await work only with Promises. You're try them with Observables. That won't work. Observables have an API that let's you convert them into Promises though. You can call a toPromise method on them in order to do that.
Try this:
async deleteRiskRatingData2() {
const data = await this.saveInformationService.deleteRiskRatingInformation().toPromise();
console.log('Deleting risk Rating');
console.log(this.riskRatingTable);
console.log('TASKS TO BE EXECUTED AFTER DROPIING COLLECTION');
}
NOTE: It's fine if you're trying this just for the sake of testing it. But I think you should not really switch back to promises just to use async-await, to make your code look synchronous.

MongoDB : does Collection.Find() support promise

I was trying to revamp an old Nodejs web service to replace callback functions with chained promises.
When querying mongodb we used the syntax below to iterate through a result set.
collection.find(filter).toArray(function(err, items) {
if (err) {
throw(err);
} else {
console.log(items);
}
If I try to replace the .toArray() section with a .then() I get the below error "col.find(...).then is not a function".
If I replace .find() with .findOne().then(), the code works perfectly.
Any help is appreciated.
find returns a Cursor, but the cursor's toArray method returns a promise. So you can do:
collection.find(filter).toArray().then(...)
I assume you're using mongoosejs.
collection.find() is just a query. To cause it to execute and return a promise, you need to call .exec() on it.:
collection.find(filter).exec()
.then(items => console.log(items))
.catch(err => { // handle error })
The mongoose docs give you more details on how to use mongoose with either callbacks or promises: https://mongoosejs.com/docs/api.html#model_Model.find

how to use mongo db.eval through Mongoose

From my mongo shell I can run functions with db.eval like:
db.eval('return 7;');
And, but for a deprecation warning, the parameter function is run properly.
Now I want to call such a 'script' from node.js, through mongoose.
I've tried to just call db.eval with the stringified function code as parameter (as in here):
mongoose.connect(PATH_TO_A_MONGO_DATABASE);
mongoose.connection.db.eval('return 9;', function(err, result){
console.log(result);
//etc.
});
This silently ignores the callback. No error is thrown, and the callback function is never called.
When doing this, I checked that mongoose.connection.db.eval is actually a function.
Since db.eval also has a Promise interface, also tried:
mongoose.connection.db.eval('return 5;').then(console.log).catch(console.log);
With the same result: no errors, just silence.
So, I guess I'm missing something with the syntax, but I can't find documentation or examples about this. Any similar questions like this or this didn't help.
PD1: I'm willing to do this because I need to call a procedure stored in system.js through mongoose. I can't do that too. However, I can't even run a silly script like return 5;, so I'm asking for the simpler task before. Any advice on how to call server scripts with mongoose is welcome.
PD2: I'm aware stored server scripts in mongo are a bad practice, and that they are deprecated for a reason and so on... but I can't just decide about that. I've been told to do that at my company, and the co-worker who set up the original code of the stored server script is not here now.
Ok, I figured out why my db.eval callbacks was being ignored. It is related with the mongoose connection.
If you start a connection like:
const conn = mongoose.connect(PATH_TO_DB);
And then just make a query:
conn.model(a_collection, a_schema).find({}).exec().then()...
Even if you just call the query right after the connection -it is, logically, an asynchronous process-, mongooose figures that it has to wait to the connection to be in a proper state, then fire the query, then call the callback with the query results.
However, this doesn't work the same way with db.eval(). just trying to call db.eval() right after the call to the connection, the callback is silently ignored:
const conn = mongoose.connect(PATH_TO_DB);
conn.db.eval('return 5;', (err, response) => {
//nothing happends
})
That was the way I was creating the connection and calling db.eval, so I couldn't get db.eval() to work.
In order to fire a db.eval and make it work, it seems that you need to wait for the connection explicitly:
const conn = mongoose.connect(PATH_TO_DB, err => {
mongoose.connection.db.eval('return 5', (err, result) => {
result; //5!
})
});
To make a query before any attemps to call db.eval also works:
const conn = mongoose.connect(PATH_TO_DB);
conn.model(a_collection, a_schema).find({}).exec()
.then(() => {
conn.db.eval('return 5', (err, result) => {
result; //5!
})
})

TypeError: Cannot call method 'then' of undefined

I have the following code executing inside a controller in Sailsjs. The underlying adapter is sails-orientdb. I get the following error back
TypeError: Cannot call method 'then' of undefined
Why is this error occurring?
User.query("select id from User where username='" + req.param("userid") + "'").then(function(userData){
console.log(userData);
return userData;
}).fail(function (err) {
console.log("Handled");
});
To expand on what #jaumard said.
Background
Usually it's recommend to use the standard waterline query methods such as find(), update(), create(), destroy(), etc. These methods support callbacks and promises and these methods will work the same way against any waterline adapter (OrientDB, MySQL, MongoDB, etc).
Sometimes you may want to do something more complex which is not supported by the standard waterline methods and for that sails-orientdb extends the waterline definition with query(), among other methods.
sails-orientdb .query()
sails-orientdb query uses callback so you can use it like this:
// Assume a model named "Friend"
Friend.query("SELECT FROM friendTable WHERE name='friend query'", function(err, retrievedUsers){
console.log(retrievedUsers);
});
UPDATE: since v0.10.53 the below is also supported:
Friend.query("SELECT FROM friendTable WHERE name='friend query'")
.then(function(retrievedUsers){
console.log(retrievedUsers);
});
If you really want to use the promise mechanism you have 2 options:
promisify query()
By using bluebird promise's promisify. Example:
var queryPromise = Promise.promisify(User.query);
queryPromise('select...').then(function(){/*...*/})
Use getDB()
Another option is to use sails-orientdb custom method getDB. This method returns an Oriento db instance which can be used for promisified queries. Example:
User.getDB().query('select from OUser where name=:name', {
params: {
name: 'Radu'
},
limit: 1
}).then(function (results){
console.log(results);
});
I don't think promise works with .query method (maybe I'm wrong).
You can use this instead :
User.findOne()
.where({ username: req.param("userid") })
.then(function(user){
return user;
}).then(function(user){
//Do what you want
}).catch(function(err){
// An error occurred
});
With sails and waterline it's not recommended to use .query when you not really need to.
See this tutorial it may help : http://documentup.com/kriskowal/q/#tutorial/chaining

Resources