I'm using MongoDB with Node.js framework.
There is a weird behavior that some documents are not getting inserted into db, though from orm's point of view there are no errors: err = null in callback of Collection.create() and fresh document with _id is returned. When I try to search by that _id in db - no document is found.
I tried to manually insert new document to db and it was successfull.
Is there a way I can trace these operations from db's point of view? Some command to list recent requests and their results..?
You can enable profiling for all operations:
db.setProfilingLevel(2)
Then, look at system.profile collection to see what's happen. system.profile is a capped collection that can be searched as any other collection. Profiling can be noisy, and eventually you should have to change the size of the system.profile collection
db.setProfilingLevel(0)
db.system.profile.drop()
db.createCollection( "system.profile", { capped: true, size:4000000 } )
db.setProfilingLevel(2)
The most notable way of tracking errors within MongoDB is to use the --diaglog option: http://docs.mongodb.org/manual/reference/program/mongod/#cmdoption--diaglog with maybe a level of 3, however 1 might be enough for you.
As noted by #Neil this has unfortunately become deprecated as of 2.6.
The only way currently is to write out ALL operations MongoDB performs, via #Rauls answer, and then use a query like:
db.system.profile.find({op:{$in:['update', 'insert', 'remove']}});
and possibly resize the capped collection used for profiling: http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/#profiler-overhead to capture the amount you want.
Related
Saving an aggregation query using "mongodb": "^3.0.6" as result with the $out operator is only working when calling .toArray().
The aggregation step(s):
let aggregationSteps = [{
$group: {
_id: '$created_at',
}
}, {'$out': 'ProjectsByCreated'}];
Executing the aggregation:
await collection.aggregate(aggregationSteps, {'allowDiskUse': true})
Expected result: New collection called ProjectsByCreated.
Result: No collection, query does not throw an exception but is not being executed? (takes only 1ms)
Appending toArray() results in the expected behaviour:
await collection.aggregate(aggregationSteps, {'allowDiskUse': true}).toArray();
Why does mongodb only create the result collection when calling .toArray() and where does the documentation tell so? How can I fix this?
The documentation doesn't seem to provide any information about this:
https://mongodb.github.io/node-mongodb-native/3.0/api/Collection.html#aggregate
https://docs.mongodb.com/manual/reference/operator/aggregation/out/index.html
MongoDB acknowledge this behaviour, but they also say this is working as designed.
It has been logged as a bug in the MongoDB JIRA, $out aggregation stage doesn't take effect, and the responses say it is not a fault:
This behavior is intentional and has not changed in some time with the node driver. When you "run" an aggregation by calling Collection.prototype.aggregate, we create an intermediary Cursor which is not executed until some sort of I/O is requested. This allows us to provide the chainable cursor API (e.g. cursor.limit(..).sort(..).project(..)), building up the find or aggregate options in a builder before executing the initial query.
... Chaining toArray in order to execute the out stage doesn't feel quite right. Is there something more natural that I haven't noticed?
Unfortunately not, the chained out method there simply continues to build your aggregation. Any of the following methods will cause the initial aggregation to be run: toArray, each, forEach, hasNext, next. We had considered adding something like exec/run for something like change streams, however it's still in our backlog. For now you could theoretically just call hasNext which should run the first aggregation and retrieve the first batch (this is likely what exec/run would do internally anyway).
So, it looks like you do have to call one of the methods to start iterating the cursor before $out will do anything. Adding .toArray(), as you're already doing, is probably safest. Note that to.Array() does not load the entire result into RAM as normal; because it includes a $out, the aggregation returns an empty cursor.
Because the Aggregation operation returns a cursor, not the results.
In order to return all the documents from the cursor, we need to use toArray method.
I'm using an ajax request from the front end to load more comments to a post from the back-end which uses NodeJS and mongoose. I won't bore you with the front-end code and the route code, but here's the query code:
Post.findById(req.params.postId).populate({
path: type, //type will either contain "comments" or "answers"
populate: {
path: 'author',
model: 'User'
},
options: {
sort: sortBy, //sortyBy contains either "-date" or "-votes"
skip: parseInt(req.params.numberLoaded), //how many are already shown
limit: 25 //i only load this many new comments at a time.
}
}).exec(function(err, foundPost){
console.log("query executed"); //code takes too long to get to this line
if (err){
res.send("database error, please try again later");
} else {
res.send(foundPost[type]);
}
});
As was mentioned in the title, everything works fine, my problem is just that this is too slow, the request is taking about 1.5-2.5 seconds. surely mongoose has a method of doing this that takes less to load. I poked around the mongoose docs and stackoverflow, but didn't really find anything useful.
Using skip-and-limit approach with mongodb is slow in its nature because it normally needs to retrieve all documents, then sort them, and after that return the desired segment of the results.
What you need to do to make it faster is to define indexes on your collections.
According to MongoDB's official documents:
Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must perform a collection scan, i.e. scan every document in a collection, to select those documents that match the query statement. If an appropriate index exists for a query, MongoDB can use the index to limit the number of documents it must inspect.
-- https://docs.mongodb.com/manual/indexes/
Using indexes may cause increased collection size but they improve the efficiency a lot.
Indexes are commonly defined on fields which are frequently used in queries. In this case, you may want to define indexes on date and/or vote fields.
Read mongoose documentation to find out how to define indexes in your schemas:
http://mongoosejs.com/docs/guide.html#indexes
Meteor Mongo and Mongodb query is doest same. I am using external Mongodb. so I need to debug my query. Is their any way to find last executed query in Mongo?
Don't know if this works in meteor mongo -but you seem to be using an external mongo - presumably you set up profiling with a capped collection, so that the collection never grows over a certain size. If you only need the last op, then you make the size pretty much smaller than this.
db.createCollection( "system.profile", { capped: true, size:4000000 } )
The mongo doc is here: http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/
From the mongo docs:
To return the most recent 10 log entries in the system.profile
collection, run a query similar to the following:
db.system.profile.find().limit(10).sort( { ts : -1 } ).pretty()
Since it's sorted inversely by time, just take the first record from the result.
Otherwise you could roll your own with a temporary client-only mongo collection:
Queries = new Mongo.Collection(null);
Create an object containing your query, cancel the last record and insert the new one.
I'm receiving:
[Error: Bad BSON Document: illegal CString]
When using the Node MongoDB driver while iterating over one of my collections with Cursor.each. It seems to make some of my documents disappear, and not found in the collection, even though they were individually accessible when I look for them using Collection.findOne().
Does this mean that my data is corrupted in some way?
Thanks to #wdberkeley for all the help in the above comment, which helped me to track down my problem.
It turns out that I did have a single corrupted document in my collection, which was inserted during an unclean shutdown of Mongo. I was unaware how that document would affect the rest of my queries though.
When you perform a collection.find(), and then start iterating with the cursor over the collection, the cursor will stop and be unable to go any further if it encounters an error, such as with [Error: Bad BSON Document: illegal CString].
This happens with both cursor.forEach or cursor.nextObject. Thus, I was unable to access any of the documents that came after the error in the collection, even though I was able to access those documents individually with collection.findOne.
The only solution in this scenario for me was to run db.repairDatabase, which removed the corrupted documents, and solved the problem for me.
There are two different methods to obtain a reference to a MongoDB collection - both of them are used throughout the official documentation.
There is
var mycollection = db.collection('mycollection)'
and there is
db.collection('mycollection', function(err, collection){
//use collection
}
I tend to use the second one because it is consistent with "db.createCollecion(collection, callback)"
What is the difference between these methods?
Is there any database interaction when using these methods?
If you look at the code for Database, currently around line 456, you'll see that the only difference between the two in the way you've used them is how the collection object is returned. If you specify a callback, then it's returned that way, otherwise, it's returned as the value to the function. If you set the options however and specifically the option strict to true, you need to use the callback. When strict is set to true, the collection is verified before continuing (asynchronously).
Given that collections can be created dynamically (and usually are upon first use), there often isn't need to use strict mode.
So, it's really matter of personal coding preference otherwise. There is normally no activity to the database when creating a Collection object via: db.collection('collectionname') with the exception I mentioned above.