Is there a way in node using the native mongodb driver to limit an update many operation only to the first 500 entries?
right now my update many line looks like this
col.updateMany({$exists : {available : false}}, {$set : {available : "na"}}, function(){});
Is there a way to limit the number of documents it searches and updates?
Thanks!
Related
I have an AWS instance with nodejs and mongod running. The collection I am trying to query from has roughly 289248 documents in there.
Here is the code I am using to query my data:
var collection = db.collection('my_collection');
collection.find({ $and: [
{"text" : {$regex : ".*"+keyword+".*"}},
{"username" : username}
] }).limit(10).toArray(function(err, docs) {
Now originally, I was having issues querying just a username collection.find({"username":username}) because there are so many entries in mongo. So I started limiting my code and in the mongo console, I can set a limit of 30 and it results the results I am looking for.
However, then when I run this application on nodejs, when I query this command, it crashes my mongod service and I have to restart it. On the node server, limit of 1 works fine but limit of 5 does not. I can't simply use limit of 1 if there are many more results in the database. What can I do?
Does not using $and make a difference? :
collection.find({text:{$regex:".*"+keyword+".*"},username: username})
I also wonder if the 'text' is a text index in which case it should be '$text'.
I also note that you use two variables in your query expression, and wonder if you have verified that those variables are defined.
My MongoDB database contains a collection with 200 k documents. I'm trying to fetch all documents in NodeJS as follows
var cursor = collection.find({}, {
"_id" : false,
}).toArray(function(err, docs) {
if (err)
throw err;
callback(null, docs);
});
The above operation is taking longer time and I could not able to get results. Is there any way to optimize find operation to get the result ?
NodeJS driver version :2.0
MongoDB version :3.2.2
I can easily load data from json raw file but I could not able to do it from MongoDB
People can't do a lot with 200k items in the UI. Google shows only 10 results per page, for good reason. Sounds like pagination can help you. Here's an example: Range query for MongoDB pagination
Meteor Mongo and Mongodb query is doest same. I am using external Mongodb. so I need to debug my query. Is their any way to find last executed query in Mongo?
Don't know if this works in meteor mongo -but you seem to be using an external mongo - presumably you set up profiling with a capped collection, so that the collection never grows over a certain size. If you only need the last op, then you make the size pretty much smaller than this.
db.createCollection( "system.profile", { capped: true, size:4000000 } )
The mongo doc is here: http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/
From the mongo docs:
To return the most recent 10 log entries in the system.profile
collection, run a query similar to the following:
db.system.profile.find().limit(10).sort( { ts : -1 } ).pretty()
Since it's sorted inversely by time, just take the first record from the result.
Otherwise you could roll your own with a temporary client-only mongo collection:
Queries = new Mongo.Collection(null);
Create an object containing your query, cancel the last record and insert the new one.
I have a large MongoDB collection, containing more than 2GB of raw data and I use a very simple query to fetch a specific document from the collection by its Id. Document sizes currently range from 10KB to 4MB, and the Id field is defined as an index.
This is the query I'm using (with the mongojs module):
db.collection('categories').find({ id: category_id },
function(err, docs) {
callback(err, docs.length ? docs[0] : false);
}).limit(1);
When I execute this query using MongoDB shell or a GUI such as Robomongo it takes approximately 1ms to fetch the document, no matter what its physical size, but when I execute the exact same query on NodeJS the response time ranges from 2ms to 2s and more depending on the amount of data. I only measure the time it takes to receive a response and even in cases where NodeJS waits for more than 500ms the MongoDB profiler (.explain()) shows it took only a single millisecond to execute the query.
Now, I'm probably doing something wrong but I can't figure out what it is. I'm rather new to NodeJS but I had experience with MongoDB and PHP in the past and I never encountered such performance issues, so I tend to think I'm probably abusing NodeJS in some way.
I also tried profiling using SpyJS on WebStorm, I saw there are a lot of bson.deserialize calls which sums up quickly into a large stack, but I couldn't investigate farther because SpyJS always crashes at this point. Probably related but I still have no idea how to deal with it.
Please advise, any leads will be appreciated.
Edit:
This is the result of db.categories.getIndexes():
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "my_db.categories"
},
{
"v" : 1,
"key" : {
"id" : 1
},
"name" : "id_1",
"ns" : "my_db.categories"
}
]
I also tried using findOne which made no difference:
db.collection('categories').findOne({ id: category_id },
function(err, doc) {
callback(err, doc || false);
});
My guess is the .limit(1) is ignored because the callback is provided early. Once find sees a callback it's going to execute the query, and only after the query has been sent to mongo will the .limit modifier try to adjust the query but it's too late. Recode as such and see if that solves it:
db.collection('categories').find({ id: category_id }).limit(1).exec(
function(err, docs) {
callback(err, docs.length ? docs[0] : false);
});
Most likely you'll need to have a combination of normalized and denormalized data in your object. Sending 4MB across the wire at a time seems pretty heavy, and likely will cause problems for any browser that's going to be doing the parsing of the data.
Most likely you should store the top 100 products, the first page of products, or some smaller subset that makes sense for your application in the category. This may be the top alphabetically, most popular, newest, or some other app-specific metric you determine.
When you go about editing a category, you'll use the $push/$slice method to ensure you avoid unbounded array growth.
Then when you actually page through the results you'll do a separate query to the individual products table by category. (Index that.)
I've written about this before here:
https://stackoverflow.com/a/27286612/68567
I am trying to run an aggregation pipeline using node.js and mongodb native driver on a sharded mongodb cluster with 2 shards. The monogdb ver. is 2.6.1. The operation runs for about 50 minutes and throws the error 'errmsg" : "exception: getMore: cursor didn't exist on server, possible restart or timeout?"' On googling I came across this link . It looks like the issue is not resolved yet. BTW, the size of the collection is about 140 million documents.
Is there a fix/workaround for this issue?
Here is the pipeline that I am trying to run. I don't know at what stage it breaks. It runs for about 50 minutes and the error happens. Same is the case with any aggregation pipeline that I try to run.
db.collection01.aggregate([
{$match:{"state_cd":"CA"}},
{$group : {"_id": "$pat_id" , count : {$sum : 1}}}
],
{out: "distinct_patid_count", allowDiskUse: true }
)
My guess is you could try to lower the batch size to make the cursor more "active".
I came across this error after our server was running for more than 2.5 months. Mongo started dropping cursors even before the timeout (I guess some sort of memory error), restart of mongo solved our problem.