I've got a query that gets quite often that I've identified as slow. It has index on every data point I query, though no compound indexes.
The query looks something like:
ExternalLead.find({
'price': {$gte:3, $lt:6},
"campaign.id":"an id",
createdOn: {$gte: new Date(moment().subtract(10, 'days')),
$lte: new Date(moment().subtract(5, 'min'))
}}).limit(10).sort({_id:-1}).select('_id').exec(function(err, docs){
if (err) console.log(err);
var st = new Date();
console.log(st - s);
});
Simple query, there are about 50k records for that query removing the price. Price is indexed, I'm 100% sure. I've verified it multiple ways. If I remove price this query finishes in about 200ms, with price it takes about 20 seconds. I've tested multiple price ranges, the first 10 it scanned should be a match. Is there something about this query that it's not using the indexes?
Also, the server is about 3x what this database needs right now, it's not a server issue. The entire database is loaded into ram.
Node 6.11.2,
Mongoose: 4.10.8,
mongodb-core: 2.1.1
MongoDb: 3.4
Turns out we needed a compound index for price and createdOn.
Related
I have a db on Dynamodb. And writing some user scores to db. Also I have a lambda function which wrote it with nodejs. I want to get first 10 users who have most points. How could I scan this users?
Thanks a lot.
Max() in NoSQL is much trickier than in SQL. And it doesn't really scale - if you want very high scalability on achieving this let me know, but let's get back to the question.
Assuming your table looks like:
User
----------
userId - hashKey
score
...
Add a dummy category attribute to your table, which will be constant (for example value "A"). Create the index:
category - hash key
score - sort key
Query this index by hash key "A" in reserve order in order to get results much faster than a scan. But this scales to max 10GB (max partition size, all data being in same partition). Also make sure you project only needed attributes for this index, in order to save space.
You can go up to 30GB for example, by setting 3 categories ("A", "B", "C"), executing 3 queries and merge programatically the results. This will affect performance a bit, but still better than a full scan.
EDIT
var params = {
TableName: 'MyTableName',
Limit: 10,
// Set ScanIndexForward to false to display most recent entries first
ScanIndexForward: false,
KeyConditionExpression: 'category = : category',
ExpressionAttributeValues: {
':category': {
S: 'category',
},
},
};
dynamo.query(params, function(err, data) {
// handle data
});
source: https://www.debassociates.com/blog/query-dynamodb-table-from-a-lambda-function-with-nodejs-and-apex-up/
I'm a beginner with CouchDB/Cloudant and I would like some expert advice on the most appropriate method of performing multidimensional queries.
Example...
My documents are like this
{
_id: 79f14b64c57461584b152123e3924516,
lat: -71.05204477,
lng: 42.36674199,
time: 1531500769,
tileX: 5,
tileY: 10,
lod: 7,
val1: 200.1,
val2: 101.5,
val3: 50
}
lat, lng, and time are the query parameters and they will be queried as ranges.
For example fetch all the documents that have
lat_startkey = -70 & lat_endkey = -72 AND
lng_startkey = 50 & lng_endkey = 40 AND
time_startkey = 1531500769 & time_endkey = 1530500000
I will also query using time as a range, and tileX, tileY, lod as exact values
For example
tileX = 5 AND
tileY = 10 AND
lod = 7 AND
time_startkey = 1531500769 & time_endkey = 1530500000
I've been reading about Views (map reduce), and I guess for the first type of query I could create a View each for time, lat, lng. My client could then perform 3 separate range queries, one against each View, and then in the client perform an intersection (inner join) of the resulting document id's. However this is obviously moving some of the processing outside of CouchDB, and I was hoping I could do this all within CouchDB itself.
I have also just found that CouchSearch (json/lucene), and n1ql exist... would these be of any help?
You should be able to use the N1QL query language for queries like this with no problems. N1QL is only available for Couchbase, not the CouchDB project that Couchbase grew out of.
For example, if I understand your first query there, you could write it like this in N1QL:
SELECT *
FROM datapoints
WHERE lat BETWEEN -72 AND -70 AND
lng BETWEEN 40 AND 50 AND
time BETWEEN 1531500769 AND 1530500000
To run such a query efficiently, you'll need an index, like this:
CREATE INDEX lat_long_time_idx ON datapoints(lat, lng, time)
You can find out more about N1QL here:
https://query-tutorial.couchbase.com/tutorial/#1
Sadly CouchDB is extremely poor at handling these sorts of multi-dimensional queries. You can have views on any of the axes but there is no easy way to retrieve the intersection, as you describe.
However an extension was written in the early days of that project to handle GeoSpatial queries (lat, long) called GeoCouch and that extension has been included in the Cloudant platform that you seem to be using. That means that you can do direct queries on the lat/long combination, just not the time axis using the GeoJSON format: https://console.bluemix.net/docs/services/Cloudant/api/cloudant-geo.html#cloudant-nosql-db-geospatial
However Cloudant also has another query system - Query: https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#query
Under this system you can build an arbitary index over your documents and then query for documents having certain criteria. For example this query selector will find documents with years in the range 1900-1903:
{
"selector": {
"year": {
"$gte": 1900
},
"year": {
"$lte": 1903
}
},
So it looks to me as if you could index the three values you care about (Lat, Long and Time) and build a 3 axis query in Cloudant. I have not tried that myself however.
I have the following parse query which times out when the number of records is large.
var query = new Parse.Query("UserLocation");
query.withinMiles("geo", geo, MAX_LOCATION_RADIUS);
query.ascending("createdAt");
if (createdAt !== undefined) {
query.greaterThan("createdAt", createdAt);
}
query.limit(1000);
it runs ok if UserLocation table is small. But the query times out from time to time when the table has ~100k records:
[2015-07-15 21:03:30.879] [ERROR] [default] - Error while querying for locations: [latitude=39.959064, longitude=-75.15846]: {"code":124,"message":"operation was slow and timed out"}
UserLocation table has a latitude,longitude pair and a radius. Given a geo point (latitude,longitude), I'm trying to find the list of UserLocations whose circle (lat,long)+radius covers the given geo point. It doesn't seem like I can use the value from another column in the table for the distance query (something like query.withinMiles("geo", inputGeo, "radius"), where "geo" and "radius" are the column names for GeoPoint and radius). It also has the limit that query "limit" combined with "skip" can only return maximum of 10,000 records (1000 records at a time and skip 10 times). So I had to do a almost full table scan by using "createdAt" as a filter criteria and keep querying until the query doesn't return results any more.
Anyway I can improve the algorithm so that it doesn't time out on large data set?
I'm using Mongoose for Node.js to interface with the mongo driver, so my query looks like:
db.Deal
.find({})
.select({
_id: 1,
name: 1,
opp: 1,
dateUploaded: 1,
status: 1
})
.sort({ dateUploaded: -1 })
And get: too much data for sort() with no index. add an index or specify a smaller limit
The number of documents in the Deal collection is quite small, maybe ~500 - but each one contains many embedded documents. The fields returned in the query above are all primitive, i.e. aren't documents.
I currently don't have any indexes setup other than the default ones - I've never had any issue until now. Should I try adding a compound key on:
{ _id: 1, name: 1, opp: 1, status: 1, dateUploaded: -1 }
Or is there a smarter way to perform the query? First time using mongodb.
From the MongoDB documentation on limits and thresholds:
MongoDB will only return sorted results on fields without an index if the combined size of all documents in the sort operation, plus a small overhead, is less than 32 megabytes.
Probably all the embedded documents are too much, you should add an index on the sorted field dateUploaded if you want to run the same query.
Otherwise you can limit you query and start paginating the results.
I've got a MongoDB database of metadata for about 300,000 photos. Each has a native unique ID that needs to be unique to protect against duplication insertions. It also has a time stamp.
I frequently need to run aggregate queries to see how many photos I have for each day, so I also have a date field in the format YYYY-MM-DD. This is obviously not unique.
Right now I only have an index on the id property, like so (using the Node driver):
collection.ensureIndex(
{ id:1 },
{ unique:true, dropDups: true },
function(err, indexName) { /* etc etc */ }
);
The group query for getting the photos by date takes quite a long time, as one can imagine:
collection.group(
{ date: 1 },
{},
{ count: 0 },
function ( curr, result ) {
result.count++;
},
function(err, grouped) { /* etc etc */ }
);
I've read through the indexing strategy, and I think I need to also index the date property. But I don't want to make it unique, of course (though I suppose it's fine to make it unique in combine with the unique id). Should I do a regular compound index, or can I chain the .ensureIndex() function and only specify uniqueness for the id field?
MongoDB does not have "mixed" type indexes which can be partially unique. On the other hand why don't you use _id instead of your id field if possible. It's already indexed and unique by definition so it will prevent you from inserting duplicates.
Mongo can only use a single index in a query clause - important to consider when creating indexes. For this particular query and requirements I would suggest to have a separate unique index on id field which you would get if you use _id. Additionally, you can create a non-unique index on date field only. If you run query like this:
db.collection.find({"date": "01/02/2013"}).count();
Mongo will be able to use index only to answer the query (covered index query) which is the best performance you can get.
Note that Mongo won't be able to use compound index on (id, date) if you are searching by date only. You query has to match index prefix first, i.e. if you search by id then (id, date) index can be used.
Another option is to pre aggregate in the schema itself. Whenever you insert a photo you can increment this counter. This way you don't need to run any aggregation jobs. You can also run some tests to determine if this approach is more performant than aggregation.