I have a the following model and indexes:
var mySchema = new Schema({
page: String,
date: Date,
data: Object,
_page: {
type: Schema.Types.ObjectId,
ref: 'Page'
},
});
mySchema.index({_page: 1, date: 1 }, {unique: true});
And this query:
mySchema.find({
_page: page._id,
date: {
$gte: date1,
$lt: date2
}
})
.sort({
date: 1
})
.exec((err, result) => {
console.log(result);
})
And it logs out like this:
myschema.find({ date: { '$gte': new Date("Thu, 16 Nov 2017 23:00:00 GMT"), '$lt': new Date("Thu, 14 Dec 2017 23:00:00 GMT") }, _page: ObjectId("5a57b30bf54be100c315f2f5") }, { sort: { date: 1 }, fields: {} })
And it takes about 1 second to return ~30 results from a table with about 4000 documents and ~60mb.
DB is a replica cluster on mlab, query speed is pretty similar weather I connect from localhost or my server (db and server in w-europe)
Is there any way to speed this up? Am I indexing or querying wrong?
Using lean() method can improve the performance of find query in mongoose. The reason for it is that using this function returns plain Javascript objects instead of extra Mongoose methods like save function, getters and setters. If you want to check the performance difference of a plain Javascript object and Mongoose Object, you can visit this link
However, you can take advantage of it for only read operations. So, if its just find operation, you can use lean().
mySchema.find(...).lean().exec(function(err, docs) {
...
});
Related
i will try to explain my problem here and what i want to complete in this task:
const getPosts = async (req, res, next) => {
let posts;
try {
posts = await Status.find({}).sort({ date: "desc" });
} catch (err) {
const error = new HttpError(
'backend_message1',
500
);
return next(error);
}
return res.json({ posts: posts });
};
With this code i fetch all posts from my database and output looks something like this:
[
{
_id: 60b39c057a5db20314c69b37,
creator: 609b176f4207251738e18aaa,
body: 'test:d',
date: 'Sun May 30 2021 16:07:01 GMT+0200 (GMT+02:00)',
__v: 0
},
{
_id: 60b39b827a5db20314c69b36,
creator: 609b176f4207251738e18aaa,
body: 'tsx:d',
date: 'Sun May 30 2021 16:04:50 GMT+0200 (GMT+02:00)',
__v: 0
},
{
_id: 60b395ec7a5db20314c69b34,
creator: 609b176f4207251738e18aaa,
body: "dddddd",
date: 'Sun May 30 2021 15:41:00 GMT+0200 (GMT+02:00)',
__v: 0
},
{
_id: 60b391ec7a5db20314c69b33,
creator: 609b176f4207251738e18aaa,
body: "lorem",
date: 'Sun May 30 2021 15:23:56 GMT+0200 (GMT+02:00)',
__v: 0
}
]
And now as you see i have here creator id, but instead of using two fatches in frontend i got idea to fetch users info from backend and just ad object for each this object and just update it then map on frontend.. I still dont know is this a good idea to do all this in backend, but when i try to do it on front i cant do it successful because my response is to slow..
So thing that i want here is when i fetch posts i also want to fetch creator info and then update every object with that creator informations like name and image.. So finall result should be something like this:
[
...
{
_id: 60b39c057a5db20314c69b37,
creator: 609b176f4207251738e18aaa,
body: 'test:d',
date: 'Sun May 30 2021 16:07:01 GMT+0200 (GMT+02:00)',
creatorinfo: {
username: Test Test,
email: test#live.com,
picture: somepictureimage.jpg
}
__v: 0
}
]
I hope I didn't complicate it too much with the question, all the best.
Whats the correct way to do this?
Use array.map and Promise.all to replace the creator id with the creator object. I don't have enough info to provide a copy-paste code solution, but this should lead you in the right direction.
My MongoDB collection have an ISODate field and I want to format the result to dd/mm/yyyy.
My model:
const Pedido = new Schema({
id: {
type: String,
required: true
},
cliente: {
type: Schema.Types.ObjectId,
ref: 'clientes',
required: true
},
date: {
type:Date
}
})
mongoose.model('pedidos',Pedido)
And that's the query and render:
var query = await Pedido.find().populate('cliente').lean().exec()
res.render("admin/pedidos",{pedidos: query})
I'm using handlebars
{{#each pedidos}}
<h5 class="ordem1">Pedido #{{id}} <small>{{date}}</small></h5>
{{/each}}
It's showing a result like that:
Wed Apr 08 2020 21:00:00 GMT-0300 (GMT-03:00)
but I want to show: 08/04/2020
Could anybody help me with this? Thank you!!
we can use $dateToString operator to format date, check the mongoDb Docs
as you can see, we can use this $dateToString operator only in aggregate pipeline, in the $project step
here is a simple example in mongo playground mongoplayground
in your example, we could do the same process, but use $lookup instead of populate
the query may be something like that
Pedido.aggregate([
{
$match: {} // add your search here
},
{
$lookup: { // this is the alternative to the populate
from: 'clientes',
localField: 'cliente',
foreignField: '_id',
as: 'clientes'
}
},
{
$project: { // add all the fields you need from the collection, if you need to omit something from the query results, just don't mention it here
id: 1,
clientes: 1,
date: { $dateToString: { format: "%d/%m/%Y", date: "$date" } } // this will return the date in the format "dd/MM/yyyy"
}
}
])
If I have different types of documents, each in their own collections, is there a way to search for posts from all collections and return them as a single list ordered by something like a datestamp?
Further, I need:
To be able to decide how many posts I need in total from all collections
The posts should be ordered by the same criteria - which means the number of posts will be different from each collection
To be able to start collecting with an offset (say, give me 100 posts starting at post no. 201).
If I saved all documents in the same collection this task would be rather easy but would also require a dynamic, largely undocumented schema since each document will be very different except for a few parameters such as the date.
So, is there a way to keep my documents in well defined schemas, each in separate collections but still being able to accomplish the above?
For argument's sake, here's how the schemas could look divided up:
var InstagramPostSchema = new Schema({
date: Date,
imageUrl: String,
...
})
var TwitterPostSchema = new Schema({
date: Date,
message: String,
...
})
And if I made one universal schema it could look like this:
var SocialPostSchema = new Schema({
date: Date,
type: String,
postData: {}
})
What's the preferred way to do this?
The ideal way would be if I could write separate schemas that inherits from a common base schema, but I'm not familiar enough with Mongoose and MongoDB to know if there's a native way to do this.
There is a good way to do this which is also a bit nicer and with some benifts over your final suggestion, and it is to use discriminators.
The basic idea is that there is a base schema with common properties or even no properties at all for which you are going to define your main collection from. Each other schema then inherrits from that and also shares the same collection.
As a basic demonstration:
var async = require('async'),
util = require('util'),
mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/test');
function BaseSchema() {
Schema.apply(this,arguments);
this.add({
date: { type: Date, default: Date.now },
name: { type: String, required: true }
});
}
util.inherits(BaseSchema,Schema);
var socialPostSchema = new BaseSchema();
var instagramPostSchema = new BaseSchema({
imageUrl: { type: String, required: true }
});
var twitterPostSchema = new BaseSchema({
message: { type: String, required: true }
});
var SocialPost = mongoose.model('SocialPost', socialPostSchema ),
InstagramPost = SocialPost.discriminator(
'InstagramPost', instagramPostSchema ),
TwitterPost = SocialPost.discriminator(
'TwitterPost', twitterPostSchema );
async.series(
[
function(callback) {
SocialPost.remove({},callback);
},
function(callback) {
InstagramPost.create({
name: 'My instagram pic',
imageUrl: '/myphoto.png'
},callback);
},
function(callback) {
setTimeout(
function() {
TwitterPost.create({
name: "My tweet",
message: "ham and cheese panini #livingthedream"
},callback);
},
1000
);
},
function(callback) {
SocialPost.find({}).sort({ "date": -1 }).exec(callback);
}
],
function(err,results) {
if (err) throw err;
results.shift();
console.dir(results);
mongoose.disconnect();
}
);
With output:
[ { __v: 0,
name: 'My instagram pic',
imageUrl: '/myphoto.png',
__t: 'InstagramPost',
date: Wed Aug 19 2015 22:53:23 GMT+1000 (AEST),
_id: 55d47c43122e5fe5063e01bc },
{ __v: 0,
name: 'My tweet',
message: 'ham and cheese panini #livingthedream',
__t: 'TwitterPost',
date: Wed Aug 19 2015 22:53:24 GMT+1000 (AEST),
_id: 55d47c44122e5fe5063e01bd },
[ { _id: 55d47c44122e5fe5063e01bd,
name: 'My tweet',
message: 'ham and cheese panini #livingthedream',
__v: 0,
__t: 'TwitterPost',
date: Wed Aug 19 2015 22:53:24 GMT+1000 (AEST) },
{ _id: 55d47c43122e5fe5063e01bc,
name: 'My instagram pic',
imageUrl: '/myphoto.png',
__v: 0,
__t: 'InstagramPost',
date: Wed Aug 19 2015 22:53:23 GMT+1000 (AEST) } ] ]
So the things to notice there are that even though we defined separate models and even seperate schemas, all items are in fact in the same collection. As part of the discriminator, each document stored has a __t field depicting it's type.
So the really nice things here are:
You can store everything in one collection and query all objects together
You can seperate validation rules per schema and/or define things in a "base" so you don't need to write it out multiple times.
The objects "explode" into their own class defintions by the attached schema to the model for each type. This includes any attached methods. So these are first class objects when you create or retrieve the data.
If you wanted to work with just a specific type such as "TwitterPost", then using that model "automatically" filters out anything else but the "twitter" posts from any query operations performed, just by using that model.
Keeping things in the one collection makes a lot of sense, especially if you want to try and aggregate data accross the information for different types.
A word of caution is that though you can have completely different objects using this pattern, it is generally wise to have as much in common as makes sense to your operations. This is particularly useful in querying or aggregating across different types.
So where possible, try to convert "legacy imported" data to a more "common" format of fields, and just keep the unique properties that are really required for each object type.
As to the first part of your question where you wanted to query "each collection" with something like different limits and then sort the overall results from each, well you can do that too.
There are various techniques, but keeping in the MongoDB form, there is nedb which you an use to both store the combined results and "sort" them as well. And all is done in a manner you are used to:
var async = require('async'),
util = require('util'),
mongoose = require('mongoose'),
DataStore = require('nedb'),
Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/test');
function BaseSchema() {
Schema.apply(this,arguments);
this.add({
date: { type: Date, default: Date.now },
name: { type: String, required: true }
});
}
util.inherits(BaseSchema,Schema);
var socialPostSchema = new BaseSchema();
var instagramPostSchema = new BaseSchema({
imageUrl: { type: String, required: true }
});
var twitterPostSchema = new BaseSchema({
message: { type: String, required: true }
});
var SocialPost = mongoose.model('SocialPost', socialPostSchema ),
InstagramPost = SocialPost.discriminator(
'InstagramPost', instagramPostSchema ),
TwitterPost = SocialPost.discriminator(
'TwitterPost', twitterPostSchema );
async.series(
[
function(callback) {
SocialPost.remove({},callback);
},
function(callback) {
InstagramPost.create({
name: 'My instagram pic',
imageUrl: '/myphoto.png'
},callback);
},
function(callback) {
setTimeout(
function() {
TwitterPost.create({
name: "My tweet",
message: "ham and cheese panini #livingthedream"
},callback);
},
1000
);
},
function(callback) {
var ds = new DataStore();
async.parallel(
[
function(callback) {
InstagramPost.find({}).limit(1).exec(function(err,posts) {
async.each(posts,function(post,callback) {
post = post.toObject();
post.id = post._id.toString();
delete post._id;
ds.insert(post,callback);
},callback);
});
},
function(callback) {
TwitterPost.find({}).limit(1).exec(function(err,posts) {
async.each(posts,function(post,callback) {
post = post.toObject();
post.id = post._id.toString();
delete post._id;
ds.insert(post,callback);
},callback);
});
}
],
function(err) {
if (err) callback(err);
ds.find({}).sort({ "date": -1 }).exec(callback);
}
);
}
],
function(err,results) {
if (err) throw err;
results.shift();
console.dir(results);
mongoose.disconnect();
}
);
Same output as before with the latest post sorted first, except that this time a query was sent to each model and we just got results from each and combined them.
If you change the query output and writes to the combined model to use "stream" processing, then you even have basically the same memory consumption and likely faster processing of results from parallel queries.
I have a mongodb database with a collection as follow:
var mongoose = require('mongoose');
var journalSchema = mongoose.Schema({
title : String,
journalid: {type:String, index: { unique: true, dropDups: true }},
articleCount : type:Number, default:1,
});
module.exports = mongoose.model('Journal', journalSchema);
Now that my database is growing, I would like to have a "articles count" field per year.
I could make an array as follow years : [{articleCount : Number}] and fill it up by accessing journal.years[X] for a specific year, where X correspond to 0 for 1997, 1 for 1998, etc..
However, my data are scrapped dynamically and I would like to have a function in my express server where articleCountis increased based on the year.
For example:
function updateYear(journalid, year, callback) {
Journal.findOneAndUpdate(
{'journalid':journalid},
{$inc : {'articleCount' : 1}}, // DO SOMETHING WITH year HERE
function() {
callback();
});
}
This does increase the article count but I don't know where to include the "year"...
What would be the fastest way of doing that, knowing that I have to fetch through quite a lot of articles (10 millions +) and I would like to be able to get/update the article count for a given year efficiently.
Hope I'm clear enough, Thanks!
Make your array a set of objects with a year and count:
journalYears: [
{
year: String, // or Number
count: Number
}
]
e.g.
journalYears: [
{ year: "2014", count: 25 },
{ year: "2015", count: 15 }
]
Then for your update:
function updateYear(journalId, year, callback) {
Journal.findOneAndUpdate(
{_id: journalId, "journalYears.year": year},
{ $inc: { "journalYears.$.count": 1 } },
callback
);
}
The index of the first match from your query is saved in $. It's then used to update that specific element in your array.
When I query a user from its email:
User.findOne({email: 'a#a.com'}).done(function(err, u){console.log(u.sessionTokens);})
I got its tokens list:
[ { token: '8dfe6aa0-2637-4b3f-9a3c-3ae8dc225b77',
issuedAt: Tue Jan 14 2014 14:40:22 GMT+0100 (CET) } ]
But when I query a token within this list it does not retrieve the user:
User.findOne({'sessionTokens.token':'8dfe6aa0-2637-4b3f-9a3c3ae8dc225b77'}).done(function(err, u){console.log(u);})
=> undefined
Any ideas ?
Does the User.findOne({'sessionTokens.token':'8d...77'}) is a correct query ?
EDIT
My User model is something like:
module.exports = {
schema: true,
attributes: {
email: {
type: 'email',
required: true
},
encryptedPassword: {
type: 'string'
},
sessionTokens: {
type: 'array'
},
...
}
}
sessionToken is an attribute of type Array. This array contains several object like:
{ token: '8dfe6aa0-2637-4b3f-9a3c-3ae8dc225b77',
issuedAt: Tue Jan 14 2014 14:40:22 GMT+0100 (CET) }
EDIT
I've tested with mongo-sails instead of mongo-disk, this is working fine but as Chad pointed this out, it depends of the adapter used.
With the help of robdubya & sfb_ in #sailsjs#freenode, we discussed a solution for this a bit. The answer depends on the adapter you are using. The adapter that I am most familiar with is the mysql adapter, and in that adapter, the 'array' attribute type gets serialized to the database as a JSON string. So in that case, using the 'contains' modifier on the query of the model would allow you to find the record you are looking for:
var criteria = {
sessionToken: {
contains: '8dfe6aa0-2637-4b3f-9a3c-3ae8dc225b77'
}
};
User.findOne(criteria).done(function (err, user) {
...
});