The best way I can describe what I'm after is by starting with a simplified demo page document:
{
name: "Page Name"
blocks: {
"top-content": 50f0ded99561a3b211000001,
"bottom-content": 50f0ded99561a3b211000002
}
}
Is there a way I can define this kind of many-to-many relation with mongoose so that I can look them up by string keys like top-content and bottom-content? The relational equivalent to this would be something like including a string on the many to many join table.
Important:
I don't want to embed the blocks I'm referencing as they could be referenced by multiple pages.
Mongoose's query population feature supports this, but top-content and bottom-content would need to be ObjectIds instead of strings (which is more efficient anyway).
After some chatting in #node.js on Freenode, it seems like I can just set this up as an array and put arbitrary key/value pairs containing the references I want.
At which point, I'll be dereferencing things myself.
{ //PAGE 1
name:"Sticks",
blocks:[
"top":[OID(5),OID(6)],
"bottom":[OID(7),OID(8)]
]
};
{ //PAGE 2
name:"Poopsicles",
blocks:[top:[OID(7)]]
};
//BLOCKS
{
OID:5,
name:"top"
};
{
OID:7,
name:"bottom rocker"
};
//QUERYING
page.findOne({name:"Sticks"}, function(e,d) {
var a = [];
for(l in d.blocks) {
a.append(d.blocks[l]);
}
blocks.find({oid:a},function (e,b) {
//do your stuff with the blocks and page here;
});
});
Related
In Mongoose, I can use a query populate to populate additional fields after a query. I can also populate multiple paths, such as
Person.find({})
.populate('books movie', 'title pages director')
.exec()
However, this would generate a lookup on book gathering the fields for title, pages and director - and also a lookup on movie gathering the fields for title, pages and director as well. What I want is to get title and pages from books only, and director from movie. I could do something like this:
Person.find({})
.populate('books', 'title pages')
.populate('movie', 'director')
.exec()
which gives me the expected result and queries.
But is there any way to have the behavior of the second snippet using a similar "single line" syntax like the first snippet? The reason for that, is that I want to programmatically determine the arguments for the populate function and feed it in. I cannot do that for multiple populate calls.
After looking into the sourcecode of mongoose, I solved this with:
var populateQuery = [{path:'books', select:'title pages'}, {path:'movie', select:'director'}];
Person.find({})
.populate(populateQuery)
.execPopulate()
you can also do something like below:
{path:'user',select:['key1','key2']}
You achieve that by simply passing object or array of objects to populate() method.
const query = [
{
path:'books',
select:'title pages'
},
{
path:'movie',
select:'director'
}
];
const result = await Person.find().populate(query).lean();
Consider that lean() method is optional, it just returns raw json rather than mongoose object and makes code execution a little bit faster! Don't forget to make your function (callback) async!
This is how it's done based on the Mongoose JS documentation http://mongoosejs.com/docs/populate.html
Let's say you have a BookCollection schema which contains users and books
In order to perform a query and get all the BookCollections with its related users and books you would do this
models.BookCollection
.find({})
.populate('user')
.populate('books')
.lean()
.exec(function (err, bookcollection) {
if (err) return console.error(err);
try {
mongoose.connection.close();
res.render('viewbookcollection', { content: bookcollection});
} catch (e) {
console.log("errror getting bookcollection"+e);
}
//Your Schema must include path
let createdData =Person.create(dataYouWant)
await createdData.populate([{path:'books', select:'title pages'},{path:'movie', select:'director'}])
Given the data structure below in firebase, i want to run a query to retrieve the blog 'efg'. I don't know the user id at this point.
{Users :
"1234567": {
name: 'Bob',
blogs: {
'abc':{..},
'zyx':{..}
}
},
"7654321": {
name: 'Frank',
blogs: {
'efg':{..},
'hij':{..}
}
}
}
The Firebase API only allows you to filter children one level deep (or with a known path) with its orderByChild and equalTo methods.
So without modifying/expanding your current data structure that just leaves the option to retrieve all data and filter it client-side:
var ref = firebase.database().ref('Users');
ref.once('value', function(snapshot) {
snapshot.forEach(function(userSnapshot) {
var blogs = userSnapshot.val().blogs;
var daBlog = blogs['efg'];
});
});
This is of course highly inefficient and won't scale when you have a non-trivial number of users/blogs.
So the common solution to that is to a so-called index to your tree that maps the key that you are looking for to the path where it resides:
{Blogs:
"abc": "1234567",
"zyx": "1234567",
"efg": "7654321",
"hij": "7654321"
}
Then you can quickly access the blog using:
var ref = firebase.database().ref();
ref.child('Blogs/efg').once('value', function(snapshot) {
var user = snapshot.val();
ref.child('Blogs/'+user+'/blogs').once('value', function(blogSnapshot) {
var daBlog = blogSnapshot.val();
});
});
You might also want to reconsider if you can restructure your data to better fit your use-case and Firebase's limitations. They have some good documentation on structuring your data, but the most important one for people new to NoSQL/hierarchical databases seems to be "avoid building nests".
Also see my answer on Firebase query if child of child contains a value for a good example. I'd also recommend reading about many-to-many relationships in Firebase, and this article on general NoSQL data modeling.
Given your current data structure you can retrieve the User that contains the blog post you are looking for.
const db = firebase.database()
const usersRef = db.ref('users')
const query = usersRef.orderByChild('blogs/efg').limitToLast(1)
query.once('value').then((ss) => {
console.log(ss.val()) //=> { '7654321': { blogs: {...}}}
})
You need to use limitToLast since Objects are sorted last when using orderByChild docs.
It's actually super easy - just use foreslash:
db.ref('Users').child("userid/name")
db.ref('Users').child("userid/blogs")
db.ref('Users').child("userid/blogs/abc")
No need of loops or anything more.
I am trying to figure out the most efficient way to find and populate documents some of which have references to another collection and some don't.
Example:
var collectionA={
{
"topic":"TopicA",
"text";"TextA")
},
{
"topic":"Topic1",
"text":"Text1"}
}
}
var collectionB={
{
"topic":"TopicB",
"text":"TextB"
"reference":["id","id"]
},
{
"topic":"Topic2",
"text":"Text2"
"reference":[]
}
}
and I have a request as follows which allows me to identify the documents I want:
req.body={"topic":["TopicA","TopicB"]}
What is the most efficient way to find the relevant documents and populate them to provide a fully populated result, in the fewest number of database calls as possible:
{"topic":"TopicA","text":"TextA"},
{"topic":"TopicB","text":"TextB","reference":[{populated document}
{populated document}]},
I am trying to use something like:
collections.find({"topic": $in req.body.top}, function(err,result){
collections.populate(result,{path:'references'}, function (err,result){
//do something
});
})
Is this on the right track?
Your help is greatly appreciated.
In order to reference other documents in mongoose you need to use Population.
If you want to reference a topic by id you would need to have something like the following in your schema: { type: Schema.Types.ObjectId, ref: 'collectionA' }
I have a tree-like schema that specifies a collection of parents, and a collection of children.
The collection of children will likely have millions of documents - each of which contains a small amount of data, and a reference to the parent that it belongs to which is stored as a string (perhaps my first mistake).
The collection of parents is much smaller, but may still be in the tens of thousands, and will slowly grow over time. Generally speaking though, a single parent may have as few as 10 children, or as many as 50,000 (possibly more, although somewhat unlikely).
A single child document might look something like this:
{
_id: ObjectId("507f191e810c19729de860ea"),
info: "Here's some info",
timestamp: 1234567890.0,
colour: "Orange",
sequence: 1000,
parent: "12a4567b909c7654d212e45f"
}
Its corresponding parent record (which lives in a separate collection) might look something like this:
{
_id: ObjectId("12a4567b909c7654d212e45f")
info: "Blah",
timestamp: 1234567890.0
}
My query in mongoose (which contains the parent ID in the request) looks like this:
/* GET all children with the specified parent ID */
module.exports.childrenFromParent = function(req, res) {
parentID = req.params.parentID;
childModel.find({
"parentid": parentID
}).sort({"sequence": "asc"}).exec(
function(err, children) {
if (!children) {
sendJSONResponse(res, 404, {
"message": "no children found"
});
return;
} else if (err) {
sendJSONResponse(res, 404, err);
return;
}
sendJSONResponse(res, 200, children);
}
);
};
So basically what's happening is that the query has to search the entire collection of children for any documents that have a parent which matches the provided ID.
I'm currently saving this parent ID as a string in the children collection schema (childModel in the code above), which is probably a bad idea, however, my API is providing the parent ID as a string in the request.
If anyone has any ideas as to how I can either fix my schema or change the query to improve the performance, it would be much appreciated!
Why are you not using .lean() before your exec? Do you really want all of your documents to be Mongoose documents or just simple JSON docs? With lean() you will not get all the extra getters and setters that come with Mongoose document. This could easily shave off at least a second or two from the response time.
Write up from the comments:
You could help speed up and optimize queries by adding an index on the parent field. You can add an (ascending) index by doing the following:
db.collection.createIndex( { parent: 1 } )
You can analyse the benefit of an index by adding .explain("executionStats") to a query. See the docs for more info.
Adding an index on a large collection may take time, you can check the status by running the following query:
db.currentOp(
{
$or: [
{ op: "query", "query.createIndexes": { $exists: true } },
{ op: "insert", ns: /\.system\.indexes\b/ }
]
}
)
Edit: If you are sorting by sequence, you might want to add a compound index for the parent and the sequence.
I am trying to generate a response that returns the same collection sorted by 3 different columns. Here's the code I currently have:
var findRoute = router.route("/find")
findRoute.get(function(req, res) {
Box.find(function(err, boxes) {
res.json(boxes)
}).sort("-itemCount");
});
As you can see, we're making a single get request, querying for the Boxes, and then sorting them by itemCount at the end. This does not work for me because the request only returns a single JSON collection that is sorted by itemCount.
What can I do if I want to return two more collections sorted by, say, name and size properties -- all in the same request?
Crete an object to encapsulate the information and chain your find queries, like:
var findRoute = router.route("/find");
var json = {};
findRoute.get(function(req, res) {
Box.find(function(err, boxes) {
json.boxes = boxes;
Collection2.find(function (error, coll2) {
json.coll2 = coll2;
Collection3.find(function (error, coll3) {
json.coll3 = coll3;
res.json(json);
}).sort("-size");
}).sort("-name");
}).sort("-itemCount");
});
Just make sure to do the appropriate error checking.
This is kind of uggly and makes your code kind of difficult to read. Try to adapt this logic using modules like async or even promises (Q and bluebird are good examples).
If I understand well, you want something like that : return Several collections with mongodb
Tell me if that helps.
Bye.
Have you tried ?
Box.find().sort("-itemCount").exec(function(err, boxes) {
res.json(boxes)
});
Also for sorting your results based on 2 or more fields you can use :
.sort({name: 1, size: -1})
Let me know if that helps.