I am learning my way around MongoDB data types and the best way to use documents and Schemas through Mongoose.
I've defined a couple of Schemas for a Navigation bar object which stores the navigation items as a list, and each item is defined by a schema with the properties name, type, url, and a list of drop downs if it has any (if the type is "dropdown").
Here are those Schemas:
var navSchema = new Schema({
id: String,
items: [Schema.ObjectId]
});
is the nav object, and
var navItemSchema = new Schema({
name: String,
type: {type: String, default: "link"},
url: {type: String, default: null},
dropdowns: {type: [Schema.ObjectId], default: null}
});
is the nav item schema, but each nav item can potentially have dropdowns, and so the dropdowns is a list of nav items, which can also potentially have dropdowns. But in this case, only a few do.
Now to create the data for these objects, I have to do something like this to create a nav item, example for "home"
var home = new navItem({
name: "home",
url: "/home"
});
but for items with dropdowns, I have to define all the items I know will be dropdowns before defining list which includes those items, and then defining the parent item and using the list with those items I just defined. Like this
var allAccessories = new navItem({
name: "all accessories",
url: "/accessories"
});
var cushions = new navItem({
name: "cushions",
url: "/accessories#cushions"
});
var cupHolders = new navItem({
name: "cup holders",
url: "/accessories#cupholders"
});
var accessoriesDropdownItems = [
allAccessories,
cushions,
cupHolders
];
var accessories = new navItem({
name: "accessories",
type: "dropdown",
dropdowns: accessoriesDropdownItems
});
So, I assume I'm doing that right..? My only gripe with this method is that in my nav.js file where I create this mongodb object/collection, I have to think about what items are going to be used in dropdowns of other items, and so I have to theoretically order them to be defined before the other variables are defined in the document.
But if I then want to use an item in two dropdown lists, and one of those dropdownlists I happened to have already defined above it in the document but now want to add to. I have to move the item definition above anywhere it's used, this ruins the organisation of the document..
But I'm okay with physically indenting to keep my work organised and sorted.
My question is how do I actually access properties of objects within lists of other objects.
Straight up I define Nav as simply an object with an id, "nav" because I don't want to use its _id ObjectId to reference it all the time...? And an items array which contains the navItemsSchema objects.
But when I try to reach into these objects through mongo shell using something like
db.navs.find({items: {$elemMatch: {name:"home"}}})
Or
db.navs.find({items: ObjectId("58d5d6df0789f718460ff278")})
Or
db.navs.find({items:{ "name" : "home"}})
I can't get any data back.. I have successfully manage to return all objects in the collection through the node app via responding found nav in navs
app.get('/nav', function(req, res) {
mongoose.model('nav').find(function(err, nav) {
res.send(nav);
});
});
But all this returns me is a data structure with object id's and not the actually data of the objects.
{"_id":"58d5d6df0789f718460ff287",
"id":"nav","__v":0,
"items":
["58d5d6df0789f718460ff278",
"58d5d6df0789f718460ff279",
"58d5d6df0789f718460ff286",
"58d5d6df0789f718460ff281",
"58d5d6df0789f718460ff282",
"58d5d6df0789f718460ff283",
"58d5d6df0789f718460ff284"
]
}
Could you please help me understand how I reach into these data hierarchies via say db.navs.nav("nav").items.findAll() and it lists all the items and their json?
Or is this not possible and you need to loop through all items, then. Wait, where are the objects stored corresponding to ObjectId's in the items list?
So I actually figured it out myself. I was using type: [Schema.ObjectId] (ie. type was a list of Schema.ObjectIds) but I didn't know that that automatically parses the inputted list of objects and extracts just their ObjectId's and stores them, all I need to do was make the type: [] (ie. just a list) and then I can traverse my nav object simply by... Oh wait, for some reason when mongoose nav.saves my nav it becomes an object inside a list. So when I mongoose.model('nav').find(function(err, nav) {} it gives me a nav object which I need to use via nav[0].items[0].name Or I can go nav = nav[0] but I wish I didn't need to do this step? Maybe there's an answer but yes. Otherwise. This is the solution I was looking for.
Related
I am designing an item inventory system for a website that I am building.
The user's inventory is loaded from a Web API. This information is then processed so that it is more suited to my web app. I am trying to combine all the item records into one MongoDB collection - so other user inventories will be cached in the same place. What I have to deal with is deleting old item records if they are missing from the user's inventory (i.e. they sold it to someone) and also upserting the new items. Please note I have looked through several Stack Overflow questions about bulk upserts but I was unable to find anything about conditional updates.
Each item has two unique identifiers (classId and instanceId) that allow me to look them up (I have to use both IDs to match it) which remain constant. Some information about the item, such as its name, can change and therefore I want to be able to update those records when I fetch new inventory information. I also want new items that my site hasn't seen before to be added to my database.
Once the data returned from the Web API has been processed, it is left in a large array of objects. This means I am able to use bulk writing, however, I am unaware of how to upsert with conditions with multiple records.
Here is part of my item schema:
const ItemSchema = new mongoose.Schema({
ownerId: {
type: String,
required: true
},
classId: {
type: String,
required: true
},
instanceId: {
type: String,
required: true
},
name: {
type: String,
required: true
}
// rest of item attributes...
});
User inventories typically contain 600 or more items, with a max count of 2500.
What is the most efficient way of upserting this much data? Thank you
Update:
I have had trouble implementing the solution to the bulk insert problem. I made a few assumptions and I don't know if they were right. I interpreted _ as lodash, response.body as the JSON returned by the API and myListOfItems also as that same array of items.
import Item from "../models/item.model";
import _ from 'lodash';
async function storeInventory(items) {
let bulkUpdate = Item.collection.initializeUnorderedBulkOp();
_.forEach(items, (data) => {
if (data !== null) {
let newItem = new Item(data);
bulkUpdate.find({
classId: newItem.classId,
instanceId: newItem.instanceId
}).upsert().updateOne(newItem);
items.push(newItem);
}
});
await bulkUpdate.execute();
}
Whenever I run this code, it throws an error that complains about an _id field being changed, when the schema objects I created don't specify anything to do with schemas, and the few nested schema objects don't make a difference to the outcome when I change them to just objects.
I understand that if no _id is sent to MongoDB it auto generates one, but if it is updating a record it wouldn't do that anyway. I also tried setting _id to null on each item but to no avail.
Have I misunderstood anything about the accepted answer? Or is my problem elsewhere in my code?
This is how I do it :
let bulkUpdate = MyModel.collection.initializeUnorderedBulkOp();
//myItems is your array of items
_.forEach(myItems, (item) => {
if (item !== null) {
let newItem = new MyModel(item);
bulkUpdate.find({ yyy: newItem.yyy }).upsert().updateOne(newItem);
}
});
await bulkUpdate.execute();
I think the code is pretty readable and understandable. You can adjust it to make it work with your case :)
Let's say I have for example:
const Stats = Item({
name: String,
value: Number
})
const Player = Schema({
name: String,
objectInventory: [Item],
petInventory: [Item]
})
Would the items somehow get mixed up? Is this safe? Are all the items unique and know where they belong to? I don't want to write Player.objectInventory and get pets in there. I'm sorry if this seems like common sense but I had that doubt.
Yes there can be two documents in one schema. This items will not get mixed up. The mongoose is nothing more than just another layer on top of the database to help you with schema. So in your case, you would just put different ids for different properties (e.g. objectInventory and petInventory) and when you would populate them, the mongoose will just make correct queries to return the results.
So I've decided to use graphql as my query engine along side with mongodb. So I created my schemas and everything looks great, BUT, one of my schemas contains a list of Strings, for instance:
exports.default = new gql.GraphQLInputObjectType({
name: 'myModel',
fields: {
type: { type: gql.GraphQLString },
workingDays: { type: new gql.GraphQLList(GraphQLString) }
}
});
So in the workingDays list I have 50 elements, and I'd like to change one of them, is there a way to do that with Graphql?
It just so happens to be a string type inside but, it could be an object as well.
Thanks.
You can add a new mutation that encodes this functionality.
For example updateWorkingDays(modelId: ID!, index: Int!, workDay: String) that updates the working day of model modelId at index to the new workDay.
Here is my Mongoose Schema:
var SchemaA = new Schema({
field1: String,
.......
fieldB : { type: Schema.Types.ObjectId, ref: 'SchemaB' }
});
var SchemaB = new Schema({
field1: String,
.......
fieldC : { type: Schema.Types.ObjectId, ref: 'SchemaC' }
});
var SchemaC = new Schema({
field1: String,
.......
.......
.......
});
While i access schemaA using find query, i want to have fields/property
of SchemaA along with SchemaB and SchemaC in the same way as we apply join operation in SQL database.
This is my approach:
SchemaA.find({})
.populate('fieldB')
.exec(function (err, result){
SchemaB.populate(result.fieldC,{path:'fieldB'},function(err, result){
.............................
});
});
The above code is working perfectly, but the problem is:
I want to have information/properties/fields of SchemaC through SchemaA, and i don't want to populate fields/properties of SchemaB.
The reason for not wanting to get the properties of SchemaB is, extra population will slows the query unnecessary.
Long story short:
I want to populate SchemaC through SchemaA without populating SchemaB.
Can you please suggest any way/approach?
As an avid mongodb fan, I suggest you use a relational database for highly relational data - that's what it's built for. You are losing all the benefits of mongodb when you have to perform 3+ queries to get a single object.
Buuuuuut, I know that comment will fall on deaf ears. Your best bet is to be as conscious as you can about performance. Your first step is to limit the fields to the minimum required. This is just good practice even with basic queries and any database engine - only get the fields you need (eg. SELECT * FROM === bad... just stop doing it!). You can also try doing lean queries to help save a lot of post-processing work mongoose does with the data. I didn't test this, but it should work...
SchemaA.find({}, 'field1 fieldB', { lean: true })
.populate({
name: 'fieldB',
select: 'fieldC',
options: { lean: true }
}).exec(function (err, result) {
// not sure how you are populating "result" in your example, as it should be an array,
// but you said your code works... so I'll let you figure out what goes here.
});
Also, a very "mongo" way of doing what you want is to save a reference in SchemaC back to SchemaA. When I say "mongo" way of doing it, you have to break away from your years of thinking about relational data queries. Do whatever it takes to perform fewer queries on the database, even if it requires two-way references and/or data duplication.
For example, if I had a Book schema and Author schema, I would likely save the authors first and last name in the Books collection, along with an _id reference to the full profile in the Authors collection. That way I can load my Books in a single query, still display the author's name, and then generate a hyperlink to the author's profile: /author/{_id}. This is known as "data denormalization", and it has been known to give people heartburn. I try and use it on data that doesn't change very often - like people's names. In the occasion that a name does change, it's trivial to write a function to update all the names in multiple places.
SchemaA.find({})
.populate({
path: "fieldB",
populate:{path:"fieldC"}
}).exec(function (err, result) {
//this is how you can get all key value pair of SchemaA, SchemaB and SchemaC
//example: result.fieldB.fieldC._id(key of SchemaC)
});
why not add a ref to SchemaC on SchemaA? there will be no way to bridge to SchemaC from SchemaA if there is no SchemaB the way you currently have it unless you populate SchemaB with no other data than a ref to SchemaC
As explained in the docs under Field Selection, you can restrict what fields are returned.
.populate('fieldB') becomes populate('fieldB', 'fieldC -_id'). The -_id is required to omit the _id field just like when using select().
I think this is not possible.Because,when a document in A referring a document in B and that document is referring another document in C, how can document in A know which document to refer from C without any help from B.
I am trying to create an array of nested objects. I am following an example from a book that does the following:
// Creates the Schema for the Features object (mimics ESRI)
var Phone = new Schema({
number: { type: Number, required: false },
...
personId: {type: Schema.Types.ObjectId}
}
);
// Creates the Schema for the Attachments object
var Person = new Schema({
name: { type: String },
phones: [Phone]
}
);
var Person = mongoose.model('Person', Person);
Which works just fine when storing multiple Phone #'s for a person. However I am not sure if there is a good/fast way to get a Phone object by _id. Since Phone is not a mongoose model you cannot go directly to Phone.findOne({...}); Right now I am stuck with getting a person by _id then looping over that persons phones and seeing if the id matches.
Then I stumbled upon this link:
http://mongoosejs.com/docs/populate.html
Is one way more right than the other? Currently when I delete a person his/her phones go away as well. Not really sure that works with 'populate', seems like I would need to delete Person and Phones.
Anyone want to attempt to explain the differences?
Thanks in advance
The general rule is that if you need to independently query Phones, then you should keep them in a separate collection and use populate to look them up from People when needed. Otherwise, embedding them is typically a better choice as it simplifies updates and deletion.
When using an embedded approach like you are now, note that Mongoose arrays provide an id method you can use to more easily look up an element by its _id value.
var phone = person.phones.id(id);