Related
I'm doing what feels like a simple query, and this is a two part question. The first is whether there's a better way to do it, given my data structure, and the second is, if not, why am I seeing this behavior in my current implementation?
I am building a quizzing app - Node.js backend with MongoDB. I have a "questions" collection - here's a sample document from that:
{
"_id": ObjectId("62742bb0dbf1a0577864bbeb"),
"category": "World History",
"results": [],
"text": "**What volcano** erupted in 79AD, burying the cities of Pompeii and Herculaneum?",
"answer": "Mt. Vesuvius",
"owner": ObjectId("605271517fce7249cc8eb436"),
"__v": 0
}
I have a "quizzes" collection, where documents can have an array of questions from the questions collection:
{
"_id":ObjectId("62800d4d1d7c821198121193"),
"type": "std",
"questions": [{
"id": ObjectId("627034e92dda695b884a632e"),
"value": 2
},
{
"id": ObjectId("627160ec79517a79180f1e41"),
"value": 2
},
{
"id": ObjectId("62742bb0dbf1a0577864bbeb"),
"value": 4,
}
],
"title": "test standard quiz",
"description": "this is a test standard quiz\n\nthis is a second line.",
"owner": ObjectId("605271517fce7249cc8eb436"),
"__v": 0
}
At least in theory, this should allow users to create questions, and add them to as many quizzes as they want with different point values, if they wish. Doing an aggregation led to some trouble, as the ObjectIDs for the questions is embedded in an object, since they need to have a point value tied to them as well. Here's my kludgy way of joining the two collections:
const getQuizDetails = async ()=> {
const Quiz = require('./models/quizModel');
const Question = require('./models/questionsModel');
const owner = '605271517fce7249cc8eb436'; //just for testing, I'm hard-coding the owner's ID here
const results = await Quiz.find({
owner,
type: 'std',
});
console.log(results); //This gives an array with the document(s) as expected.
const vals = results.map(r=> {
return r.questions.map(q=> {
return q.value;
}
}
console.log(vals); //This gives an array of arrays, containing the values of each question in each of the results above (e.g. [[2,2,4]] if we just returned the sample document above)
let qs;
for (var i = 0; i < results.length; i++) {
qs = await Promise.all(
results[i].questions.map((q) => {
return Question.findById(q.id);
})
);
q2 = qs.map((q, j) => {
return {
...q,
value: vals[i][j],
};
});
console.log(qs); //This gives an array of questions, with the text and answer of each one, which is what I want.
console.log(q2); //This gives something unexpected.
}
}
Above, when I log the variable qs, I get an array with the question details, with the question text, category, and answer:
{
category: 'Pop Music',
results: [],
_id: 627160ec79517a79180f1e41,
text: `Dua Lipa's 2021 hit "Cold Heart" features **what other artist**? His song "Sacrifice" provides both the title and many of the lyrics of "Cold Heart".`,
answer: 'Elton John',
owner: 605271517fce7249cc8eb436,
__v: 0
},
{
category: 'World History',
results: [],
_id: 62742bb0dbf1a0577864bbeb,
text: '**What volcano** erupted in 79AD, burying the cities of Pompeii and Herculaneum?',
answer: 'Mt. Vesuvius',
owner: 605271517fce7249cc8eb436,
__v: 0
},
{
category: 'Art',
results: [],
_id: 627435e1bd9f6b4bcc057761,
text: 'Who painted the Mona Lisa?',
answer: 'Leonardo da Vinci',
owner: 605271517fce7249cc8eb436,
__v: 0
},
I would expect, since I have an array of objects (and not promises anymore...are they?), that mapping it with another array (of numbers) would be straightforward. However, this is not the case. When I log q2, I get an array of objects similar to this:
{
'$__': InternalCache {
strictMode: true,
selected: {},
shardval: undefined,
saveError: undefined,
validationError: undefined,
adhocPaths: undefined,
removing: undefined,
inserting: undefined,
saving: undefined,
version: undefined,
getters: {},
_id: 627ad0c89041f24a60469288,
populate: undefined,
populated: undefined,
wasPopulated: false,
scope: undefined,
activePaths: [StateMachine],
pathsToScopes: {},
cachedRequired: {},
session: undefined,
'$setCalled': Set(0) {},
ownerDocument: undefined,
fullPath: undefined,
emitter: [EventEmitter],
'$options': [Object]
},
isNew: false,
errors: undefined,
'$locals': {},
'$op': null,
_doc: {
category: 'Geography',
results: [],
_id: 627ad0c89041f24a60469288,
text: 'What is the capital of Georgia?',
answer: 'Tblisi',
owner: 605271517fce7249cc8eb436,
__v: 0
},
'$init': true,
value: 10
}
All of the information that I need is there (the question information plus the point value), but I have a lot of other stuff as well - if I were to guess, it's the structure of a promise or something, but there shouldn't be any unresolved promises here - I've awaited them all, and they logged to the console as resolved objects. What's going on here?
I am trying to connect my node with my mongodb.i have no issue in that. Here i am getting my value as well as model in output, but I only like to have my name and tag in terminal, so is there a way.
i am doing it in window OS. so is it an window setting issue. so please let me know what is the way to get tags and name.
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/playground')
.then(()=>console.log('Connected to MongoDB...'))
.catch(err =>console.error('Could not connect to MongoDB...',err));
const courseSchema = new mongoose.Schema({
name : String,
author : String,
tags : [String],
date : {type : Date, default : Date.now},
isPublished : Boolean
});
const Course = mongoose.model('Course',courseSchema);
async function createCourse(){
const course = new Course({
name : 'Angular.js Course',
author: 'Mosh',
tags : ['Angular','Frontend'],
isPublished : true
});
const result = await course.save();
// console.log(result);
}
async function getCourses(){
const courses = await Course
.find({ author: 'Kunal', isPublished : true})
.limit(10)
.sort({name : 1})
.select({name : 1, tags : 1});
console.log(courses);
}
getCourses();
output
Connected to MongoDB...
[
model {
'$__': InternalCache {
strictMode: true,
selected: [Object],
shardval: undefined,
saveError: undefined,
validationError: undefined,
adhocPaths: undefined,
removing: undefined,
inserting: undefined,
version: undefined,
getters: {},
_id: 5fc6f9ba8059e13514a7de87,
populate: undefined,
populated: undefined,
wasPopulated: false,
scope: undefined,
activePaths: [StateMachine],
pathsToScopes: {},
ownerDocument: undefined,
fullPath: undefined,
emitter: [EventEmitter],
'$options': true
},
isNew: false,
errors: undefined,
_doc: {
tags: [Array],
_id: 5fc6f9ba8059e13514a7de87,
name: 'node.js Course'
},
'$init': true
}
]
i think you need to add .lean() to achieve what you want :
async function getCourses(){
const courses = await Course
.find({ author: 'Kunal', isPublished : true})
.limit(10)
.sort({name : 1})
.select({name : 1, tags : 1}).lean();
console.log(courses);
}
The lean() function tells mongoose to not hydrate query results. In
other words, the results of your queries will be the same plain
JavaScript objects that you would get from using the Node. js MongoDB
driver directly, with none of the mongoose magic.
I can't get this very simple virtual column to work (surnameName). It is not returned in query results, but it does not throw any error either.
My model (I removed irrelevant fields):
const Person = connectionPool.define('person', {
ID: {
type: Sequelize.INTEGER,
autoIncrement: true,
allowNull: false,
primaryKey: true
},
name: Sequelize.STRING,
surname: Sequelize.STRING,
surnameName: {
type: Sequelize.VIRTUAL(Sequelize.STRING, ['surname', 'name']),
get() {
return this.getDataValue('surname') + ' ' + this.getDataValue('name');
}
}
});
This is how I query the model:
const cfg = {
where: {},
limit: 10,
raw: false, // tried with and without this line
attributes: ['surnameName']
}
models.Person.findAll(cfg)
.then(results => {
console.log(results[0]);
})
And this is what I get in the console log:
person {
dataValues: { surname: 'Baggins', name: 'Frodo' }, // all other fields are autoexcluded by Sequelize
...
_options:
{ isNewRecord: false,
_schema: null,
_schemaDelimiter: '',
raw: true, // is true even if I set 'raw' to false in findAll options
attributes: [ 'surnameName', 'surname', 'name' ] // <= surnameName is there!
}
}
Virtual column is not returned in the results, however the logged instance shows that the internal _options.attributes array does contain the field, so Sequelize somehow acknowledges that it should be added. I tried explicitly turning raw=false, as I read that raw excludes virtual columns, but it has no effect. The results are definitely not raw.
What can be wrong here? Any help will be appreciated!
It is possible to hide properties javascript object. Here is an example
function Person(fName, lName) {
this.fName = fName;
this.lName = lName;
Object.defineProperties(this, {
fullName: {
get : function () {
return this.fName + " " + this.lName;
}
}
});
}
const ratul = new Person("Ratul", "sharker");
console.log(ratul);
console.log(ratul.fullName);
Look closely that console.log(ratul) does not print fullName, but fullName is sitting here, returning it's value seen in console.log(ratul.fullName).
Similar thing can be found in this answer.
I'm writing alexa skill and would like to check if user exists in MongoDB. My code works but I don't know how to define situation if user is already in a database :(
Everytime when I execute code I get:
"Hello Anna you are new here"
My user Anna is saved in MongoDB
But I would like to distinguish when my user is already in a database and react for that.
Does anybody smart has a solution for my problem?
var myName = "Anan1";
var userID = this.event.session.user.userId;
console.log(userID);
self = this;
User.findOneAndUpdate(
{userId: userID},
{$set:{name:myName}},
{upsert: true, new: false, runValidators: true},
function(err, doc){
if(err){
console.log("eeoror");
}
console.log(doc);
if (doc==null){
self.emit(':ask',
"Hello "+ myName +"you are new here")
}else {
self.emit(':ask',
"Hello "+ myName +"you are not new here")
}
});
It sounds like what you really want is a unique key constraint and not an upsert.
The unique key can be set in [mongoose] with either the schema field options:
const s = new Schema({ name: { type: String, unique: true }});
or by the index method:
Schema.path('name').index({ unique: true });
If an attempt is made to create a document that already has an entry for that key then an error will be thrown:
NOTE: violating the constraint returns an E11000 error from MongoDB when saving, not a Mongoose validation error.
As noted in comment earlier, you have two basic approaches to work out whether something was "created" or not. These are either to:
Return the rawResult in the response and check the updatedExisting property which tells you if it's an "upsert" or not
Set new: false so that "no document" is actually returned in result when it's actually an "upsert"
As a listing to demonstrate:
const { Schema } = mongoose = require('mongoose');
const uri = 'mongodb://localhost/thereornot';
mongoose.set('debug', true);
mongoose.Promise = global.Promise;
const userSchema = new Schema({
username: { type: String, unique: true }, // Just to prove a point really
password: String
});
const User = mongoose.model('User', userSchema);
const log = data => console.log(JSON.stringify(data, undefined, 2));
(async function() {
try {
const conn = await mongoose.connect(uri);
await Promise.all(Object.entries(conn.models).map(([k,m]) => m.remove()));
// Shows updatedExisting as false - Therefore "created"
let bill1 = await User.findOneAndUpdate(
{ username: 'Bill' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: true, rawResult: true }
);
log(bill1);
// Shows updatedExisting as true - Therefore "existing"
let bill2 = await User.findOneAndUpdate(
{ username: 'Bill' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: true, rawResult: true }
);
log(bill2);
// Test with something like:
// if ( bill2.lastErrorObject.updatedExisting ) throw new Error("already there");
// Return will be null on "created"
let ted1 = await User.findOneAndUpdate(
{ username: 'Ted' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: false }
);
log(ted1);
// Return will be an object where "existing" and found
let ted2 = await User.findOneAndUpdate(
{ username: 'Ted' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: false }
);
log(ted2);
// Test with something like:
// if (ted2 !== null) throw new Error("already there");
// Demonstrating "why" we reserve the "Duplicate" error
let fred1 = await User.findOneAndUpdate(
{ username: 'Fred', password: 'password' },
{ $setOnInsert: { } },
{ upsert: true, new: false }
);
log(fred1); // null - so okay
let fred2 = await User.findOneAndUpdate(
{ username: 'Fred', password: 'badpassword' }, // <-- dup key for wrong password
{ $setOnInsert: { } },
{ upsert: true, new: false }
);
mongoose.disconnect();
} catch(e) {
console.error(e)
} finally {
process.exit()
}
})()
And the output:
Mongoose: users.remove({}, {})
Mongoose: users.findAndModify({ username: 'Bill' }, [], { '$setOnInsert': { password: 'password', __v: 0 } }, { upsert: true, new: true, rawResult: true, remove: false, fields: {} })
{
"lastErrorObject": {
"n": 1,
"updatedExisting": false,
"upserted": "5adfc8696878cfc4992e7634"
},
"value": {
"_id": "5adfc8696878cfc4992e7634",
"username": "Bill",
"__v": 0,
"password": "password"
},
"ok": 1,
"operationTime": "6548172736517111811",
"$clusterTime": {
"clusterTime": "6548172736517111811",
"signature": {
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
"keyId": 0
}
}
}
Mongoose: users.findAndModify({ username: 'Bill' }, [], { '$setOnInsert': { password: 'password', __v: 0 } }, { upsert: true, new: true, rawResult: true, remove: false, fields: {} })
{
"lastErrorObject": {
"n": 1,
"updatedExisting": true
},
"value": {
"_id": "5adfc8696878cfc4992e7634",
"username": "Bill",
"__v": 0,
"password": "password"
},
"ok": 1,
"operationTime": "6548172736517111811",
"$clusterTime": {
"clusterTime": "6548172736517111811",
"signature": {
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
"keyId": 0
}
}
}
Mongoose: users.findAndModify({ username: 'Ted' }, [], { '$setOnInsert': { password: 'password', __v: 0 } }, { upsert: true, new: false, remove: false, fields: {} })
null
Mongoose: users.findAndModify({ username: 'Ted' }, [], { '$setOnInsert': { password: 'password', __v: 0 } }, { upsert: true, new: false, remove: false, fields: {} })
{
"_id": "5adfc8696878cfc4992e7639",
"username": "Ted",
"__v": 0,
"password": "password"
}
So the first case actually considers this code:
User.findOneAndUpdate(
{ username: 'Bill' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: true, rawResult: true }
)
Most options are standard here as "all" "upsert" actions will result in the field content being used to "match" ( i.e the username ) is "always" created in the new document, so you don't need to $set that field. In order to not actually "modify" other fields on subsequent requests you can use $setOnInsert, which only adds these properties during an "upsert" action where no match is found.
Here the standard new: true is used to return the "modified" document from the action, but the difference is in the rawResult as is shown in the returned response:
{
"lastErrorObject": {
"n": 1,
"updatedExisting": false,
"upserted": "5adfc8696878cfc4992e7634"
},
"value": {
"_id": "5adfc8696878cfc4992e7634",
"username": "Bill",
"__v": 0,
"password": "password"
},
"ok": 1,
"operationTime": "6548172736517111811",
"$clusterTime": {
"clusterTime": "6548172736517111811",
"signature": {
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
"keyId": 0
}
}
}
Instead of a "mongoose document" you get the actual "raw" response from the driver. The actual document content is under the "value" property, but it's the "lastErrorObject" we are interested in.
Here we see the property updatedExisting: false. This indicates that "no match" was actually found, thus a new document was "created". So you can use this to determine that creation actually happened.
When you issue the same query options again, the result will be different:
{
"lastErrorObject": {
"n": 1,
"updatedExisting": true // <--- Now I'm true
},
"value": {
"_id": "5adfc8696878cfc4992e7634",
"username": "Bill",
"__v": 0,
"password": "password"
},
"ok": 1,
"operationTime": "6548172736517111811",
"$clusterTime": {
"clusterTime": "6548172736517111811",
"signature": {
"hash": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
"keyId": 0
}
}
}
The updatedExisting value is now true, and this is because there already was a document that matched the username: 'Bill' in the query statement. This tells you the document was already there, so you can then branch your logic to return an "Error" or whatever response you want.
In the other case, it may be desirable to "not" return the "raw" response and use a returned "mongoose document" instead. In this case we vary the value to be new: false without the rawResult option.
User.findOneAndUpdate(
{ username: 'Ted' },
{ $setOnInsert: { password: 'password' } },
{ upsert: true, new: false }
)
Most of the same things apply except that now the action is the original state of the document is returned as opposed to the "modified" state of the document "after" the action. Therefore when there is no document that actually matches the "query" statement, the returned result is null:
Mongoose: users.findAndModify({ username: 'Ted' }, [], { '$setOnInsert': { password: 'password', __v: 0 } }, { upsert: true, new: false, remove: false, fields: {} })
null // <-- Got null in response :(
This tells you the document was "created", and it's arguable that you already know what the content of the document should be since you sent that data with the statement ( ideally in the $setOnInsert ). Point being, you already know what to return "should" you require to actually return the document content.
By contrast, a "found" document returns the "original state" showing the document "before" it was modified:
{
"_id": "5adfc8696878cfc4992e7639",
"username": "Ted",
"__v": 0,
"password": "password"
}
Therefore any response which is "not null" is therefore an indication that the document was already present, and again you can branch your logic depending on what was actually received in response.
So those are the two basic approaches to what you are asking, and they most certainly "do work"! And just as is demonstrated and reproducible with the same statements here.
Addendum - Reserve Duplicate Key for bad passwords
There is one more valid approach that is hinted at in the full listing as well, which is essentially to simply .insert() ( or .create() from mongoose models ) new data and have a "duplicate key" error throw where the "unique" property by index is actually encountered. It's a valid approach but there is one particular use case in "user validation" which is a handy piece of logic handling, and that is "validating passwords".
So it's a pretty common pattern to retrieve user information by the username and password combination. In the case of an "upsert" this combination justifies as "unique" and therefore an "insert" is attempted if no match is found. This is exactly what makes matching the password a useful implementation here.
Consider the following:
// Demonstrating "why" we reserve the "Duplicate" error
let fred1 = await User.findOneAndUpdate(
{ username: 'Fred', password: 'password' },
{ $setOnInsert: { } },
{ upsert: true, new: false }
);
log(fred1); // null - so okay
let fred2 = await User.findOneAndUpdate(
{ username: 'Fred', password: 'badpassword' }, // <-- dup key for wrong password
{ $setOnInsert: { } },
{ upsert: true, new: false }
);
On the first attempt we don't actually have a username for "Fred", so the "upsert" would occur and all the other things as already described above happen to identify whether it was a creation or a found document.
The statement that follows uses the same username value but provides a different password to what is recorded. Here MongoDB attempts to "create" the new document since it did not match on the combination, but because the username is expected to be "unique" you receive a "Duplicate key error":
{ MongoError: E11000 duplicate key error collection: thereornot.users index: username_1 dup key: { : "Fred" }
So what you should realize is you now get three conditions to evaluate for "free". Being:
The "upsert" was recorded by either the updatedExisting: false or null result depending on the method.
You know the document ( by combination ) "exists" via either the updatedExisting: true or where the document returns was "not null".
If the password provided was not a match for what already existed for the username, then you would get the "duplicate key error" which you can trap and respond accordingly, advising the user in response that the "password is incorrect".
All of that from one request.
That's the main reasoning for using "upserts" as opposed to simply throwing inserts at a collection, as you can get different branching of the logic without making additional requests to the database to determine "which" of those conditions should be the actual response.
MessageThread.findById(req.body._id)
.populate({ path: "messages" })
.exec((err, foundMessageThread) => {
var filtered = foundMessageThread.messages.map(message=>{
return Object.assign({}, message, {isRead: true});
})
console.log("filtered", filtered);
});
console.log shows:
{ '$__':
InternalCache {
strictMode: true,
selected: {},
shardval: undefined,
saveError: undefined,
validationError: undefined,
adhocPaths: undefined,
removing: undefined,
inserting: undefined,
version: undefined,
getters: {},
_id: 5a4c7f2d8b49fc260c396f55,
populate: undefined,
populated: undefined,
wasPopulated: true,
scope: undefined,
activePaths: [Object],
pathsToScopes: {},
ownerDocument: undefined,
fullPath: undefined,
emitter: [Object],
'$options': true },
isNew: false,
errors: undefined,
_doc:
{ sentAt: 2018-01-03T06:58:53.188Z,
isRead: false,
_id: 5a4c7f2d8b49fc260c396f55,
sender: 5a4b77767251b44cd870219f,
reciever: 5a4b780a7251b44cd87021a1,
text: 'qwe',
__v: 0 },
'$init': true,
isRead: true },
......
it repeats many times.
I suppose it (InternalCache { strictMode: true...) relates to message that is taken from foundMessageThread. And it reveals its metadata(in my term) while assigning. Because:
MessageThread.findById(req.body._id)
.populate({ path: "messages" })
.exec((err, foundMessageThread) => {
var filtered = foundMessageThread.messages.map(message=>{
console.log("message", message)
return Object.assign({}, message, {isRead: true});
})
console.log("filtered", filtered);
});
console.log shows
{ sentAt: 2018-01-03T06:58:53.188Z,
isRead: false,
_id: 5a4c7f2d8b49fc260c396f55,
sender: 5a4b77767251b44cd870219f,
reciever: 5a4b780a7251b44cd87021a1,
text: 'qwe',
__v: 0 },
....
My question:
Is it normal behavior?
If it is how to fix it? Because "metadata" prevents objects being assigned.
P.S. I've tried:
MessageThread.findById(req.body._id)
.populate({ path: "messages" })
.exec((err, foundMessageThread) => {
var filtered = foundMessageThread.messages.map(message=>{
return **Object.assign({}, message._doc, {isRead: true})**;
})
console.log("filtered", filtered);
});
This is a normal behaviour with mongoose. Objects returned by mongoose wrap the actual data, so as to add behaviours (methods) to it.
You can get to the actual data object by using toObject method, for eg, message.toObject().
However there are properties like __v, which are used by mongoose for house keeping purposes. If you don't want them, you can modify the toObject method like this
messageSchema.set('toObject', {
versionKey: false,
transform: (doc, ret) => {
delete ret.__v;
return ret;
},
});
You can also use .lean() method with mongoose request. This allows to get less cumbersome response and process it easyly:
try {
const assets = await myModel.Assets
.find({ isActive: true }, { __v: 0, _id: 0 })
.lean()
.exec()
// do something
}
catch(error) {
throw error
}
It appears that the _doc property of mongoose objects should be referenced if you want to either assign to or from those objects. In fact, I was unable to set additional properties in a normal manner like so mongoDoc.newProp = propValue;. It was not altering the original object.
For assignment, instead of:
Object.assign(mongoDoc, {a: 1, b: 2});
You'd want:
Object.assign(mongoDoc._doc, {a: 1, b: 2});
Or if you're assigning the mongoDoc properties to another object, you'd want
Object.assign({a: 1, b: 2}, mongoDoc._doc);
// add .lean() in your query like below
const documents = await DocumentModel.find().lean();
// at debug now ,
// now you will get your result;