Mongoose response showing cached collection - node.js

I'm running an Express app with Mongoose communicating with a Mongo DB. I have a simple page CMS to update values. On my staging environment when I update the page I can see the update reflected in my Mongo console, so I know the change is persisting to the DB. However, when I make a call to my API I see a cached response. If I restart Node I'll see the correct updated value.
Oddly enough I have other Mongoose models which seem to update fine. This also does not happen in my local development environment. Next steps to debug this would be very helpful as I can't track down where the issue would lie. I can only assume my staging environment Mongo DB has some sort of caching for this single collection and not the rest, is this a possibility?
Here's my model schema:
import mongoose, { Schema } from 'mongoose';
const HomePage = new Schema({
marquee: {
image: String,
label: String,
headline: String,
copy: String,
linkUrl: String,
linkText: String,
videoText: String,
videoUrl: String
},
updatedAt: {
type: Date,
default: Date.now
},
createdAt: {
type: Date
}
}, {
collection: 'homepage'
});
export default mongoose.model('HomePage', HomePage);
Additionally, in my API's response header, I have the cache set to: Cache-Control:max-age=0

I narrowed this down to the browser caching the response, which is odd as I've built other applications on this same platform without the issue. To resolve the issue I simply added a ? to my URL path. So my final get request is now as follows: http://myapp.com/api/homepage?

Related

why mongoose query works fine on localhost but not on prod server instead returns an empty []?

So i have a few days trying to debug my restful API, specifically mongoose and MongoDB on prod server (aws server). There is a service which query Users collection and returns the model, simple enough. But the question is this why would this query criteria return an empty [] on prod server but returns the model on localhost and using Postman for testing purposes.On prod server i will have to be explicit on what data i want from the document in order to return the model. Below i will leave my codebase such as the schema, services.
NOTE -> my prod server and local machine are both Ubuntu. Prod server (Ubuntu 18.04.3, codename Bionic) and local machine (Ubuntu 21.10, codename Impish).
Schema
userSchema: {
name: {
type:Schema.Types.String,
required: true
},
carsOwned: [{
name: {
type: Schema.Types.String,
},
brand: [{
name: {
type: Schema.Types.String,
},
}]
}],
totalNumberOfCarsOwned: {
type: Schema.Types.Number,
default: function () {
let carsCount = 0;
for (let car of this.carsOwned) {
carsCount += car.brand.length;
}
return carsCount
}
}
Sevice
userService.getCarById = (id) => {
// this is the original query which works perfectly locally but not on live server (aws)
return userModel.findOne({carsOwned: {$elemMatch: {_id: id}}})
// This is the updated query which works perfectly both locally and live server (aws)
return userModel.findOne({carsOwned: {$elemMatch: {_id: id} } }, {"carsOwned._id":1, "carsOwned.brand":1 });
// noticed how i exclude totalNumberOfCarsOwned in the updated query criteria. If i included the property totalNumberOfCarsOwned then it returns an empty array like user = []
};
Both versions of my MongoDB (5.0.8) and Mongoose(5.4.16) are the same locally and on prod server. I simply don't understand why others query works fine and this is the only service that has a tricky and weird temporary solution.
I just need an guide that points me to an answer that explain this issue, please.

How to insert Array of objects in mongoDB?

I am very new to MONGO DB so please bear with me.I am having a problem my array of objects is not working properly .
Here is my schema
const playerSchema = new mongoose.Schema({
name: String,
stats :{
wins:Number,
losses:Number,
xp:Number
},
achievement:[
{
name:String,
date: String
}
] });
Here is my document
const fluffy = new playerModel({
"name":"nic raboy",
"stats":{
"wins":5,
"losses":10,
"xp":300
},
"achievements":[
{"name":"Massive XP","date" :"25-08-21"},
{"name":"instant loss","date":"24-08-21"}
]
});
however in mongodb atlas its only showing array...and i cant see the objects inside...
SCREENSHOT
Your schema is correct, it seems your input is wrong,
In schema definition you named it achievement, whereas in input document it is achievements. Correct this everything will work as you expected.
Explanation
The schema is expecting achievement and you inserted achievements, that is why it is shown as an empty array in the database. To avoids this kind of typos in the future, use the required flag.
const playerSchema = new mongoose.Schema({
name: String,
stats: {
wins: Number,
losses: Number,
xp: Number
},
achievements: [
{
name: {
type: String,
required : true,
},
date: {
type: String,
required : true, // required informs for missing fields
}
}
]
})
Refer this link for more on validation
You can use insertMany see the doc here.
Of course, a while loop should work find calling multiple times insertOne, though I advise you to use the insertMany() method.
If you're new to MongoDB, I strongly encourage you to have a look at MongoDB University's MongoDB basics course as well as the MongoDB for JavaScript Developers course.

Mongoose with CosmosDB: Getting error `Shared throughput collection should have a partition key`

I have a node-express application that currently uses Mongoose to connect to MongoDB, and am attempting to migrate it to Azure Cosmos DB.
When I simply allow Mongoose to create the database, the application works fine, however the database is created with individual collection RU pricing.
If I create a new database with Shared throughput enabled and attempt to use that, I get the error Shared throughput collection should have a partition key
I have tried updating the collection schema to include a shard key like this:
const mongoose = require('mongoose');
module.exports = function() {
const types = mongoose.Schema.Types;
const messages = new mongoose.Schema({
order: { type: types.ObjectId, required: true, ref: 'orders' },
createdAt: { type: Date, default: Date.now },
sender: { type: types.ObjectId, required: true, ref: 'users' },
recipient: { type: types.ObjectId, ref: 'users' },
text: { type: String, required: true },
seen: { type: Boolean, default: false },
}, { shardKey: { order: 1 } });
return mongoose.model('messages', messages);
};
However this does not work.
Any ideas on how to create/use a partition key? Alternatively the database is small, so if its possible to remove the requirement for the partition key that would also be fine.
Now I don't have an exact answer for this question so no need to accept this unless you feel it's correct.
The best solution I've found so far is that this is due to "Provision Throughput" being checked when the database is created in the Azure console. If you delete and recreate the database with this box not checked (it's right below the input for the database name) then you should no longer encounter this error.
You specify it when you're creating a collection in the DB that you've opted in for Shared Throughput.
Collection vs Database
If you're using individual collection pricing, you can set the throughput on the individual collections. If you're using the lesser pricing option, you'd get the shared throughput (at the database level) which is less granular but less expensive.
Details here: https://azure.microsoft.com/en-us/blog/sharing-provisioned-throughput-across-multiple-containers-in-azure-cosmosdb/
Partition keys
If you're using the Shared throughput, you'll need a partition Id for the collection that you're adding.
So - create a DB with Shared throughput (check the checkbox below)
After that when you're attempting to add a new document you should be able to create a partition key.
I have yet another not-quite-complete answer for you. It seems like, yes, it is required to use partitioned collections if you are using the shared/db-level throughput model in Cosmos. But it turns out it is possible to create a CosmosDb collection with a partition key using only the MongoDb wire protocol (meaning no dependency on an Azure SDK, and no need to pre-create every collection via the Azure Portal).
The only remaining catch is, I don't think it's possible to run this command via Mongoose, it will probably have to be run directly via the MongoDb Node.js Driver, but at least it can still be run from code.
From a MongoDb Shell:
db.runCommand({shardCollection: "myDbName.nameOfCollectionToCreate",
key: {nameOfDesiredPartitionKey: "hashed"}})
This command is meant to set the sharding key for a collection and start sharding the collection, but in CosmosDb it works to create the collection with the desired partitionKey already set.
I have an even little more complete answer. You actually can do it with mongoose. I usually do it like this in an Azure Function:
mongoose.connect(process.env.COSMOSDB_CONNSTR, {
useUnifiedTopology: true,
useNewUrlParser: true,
auth: {
user: process.env.COSMODDB_USER,
password: process.env.COSMOSDB_PASSWORD,
},
})
.then(() => {
mongoose.connection.db.admin().command({
shardCollection: "mydb.mycollection",
key: { _id: "hashed" }
})
console.log('Connection to CosmosDB successful 🚀')
})
.catch((err) => console.error(err))

MongoDB private fields

I have a product model, it has many fields. Some of them are dedicated to front-end application, ex:
var GameSchema = new Schema({
likes: {
type: [{
type: Schema.ObjectId,
ref: 'User'
}]
},
likes_count: {
type: Number
}
});
I don't need likes_count field in Db, but controller returns only fields that model have, so i add likes_count field to db model
exports.some_method = function(req, res){
var game = req.game;
game.likes_count = game.likes.length
res.json(game);
}
Is there a way to add extra data to db model when sending request without having them in db?
Please note, problem is not in likes_count field itself, i have different models, but the point is having extra data on db model.
For those who still interested, mongo_db mongoose(#robertklep) has virtual fields, that can be used as temporary data field, that doesn't exist in database
GameSchema.virtual('likes_count').get(function () {
return this.likes.length;
});
And note, your model must have permission for virtuals like this, so that you can use it inside controllers
var UserSchema = new Schema({
username: {
type: String
}
}, {
toObject: { virtuals: true },
toJSON: { virtuals: true }
});
"Is there a way to add extra data to db model when sending request without having them in db?"
You may be able to do so from a driver's perspective and I'll leave that to those who know abut such things. Check out the following post Mapping a private backing field with MongoDB C#.
I can answer from the MongoDB engine & server processes aspect; if you are looking for a way to flag a field in the JSON document to make it private when sent to the actual CRUD request the MongoDB engine receives then no.
However, you could intercept the JSON prior to the actual CRUD request and transform it. The JSON you are generating is not inserted until you execute one of the INSERT, Modify, or Update statements. The pseudo steps would be to generate a JSON document, send it to a broker\wrapper etc in front of MongoDB, and then transform it by removing the fields in question, then send the new object as a CRUD request to the MongoDB engine.

Mongoose Changing Schema Format

We're rapidly developing an application that's using Mongoose, and our schema's are changing often. I can't seem to figure out the proper way to update a schema for existing documents, without blowing them away and completely re-recreating them from scratch.
I came across http://mongoosejs.com/docs/api.html#schema_Schema-add, which looks to be right. There's little to no documentation on how to actually implement this, making it very hard for someone who is new to MongoDB.
I simply want to add a new field called enabled. My schema definition is:
var sweepstakesSchema = new Schema({
client_id: {
type: Schema.Types.ObjectId,
ref: 'Client',
index: true
},
name: {
type: String,
default: 'Sweepstakes',
},
design: {
images: {
type: [],
default: []
},
elements: {
type: [],
default: []
}
},
enabled: {
type: Boolean,
default: false
},
schedule: {
start: {
type: Date,
default: Date.now
},
end: {
type: Date,
default: Date.now
}
},
submissions: {
type: Number,
default: 0
}
});
Considering your Mongoose model name as sweepstakesModel,
this code would add enabled field with boolean value false to all the pre-existing documents in your collection:
db.sweepstakesModel.find( { enabled : { $exists : false } } ).forEach(
function (doc) {
doc.enabled = false;
db.sweepstakesModel.save(doc);
}
)
There's nothing built into Mongoose regarding migrating existing documents to comply with a schema change. You need to do that in your own code, as needed. In a case like the new enabled field, it's probably cleanest to write your code so that it treats a missing enabled field as if it was set to false so you don't have to touch the existing docs.
As far as the schema change itself, you just update your Schema definition as you've shown, but changes like new fields with default values will only affect new documents going forward.
I was also searching for something like migrations, but didn't find it. As an alternative you could use defaults. If a key has a default and the key doesn't exist, it will use the default.
Mongoose Defaults
Default values are applied when the document skeleton is constructed. This means that if you create a new document (new MyModel) or if you find an existing document (MyModel.findById), both will have defaults provided that a certain key is missing.
I had the exact same issue, and found that using findOneAndUpdate() rather than calling save allowed us to update the schema file, without having to delete all the old documents first.
I can post a code snippet if requested.
You might use mongo shell to update the existing documents in a specific collection
db.SweeptakesModel.update({}, {$set: {"enabled": false}}, {upsert:false, multi:true})
I had a similar requirement of having to add to an existing schema when building an app with Node, and only found this (long ago posted) query to help.
The schema I added to by introducing the line in the original description of the schema and then running something similar to the following line, just the once, to update existing records:
myModelObject.updateMany( { enabled : { $exists : false } }, { enabled : false } )
'updateMany' being the function I wanted to mention here.
just addition to what Vickar was suggesting, here Mongoose Example written on Javascript (Nodejs):
const mongoose = require('mongoose');
const SweeptakesModel = mongoose.model(Constants.SWEEPTAKES,sweepstakesSchema);
SweeptakesModel.find( { enabled : { $exists : false } }).then(
function(doc){
doc.enabled = false;
doc.save();
}
)

Resources