I'm using MongoDB and Mongoose in a REST API. Some deployments require a replica set, thus separate read/write databases, so as a result I have separate read/write connections in the API. However, more simple deployments don't need a replica-set, and in those cases I point my read/write connections to the same MongoDB instance and database.
My general approach is to create all models for both connections at API start up. Even when read/write conns are connecting to same database, I am able to create the same models on both connections without error.
let ReadUser = dbRead.model('User', userSchema);
let WriteUser = dbWrite.model('User', userSchema);
// no error even when dbRead and dbWrite point to same DB
Trouble comes when until I start using Mongoose Discriminators.
let ReadSpecialUser = ReadUser.discriminator('SpecialUser', specialUserSchema);
let WriteSpecialUser = WriteUser.discriminator('SpecialUser', specialUserSchema);
// Results in this Error when read and write point to same DB:
// Error: Discriminator with name "SpecialUser" already exists
I'm look for an elegant way to deal with this. Is there a way to query the db for discriminators that are already in use?
According to the Mongoose API docs the way to do this is to use Model.discriminators. So in the case above it would be
ReadUser.discriminators
or
WriteUser.discriminators
However this doesn't return anything for me. What does work is using
Object.keys(Model.discriminators)
As expected this gets you an array of strings of the discriminator names you've set previously.
If you want to use the existing discriminator model and know its name what you can do is use Model.discriminators.discriminatorName. In your example it would be:
let ReadSpecialUserDocument = new ReadUser.discriminators.SpecialUser({
key: value,
key: value,
});
ReadSpecialUserDocument.save()
This can be useful when you need to reuse the discriminator at different times, and its name is tied to your data in some way.
Related
Recently I have decided to replace arrays with priority queues for storing my list of jobs for a user into MongoDB. I use NodeJS and ExpressJS for backend. The priority queue I attempted to store is from an external package which can be installed by running the following command in terminal:
yarn add js-priority-queue
For some reason the priority queue works perfectly prior to storing it into MongoDB. However, the next time I attempt to take it out of MongoDB and use it, its functionality is missing. I declare its type as Schema.Types.Mixed in the Schema. Am I doing something wrong or is it not possible to store instantiated class objects into MongoDB?
As far as I know, when you store things in MongoDB they are stored as extended JSON (EJSON) in binary format (BSON)
const { EJSON } = require('bson');
const test = EJSON.stringify({a: new Date(), foo:function(){console.log('foo');}})
console.log(test) // "{"a":{"$date":"2020-07-07T14:45:49.475Z"}}"
So any sort of function is lost.
I recently started using Node.js + Express.js (generated with pug) + pg-promise for handling db.
My first target is to obtain data from Postgres (already set up) and display it pretty using render and pug. Let's say it is user list from Users table.
On this restful tutorial I have learned how to get data and return it as JSON - it worked.
Based on Mozilla's tutorial I seperated my code:
routes/users.js: where for '/' I call user_controller.user_list method (using router.get)
controllers/userController.js I have exported user_list where I would like to ask model for data and call render if I have results
queries.js which is kinda my model? But I'm not sure. It has API: connection to db with promises and one function for every query I am going to use in Controllers. I believe I should have like one Model file per table (or any logical entity) but where to store pgp connections?
This file is based on first tutorial I mentioned
// queries.js (connectionString is set properly to my postgres)
var pgp = require('pg-promise')(options);
var db = pgp(connectionString);
function getUsers(req, res, next) {
db.any('SELECT (user_id, username) FROM public.users ORDER BY user_id ASC LIMIT 1000')
.then(function (data) {
res.json({ data: data });
})
.catch(function (err) {
return next(err);
});
}
module.exports = {
getUsers: getUsers
};
Here starts my problem as most tutorials uses mongoose which is very model-db-schema-friendly and what I have is simple 'SELECT ...' string I pass to pg-promise's any() function.
Therefore I have no model class like User.
In userControllers.js I don't know how to call getUsers() to handle its data. Returning JS object from getUsers() would be nice.
Also: where should I call render? In controller or only in
db.any(...).then(function (data) { <--here--> })
Before, I also tried to embed whole Postgres handling into Controller but from db.any() I got this array for handling:
[{ row: '(1,John)' },{ row: '(2,Amy)' },{ row: '(50,Peter)' } ]
Didn't know how go from there as I probably lost my API functionality as well ;-)
I am browsing through multiple tutorials how to handle MVC but usually they handle MongoDB and
satisfy readers with res.send() not render().
I am not sure that I understand what your question is exactly about, but since I do not have enough reputation to comment, I'll do my best to help you with your interrogations. :)
First, regarding the queries.js file, it is IMO not exactly a model, but rather a DAO (Data Access Object) file. DAO comes between you Model (which is actually you database) and your Controller layers. There usually is a DAO file per object (User, Pet, whatever you want) in your data model.
When the data model is rather complex, it can be useful to use an Object Relational Mapping (ORM) such as Mongoose to map your database and execute complexe processes on your objects. In such a case, you might need a specific file per object so as to describe your model and store your queries. But since you don't need an ORM, you DAO can directly interact with your database. That is why you do not have a User.js file.
Regarding the way the db object should be used, I think you should refer directly to pg-promise documentation on the matter.
IMPORTANT: For any given connection, you should only create a single
Database object in a separate module, to be shared in your application
(see the code example below). If instead you keep creating the
Database object dynamically, your application will suffer from loss in
performance, and will be getting a warning in a development
environment (when NODE_ENV = development)
As a matter of fact, a db object in pg-promise sort of represents the database itself and is actually designed for the simultaneous use of several databases, which does not seem to be your case for the moment.
Finally, when it comes to the render function, I believe it should be in the controller, as your DAO is not supposed to know how the data it has gathered is going to be used.
Modularity is always a time-saving choice on the long-term.
Furthermore, note that you might later need a Business Layer between your DAO and your controller, in order to preprocess and postprocess data you are going to persist or to display. In such a case, if you need for instance to ask for data from your database, you will need to render data after it is processed by the Business layer. If the render is made in the DAO layer, it will not be possible.
In the link I provided earlier to pg-promise's db object connection, you will also find documentation on the any() method. You might already have looked it up.
It specifically states that it returns
A promise object that represents the query result:
When no rows are returned, it resolves with an empty array.
When 1 or more rows are returned, it resolves with the array of rows.
so your returned data is a JS Array. If you want to make it a JS object, just use
JSON.stringify(yourArray) to process your data before rendering it in your controller.
But I wonder if Pug is not able to use your data directly.
Also, if you cannot get any data out of your DAO, maybe you should check that your data object is not empty, as such a case is tolerated by the any() method. If you expect your query to always return something, you might want to consider using the many() or the one() methods.
I hope this helps you.
I am working with SailsJs+MongoDB API. I have to create New colletion in mongoDB .Name of colletion will be in request Parameter.
example:
Suppose I want to create 'Users' collection in 'mongoDbDatabase' database
by following request.
{
"collectionName" : "Users",
"dbName" :"mongoDbDatabase"
}
Now is there any way to create dynamic collection in mongoDB using req.param('collectionName) variable ?
To use all the tools that Sails provides, you have to add code (before you start your app) for each Model / Collection you are planning to interact with. As a result, creating a collection dynamically will mean you can't take advantage of the whole data - framework sails provides.
Are you sure you need a dynamic collection? Why not a defined collection differentiated by attributes?
If you really need to do this from Sails, it looks like you can get access to the underlying raw mongo database:
var db = AnyModel.getDatastore().manager; // the database will be as defined in config/models.js or config/connections.js
var collectionName = 'Widgets';
db.createCollection(collectionName);
// note, even if this works, something like 'Widgets.find' will not.
I have a mongodb replica set from which I want to read data from primary and secondary db.
I have used this command to connect to the db:
mongoose.connect('mongodb://user:password#54.230.1.1,user:password#54.230.1.2,user:password#54.230.1.3/PanPanDB?replicaSet=rs0&readPreference=nearest');
It doesn't work.. My application continues to read from the primary.. Any suggestion please?
If you want to read from a secondary, you should set your read preference to either of:
secondaryPreferred - In most situations, operations read from secondary members but if no secondary members are available, operations read from the primary.
secondary - All operations read from the secondary members of the replica set.
Reading from nearest as per your example will select the nearest member by ping time (which could be either the primary or a secondary).
Caveats
When using any read preference other than primary, you need to be aware of potential issues with eventual consistency that may affect your application logic. For example, if you are reading from a secondary there may be changes on the primary that have not replicated to that secondary yet.
If you are concerned about stronger consistency when reading from secondaries you should review the Write Concern for Replica Sets documentation.
Since secondaries have to write the same data as the primary, reading from secondaries may not improve performance unless your application is very read heavy or is fine with eventual consistency.
Following the documentation found on MongoDB website and on Mongoose web site, you can add this instruction for configuring the ReadPreference on Mongoose:
var opts = { replSet: {readPreference: 'ReadPreference.NEAREST'} };
mongoose.connect('mongodb://###:#######:###/###', opts);
This has been tested using Mongoose version 3.8.9
As well as setting the connection URI (as you did) and the connection options (as Emas did), I also had to explicitly choose the server for each query, e.g.
var query = User.find({}).read("nearest");
query.exec(function(err, users) {
// ...
});
Mongoose use node package "mongodb", connection uri or opts is parsed by "mongodb". Here is mongodb connect opts and mongodb readPreference source code.
So, we can use mongoose like this:
var opts = {db: {readPreference: 'nearest'};
mongoose.connect(uri, opts);
Also, just use uri like this:
var uri = 'mongodb://###?readPreference=nearest';
mongoose.connect(uri, opts);
In mongoose 4.3.4 above take effect.
This is the proper instantiation in Mongoose v5.9.5:
const opts = {
readPreference: 'nearest',
}
mongoose.connect(MONGODB_CONNECTION, opts)
These are the different string values depending on the preference type you're looking for:
ReadPreference.PRIMARY = 'primary';
ReadPreference.PRIMARY_PREFERRED = 'primaryPreferred';
ReadPreference.SECONDARY = 'secondary';
ReadPreference.SECONDARY_PREFERRED = 'secondaryPreferred';
ReadPreference.NEAREST = 'nearest'
You can simply do that by using below code
var collection = db.collection(collectionName,{readPreference:'secondaryPreferred'});
http://p1bugs.blogspot.in/2016/06/scaling-read-query-load-on-mongodb.html
I just want to know if when I set a mongoose connection and I define some models, (previously adding their appropriate requires on app.js, or wathever), the model, if not exist, will be created automatically the first time when I run node app.js?
Is this kind of logic correct?
If not, do I have to create before my mongoDB collections, models and so on?
I was thinking to an automatic creation of the mongo db collection when I first run the app.js
Thanks!
Michele Prandina
Schemas (and models) are a client-side (node.js) manifestation of your data model. A few things, like the indexes you've defined, are created upon first use (like saving a document for example). Nearly everything else is delay created, including collections.
If you want consistent behavior regarding your models (and their associated schemas), you'll need to make sure they're loaded prior to any access of the associated database. It doesn't really matter where you put them, as long as they are created/executed prior to usage. You might for example:
app.js
models\Cheese.js
\Cracker.js
Then, in app.js:
var Cheese = require('Cheese.js');
var Cracker = require('Cracker.js');
Assuming, of course, you've exported the models:
model.exports = mongoose.model('Cheese',
new mongoose.Schema({
name: String,
color: String
})
);