Is it possible to connect to MongoDB in a read-only mode?
I'm currently using the driver for Node.js to create a new client with MongoClient.connect:
require('mongodb').MongoClient.connect(url, {
// options object
}, function(err, client) {
// ...
});
I don't see anywhere in the docs how to create a client in read-only mode.
It is possible? how?
Background:
I'm building an app which connects to a MongoDB. Other developers in my team extend this app with plugins that consume data. A plugin is supplied with a client object to access the databse. I want to prevent other developers from accidentally making changes to the database.
One of the most clever workarounds for this scenario requires the usage of replica set. Simply connect only to one of the secondary node(s) will prevent write operation and achieve a read-only behaviour.
Related
I have access to two separate databases that I'd like to keep in sync, a new one and an existing one, which will be in separate physical locations. The new one is going to be used to service an external API, so to cut down on request time, I think it makes sense to only query the local database for API requests.
My initial approach was to use mongoose.createConnection and limit the local collection to minor metadata and directly access the remote collection, but that's what I'm now looking to avoid.
Another approach might be to use mongoose.createConnection to periodically query the remote db and update the local one, but it could be costly if I want to do make frequent updates.
There are ways to cut down the cost - for example, there is a lastUpdated property in the relevant collection on the existing database, which could be used to limit the remote query to recently updated records such as:
RemoteCollection.find({
lastUpdated: {$gte: Date.now() - lookbackPeriod}
})
But I'm wondering if there's any native functionality of mongoose/mongoDB that can be used more efficiently make the updates. I also thought about mongodump and mongorestore to keep a full local copy of the records I needed, but that also seems costly.
Any help is appreciated.
After a bit of reading and thanks to Jake's comment, it looks like it's working. I need to do some more setup, but the code below should work and is based on this section from the docs:
https://mongoosejs.com/docs/models.html#change-streams
The first step would be start mongod with the --replSet flag:
mongod --replSet "rs0" --bind_ip localhost,<hostname(s)|ip address(es)>
Then close and restart mongo and run rs.initiate() on the database. You can then check the status of the replica set with rs.status(). If that command works and returns a result, the replica set functionality should be there.
Then within Node, you can do something like this:
// The docs reference creating a new model but you can just import an existing one
const RemotePerson = require('./models/RemotePerson');
const LocalPerson = require('./models/LocalPerson');
RemotePerson.watch().on('change', data => {
if (data.operationType === "insert") {
LocalPerson.create(data.fullDocument);
} else if (data.operationType === "update") {
LocalPerson.findByIdAndUpdate(data.documentKey, {
$set: data.updateDescription.updatedFields
});
}
});
As per the title, can a Node.js package require a database connection?
For example, I have written a specific piece of middlware functionality that I plan to publish via NPM, however, it requires a connection to a NoSQL database. The functionality in its current state uses Mongoose to save data in a specific format and returns a boolean value.
Is this considered bad practice?
It is not a bad practice as long as you require the DB needed and also explicitly state it clearly in your Readme.md file, it's only a bad practice when you go ahead and work without provide a comment in your codes or a readme.md file that will guide any other person going through your codes.
Example:
//require your NoSQL database eg MongoDB
const mongoose = require('mongoose');
// to connect to the database. **boy** is the database name
mongoose.connect('mongodb://localhost/boy', function(err) {
if (err) {
console.log(err);
} else {
console.log("Success");
}
});
You generally have two choices when your module needs a database and wants to remain as independently useful as possible:
You can load a preferred database in your code and use it.
You can provide the developer using your module a means of passing in a database that meets your specification to be used by your module. The usual way of passing in the database would be for the developer using your module to pass your module the data in a module constructor function.
In the first case, you may need to allow the developer to specify a disk store path to be used. In the second case, you have to be very specific in your documentation about what kind of database interface is required.
There's also a hybrid option where you offer the developer the option of configuring and passing you a database, but if not provided, then you load your own.
The functionality in its current state uses Mongoose to save data in a specific format and returns a boolean value. Is this considered bad practice?
No, it's not a bad practice. This would be an implementation of option number 1 above. As long as your customers (the developers using your module) don't mind you loading and using Mongoose, then this is perfectly fine.
I'm facing a problem with my MongoDB environment - the setup is as follows:
My node app provides a restify API which handles user registration (look up if a user exists in a collection based on his mail, and if not, insert him (note - insert uses bcrypt to hash the passwords, so probably is a bit slower)). It uses restify and Mongoose ORM.
A second benchmark script (also written in node, running on the same machine) accesses this restify API using HTTP PUT.
I'm starting around 20-30 of these requests in the benchmark (with random data) and only some of the API requests correctly insert the new users. For the other, MongoDB produces errors similar to the following:
not authorized on ... to execute command { find: "users", filter: { mail: "rroouksl#hddngrau.de" } }
not authorized on ... to execute command { insert: "users", documents: [ { ... } ], ordered: false, writeConcern: { w: 1 } }
Some other users get inserted perfectly fine. Especially with a low number of requests at the same time (1-5) no problems occur. Shouldn't Mongo be able to handle these "low" amount of requests? Is it a problem because it's running on the same machine? Hasn't the user I created in Mongo for this project got enough txns/second allowed?
Best regards,
Zahlii
It turned out that mongo was still using the "old" storage engine and not WiredTiger. Since my queries included updating records, the old engine performed collection-based locks which means that the errors were solely based on read-write locks.
I migrated to WiredTiger which performs document-based locking and since then, the database handles many parallel requests without these errors (although sometimes under heavy load they appear again - but this is part of mongo being NoSQL I guess)
You can try:
Db.authenticate(user, password, function(err, res) {
// callback
});
Also see the source.
During sails lift I don't yet have all the connection information for my DB.
Is there a way to either have config values dependent on promises or dynamically create a connection after sails lift has completed?
I would obviously have to add a policy or hook to handle requests to routes needing the model if it wasn't available yet, but at this point I don't see how to even let the sails lift until I already know the connection info (it must be in the configs).
I'm hoping I'm missing a way to dynamically create connections and wire models to them.
Update: In Sails v1.0 / Waterline v0.13, this can be accomplished by accessing the stateless, underlying driver; e.g. sails.getDatastore().driver. This can be used in any database adapter that supports the new stateless driver interface, including MySQL, PostgreSQL, and MongoDB.
Prior to Sails v1.0, this was not officially supported in Sails or Waterline directly, but depending on your use case there are a couple of good solutions for this. If your use case is a handful of dynamic connections for the purpose of development (e.g. in an auto-reload plugin), and you're willing to live on the edge, you can take advantage of a private API as an immediate-term workaround: sails.hook.orm.reload(). However you definitely don't want to use that in production since it literally flushes the entire ORM.
If you are going to be dealing with a larger number (let's say > 10 unique configurations) of runtime-dynamic datastore configurations during the lifetime of the running Node process, that's a different story. In that case, I would recommend using the relevant raw driver (e.g. https://github.com/felixge/node-mysql) to summon/release those dynamic connections from a pool directly via a service. You can still use your normal models in your app for connections which are static-- you will just be best off implementing dynamic database connections separately in your service. For example, if you were building a hosted version of phpMyAdmin you might use a lower-level NPM package to dynamically fetch information about users' tables, but you'd still probably want to have Account and Database models that refer to tables/collections stored in your own database.
A more integrated solution for Sails is in the works. This ability to tap into the raw connection lifecycle and access it from userland is a prerequisite for built-in transaction support, which is something we expect to land in Sails/Waterline some time in the second half of 2016. In the mean time, if you encapsulate your logic to summon/release connections via service methods as suggested above, you'll have a working solution for now and your business logic should be more or less future proof (when you upgrade, you'll just need to swap out the implementation in your service). Hope that helps!
Yes; two things in sails.js allow you to do this. One currently exists, and one is upcoming.
https://github.com/sgress454/sails-hook-autoreload. This module watches for config file changes on disk and will re-load your ORM models when a file changes.
I am working on this exact feature right now, and my plan is to publish my work at the end of next week. I agree that it will be very useful.
The API will allow you to define new Models and Connections in the database on the fly. sails.js lifecycle callbacks handle updating the ORM and adapters and so forth. It is event-based and will allow you to manually fire events to update the ORM/Connections, like so:
sails.emit('hook:dynamic-orm:reload')
Is this what you need?
I have found a workaround for MySql DB
Important: In my case, I will be changing database but all database would have the same schema only difference is in their name and data they contain and make sure to add any error handling you need
In config/connection.js --------
disable Pooling
mysql_database: {
adapter: 'sails-mysql',
host: 'localhost',
user: 'root', //optional
password: '12345', //optional
database: 'db1', //optional
pool: false
},
Now navigate to
node_modules/sails-mysql/lib/connections/spawn.js
Add connectionObject.config = sails.SwitchDbConfig
connectionObject.config = sails.SwitchDbConfig
var conn = mysql.createConnection(connectionObject.config);
conn.connect(function (err) {
afterwards(err, conn);
});
Now Finally Set sails.SwitchDbConfig form anywhere (service , controller ,etc)
as
sails.SwitchDbConfig = {
pool: false,
connectionLimit: 5,
waitForConnections: true,
adapter: 'sails-mysql',
host: 'localhost',
user: 'root',
password: '12345',
database: sails.DB_NAME,
identity: 'mysql_database'
}
And at last if you find something wrong for needs to be updated .... please ping
I am currently working on an inventory management software in Node js and MongoDB. I am pretty new to MongoDB, having worked in Oracle and MySQL for most of my projects.
Is it possible to create a separate database schema for every client who uses my software, with each client having access only to his copy of the database schema and collections?
The equivalent of selecting data in Oracle database would be
Select * from User1.table,
Select * from User2.table etc
Also, if it were possible, how would it be implemented using a node js mongo db client like mongoose?
I looked at MongoDB documentation, but it talks mainly about adding users to a database for authorization.
I apologize if it seems like a silly question, but id appreciate it if someone could point me in the right direction for this.
Before starting to invest a lot of time in the development of your project, check out other possible approaches to the scenario that you are trying to build.
I did a quick search on SO and found some additional threads with similar scenarios:
MongoDB Database vs. Collection
MongoDB Web App - Database per User
Additional info about mongoose database creation
Whenever you call the connect method on the mongoose object, you are either connecting to an existing database or you are creating it in case it doesn't already exist.
You could have a function that allows you to pass in a name argument with the name and create databases programmatically:
function createDatabase(name) {
var conn_string = 'mongodb://localhost/';
if (typeof name == 'string') {
conn_string += name;
}else{
return false;
}
mongoose.connect(conn_string);
}
Also, be aware that a database will be created when you first insert a record in a collection of that particular database.
It is not sufficient to only connect to the database, you also have to insert a record.
As per my previous example, you could also pass a schema parameter to the function, tailored to each user's profile and fire an insert statement after you connect to that database.