After reading through the mLab -> Atlas migration plan a few times, I decided I'd try a different way. My coding background is mainly asm on mcs51 so I'm something of a n00b in the node.js/mongo/heroku world. I barely understood half of the migration process.
So I wrote a small test app following this blog entry and then used what I'd learned to modify my actual app to talk to Atlas directly. I exported the collections from the old db to JSON, then imported them into the Atlas version to recreate the database. Everything appears to be working correctly; I don't see any data going into the old db and it looks like the new Atlas db is getting all the action.
But I'm leery of deleting the mLab add-on from Heroku until I've verified that it's truly not needed any more, because I'm pretty sure that I won't be able to recreate it if it turns out I've missed something.
So my question is, how can I ensure I'm no longer using the mLab add-on? I don't really understand what it was doing for me in the first place so I'm not sure how to verify I'm not using it any more.
Here are the relevant code snippets I'm using to access the Atlas db...
function myEncode(str) { // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent
return encodeURIComponent(str).replace(/[!'()*]/g, function(c) {
return '%' + c.charCodeAt(0).toString(16);
});
}
const ATLASURI = process.env.ATLASURI;
const ATLASDB = process.env.ATLASDB;
const ATLASUSER = process.env.ATLASUSER;
const ATLASPW = myEncode(process.env.ATLASPW); // wrapper needed to handle strong paswords...
const dbURL = "mongodb+srv://"+ATLASUSER+":"+ATLASPW+"#"+ATLASURI+"/"+ATLASDB+"?retryWrites=true&w=majority";
var GoogleStrategy = require('passport-google-oauth20').Strategy;
const {MongoClient} = require('mongodb');
const client = new MongoClient(dbURL, { useNewUrlParser: true, useUnifiedTopology: true });
var store = new MongoDBStore({uri: dbURL,collection: 'Sessions'});
var db = undefined;
client.connect(async function(err) {
if(err) {console.log("Error:\n"+String(err));}
db = await client.db(ATLASDB);
console.log("Connected to db!");
banner();
});
Related
I am trying to implement a common module for MongoDB connection using mongoose. and want to use the connection in other application for database operation. but facing issue when trying to use common database module. operation is halted / hanging after creating db connection. here is my codebase.
When I am using module specific dababase connection, then it is working fine, but when I am using common database connection it is hanging
Common DB Module
'use strict'
const mongoose = require('mongoose');
const DBOptions = require('./DBOption');
require("dotenv").config();
mongoose.Promise = global.Promise;
let isConnected;
const connectToDatabase = (MONGODB_URL) => {
if (isConnected) {
console.log('using existing database connection');
return Promise.resolve();
}
console.log('using new database connection');
console.log('DBOptions >> '+JSON.stringify(DBOptions));
return mongoose.connect(MONGODB_URL, DBOptions)
.then(db => {
console.log('db.connections[0].readyState >> '+db.connections[0].readyState);
isConnected = db.connections[0].readyState;
});
};
module.exports = connectToDatabase;
API Controller
const dbConnection = require('../DB/connection') // Internal Class
const DBConnection = require('as-common-util').connectToDatabase; // Common Class
/**
*
*/
app.get('/usr/alluser', async (req, res) => {
try {
//await dbConnection(process.env.MONGODB_URL) // This is working
await DBConnection(process.env.MONGODB_URL) // Code is hanging for this
let allUsers = await UserService.getAllUser()
console.log("All Users >> " + allUsers)
if (allUsers) {
return res.status(200).send(
new APIResponse({
success: true,
obj: allUsers
})
)
}
} catch (error) {
console.log(error)
}
})
It is hanging at following position
using new database connection
DBOptions >>
{"useNewUrlParser":true,"useUnifiedTopology":true,"useCreateIndex":true,"useFindAndModify":false,"autoIndex":false,"poolSize":10,"serverSelectionTimeoutMS":5000,"socketTimeoutMS":45000,"family":4}
db.connections[0].readyState >> 1
I am confused why same code is not working for common module.
This kind of pattern is not how Mongoose is meant to be used. Under the hood, Mongoose passes the underlying connection to the models in your module without the user really knowing anything about what is going on. That's why you can do magic stuff like MyModel.find() without ever having to create a model object yourself, or pass a db connection object to it.
If your db connection is in another module though, Mongoose won't be able to make those connections between your models and the MongoDB client connection since the models are no longer being registered on the mongoose object that is actually connected, and as a result, any requests you make using your models will break, since they will always be trying to connect through the object in your module.
There are other reasons why this won't, and shouldn't, work though. You're not supposed to be able to split a client. Doing so would make it unclear where communication along a client is coming from, or going to. You could change your function to make it return an established client connection. But your Mongoose models still wouldn't work. You would just be left with raw mongodb. If you want to do that, you might as well just uninstall Mongoose and use the mongodb library. Ultimately, you don't really gain anything from initializing the connection in a shared module. Initializing a connection is just a couple lines of code.
I doubt it's the connection that you want to share, rather it's the models (I'm guessing). You can put those in a shared module, and export them as a kind of connector function that injects the a given Mongoose instance into the models. See: Defining Mongoose Models in Separate Module.
Here is my code in my NodeJS application, to connect to my MongoDB engine :
const collection = 'mynewcollection';
const password = 'passwordwithan#';
const mongoUrl = `mongodb://admin:${encodeURIComponent(password)}#mymongobase.net/${collection}`;
// Connect using the connection string
MongoClient.connect(mongoUrl, {useNewUrlParser: true}, function(err, db) {
console.log(err.toString())
});
I get an authentication error. I tried several things to handle the '#' character and reading the documentation I thought that it was the good one...
But it is still failing even if the user and password are the good one.
Is the API correctly used ? Do you understand what is wrong ?
Thanks in advance.
OK I found the solution.
If you use :
const mongoUrl = `mongodb://admin:${encodeURIComponent(password)}#mymongobase.net/${collection}`;
Mongo will try to connect with admin/password defined on the collection.
If you don't put the collection in the mongo url :
const mongoUrl = `mongodb://admin:${encodeURIComponent(password)}#mymongobase.net`;
Then you can use the super user defined on all the mongo engine.
I am having major performance problems with MongoDB. Simple find() queries are sometimes taking 2,000-3,000 ms to complete in a database with less than 100 documents.
I am seeing this both with a MongoDB Atlas M10 instance and with a cluster that I setup on Digital Ocean on VMs with 4GB of RAM. When I restart my Node.js app on Heroku, the queries perform well (less than 100 ms) for 10-15 minutes, but then they slow down.
Am I connecting to MongoDB incorrectly or querying incorrectly from Node.js? Please see my application code below. Or is this a lack of hardware resources in a shared VM environment?
Any help will be greatly appreciated. I've done all the troubleshooting I know how with Explain query and the Mongo shell.
var Koa = require('koa'); //v2.4.1
var Router = require('koa-router'); //v7.3.0
var MongoClient = require('mongodb').MongoClient; //v3.1.3
var app = new Koa();
var router = new Router();
app.use(router.routes());
//Connect to MongoDB
async function connect() {
try {
var client = await MongoClient.connect(process.env.MONGODB_URI, {
readConcern: { level: 'local' }
});
var db = client.db(process.env.MONGODB_DATABASE);
return db;
}
catch (error) {
console.log(error);
}
}
//Add MongoDB to Koa's ctx object
connect().then(db => {
app.context.db = db;
});
//Get company's collection in MongoDB
router.get('/documents/:collection', async (ctx) => {
try {
var query = { company_id: ctx.state.session.company_id };
var res = await ctx.db.collection(ctx.params.collection).find(query).toArray();
ctx.body = { ok: true, docs: res };
}
catch (error) {
ctx.status = 500;
ctx.body = { ok: false };
}
});
app.listen(process.env.PORT || 3000);
UPDATE
I am using MongoDB Change Streams and standard Server Sent Events to provide real-time updates to the application UI. I turned these off and now MongoDB appears to be performing well again.
Are MongoDB Change Streams known to impact read/write performance?
Change Streams indeed affect the performance of your server. As noted in this SO question.
As mentioned in the accepted answer there,
The default connection pool size in the Node.js client for MongoDB is 5. Since each change stream cursor opens a new connection, the connection pool needs to be at least as large as the number of cursors.
const mongoConnection = await MongoClient.connect(URL, {poolSize: 100});
(Thanks to MongoDB Inc. for investigating this issue.)
You need to increase your pool size to get back your normal performance.
I'd suggest you do more log works. Slow queries after restarted for a while might be worse than you might think.
For a modern database/web app running on a normal machine, it's not very easy to encounter with performance issues if you are doing right. There might be a memory leak or other unreleased resources, or network congestion.
IMHO, you might want to determine whether it's a network problem first, and by enabling slow query log on MongoDB and logging in your code where the query begins and ends, you could achieve this.
If the network is totally fine and you see no MongoDB slow queries, that means something goes wrong in your own application. Detailed logging might really help where query goes slow.
Hope this would help.
I am having big problems trying to wrap my head around mongodb.
First of all, I tried using mongoose, but I really don't like its schemas, I want to have my own classes, and perform CRUD operations in a classic style.
So I moved on and started using mongodb. I think I already gonna give up and use MySQL, because I find their documentation very very obscure, and also no public issue tracker on github.
Anyway, my problem is that if I connect to a non existing db or to a non existing host, mongodb isn't showing any error.
My code:
const mongodb = require( 'mongodb' ),
MongoClient = mongodb.MongoClient;
let db_uri = `mongodb://${config.host}:${config.port}/${config.name}`;
MongoClient.connect( db_uri, function( err, mongoclient ) {
console.log( err );
});
Assume that config.host or config.port are not correct.
err is alway null.
Why is this ?
Mongodb does this great(and in some cases really annoying thing if you have fat fingers) where if you attempt to connect a database that doesn't exist, it will create it for you. I knew it did this in command line, but it appears it does it with this code as well. Using the insert code from the npm node instructions I was able to test and verify this.
Edit- removing example and adding something actually helpful
It may be cumbersome, but there is a way to test if a collection exists. You ask if an index exists.
const mongodb = require( 'mongodb' ),
MongoClient = mongodb.MongoClient;
var db_uri = "mongodb://localhost:27017/newDb";
MongoClient.connect( db_uri, function( err, db ) {
var collection = db.collection('documents');
collection.indexExists(0, function(err, indexExists){
});
});
I want to Sync() the PouchDb on my Nodejs-server with the IndexedDb at frontend.
But : I dont use Couch or Pouchdb-server
on Backend I'm runnig :
var pouch = require('pouchdb')
var db = new pouch('fuu');
test.app.use('/sync'),function(req,res,next){console.log('Woop woop');
db.info(function)(err,info) {return res.send('info')})
});
// same problem with : db.allDocs(..);
on frontend:
var db = new PouchDB('ba');
var remoteCouch = ("http://localhost:3000/sync")
var sync = function() {
var opts = {live: true};
db.sync(remoteCouch, opts);}
sync();
But now there is an endless call of 'Woob woob' in the console and nothing sync's ..
Have someone an idea what I'm doing wrong?
You need to correctly map all the urls that PouchDB will use to be able to sync. I aassume your backend is a typo and you are using var db = new PouchDB('fuu') on the backend right?
PouchDB-Server has extracted the url routing logic it uses into another module, https://github.com/pouchdb/express-pouchdb, the README should give you an example of how to do that and you dont need the extra functionality provided by pouchdb-server