Using PostgreSQL.
I'm trying to set up a proper test suite for an API. So far it works but the tests are done on the main database directly. I then have to remove everything my tests created, or edited in such case.
I know it's a bad thing to do (because I can forget to revert a change, or mess up the sequences). So I would like to create a test database with the same structure and base data, and then delete it afterwards. Is this approach the good one to take in this case?
And if I want to do it this way, how should I do it? Is there a way in NodeJS to execute an SQL script? I tried with a shell script but so far it's a been complete mess with the permissions, so I figured it would be easier with NodeJS directly.
I'm using Mocha for my tests.
I would suggest a separate test database. It can be light, and you will want to know the data that is in there (so you can test it!). A base dataset that can handle all your business rules can be exported as a SQL file (or some export method).
Typically your application will have a connection to the database, and your test framework will have some approach to run a method prior to tests starting. It is here that you specify the test DB. Your database access objects (DAOs), or scripts, methods, will utilize the main connection in some way, either as a method parameter, or require statement, etc.
As an example, I'm using the knex module to connect to the DB and build queries. I initialize and reference my single DB connection as specified in their docs.
var Knex = require( 'knex' );
Knex.knex = Knex.initialize( {
client : 'mysql',
connection : {
host : 'my.domain.com',
user : 'dbrole',
password : 'password',
database : 'productiondb',
charset : 'utf8'
}
} );
My DAOs get the connection like this:
var knex = require('knex').knex;
Now in my unit tests, before a test suite is run, I can set my connection to be the test DB
var Knex = require( 'knex' );
Knex.knex = Knex.initialize( {
client : 'mysql',
connection : {
host : '127.0.0.1',
user : 'root',
password : 'root',
database : 'testdb',
charset : 'utf8'
}
} );
And there you have it! Exact same code is used in test and production, and your production DB is de-coupled from your tests. This pattern can work with a lot of frameworks, so you'll have to adapt (and clean up your tests if they are junking up the test DB, maybe a restore to default when all tests are complete).
Edit: By the way, knex works with postgre and is a fun way to build queries in pure node JS. It can also execute raw SQL.
Related
I am super new to node express and postgres and wondering the following:
const pg=require('pg').native
const client=new pg.Clirnt('postgres ...')
what is const?
pg is used to create a client to connect to the Postgres database-correct?
If so
var db = new Sequelize('postgres://localhost:5432/mydb')
would work too or would I just have created a database without connecting it?
Why exactly do I need to connect at all-to do what?
Thanks a lot!
const is constant in javascript which was introduced in ES6 specification.
node-postgres is a client for PostgreSQL.
Sequelize is using node-postgres for working with PostgreSQL database, so yes, in the nutshell, it will act like node-postgres.
Imagine the warehouse, that's your database, where you have different shelves, that's your tables, to take or put different items into warehouse you need workers that will do your instructions like - INSERT someitem INTO items_shelf;. So worker is the client like Sequelize or node-postgres. The important part, warehouse should be open, otherwise, workers couldn't access to it, so your database should be turned on.
Hope I'm explained understandable enough.
what is const?
TLDR; variables that can't be re-assigned. scoped the same way as var. Part of es6.
pg is used to create a client to connect to the postgres database?
yes, note you need to do npm install --save pg as well as npm install --save sequelize. the save flag adds the packages to the package.json file for your convenience.
would I just have created a database without connecting it?
That bit of code should instantiate a connector - you haven't modified the database, and you also don't really know if the connection works yet.
why exactly do I need to connect at all?
The pg library looks to use a connection pool; this means you set it up once, and then you use it repeatedly as desired and it handles the connections for you. You connect now so you can run queries against the database later.
This snippet of code connects to a postgres instance running locally on my machine, and tests that it can connect - per the docs
const Sequelize = require('sequelize');
var sequelize = new Sequelize('postgres://localhost:5432/postgres');
sequelize.authenticate().then(() => {
console.log('yay');
}).catch((e) => {
console.log('nooo', e);
});
I'm writing functions for my node.js server using TDD(Mocha). For connecting to the database I'm doing
before(function(done){
db.connect(function(){
done();
});
});
and I'm running the test cases using make test and have configured my makefile to run all the js files in that particular folder using mocha *.js
But for each js file I'll have to make a separate connection to the database, otherwise my test cases fail since they do not share common scope with other test files.
So the question is, Is there anything like beforeAll() that would just simply connect once to the database and then run all the test cases? Any help/suggestion appreciated.
You can setup your db connection as a module that each of the Mocha test modules imports.
var db = require('./db');
A good database interface will queue commands you send to it before it has finished connecting. You can use that to your advantage here.
In your before call, simply do something that amounts to a no op. In SQL that would be something simple like a raw query of SELECT 1. You don't care about the result. The return of the query just signifies that the database is ready.
Since each Mocha module uses the same database module, it'll only connect once.
Use this in each of your test modules:
before(function(done) {
db.no_op(done);
});
Then define db.no_op to be a function that performs the no op and takes a callback function.
I'm a bit new to developing in nodejs, so this is probably a simple problem. I'm building a typical webapp based on express + sequelize. I'm using sqlite in-memory since I'm just prototyping at the moment. I understand if I were to use a persistent sqlite file, this may not be a problem, but that's not my goal at the moment. Consider the following:
var User = sequelize.define("User", {
"username": DataTypes.STRING,
// etc, etc, etc
});
sequelize.sync();
User.build({
"username": "mykospark"
});
At first, I got an error on User.build() about the Users table not existing yet. I realized that sequelize.sync() was being called async, and the insert was happening before the table was created. I then re-arranged my code so that the User.build() call was inside of sequelize.sync().complete() which fixes the problem, but I'm not sure how to apply this to the rest of my project.
My project uses models in a bunch of different places. It is my understanding that I just want to call sequelize.sync() once after my models are defined, then they can be used freely. I could probably find some way to block the entire nodejs app until sequelize.sync() finishes, but that doesn't seem like good form. I suppose I could wrap every single model operation into a sequelize.sync().complete() call, but that doesn't seem right either.
So how do people usually deal with this?
Your .sync() call should be called once within your app.js file. However, you might have additional calls if you manage multiple databases in one server. Typically your .sync() call will be in your server file and the var User = sequelize.define("ModelName"... will be in your models/modelName.js file. Sequelize suggests this type of guidance to "create a maintainable application where the database logic is collected in the models folder". This will help you as your development grows. Later in the answer, I'll provide an easy step to follow for initializing the file structure.
So for your case, you would have app.js, models/index.js and models/users.js. Where app.js would be your server running the .sync() method. In the models folder you will have the required index.js folder where you configure a connection to the database and collect all the model definitions. Finally you have your user.js files where you add your model with class and instance methods. Below is an example of the models/user.js file you might find helpful.
user.js
module.exports = function(sequelize, DataTypes) {
return sequelize.define('User', {
username: DataTypes.STRING,
},{
classMethods: {
doSomething: function(successcb, errcb, request) {}
},
instanceMethods: {
someThingElse: function(successcb, errcb, request) {}
}
});
};
models/index.js --> See here
EDIT 03/14/17
Now the best option to setup your node app with sequelize is to use sequelize-cli. This is sequelize migrations and has very useful functionality in development and production environments. For the scope of this question and revision to the answer, the best approach is the following:
npm install sequelize-cli
Use npm install sequelize-cli -g if you want it installed globally.
Then initialize sequelize migrations:
sequelize init
It should install the following folders and files structure in the folder you initiated the command:
config:
-config.json
models:
-index.js
seeders:
migrations:
If you want to create a model you can run the following command and it will auto generate the file structure for you. Here is an example
sequelize model:create --name User --attributes "user:string email:string"
Next you should be able to see the new model page in models/page.js.
config:
-config.json
models:
-index.js
-user.js
-page.js
seeders:
migrations:
You'll need to then go into you models/index.js and define your new model for your database to access the correct path for that model. Here is an example:
models/index.js
var sq = new Sequelize(dbname, user, password, config);
db = {
Sequelize: Sequelize,
sequelize: sq,
page: sq.import(__dirname + '/page.js'),
user: sq.import(__dirname + '/user.js')
}
module.exports = db;
If you need to make changes to the model you can go into the migrations folder and add methods. Follow the sequelize migration docs here. Now, about the app.js server. Before you run your server you need to initialize your databases. I use the following script to initialize the database before running the server to setup a postgres db:
postgresInit.sh
[...]
`sudo -u postgres createdb -U postgres -O $PG_USER $PG_DB. `
If you prefer a javascript solution, there is an SO solution here
app.js
[...]
console.log('this will sync your table to your database')
console.log('and the console should read out Executing (default): CREATE TABLE IF NOT EXISTS "TABLE NAME"....')
db.sequelize.sync(function(err){});
I am using mongodb-native-driver in express.js app. I have around 6 collections in the database, so I have created 6 js files with each having a collection as a javascript object (e.g function collection(){}) and the prototypes functions handling all the manipulation on those collections. I thought this would be a good architecture.
But the problem I am having is how to connect to the database? Should I create a connection in each of this files and use them? I think that would be an overkill as the connect in mongodb-native-driver creates a pool of connections and having several of them would not be justified.
So how do I create a single connection pool and use it in all the collections.js files? I want to have the connection like its implemented in mongoose. Let me know if any of my thought process in architecture of the app is wrong.
Using Mongoose would solve these problems, but I have read in several places thats it slower than native-driver and also I would prefer a schema-less models.
Edit: I created a module out of models. Each collection was in a file and it took the database as an argument. Now in the index.js file I called the database connection and kept a variable db after I got the database from the connection. (I used the auto-reconnect feature to make sure that the connection wasn't lost). In the same index.js file I exported each of the collections like this
exports.model1 = require('./model1').(db)
exprorts.model2 = require('./model2').(db)
This ensured that the database part was handled in just one module and the app would just call function that each model.js file exported like save(), fincdbyid() etc (whatever you do in the function is upto you to implement).
how to connect to the database?
In order to connect using the MongoDB native driver you need to do something like the following:
var util = require('util');
var mongodb = require('mongodb');
var client = mongodb.MongoClient;
var auth = {
user: 'username',
pass: 'password',
host: 'hostname',
port: 1337,
name: 'databaseName'
};
var uri = util.format('mongodb://%s:%s#%s:%d/%s',
auth.user, auth.pass, auth.host, auth.port, auth.name);
/** Connect to the Mongo database at the URI using the client */
client.connect(uri, { auto_reconnect: true }, function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server at:');
console.log('\n\t%s\n', uri);
// Create or access collections, etc here using the database object
}
});
A basic connection is setup like this. This is all I can give you going on just the basic description of what you want. Post up some code you've got so far to get more specific help.
Should I create a connection in each of this files and use them?
No.
So how do I create a single connection pool and use it in all the collections.js files?
You can create a single file with code like the above, lets call it dbmanager.js connecting to the database. Export functions like createUser, deleteUser, etc. which operate on your database, then export functionality like so:
module.exports = {
createUser: function () { ; },
deleteUser: function () { ; }
};
which you could then require from another file like so:
var dbman = require('./dbmanager');
dbman.createUser(userData); // using connection established in `dbmanager.js`
EDIT: Because we're dealing with JavaScript and a single thread, the native driver indeed automatically handles connection pooling for you. You can look for this in the StackOverflow links below for more confirmation of this. The OP does state this in the question as well. This means that client.connect should be called only once by an instance of your server. After the database object is successfully retrieved from a call to client.connect, that database object should be reused throughout the entire instance of your app. This is easily accomplished by using the module pattern that Node.JS provides.
My suggestion is to create a module or set of modules which serves as a single point of contact for interacting with the database. In my apps I usually have a single module which depends on the native driver, calling require('mongodb'). All other modules in my app will not directly access the database, but instead all manipulations must be coordinated by this database module.
This encapsulates all of the code dealing with the native driver into a single module or set of modules. The OP seems to think there is a problem with the simple code example I've posted, describing a problem with a "single large closure" in my example. This is all pretty basic stuff, so I'm adding clarification as to the basic architecture at work here, but I still do not feel the need to change any code.
The OP also seems to think that multiple connections could possibly be made here. This is not possible with this setup. If you created a module like I suggest above then the first time require('./dbmanager') is called it will execute the code in the file dbmanager.js and return the module.exports object. The exports object is cached and is also returned on each subsequent call to require('./dbmanager'), however, the code in dbmanager.js will only be executed the first require.
If you don't want to create a module like this then the other option would be to export only the database passed to the callback for client.connect and use it directly in different places throughout your app. I recommend against this however, regardless of the OPs concerns.
Similar, possibly duplicate Stackoverflow questions, among others:
How to manage mongodb connections in nodejs webapp
Node.JS and MongoDB, reusing the DB object
Node.JS - What is the right way to deal with MongoDB connections
As accepted answer says - you should create only one connection for all incoming requests and reuse it, but answer is missing solution, that will create and cache connection. I wrote express middleware to achieve this - express-mongo-db. At first sight this task is trivial, and most people use this kind of code:
var db;
function createConnection(req, res, next) {
if (db) { req.db = db; next(); }
client.connect(uri, { auto_reconnect: true }, function (err, database) {
req.db = db = databse;
next();
});
}
app.use(createConnection);
But this code lead you to connection-leak, when multiple request arrives at the same time, and db is undefined. express-mongo-db solving this by holding incoming clients and calling connect only once, when module is required (not when first request arrives).
Hope you find it useful.
I just thought I would add in my own method of MongoDB connection for others interested or having problems with different methods
This method assumes you don't need authentication(I use this on localhost)
Authentication is still easy to implement
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var client = new MongoClient(new Server('localhost',27017,{
socketOptions: {connectTimeoutMS: 500},
poolSize:5,
auto_reconnect:true
}, {
numberOfRetries:3,
retryMilliseconds: 500
}));
client.open(function(err, client) {
if(err) {
console.log("Connection Failed Via Client Object.");
} else {
var db = client.db("theDbName");
if(db) {
console.log("Connected Via Client Object . . .");
db.logout(function(err,result) {
if(!err) {
console.log("Logged out successfully");
}
client.close();
console.log("Connection closed");
});
}
}
});
Credit goes to Brad Davley which goes over this method in his book (page 231-232)
What is the correct way to access a mongodb database from Express ?
Right now, I am including my database handler db.js which contains mongoose.connect( 'mongodb://localhost/db' ); every time I need to do a database call.
Should I use the same connection and passing my db object through callbacks or I can just include my db file every time ?
In other words, is mongoose.connect always re-using the same connection ?
Edit: my source code is public here, I am fairly new to nodejs/express applications and I am not sure if my application is structured properly...
You only need to connect to your database once. In your other files, you need to include your models and use them to read / write to your database collections.
Edit: Looking at your code -- why don't you move your connect into your initialization script, and then include db.js to access your models?