Knex migration not working when using migration API - node.js

I am new to knex migrations and for the past 2 days I have been struggling to get it working but nothing happen. I am trying to run my migrations programmatically using the knex.migration object.
First using the cli, I create a migration file in the migrations directory. Here is its content:
exports.up = function(knex, Promise) {
return Promise.all([
knex.schema.createTable('users', function (table) {
table.increments('id').primary();
table.string('username');
table.string('password');
table.string('email');
table.string('name');
table.timestamp('date');
}),
]);
};
exports.down = function(knex, Promise) {
};
Then from my code I initialize the Knex object:
var knex = Knex({
client:'sqlite3',
connection:{
filename: './knex.sqlite'
}
});
Then I execute the migration:
knex.migrate.latest().then(()=>{
// console.log()
}).catch(err =>{
//
});
But absolutely nothing happens. My migration file is never executed and there is no error or warning message. So I don't know where to look at to start searching for the problem. When I look at my sqlite database, I can see that tables knex_migrations, knex_migrations_lock and sqlite_sequence have been created.
So what I am doing wrong here? Is there something I am missing?
Thanks for any suggestion

There's no requirement to use the CLI tools. Sometimes it's not possible to use it due to its limitations and in this case it's indeed possible to use the migration API directly, like so:
const knex = require('knex')({
// Here goes the Knex config
});
const migrationConfig = {
directory: __dirname + './migrations',
}
console.info('Running migrations in: ' + migrationConfig.directory);
knex.migrate.latest(migrationConfig).then(([batchNo, log]) => {
if (!log.length) {
console.info('Database is already up to date');
} else {
console.info('Ran migrations: ' + log.join(', '));
}
// Important to destroy the database, otherwise Node script won't exit
// because Knex keeps open handles.
knex.destroy();
});
There was two issues in the original question:
The migration directory was not specified - in this case Knex is not smart and will simply not do anything instead of throwing an error. Most likely the default used by Knex is not right so it's best to specify it.
knex.destroy() was missing. In this case, the script will never exit because Knex keeps open handles on the database, so it just looks like it's stuck doing nothing.
The above script also outputs more log info to see what's exactly happening.

Knex migrations are supposed to run by Knex CLI,FYI: https://knexjs.org/#Migrations
As your code described, I found a strange issue:
knex.migrate is actually undefined, it's not a property of knex.

Related

How to update postgres jsonb with dynamic variables in nodejs? [duplicate]

I have following setup for my project, using the pg node-postgres package:
The simple table 'tmp' looks like this:
According to jsonORG and the postgres docs the object:
{"foo" : true}
is syntactically valid JSON, and when using the pgAdmin Query-tool with:
UPDATE tmp SET data = '{"foo": false}' WHERE id = '1'
works fine, but when i try updating my table through my express route using pg:
router.put('/updateTMP', (req, res) => {
// I use dummies in this case instead of req.body.someKey for testing purposes
let dummyJSON = {"foo":true};
let dummyID = 1;
pg.query(`UPDATE tmp SET data = '${dummyJSON}' WHERE id = '${dummyID}'`, (errUpdate, responseUpdate) => {
if (!errUpdate) { // NO ERROR
res.json({success: true, message: responseUpdate});
}
else { // ERROR
console.log(dummyJSON);
console.log(errUpdate);
res.json({success: false, message: errUpdate});
}
})
})
I get the following error from the database:
error: invalid input syntax for type json
I've tried the to_json function from postgresql and the to-json package from npm in the express route - all with the same negative result.
Am i missing some fundamental understanding or is it some formating/quoting-issue?
Thanks in advance for your ideas! ;)
ps: And yes - I've read through this, and that article..
I had the same problem. Try converting your JS object to string using JSON.stringify() before passing it into the query as pg won't always do that for you automatically.
See this issue on GitHub for more info.

Mongodb Shell command execution to drop collection via nodejs

I have a test setup in which mongoimport and mongoexportcommands are used to populate an exiting mongoDB database, say testDB from folder testDump.
The problem occurs for the files which are empty in the folder from which testDB is initially populated and then restored.
Eg. a collection file called abcInstance.json is empty in the testDump.
$ cat abcInstance.json
[]
Now when I run some test, this collection gets populated in testDB but at the end when I restore all collections from the testDump folder using mongoimport command it fails for empty files.
So, I am trying to drop those collections using mongo and spawn command.
if (statSync(collectionFile).size === 4) {
const options = [
'testDB',
'--eval',
'"db.abcInstance.drop()"'
];
const dropDB = spawn('mongo', options, { stdio: 'inherit' });
if (dropDB.status !== 0) {
throw new Error('failed to drop collection ');
}}
But this is also failing and I cannot figure out the error.
I have tested that the same command runs successfully on command line:
$ mongo testDB --eval "db.abcInstance.drop()"
MongoDB shell version v3.6.4
connecting to: mongodb://127.0.0.1:27017/alyneKickStartDB
MongoDB server version: 3.6.4
true
Any idea where I am going wrong?
So, I was able to solve the problem of executing the mongo command by a slightly different approach as stated here.
Basically, the problem I figured out was that my parent process was exiting without waiting for the child process to finish execution as both spawn and exec are asynchronous functions. So I modified my code as follows:
const { promisify } = require('util');
const exec = promisify(require('child_process').exec)
async func test () {
const res = await exec('mongo testDB --eval "db.abcInstance.drop()" --quiet')
return { res }
}
Now, when I call this test() function, my collection is successfully dropped.
Does anybody know of any problem with this approach or a better way?

Is DB a reserved word in Node.js

I am building some tests with mocha and chai(expect).
Keeping it simple as I am learning about testing methodology as I go along.
I have a mysql db layer in a config file.
Testing the db parameters, I ran into a weird issue.
These db parameters test fine:
host= 'localhost',
user='foo',
password='bar',
The tests:
var expect = require('chai').expect;
var db = require('../db/config.ini');
describe('Database Access', function() {
it('HOST parameter should be a string', function() {
expect(host).to.be.a('string');
});
it('USER parameter should be a string', function() {
expect(user).to.be.a('string');
});
it('PASSWORD parameter should be a string', function() {
expect(password).to.be.a('string');
});
it('DB parameter should be a string', function() {
expect(db).to.be.a('string');
});
it('HOST parameter should equal localhost', function() {
expect(host).to.equal('localhost');
});
it('USER parameter should equal foo', function() {
expect(user).to.equal('foo');
});
it('PASSWORD parameter should equal bar', function() {
expect(password).to.equal('bar');
});
it('DB parameter should equal thatone', function() {
expect(context).to.equal('thatone');
});
});
When I add the database to choose,
db='thatone';
The test fails the parameter because it reads it as an object.
1) Database Access DB parameter should be a string:
AssertionError: expected {} to be a string
at Context.<anonymous> (test/db_tests.js:21:20)
If I change the variable name to "context" the test passes as expected.
I'm wondering if there is something obvious I am missing about using "db" as a variable.
UPDATE
Really stupid, novice level mistake.
So focused on learning testing methodology I didn't realize I had created the
'db' var as a require to the 'ini' and then referenced it later as though it was unique.
Really dumb. Rushing through this recklessly to get to a destination, and failing to follow some good methodology.
The result of executing this is not a string:
var db = require('../db/config.ini');
It seems you are trying to get a file in some INI dialect to be meaningfully interpreted by Node. Node does not support this by default. If you do not get an error while loading it, the most likely reason is that the text you have in there happens to also be valid JavaScript but since INI files do not contain proper code to export something (i.e. the file does not contain exports.db = "something" or module.exports = { ... } or something similar), then the module has the value {}.
You need to add one of the multiple npm packages that will automatically interpret an INI file and provide a meaningful value. I cannot recommend one as I don't use INI files in my software but you can search npm for a package that will perform the translation for you.

Timeout when attempting to connect to mongo from jest unit tests

I want to write some unit tests with jest and mongoose to validate data interaction with mongo.
I don't want to mock mongoose here because I specifically want to validate the way that mongo documents are created/modified/handled.
package.json is configured to leave node modules unmocked:
{
"jest": {
"unmockedModulePathPatterns": [
"node_modules"
]
}
}
In my actual test, I have set up a beforeAll() hook to take care of connecting to mongo:
const mongoose = require('mongoose');
describe('MyTest', () => {
beforeAll((done) => {
mongoose.connect('mongodb://127.0.0.1:27017/test');
let db = mongoose.connection;
db.on('error', (err) => {
done.fail(err);
});
db.once('open', () => {
done();
});
});
it('has some property', () => {
// should pass but actually never gets run
expect(1).toBe(1);
});
});
Here's the output:
/usr/local/bin/node node_modules/jest-cli/bin/jest.js --verbose
Using Jest CLI v0.10.0, jasmine2
FAIL src/lib/controllers/my-controller/__tests__/my-test.js (5.403s)
MyTest
✕ it has some property
MyTest › it has some property
- Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
at Timer.listOnTimeout (timers.js:92:15)
1 test failed, 0 tests passed (1 total in 1 test suite, run time 5.903s)
Process finished with exit code 1
The test times out every time because done() is never called in the beforeAll() hook - no errors thrown nor anything printed to console. I can place breakpoints in the beforeAll() hook to validate the code is being run. It appears that mongoose is hanging whilst attempting to open a connection to Mongo, and then the jest tests are timing out.
When I'm running similar code outside of the jest environment, it connects as expected (nigh on instantly). Suspecting that it could be causing problems, I've experimented with disabling jest's automock feature completely, but the behaviour remains unchanged.
I imagine I've missed something incredibly obvious... any ideas what could be going on?
jest-cli v. 0.10.0
mongoose v. 4.4.11
Updates:
Tried replacing ES6 arrow function syntax with plain function(done) {}. No difference.
Tried making the test async by passing done parameter, and calling it on test completion. No difference.
Tried calling mongoose.connect() after the declaration of error and connected event handlers
Tried commenting out all mongoose-related code to check that the beforeAll() hook is working correctly - it is.
Jest-jasmine is different from Jasmine.
The syntax you used actually belongs to Jasmine 2.0+, not Jest-jasmine. So if you want to continue to use jest, you have to look into jest docs to find the answer.
Even more, "beforeAll" is not a standard jest injected variables, see jest api.
I got your code run with Jasmine 2.3.4, it runs just fine. I tried Promise/pit in jest to finish this job but failed.
First, install jasmine.
npm install -g jasmine
mkdir jasmine-test
cd jasmine-test
jasmine init
jasmine examples
touch spec/mongodbspec.js
Here is my directory structure:
fenqideMacBook-Pro:jasmine-test fenqi$ ls -R
.:
spec/
./spec:
mongodbspec.js support/
./spec/support:
jasmine.json
Then, edit spec/mongodbspec.js, I just add one line " 'use strict'; " to your code.
'use strict';
const mongoose = require('mongoose');
describe('MyTest', () => {
beforeAll((done) => {
mongoose.connect('mongodb://127.0.0.1:27017/test');
let db = mongoose.connection;
db.on('error', (err) => {
done.fail(err);
});
db.once('open', () => {
done();
});
});
it('has some property', () => {
// should pass but actually never gets run
expect(1).toBe(1);
});
});
Last, run 'jasmine' :
fenqideMacBook-Pro:jasmine-test fenqi$ jasmine
Started
.
1 spec, 0 failures
Finished in 0.023 seconds

Mongoose Trying to open unclosed connection

This is a simplified version of the problem, but basically I'm trying to open 2 mongodb connections with mongoose and it's giving me "Trying to open unclosed connection." error.
Code sample:
var db1 = require('mongoose');
db1.connect('my.db.ip.address', 'my-db');
var db2 = require('mongoose');
db2.connect('my.db.ip.address', 'my-db');
db2.connection.close();
db1.connection.close();
Any idea how to make it work?
connect() opens the default connection to the db. Since you want two different connections, use createConnection().
API link: http://mongoosejs.com/docs/api.html#index_Mongoose-createConnection
To add on Raghuveer answer :
I would also mention that instead of using mongoose directly (you are probably using it this way you end up on this post) :
require('mongoose').model(...);
You would use the returned connection :
var db = require('mongoose').connect('xxx', 'yyy');
db.model(...);
I get this issue while running my tests.
This is what I did to solve it.
//- in my app.js file.
try {
mongoose.connect('mongodb://localhost/userApi2'); //- starting a db connection
}catch(err) {
mongoose.createConnection('mongodb://localhost/userApi2'); //- starting another db connection
}
I had this problem doing unit test with mocha.
The problem came when I added a second test because beforeEach is called twice.
I've solved this with this code:
const mongoose = require('mongoose');
describe('Your test suite', () => {
beforeEach( () => {
if (mongoose.connection.db) {
return; // or done();
} else {
// connect to mongodb
});
describe('GET /some-path', () => {
it('It should...', () => {
});
});
describe('POST /some-path', () => {
it('It should...', () => {
});
});
});
Hope it helps you!
You are attempting to open the default connection ( which is not yet closed ) a 2nd time.
do the following instead
var db = require('mongoose'); //note only one 'require' needed.
var connectionToDb1 = db.createConnection('my.db1.ip.address', 'my-db1');
var connectionToDb2 = db.createConnection('my.db2.ip.address', 'my-db2');
Using mongoose.disconnect(fn):
mongoose.disconnect(() => {
// here it would be possible "reset" models to fix
// OverwriteModelError errors
mongoose.models = {};
// here comes your logic like registering Hapi plugins
server.register(somePlugin, callback);
});
I found this question typing the error message and despite my problem is a bit different I believe it could be useful for those using Hapi. More specifically Hapi + rest-hapi + mocha.
When running mocha with --watch option I was facing both: OverwriteModelError and Error: Trying to open unclosed connection errors.
Simple Solution -
Use mongoose.createConnection() instead of mongoose.connect()
Its occurs because of version issue

Resources