Sequelize CLI & DB migrations with Typescript - node.js

Sequelize docs claim that it works with Typescript but in order for it to be useful in production one needs to use DB migration scripts. These can only be executed using the Sequelize CLI but this CLI seems to have no regard for Typescript at all. It only generates (and apparently runs) JS files. There is a "sequelize-cli-typescript" project on NPM but it's 4 years old! And the only answer on Stack Overflow to a related question is literally 6 words long, none of them useful: Using sequelize cli with typescript
My setup is like so:
I created a basic Node app with Typescript and a simple tsconfig.js like so:
{
"compilerOptions": {
"allowJs": true,
"module": "commonjs",
"esModuleInterop": true,
"target": "es6",
"noImplicitAny": true,
"moduleResolution": "node",
"sourceMap": true,
"outDir": "dist",
"baseUrl": ".",
"paths": {
"*": [
"node_modules/*"
]
}
},
"include": [
"*"
]
}
I'm also using ESLint with the following .eslintrc:
{
"root": true,
"parser": "#typescript-eslint/parser",
"plugins": [
"#typescript-eslint"
],
"extends": [
"airbnb-base", "airbnb-typescript/base"
],
"parserOptions": {
"project": "./tsconfig.json"
}
}
I then tried running the Sequelize CLI init command, npx sequelize-cli init as described in the DB migrations documentation. This caused a file models/index.js to be created with the following content:
'use strict';
const fs = require('fs');
const path = require('path');
const Sequelize = require('sequelize');
const basename = path.basename(__filename);
const env = process.env.NODE_ENV || 'development';
const config = require(__dirname + '/../config/config.json')[env];
const db = {};
let sequelize;
if (config.use_env_variable) {
sequelize = new Sequelize(process.env[config.use_env_variable], config);
} else {
sequelize = new Sequelize(config.database, config.username, config.password, config);
}
fs
.readdirSync(__dirname)
.filter(file => {
return (file.indexOf('.') !== 0) && (file !== basename) && (file.slice(-3) === '.js');
})
.forEach(file => {
const model = require(path.join(__dirname, file))(sequelize, Sequelize.DataTypes);
db[model.name] = model;
});
Object.keys(db).forEach(modelName => {
if (db[modelName].associate) {
db[modelName].associate(db);
}
});
db.sequelize = sequelize;
db.Sequelize = Sequelize;
module.exports = db;
This causes my IDE (VS Code) to show an error:
Parsing error: "parserOptions.project" has been set for #typescript-eslint/parser.
The file does not match your project config: models/index.js.
The file must be included in at least one of the projects provided.
I have no idea what this is supposed to mean. I've read numerous explanations and none makes sense to me. I have allowJs set to true in tsconfig and include all. What am I missing? More importantly how can I use Typescript for DB migrations and CLI generated code?

As for generated code for connecting to DB and registering models you should convert it manually to ts (a one-time thing usually).
As for migrations I use the following workaround:
configure migrations path in .sequelizerc to a subfolder in the folder where the compiled app will be placed by TS (it's important in order to be able to apply migrations on testing and prod environments), something like:
const { resolve } = require('path');
module.exports = {
config: resolve('build/application/config.js'),
'seeders-path': resolve('build/application/seeders'),
'migrations-path': resolve('build/application/migrations'),
'models-path': resolve('application/models')
};
generate a migration file with a given name:
sequelize migration:create --name add-some-table
move the generated file to application/migrations and change its extension to .ts
replace the content of the file with some kind of a migration Typescript-template:
import { QueryInterface, DataTypes, QueryTypes } from 'sequelize';
module.exports = {
up: (queryInterface: QueryInterface): Promise<void> => queryInterface.sequelize.transaction(
async (transaction) => {
// here go all migration changes
}
),
down: (queryInterface: QueryInterface): Promise<void> => queryInterface.sequelize.transaction(
async (transaction) => {
// here go all migration undo changes
}
)
};
add necessary changes to the migration file
Basically, it's possible to write some script file to simply generate .ts migration file copying a predefined template file in application/migrations folder directly with a given name to combine steps 2-4 into a single step.
If you will come up with a better solution eventually feel free to share it in the comments or even as a separate answer here.

Related

Is it possible to collect coverage on code loaded with vm.runInContext with Jest?

I'm working on a legacy JS project which is not using any require/import. When deploying, the files are just concatenated and the result is sent to a server.
In order to write tests with jest, I created a custom environment to load all the JS files in the global context so that I can call the functions in the test file.
For example:
src/index.js
function sum(x, y) {
return x + y;
}
src/index.spec.js
it('should sum two numbers', () => {
expect(sum(1, 2)).toBe(3);
});
jest.config.js
module.exports = {
clearMocks: true,
collectCoverage: true,
collectCoverageFrom: [
"src/**/*.js",
],
coverageDirectory: "coverage",
coverageProvider: "v8",
testEnvironment: "./jest.env.js",
};
jest.env.js
const NodeEnvironment = require('jest-environment-node').TestEnvironment;
const fs = require('fs');
const vm = require("vm");
const path = require("path");
class CustomEnv extends NodeEnvironment {
constructor(config, context) {
super(config, context);
this.loadContext();
}
loadContext() {
const js = fs.readFileSync('./src/index.js', 'utf8');
const context = vm.createContext(this.global);
vm.runInContext(js, context, {
filename: path.resolve('./src/index.js'),
displayErrors: true,
});
Object.assign(this.global, context);
}
}
module.exports = CustomEnv;
When I run npx jest, the test is executed but the coverage is empty...
Any idea on how to fix the coverage?
I've created a minimal reproducible repo here: https://github.com/GP4cK/jest-coverage-run-in-context/tree/main. You can just clone it, run npm i and npm t.
Note: I'm happy to change v8 to babel or load the context differently if it makes it easier.

Sequelize on NodeJS/ExpressJS return an error when running CLI command db:migrate

I'm following this tutorial Using PostgreSQL and Sequelize to persist our data on medium, and right now I'm stuck at the db:migrate. it's returning this error
Sequelize CLI [Node: 12.1.0, CLI: 5.4.0, ORM: 5.8.2]
Loaded configuration file "config.json".
Using environment "development".
ERROR: Error parsing url: undefined
as you can see I'm using NodeJS version 12.1.0 and Sequelize CLI version 5.4.0 and Sequelize version 5.8.2 and all of them were the latest version.
and before running sequelize db:migrate, I'm running this command first SET DATABASE_URL=postgresql://[user[:password]#][netlocation][:port][/dbname] and it does not returns any error.
but it's returning error after I ran db:migrate
I already tried to find the problem, but I can't found the answer yet.
Here is my ./Models/Index.js file.
'use strict';
require('dotenv').config();
import { readdirSync } from 'fs';
import { basename as _basename, join } from 'path';
import Sequelize from 'sequelize';
const basename = _basename(__filename);
const env = process.env.NODE_ENV || 'development';
const config = require(__dirname + '/../../config.json')[env];
const db = {};
let sequelize;
if (config.use_env_variable) {
sequelize = new Sequelize(process.env[config.use_env_variable], config);
} else {
sequelize = new Sequelize(config.database, config.username, config.password, config);
}
readdirSync(__dirname)
.filter(file => {
return (file.indexOf('.') !== 0) && (file !== basename) && (file.slice(-3) === '.js');
})
.forEach(file => {
const model = sequelize['import'](join(__dirname, file));
db[model.name] = model;
});
Object.keys(db).forEach(modelName => {
if (db[modelName].associate) {
db[modelName].associate(db);
}
});
db.sequelize = sequelize;
db.Sequelize = Sequelize;
export default db;
if you realize I just changed it to ES6 format which change some codes, but before I change it to ES6, it doesn't work either. and for all the rest of the files I following the tutorial.
Here are the files that I think have a connection:
.env
DATABASE_URL=postgres://postgres:admin#localhost:5432/test_app
.sequelizerc
const path = require('path');
module.exports = {
"config": path.resolve('./config.json'),
"models-path": path.resolve('./app/Models'),
"migrations-path": path.resolve('./migrations')
};
config.json
{
"development": {
"use_env_variable": "DATABASE_URL"
},
"test": {
"use_env_variable": "DATABASE_URL"
},
"production": {
"use_env_variable": "DATABASE_URL"
}
}
If there are some files that I haven't included yet please tell me, and please help me to fix find the solution for this problem. Thank you
OS: Windows 10
Basically you are unable to set environment variable DATABASE_URL successfully.
I am not a Windows guy, but this should do your job.
If you are using GitBash, then it is as simple as:
export DATABASE_URL=postgres://postgres#localhost:5432/database_name
and after that:
node_modules/.bin/sequelize db:migrate
EDIT:
I am not sure how to set this variable in gitbash and cmd.
Here is an alternate.
in config/config.json
"development": {
"username": "postgres"
"password": "postgres",
"database": "your_db_here",
"host": "127.0.0.1",
"dialect": "postgres"
},
update these variables according to your postgres db.
and run:
node_modules/.bin/sequelize db:migrate
You cannot fetch values at runtime inside config.json. It has to be static.
You should either use config.json or env variables or roll your own like mentioned in another answer.
To use env variables, you will eschew config.json. Instead, in models/index.js, set
if (config.use_env_variable) {
sequelize = new Sequelize(process.env[config.use_env_variable], config);
} else {
sequelize = new Sequelize(config.database, config.username, config.password, config);
}
to
sequelize = new Sequelize(process.env.DATABASE_URL)
AFAIK Sequelize migrations are a different beast than the normal sequelize workflow.
It is reading config/config.json when it loads - so you cannot use system environment variables - it has to be a static json file.
What I do in my projects, is having my config.js file making sure the config file is up to date with whatever settings I have.
I do this when the main program starts and also in package.json as follows:
(make sure to add npm-run-all to your package.json)
"scripts": {
"config": "node src/config.js",
"_migrate": "sequelize db:migrate",
"_migrate:status": "sequelize db:migrate:status",
"_migrate:undo": "sequelize db:migrate:undo",
"_seed": "sequelize db:seed:all",
"migrate": "npm-run-all config _migrate",
"migrate:status": "npm-run-all config _migrate:status",
"migrate:undo": "npm-run-all config _migrate:undo",
"seed": "npm-run-all config _seed"
},
config.js simply does something similar to this at the end of the file:
// Export sequelize config/config.json for easy compatibality with sequelize-cli
const filepath = path.resolve(__dirname, '../../config');
const filename = path.join(filepath, 'config.json');
fs.ensureDir(filepath)
.then(() => fs.writeFileSync(filename, JSON.stringify(sequelizeConfig, 2) + '\n'))
.catch((err) => console.error(`Failed to write config: ${err}`));
sequelizeConfig is should be the fully generated sequelize config object. You can also have a generic one like you have now, and build upon it.

Migrate Node.js project to TypeScript from plain ES6

Is started migrating a Node.js project from plain ES6 to TypeScript.
What I did:
npm install -g typescript
npm install #types/node --save-dev
Setup tsconfig.json:
{
"compilerOptions": {
"emitDecoratorMetadata": true,
"experimentalDecorators": true,
"moduleResolution": "node",
"module": "commonjs",
"target": "es6",
"sourceMap": true,
"outDir": "dist",
"allowJs": true,
"forceConsistentCasingInFileNames": true
},
"exclude": [
"node_modules",
"dist",
"docs"
]
}
Change all file extensions from .js to .ts (except in node_modules):
find . -not \( -path node_modules -prune \) -iname "*.js" -exec bash -c 'mv "$1" "${1%.js}".ts' - '{}' \;
Running tsc now leads to a ton of errors like these:
server.ts:13:7 - error TS2451: Cannot redeclare block-scoped variable 'async'.
13 const async = require('async');
~~~~~
Or these:
bootstrap/index.ts:8:7
8 const async = require('async');
~~~~~
'async' was also declared here.
Update:
The same happens for retry and other npm packages:
const retry = require('retry');
Changing require statements to ES6 import statements mostly fixed these but having to migrate a few thousands files at once is not feasable so I need a way to stick with require for a while. Is this possible?
It's possible, but you'll still have to edit those files.
Either of those methods will be enough.
Replace const ... = require() with import ... = require():
import async = require('async');
...
Add export {} to the top of the file:
export {};
const async = require('async');
...
The reason of initial issue is that in TS different files are not modules unless they explicitly declared as modules, thus they are compiled/executed in the same global scope, and that's why tsc is reporting you that async variable can't be redeclared.
From documentation:
In TypeScript, just as in ECMAScript 2015, any file containing a top-level import or export is considered a module. Conversely, a file without any top-level import or export declarations is treated as a script whose contents are available in the global scope (and therefore to modules as well).
This is the same problem as this one.
In order to be treated as ES module, a file should contain either import or export statement, otherwise a variable is considered to be declared in global scope by TypeScript compiler (even if this is not so at runtime).
The solution is same as in linked question, to add dummy export {}. This could be done in batch with regex replacement but in case CommonJS , module.exports = ... exports are already in use, there may be a conflict between them.
The use of CommonJS require() imports results in untyped code. All major libraries already have according #types/... or built-in typings. Existing NPM packages can be matched with a regex from code base in order to install relevant #types/... packages in batch, imports like const async = require('async') can be replaced in batch with import async from 'async'. This requires esModuleInterop and allowSyntheticDefaultImports options to be set.
async is a protected keyword. When you use async/await you might skip the 'async' package. If you made ES6+ properly with ECMAScript modules (ESM) you also renamed all your files *.mjs, for example index.mjs. If you have the filename index.js it is most often assumed NOT to be ESM. You have to add types / interfaces to all your ES6 code, so depending on your case it might not be feasible to make all at once, that's why I give the example in ES2015+ ESM notation.
For TypeScript you should be able to use ESM because I guess you want more up to date notation. In order to use async at top level, the async function exist for doing that. Example code for index.mjs that include ES2015+ import from ES5/CommonJS *.js with module.exports and ESM import/export and finally dynamic import:
import { createRequireFromPath } from 'module'; // ESM import
import { fileURLToPath } from 'url';
const require = createRequireFromPath(fileURLToPath(import.meta.url));
// const untypedAsync = require('async');
class Index {
constructor() {
this._server = null;
this.host = `localhost`;
this.port = 8080;
}
set server(value) { this._server = value; }
get server() { return this._server; }
async start() {
const http = await import(`http`); // dynamic import
this.server = http.createServer(this.handleRequest);
this.server.on(`error`, (err) => {
console.error(`start error:`, err);
});
this.server.on(`clientError`, (err, socket) => {
console.error(`start clientError:`, err);
if (socket.writable) {
return socket.end(`HTTP/1.1 400 Bad Request\r\n\r\n`);
}
socket.destroy();
});
this.server.on(`connection`, (socket) => {
const arrival = new Date().toJSON();
const ip = socket.remoteAddress;
const port = socket.localPort;
console.log(`Request from IP-Address ${ip} and source port ${port} at ${arrival}`);
});
this.server.listen(this.port, this.host, () => {
console.log(`http server listening at ${this.host}:${this.port}`);
});
}
handleRequest(req, res) {
console.log(`url:`, req.url);
res.setHeader(`Content-Type`, `application/json`);
res.writeHead(200);
res.end(JSON.stringify({ url: req.url }));
}
}
export default Index; // ESM export
export const randomName = new Index(); // Usage: import { randomName } from './index.mjs';
async function main() {
const index = new Index();
const cjs = require(`./otherfile.js`); // ES5/CommonJS import
console.log(`otherfile:`, cjs);
// 'async' can be used by using: cjs.untypedAsync
await index.start();
}
main();
// in otherfile.js
const untypedAsync = require('async');
const test = {
url: "url test",
title: "title test",
};
module.exports = { test, untypedAsync }; // ES5/CommonJS export.
However, to use .mjs with typescript currently have some issues. Please look at the related typescript issues that are still open: .mjs input files and .mjs output files. You should at least transpile your .ts to .mjs to solve your problems. The scripts might look like (es6 to ts source):
// in package.json
"files": [ "dist" ],
"main": "dist/index",
"types": "dist/index.d.ts",
"scripts": {
"mjs": "tsc -d && mv dist/index.js dist/index.mjs",
"cjs": "tsc -m commonjs",
"start": "node --no-warnings --experimental-modules ./dist/index.mjs"
"build": "npm run mjs && npm run cjs"
},
"devDependencies": {
"typescript": "^3.2.2"
}
// in tsconfig.json
"compilerOptions": {
"module": "es2015",
"target": "ES2017",
"rootDir": "src",
"outDir": "dist",
"sourceMap": false,
"strict": true
}
Since you are migrating a large project to TypeScript, I would suggest using some tool like this package (https://www.npmjs.com/package/javascript-to-typescript) which could automate some of the work.
You can write a script to open each file in the project and add export {} at the top as suggested by #Styx in his answer.

typescript error running navalia example

I am trying to run this example from https://github.com/joelgriffith/navalia but for the light of me, I couldn't get it to work without error:
navaliatest.ts
/// <reference path="typings.d.ts" />
import { Chrome } from 'navalia';
const chrome = new Chrome();
async function buyItOnAmazon() {
const url = await chrome.goto('https://amazon.com');
const typed = await chrome.type('input', 'Kindle');
const clicked = await chrome.click('.nav-search-submit input');
chrome.done();
console.log(url, typed, clicked); // 'https://www.amazon.com/', true, true
}
buyItOnAmazon();
tsconfig.json
{
"files": [
"navaliatest.ts"
],
"compilerOptions": {
"noImplicitAny": false,
"target": "es6",
"moduleResolution": "node",
"paths": {
"*" : ["/usr/local/lib/node_modules/*"]
}
}
}
typings.d.ts
/// <reference path="/usr/local/lib/node_modules/navalia/build/Chrome.d.ts" />
declare module 'navalia' {
var Chrome: any;
export = Chrome;
}
Below are the versions:
MacBook-Pro:testcasperjs myusername$ node --version
v6.11.2MacBook-Pro:testcasperjs myusername$ npm --version
3.10.10
MacBook-Pro:testcasperjs myusername$ tsc --version
Version 2.4.2
This is the error I got although I do get .js file output:
MacBook-Pro:testcasperjs myusername$ tsc navaliatest.ts
../../../usr/local/lib/node_modules/navalia/node_modules/chrome-launcher/chrome-finder.ts(203,16): error TS2339: Property 'from' does not exist on type 'ArrayConstructor'.
../../../usr/local/lib/node_modules/navalia/node_modules/chrome-launcher/chrome-launcher.ts(99,15): error TS1056: Accessors are only available when targeting ECMAScript 5 and higher.
navaliatest.ts(3,10): error TS2305: Module ''navalia'' has no exported member 'Chrome'.
I am sure there is a stupid mistake somewhere but please could someone help me and take a look? Thanks.
You don't need to redeclare navalia. It has already been done for you at node_modules/navalia/build/index.d.ts given that moduleResolution is set to Node
You'll need to set module to commonjs so that you can run it in node
tsconfig.json
{
"files": [
"navaliatest.ts"
],
"compilerOptions": {
"noImplicitAny": false,
"target": "es6",
"module": "commonjs",
"moduleResolution": "Node"
}
}
navaliatest.ts (No change)
import { Chrome } from 'navalia';
const chrome = new Chrome();
async function buyItOnAmazon() {
const url = await chrome.goto('https://amazon.com');
const typed = await chrome.type('input', 'Kindle');
const clicked = await chrome.click('.nav-search-submit input');
chrome.done();
console.log(url, typed, clicked); // 'https://www.amazon.com/', true, true
}
buyItOnAmazon();
It'll create navaliatest.js with no errors, which can be run in node.

Typescript - tsc compiler breaks my code

I wrote this code in Typescript
import redis from 'redis';
import Promise from 'bluebird';
const DEFAULT_REDIS_TTL = 7200; // 2 hours
export default class Redis {
readonly client : any;
ttl : number = DEFAULT_REDIS_TTL;
constructor(uri? : string, ttl : number = DEFAULT_REDIS_TTL) {
this.client = redis.createClient(uri);
}
...
}
export { Redis };
the compiler gives me this
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
var redis_1 = require("redis");
var bluebird_1 = require("bluebird");
var DEFAULT_REDIS_TTL = 7200; // 2 hours
var Redis = (function () {
function Redis(uri, ttl) {
if (ttl === void 0) { ttl = DEFAULT_REDIS_TTL; }
this.ttl = DEFAULT_REDIS_TTL;
this.client = redis_1.default.createClient(uri);
this.client.on('error', function (error) {
console.error(error);
});
...
exports.Redis = Redis;
exports.default = Redis
I don't know why 'redis.createClient(uri);just becomeredis_1.default.createClient(uri);`
I get the following error when trying to run my code in node
build/Lib/Cache/Redis.js:11
this.client = redis_1.default.createClient(uri);
^
TypeError: Cannot read property 'createClient' of undefined
my tsconfig looks like this
{
"compilerOptions": {
"module": "mymodule",
"target": "es5",
"noImplicitAny": false,
"sourceMap": false,
"module": "commonjs",
"outDir": "build"
},
"include": [
"src/**/*.ts"
],
"exclude": [
"node_modules"
]
}
I run the compiler in main directory
tsc
I'm using node 6.7.2
Change your import to:
import * as redis from 'redis';
I don't think the typings for redis has a default export. Make sure you have the latest typings.
If you have the latest typings, import redis from 'redis'; should throw a compile time error.

Resources