serverless webpack config, manually add directory - node.js

I have the following code I am running that is failing to work because it can not find the models folder when I upload to amazon.
exports.setModels = function(connection,modelPath){
//Import all the known models for the project.
//Proof of Stage being set.
console.log("stage for models="+stage);
const fs = require('fs');
const dir = modelPath;
var models = {};
//#JA - Wait until this function finishes ~ hence readdirSync vs regular readdir which is async
fs.readdirSync(dir).forEach(file => {
console.log("file="+file);
//Split the .js part of the filename
var arr = file.split(".");
var name = arr[0].toLowerCase();
//Create a modle object using the filename as the reference without the .js pointing to a created sequelize instance of the file.
var modelPath = "../models/"+file; //default assumes same directory that was used to scan.
if(process.env.DOMAIN_NAME){ //If this enviroment variable is detected then change the modelPath.
modelPath = "../../../../models/"+file;
}
models[name] = connection.import(modelPath);
})
return models;
}
After investigating the issue, I found out its because the models folder is not being packaged by the serverless webpack plugin.
I found out recently how to force certain packages to upload using this code in my serverless file.
webpackIncludeModules:
forceInclude:
- mysql
- mysql2
This will only seem to include packages however, and when I tried to reference the models folder to automatically include all my sequelize models I got an error saying it was not a package, which of course makes sense.
This does leave me with the question of how I can get it to package the models directory without manually doing a require for each model. I wrote a dynamic function to get them at runtime and import it to sequelize.
Information on the plugin is here (https://github.com/serverless-heaven/serverless-webpack), I looked through it all but can't seem to find the answer.
The output of packaging with serverless looks like this, missing the models folder with all my sequelize models.
My main directory before doing webpack looks like this.
Here is my webpack.config.js as well.
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");
module.exports = {
entry: slsw.lib.entries,
target: "node",
// Since 'aws-sdk' is not compatible with webpack,
// we exclude all node dependencies
externals: [nodeExternals()],
// Run babel on all .js files and skip those in node_modules
module: {
rules: [
{
test: /\.js$/,
loader: "babel-loader",
include: __dirname,
exclude: /node_modules/
}
]
}
};

you could use the copy-webpack-plugin to include the models instead of referencing.
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");
const CopyWebpackPlugin = require('copy-webpack-plugin');
module.exports = {
entry: slsw.lib.entries,
target: "node",
// Since 'aws-sdk' is not compatible with webpack,
// we exclude all node dependencies
externals: [nodeExternals()],
// Run babel on all .js files and skip those in node_modules
module: {
rules: [
{
test: /\.js$/,
loader: "babel-loader",
include: __dirname,
exclude: /node_modules/
}
]
},
plugin: [
new CopyWebpackPlugin(filesToCopy, {})
]
};
make sure you import the models from the right folder since you are importing from relative path. This might change if you're bundle is in a different folder.

Related

Webpack generates js files with css / scss files

Description
In webpack I am using mini-css-extract-plugin:
plugins: [
new MiniCssExtractPlugin({
filename: '[name].[hash].css',
chunkFilename: '[name].[hash].css',
})
]
To load scss files in chunk files:
{
test: /\.scss$/,
use: [
{ loader: MiniCssExtractPlugin.loader, options: {
hmr: isdev,
reloadAll: true
}
},
"css-loader",
"sass-loader",
]
}
When I load a scss with an dynamic import:
import(/* webpackChunkName: "test" */ 'test.scss')
It will generate a test.[hash].css containing the styles and a test.[hash].js:
(window["webpackJsonp"] = window["webpackJsonp"] || []).push([[15],{
/***/ 81:
/***/ (function(module, exports, __webpack_require__) {
// extracted by mini-css-extract-plugin
/***/ })
}]);
Problem
I want to minimize the delay and loaded files so I find it redundant to have a nearly empty test.[hash].js file.
Do you have a way to either include the scss in the js file (see Idea 1) or to not emit/use the nearly empty js file?
Idea 1: not using mini-css-extract-plugin
My first idea was not using mini-css-extract-plugin for dynamic imported scss, but this will include a lot css-base stuff in the js (https://github.com/webpack-contrib/extract-text-webpack-plugin/issues/255).
Here is an extract of code that could interrest you. It's coded in live here, so there is maybe some error I don't know.
I use an alternative way but nearby inside my own project.
The behaviour is :
Use the Event Hook Plugin and call it when the webpack is done
Loop through each file
If file is css and have the same name as with js extension
Then remove the js file
const EventHooksPlugin = require('event-hooks-webpack-plugin');
const path = require('path');
const fs = require('fs');
const _ = require('underscore');
plugins: [
new EventHooksPlugin({
done: () => {
const publicDir = __dirname + '/public';
const files = fs.readdirSync(publicDir);
_.each(files, file => {
if (path.extname(file) !== '.css') { return ;}
const fileJs = file.replace('.css', '.js');
if (!fs.existsSync(fileJs)) {return;}
fs.unlinkSync(fileJs);
});
}
})
]

Webpack Usage of chunkFilename

Iam having trouble understand the use of chunkFilename property. Where do we need it?
I cant find any usage in webpack docs.
const path = require('path');
const config = {
entry:['./util.js','./index.js'],
output: {
path: path.resolve(__dirname, 'public'),
filename: '[name].bundle.js',
chunkFilename: '[name].chunk.js'
},
module: {
rules: [
{ test: /\.js$/, exclude:/node_modules/ }
]
}
};
module.exports = config;
chunkFilename is used for naming chunks :]
Chunks are created using dynamic import such as import('./file') or webpacks require.ensure() syntax.

How to include a few node_modules package in babel-node

I'm trying to include #mycompany/package1, and #mycompany/package2 to be compiled along with the rest of my code using babel-node. Since package1 and package2 are in ES6. (Also note I'm not using Webpack)
In my jest config I added the below option into my jest config which works fine. When testing the code will compile the packages correctly
"transformIgnorePatterns": [
"/node_modules/(?!(#mycompany)/).*/"
],
But when trying to run babel-node I get errors.
In my babel.config.js
module.exports = {
presets: [
'#babel/preset-flow',
[
'#babel/preset-env',
{
targets: {
node: 8
}
}
]
],
plugins: ['#babel/plugin-proposal-class-properties']
};
I tried adding the below code to my babel.config.js but it still complains about ES6 errors within my node_modules/#mycompany/package1
I tried to include the viz package but then babel wouldn't compile my src files
include: [path.resolve(__dirname, 'node_modules/#mycompany/package1')]
include: ['/node_modules/((#mycompany)/).*/']
I tried to exclude everything but #mycompany packages but I still get transpile errors in my package1
exclude: [/node_modules\/(?!(#mycompany)\/).*/],
I tried playing with ignore but those don't seem like they are the right options based on reading the docs
I found out that we can do this with webpack to help bundle the packages with the rest of your code.
This is my webpack file for NodeJS.
const path = require('path');
const nodeExternals = require('webpack-node-externals');
const webpack = require('webpack');
const spawn = require('child_process').spawn;
const nodeEnv = process.env.NODE_ENV;
const isProduction = nodeEnv === 'production';
const compiler = webpack({
entry: ['#babel/polyfill', './src/server.js'],
output: {
path: path.resolve(__dirname, 'lib'),
filename: 'server.bundle.js',
libraryTarget: 'commonjs2'
},
externals: [
nodeExternals({
whitelist: [/#mycompany\/.*/]
})
],
plugins: plugins,
target: 'node',
mode: 'development',
module: {
rules: [
{
test: /\.js$/,
exclude: /node_modules\/(?!(#mycompany)\/).*/,
use: {
loader: 'babel-loader',
options: {
configFile: './babel.config.js'
}
}
}
]
}
});
if (isProduction) {
compiler.run((err, stats) => {
if (err) {
console.error(err);
return;
}
console.log(
stats.toString({
colors: true
})
);
});
} else {
let serverControl;
compiler.watch(
{
aggregateTimeout: 300,
poll: 1000
},
(err, stats) => {
if (serverControl) {
serverControl.kill();
}
if (err) {
console.error(err);
return;
}
console.log(
stats.toString({
colors: true
})
);
// change app.js to the relative path to the bundle created by webpack, if necessary
serverControl = spawn('node', [
path.resolve(__dirname, 'lib/server.bundle.js')
]);
serverControl.stdout.on('data', data => console.log(data.toString()));
serverControl.stderr.on('data', data => console.error(data.toString()));
}
);
}
Note the most important part is
Adding webpack-node-externals. Since this is a node.js server we don't need to bundle the node_modules.
Make sure you whitelist your package that you need to be compiled/bundled and also make sure you have your packages included to be compiled in your babel-loader
nodeExternal tells webpack know not to bundle ANY node_modules.
whitelist is saying that we should bundle the packages we listed
externals: [
nodeExternals({
whitelist: [/#mycompany\/.*/]
})
]
This line means to exclude all node_modules EXCEPT #mycompany/* packages
exclude: /node_modules\/(?!(#mycompany)\/).*/,

Webpack dll import is not defined

I am trying to put a bunch of libs on the DLL following guides like react-boilerplate and this one.
When I build and run, the DLL files are given as not defined.
I'm probably missing something I did a separated webpack to build the dll:
import webpack from 'webpack'
const library = '[name]'
export default {
entry: {
'lokka': ['lokka', 'lokka-transport-http', 'socket.io-client']
/** Other libs **/
},
output: {
filename: '[name].dll.js',
path: 'build/',
library: library
},
plugins: [
new webpack.DllPlugin({
path: 'build/[name]-manifest.json',
name: library
})
]
}
And added the references to the manifest.json
import webpack from 'webpack'
const desiredLibs = [
'lokka'
]
const plugins = desiredLibs.map((lib) => {
return new webpack.DllReferencePlugin({
context: process.cwd(),
manifest: require(`../build/${lib}-manifest.json`)
})
})
export const dllReference = () => {
return { plugins }
}
export default dllReference
Was there anything else I should do?
On my case, it is complaining that lokka is not found when the code is run.
Turns out I (obviously) need to include the generated DLL on my scripts src AND copy it in the case of dev, since hot reloading would only serve the entry of it and it's dependencies, so for the dllReference and copy part it became:
import webpack from 'webpack'
import CopyWebpackPlugin from 'copy-webpack-plugin'
import path from 'path'
const desiredLibs = ['lokka', 'react', 'moment']
const copies = []
const plugins = desiredLibs.map((lib) => {
copies.push({
from: path.join(__dirname, `../compileResources/${lib}.dll.js`),
to: `dll`
})
return new webpack.DllReferencePlugin({
context: process.cwd(),
manifest: require(`../compileResources/${lib}-manifest.json`)
})
})
plugins.push(
new CopyWebpackPlugin(copies)
)
/**
* Adds the dll references and copies the file
*/
export const dllReference = () => {
return { plugins }
}
export default dllReference
And then since I copied the dll's using the copy plugin I needed to add the scripts on the html. Really obvious on hindsight

Webpack bundling doesn't include specific module such as 'sequelize'

I'm developing nodejs with webpack, seqelize and other modules with below webpack configuration.
const path = require('path');
const webpack = require('webpack');
const fs = require('fs');
const glob = require('glob');
const CleanWebpackPlugin = require('clean-webpack-plugin');
var nodeModules = {};
fs.readdirSync('node_modules')
.filter(function(x) {
return ['.bin'].indexOf(x) === -1;
})
.forEach(function(mod) {
nodeModules[mod] = 'commonjs ' + mod;
});
module.exports = {
// entry: [ path.resolve(__dirname, 'server.js'), models ],
entry: glob.sync(path.resolve(__dirname, 'src/**/*.js')),
resolve: {
// Add `.ts` and `.tsx` as a resolvable extension.
root : [path.resolve(__dirname, '')],
extensions: ['', '.webpack.js', '.web.js', '.ts', '.tsx', '.js'],
modulesDirectories: ['node_modules']
},
module: {
loaders: [
{
test: /\.js$/,
loader: 'babel',
exclude: /node_modules/,
query: {
cacheDirectory: true,
presets: ['es2015']
}
}, {
test: /\.json$/,
loader: 'json'
}
]
},
plugins: [
new CleanWebpackPlugin(['build'], {
root: path.resolve(__dirname, ''),
verbose: true,
dry: false,
exclude: [],
watch: true
})
],
node: {
__filename: true,
__dirname: true
},
target: 'node',
externals: nodeModules,
output: {
path: path.resolve(__dirname, 'build'),
filename: 'server.[chunkhash].js',
libraryTarget: 'commonjs'
}
}
In this case, I try to bundle the whole sources with the config and then get the bundled file named server.[chunkhash].js.
I want to move the file to server and just make work with a command like node server.[chuckhash].js, however, I got the message like below.
module.js:472
throw err;
^
Error: Cannot find module 'sequelize'
at Function.Module._resolveFilename (module.js:470:15)
at Function.Module._load (module.js:418:25)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
...
So, I tried to find the specific point to make the error, and then found out my models/index.js use the seqelize module for below codes.
import fs from 'fs';
import path from 'path';
import Sequelize from 'sequelize';
import config from 'config/env';
const sequalize = new Sequelize(config.mysql.database, config.mysql.username, config.mysql.password, config.mysql.params.options);
const db = {};
fs.readdirSync(__dirname)
.filter(file => {
return (file.indexOf('.') !== 0) && (file !== 'index.js');
})
.forEach(file => {
const model = sequalize.import(path.resolve(__dirname, file));
db[model.name] = model;
});
Object.keys(db).forEach(modelName => {
if ('associate' in db[modelName]) {
db[modelName].associate(db);
}
});
db.sequelize = sequalize;
db.Sequelize = Sequelize;
export default db;
How could I fix this issue?
Actually, with the nodemodule in the same folder, there has been no error, but, make the bundled file, it will make the error.
You defined all your node_modules as externals (externals: nodeModules,). This means that webpack won't bundle any module that comes from node_modules and will just leave the imports to be resolved at runtime, just like it would when running it in Node without using webpack. For this to work you need to have the modules available wherever you run the bundle.
If you want webpack to bundle the node_modules as well, you can remove the externals option.
The externals config you're using likely came (directly or indirectly) from Backend Apps with Webpack (Part I) and you should read that blog post to understand what it's really doing and whether you need it.

Resources