Warnings when building backend Express/WS Node app with Webpack - node.js

I am getting some confusing warnings when building a backend Node server with Webpack. I want to use Webpack to build my backend primarily for two reasons:
Webpack creates a single executable file, which is easier to deploy
Webpack includes all of my app's dependencies, so I can deploy my app to any compatible Node environment without needing to install dependencies first
Here are the warnings I'm getting:
WARNING in ./~/ws/lib/BufferUtil.js
Module not found: Error: Can't resolve 'bufferutil' in .../node_modules/ws/lib
# ./~/ws/lib/BufferUtil.js 35:21-42
# ./~/ws/lib/Receiver.js
# ./~/ws/index.js
# ./src/main.js
WARNING in ./~/ws/lib/Validation.js
Module not found: Error: Can't resolve 'utf-8-validate' in .../node_modules/ws/lib
# ./~/ws/lib/Validation.js 10:22-47
# ./~/ws/lib/Receiver.js
# ./~/ws/index.js
# ./src/main.js
WARNING in ./~/express/lib/view.js
80:29-41 Critical dependency: the request of a dependency is an expression
For the Critical dependency warning, I've found a good example explaining the problem and some documentation on how to use the ContextReplacementPlugin, although it's still unclear to me how to apply it to this situation. It looks like the warning is being caused by line 80 in node_modules/express/lib/view.js:
opts.engines[this.ext] = require(mod).__express
It is clear to me that the dependency cannot be resolved at build time, so how can I use the ContextReplacementPlugin to fix this dependency?
As for the Module not found warnings in the ws package, it's unclear to me what's going on. It looks like those dependencies exist in my global node_modules, and maybe they're not being pulled in by Webpack. I've tried adding them to my project's devDependencies, but then I just get Critical dependency warnings for them instead.
My application still runs after being built, so I suppose I could technically ignore the warnings, but I figure that these are widely used Node packages and Webpack is a popular build tool, so there must be a reasonable solution available.
Here are my dependencies in my package.json:
"devDependencies": {
"#types/cassandra-driver": "^0.8.10",
"#types/express": "^4.0.35",
"#types/uuid": "^2.0.29",
"#types/ws": "0.0.40",
"nodemon": "^1.11.0",
"typescript": "^2.3.1",
"webpack": "^2.5.1"
},
"dependencies": {
"cassandra-driver": "^3.2.1",
"express": "^4.15.2",
"uuid": "^3.0.1",
"ws": "^2.3.1"
}
And here's my webpack.config.js:
const path = require('path');
module.exports = {
entry: './src/main.js',
output: {
path: path.join(__dirname, 'dist'),
filename: 'main.js'
},
target: 'node',
node: {
__dirname: false,
__filename: false
}
};
I like keeping things minimal if possible. Thanks for reading.

The short answer
webpack can work with node, but it cannot extract follow require() statements. Modifications have to be made to ignore require() in order for it to work.
The long answer
It is actually possible to pull some files into a master file and run it in some instances.
One instance is if all the modules required are written in typescript and the modules are written in a away that the typescript plugin can parse the module in.
Another instance would be if you are using es6 babel plugins and using es6 style imports.
Even in the above scenarios the blunder may choose not to pull in certain files
The ultimate answer
It really should not matter too much about trying to perform the long answer because modules are stored in memory at boot and then referenced later from the cache. See the article below for more information.
http://fredkschott.com/post/2014/06/require-and-the-module-system/

Related

Yarn workspaces -- package alias

TL;DR How can I create an alias for a local yarn workspace dependency?
I've tried yarn workspaces before and never succeeded, and I'm giving it another try.
I've set "workspaces": ["packages/*"] in package.json.
For each package, I decided to use the naming convention #-/package-name to prevent naming conflicts and without worrying about having namespaces for internal packages.
When adding packages as dependencies, I've been following a style where I use an interface name for resolution, but point that towards a concrete implementation. This is what I did before using yarn workspaces:
"dependencies": {
"my-interface-name": "file:some/path/to/packages/some-concrete-implementation"
}
This is basically to allow what I like to call compile-time static dependency injection. And it also means each package can individually name their interface dependencies appropriately to their need and to prevent naming conflicts.
However, I can't figure out how to accomplish this with yarn workspaces. How do I create an alias for my yarn workspaces package #-/some-concrete-implementation called my-interface-name?
What I've already tried with no success:
Defining the dependency like "my-interface-name": "#-/some-concrete-implementation"} - for some reason this causes yarn to look for #-/some-concrete-implementation on the npm registry instead of in the local workspace
I've also tried to use the workspace protocol: "my-interface-name": "workspace:#-/some-concrete-implementation"} but it still looks for the package on the npm registry!
What I haven't yet tried and could work but removes benefits of using yarn workspaces in the first place:
"dependencies": {"my-interface-name": "file:../../node_modules/#-/some-concrete-implementation"}"
Have you seen the resolutions package.json key? Is it what you need?
I've used it for aliasing/overriding external packages but the example in the docs shows it working with local packages.
Resolutions
Allows you to override a version of a particular nested dependency. See the Selective Versions Resolutions RFC for the full spec.
{
"resolutions": {
"transitive-package-1": "0.0.29",
"transitive-package-2": "file:./local-forks/transitive-package-2",
"dependencies-package-1/transitive-package-3": "^2.1.1"
}
}
From the RFC:
"**/a" denotes all the nested dependencies a of the project.
"a" is an alias for **/a (for retro-compatibility, see below, and because if it wasn't such an alias, it wouldn't mean anything as it would represent one of the non-nested project dependencies, which can't be overridden as explained below).
So, I believe the rule you need is:
"**/my-interface-name": "file:some/path/to/packages/some-concrete-implementation"
// OR equivalent
"my-interface-name": "file:some/path/to/packages/some-concrete-implementation"
I believe it works in the package's package.json. Worst case you can hoist it to the workspace root and make the rule specific to the workspace e.g. "a/b".
The workspace: alias protocol (available in pnpm too) seems the direction to take.
I've also tried to use the workspace protocol: "my-interface-name": "workspace:#-/some-concrete-implementation"} but it still looks for the package on the npm registry!
Be sure to have yarn 3 installed, otherwise you'll run into weird issues.
Note that the syntax of "my-interface-name": "workspace:#-/some-concrete-implementation" looks incorrect.
It should be "#xxx/some-concrete-implementation": "workspace:*", assuming the name of linked the package is "name": "#xxx/some-concrete-implementation".
With this in mind you don't even need to create a specific #-/name. With workspace protocol, yarn will ensure it's never downloaded from npm. It becomes an internal workspace dependency.
PS:
Yarn 3 installation
Generally a simple yarn set version 3.0.2 && yarn plugin import workspace-tools) will work.
To avoid pnp current limitation, check the generated config .yarnrc.yml and ensure nmLinker is set to 'node-modules'
# Yarn 2+ supports pnp or regular node_modules installs. Use node-modules one.
nodeLinker: node-modules
plugins:
- path: .yarn/plugins/#yarnpkg/plugin-workspace-tools.cjs
spec: "#yarnpkg/plugin-workspace-tools"
yarnPath: .yarn/releases/yarn-3.0.2.cjs
PS: you might want to add this to .gitignore too
.yarn/*
!.yarn/patches
!.yarn/releases
!.yarn/plugins
!.yarn/sdks
!.yarn/versions
.pnp.*
Run a yarn install just after.
About package.json's
Like you did, the root package.json will define the workspace paths:
{
"name": "monorepo",
"workspaces": [
"packages/*" // Enable package discovery in packages/* directory.
],
// ...
"devDependencies": {
"husky": "7.0.2", // Only what's needed for monorepo management
}
In your app packages/app/package.json
{
"name": "my-app",
"devDependencies": {
"#types/node": "16.10.1",
//...
},
"dependencies": {
// Assuming the name of packages/shared is "#your-org/some-concrete-implementation",
// we explicitly declare the dependency on it through
// workspace: alias (package-manager perspective)
"#your-org/some-concrete-implementation": "workspace:*",
}
}
You consumed package should declare the same name
{
"name": "#your-org/some-concrete-implementation",
}
Bonus: Typescript aliases
If your project is written in ts, you can even replicate your paths through
typescript path mapping. It will allow to include the files just as is (no prior compilation needed).
Following your example, just edit a ./packages/xxx/tsconfig.json in this way
{
"compilerOptions": {
// here baseUrl is set at ./src (good practice), can
// be set to '.'
"baseUrl": "./src",
"paths": {
// Declare deps here (keep them in sync with what
// you defined in the package.json)
// PS: path are relative to baseUrl
"#your-org/some-concrete-implementation/*": ["../../some-concrete-implementation/src/*"],
// if you have a barrel in ui-lib
"#your-org/some-concrete-implementation": ["../../some-concrete-implementation/src/index"],
}
},
}
PS: for non typescript: babel/plugin-module-resolver can be used in a similar manner.

async / await breaks my Webpack build for AWS Lambda; how can I migrate to Node 8.10?

Note: this is a Q&A on migrating AWS Lambda Webpack builds from v6.10 to v8.10 — no help is needed, but even better answers are of course always encouraged!
For a while now I have been a disciple of using Webpack to build my back-end Lambdas after reading James Long's excellent series entitled "Backend Apps with Webpack" (part 1, part2, and part3).
Up until recently, the only version of Node.js that Amazon Web Services offered was 6.10; you had to write your Lambda fn in the "callback" style. But on April 2, 2018 AWS announced that 8.10 was now supported, and with it the suggested pattern is async / await which is great! Until it immediately broke my Webpack build. After a bit of debugging, I can break my build just by adding one async fn to the Lambda handler (I don't even need to call it):
async function firstAsync() {
return true;
}
exports.handler = async (event) => {
// TODO implement
return 'Hello from Lambda!'
};
To be clear, doing this in the AWS Lambda console is perfectly fine, runs great. Webpack even builds it successfully, but upon uploading to AWS Lambda, I get the following error message: regeneratorRuntime is not defined. My code is below. What am I needing to do?
webpack.config.js
const nodeExternals = require('webpack-node-externals');
const path = require('path');
const UglifyJsPlugin = require('uglifyjs-webpack-plugin')
const webpack = require('webpack');
const config = {
entry: './src/index.js',
output: {
path: path.resolve(__dirname, 'dist'),
library: 'index',
libraryTarget: 'commonjs2',
filename: 'index.js'
},
target: 'node', // building for a Node environment
externals: [nodeExternals()], // in order to ignore all modules in node_modules folder
module: {
rules: [{
test: /\.(js|jsx)$/,
use: {
loader: 'babel-loader',
options: {
presets: ['env']
}
}
}]
},
plugins: [
new UglifyJsPlugin()
]
};
module.exports = config;
package.json
{
"name": "lambda-webpack",
"version": "1.0.0",
"description": "An empty project scaffold to enable webpack builds in AWS Lambda",
"main": "index.js",
"scripts": {
"build": "webpack",
"upload": "upload.bat"
},
"author": "Geek Stocks®",
"license": "MIT",
"devDependencies": {
"aws-sdk": "^2.179.0",
"babel-core": "^6.26.0",
"babel-loader": "^7.1.2",
"babel-preset-env": "^1.6.1",
"uglifyjs-webpack-plugin": "^1.1.6",
"webpack": "^3.10.0",
"webpack-node-externals": "^1.6.0"
}
}
Geek Stock's answer above is only necessary when using the Node v6.10 runtime. It allows you to use async/await syntax and babel-runtime/regenerator converts your async functions to ES6 generators which is supported on Node v4.3.2 and newer runtimes.
In Node v8.10 though, it is different. async/await is already supported so you don't need Babel to convert your async functions to generators. You just use it correctly and it works.
As for your specific situation, I assume you simply changed your Javascript code to use async/await code but you didn't tell Babel NOT to convert your async/await code to generators. That is why babel-runtime is complaining about a missing plugin.
If this is the case, you simply need to configure Babel to simply target Node v8.10. You can do it like this in your .babelrc...
{
"presets": [
[
"env",
{
"targets": {
"node": "8.10"
}
}
]
]
}
babel will then stop converting those async functions. And since it's not doing a conversion, it will not need the regenerator-runtime anymore.
In my case I'm using it with nodejs12.x. I'm receiving the error regeneratorRuntime is not defined.
I was using preset es2015 and stage-0 with a node12.x code. Also I'm using an older version of babel.
What I did, installed:
"#babel/cli": "^7.10.1",
"#babel/core": "^7.10.2",
"#babel/preset-env": "^7.10.2"
And then set this babel preset:
"presets": [
["#babel/preset-env", {"targets": { "node": "current" }}]
]
Thanks,
To begin using async / await in your Webpack-built code, you need a little help from Babel, specfically the babel-runtime and the babel-plugin-transform-runtime. Fortunately there is a really good writeup on the installation and use of both here on the Babel website. You need Babel to do the following for you:
Automatically requires babel-runtime/regenerator when you use
generators/async functions.
I won't repeat what has been written there, however, of particular note is HOW you need to install these, because while their writeup certainly works for most, there is a needed tweak for some AWS Lambda devs who have not needed runtime dependencies before now.
The first part is pretty normal, you need new devDependencies:
npm install --save-dev babel-plugin-transform-runtime
And also you need to tweak your .babelrc file like they describe:
{
"plugins": [
["transform-runtime", {
"helpers": false,
"polyfill": false,
"regenerator": true,
"moduleName": "babel-runtime"
}]
]
}
But here is the kicker, and this is new to the code above: the babel-runtime is a regular dependencies, not a devDependencies:
npm install --save babel-runtime
These 2 installs and the .babelrc tweaks do SOLVE the async / await problem described above, but it introduces a new problem to the build above: Cannot find module 'babel-runtime/regenerator'.
If you are at all concerned with keeping your Webpack built code small, you are likely also using webpack-node-externals like the above code does. For example, the aws-sdk for JavaScript in Node is very large and is already available in the Lambda environment, so its superfluous to bundle it again. The webpack-node-externals configuration above configures Webpack to ignore ALL the modules in node-modules by default, and there is the root of the new problem. The newly installed babel-runtime is a module that needs to be bundled in the build in order for Lambda to run correctly.
Understanding the problem then, the answer becomes simple: don't use the default configuration. Instead of passing in nothing to webpack-node-externals, configure it like this:
externals: [nodeExternals({
whitelist: [
'babel-runtime/regenerator',
'regenerator-runtime'
]
})], // ignore all modules in node_modules folder EXCEPT the whitelist
And that solves both the original async / await problem, and the (possible) new problem you may be encountering if you had no previous dependencies in your build. Hope that helps — happy Lambda awaiting!

How to remove eval and Function constructor from webpack build to avoid CSP issues

Problem is with Webpack which uses eval in compiled code. Due to this, Chrome extension and Firefox addons does not work as it requires 'unsafe-eval' directive in CSP property which is not allowed. I am using Vue.js for frontend and webpack and vue-loader for build process
Package.json file
{
"webpack": "^3.10.0",
"babel-core": "^6.18.2",
"babel-loader": "^7.1.2",
"babel-preset-es2015": "^6.18.0",
"babel-preset-stage-2": "^6.24.1",
"file-loader": "^0.9.0",
"style-loader": "^0.18.2",
"vue-loader": "^10.0.2"
}
This is what contained within build.js file from webpack. Both Function constructor and eval usages.
try {
// This works if eval is allowed (see CSP)
g = g || Function("return this")() || (1,eval)("this");
} catch(e) {
// This works if the window reference is available
if(typeof window === "object")
g = window;
}
// Another method of build
function setImmediate(callback) {
// Callback can either be a function or a string
if (typeof callback !== "function") {
callback = new Function("" + callback);
}
This is the result of the web-ext lint which checks for issues in addon
Code Message File Line Column
DANGEROUS_EVAL The Function build.js 433 11
constructor is
eval.
DANGEROUS_EVAL eval can be build.js 433 43
harmful.
DANGEROUS_EVAL The Function build.js 8814 20
constructor is
eval.
Is there any way I can just build without Webpack using build because from side of Vue there is a support to use runtime code of Vue but Webpack has no flat to build as per CSP policy. Please help as I don't need especially this line in build
The reason is that, Webpack checks for global variables and it needed, node:false in Webpack config file which actually removes the above mentioned code. The reason, the above code is not an issue on web application because it won't run the code but in case of Chrome Extension or Firefox plugin, the code is scanned irrespective of execution and this was creating problem.
This is present in webpack source code here. More info about globals is present here.
Tried it with webpack: ^3.11.0 version, worked like charm.
This happens because of Vue.js, not Webpack.
According to the vue docs:
Some environments, such as Google Chrome Apps, enforce Content Security Policy (CSP), which prohibits the use of new Function() for evaluating expressions. The full build depends on this feature to compile templates, so is unusable in these environments.
On the other hand, the runtime-only build is fully CSP-compliant. When using the runtime-only build with Webpack + vue-loader or Browserify + vueify, your templates will be precompiled into render functions which work perfectly in CSP environments.
Unfortunately, Vue doesn't have something like ng-csp in Angular. So the only way to run your extension – to use runtime build.
There are good answers how to such build:
Vue.js in Chrome extension
Vuejs browser extension without using 'unsafe-eval' in CSP

webpack import error with node-postgres ('pg'.Client)

Trying to bundle the following file with Webpack fails with
ERROR in ./~/pg/lib/native/index.js Module not found: Error: Cannot
resolve module 'pg-native' in
.../node_modules/pg/lib/native
# ./~/pg/lib/native/index.js 9:13-33
I tried several ignore statements in the .babelrc but didnt get it running...
The test-file i want to bundle: handler.js
const Client = require('pg').Client;
console.log("done");
webpack.config.js
module.exports = {
entry: './handler.js',
target: 'node',
module: {
loaders: [{
test: /\.js$/,
loaders: ['babel'],
include: __dirname,
exclude: /node_modules/,
}]
}
};
.babelrc
{
"plugins": ["transform-runtime"],
"presets": ["es2015", "stage-1"]
}
package.json
"dependencies": {
"postgraphql": "^2.4.0",
"babel-runtime": "6.11.6"
},
"devDependencies": {
"babel-core": "^6.13.2",
"babel-loader": "^6.2.4",
"babel-plugin-transform-runtime": "^6.12.0",
"babel-preset-es2015": "^6.13.2",
"babel-preset-stage-0": "^6.5.0",
"babel-polyfill": "6.13.0",
"serverless-webpack": "^1.0.0-rc.3",
"webpack": "^1.13.1"
}
Somewhat related github-issues:
https://github.com/brianc/node-postgres/issues/1187
https://github.com/serverless/serverless-runtime-babel/issues/8
This is indeed an old thread, but one that helped me nonetheless.
The solution provided by Steve Schafer 1 is good, but not the simplest.
Instead, the one provided by Marco Lüthy 2 in the linked issue is probably the easiest to set up because it is pure configuration, without even the need for a dummy file to be created.
It consists of modifying your Webpack config plugins array as follows:
const webpack = require('webpack');
const webpackConfig = {
...
resolve: { ... },
plugins: [
new webpack.IgnorePlugin(/^pg-native$/)
// Or, for WebPack 4+:
new webpack.IgnorePlugin({ resourceRegExp: /^pg-native$/ })
],
output: { ... },
...
}
Updated to include a change suggested in the comments.
This is an old thread but the problem still exists, so for anyone experiencing it, there is a workaround. The problem is an interaction between the way that node-postgres is written and how babel rewrites the code, which forces pg-native to be loaded even when you don't explicitly import/require it.
The simplest workaround is to add a couple of aliases to your webpack.config.js to cause it to link in a dummy do-nothing file instead:
{
...
resolve: {
alias: {
...
'pg-native': path-to-dummy-js-file,
'dns': path-to-dummy-js-file
}
}
...
}
where the dummy file contains a single line:
export default null
See https://github.com/brianc/node-postgres/issues/838 for further discussion and alternative workarounds.
I know that this is an old topic but I'm compelled to share how I solved it. It was maddening to deal with.
So, here is the readers digest version as based on the recollection from the last brain cell that I have.
Error:
Webpack Compilation Error ./node_modules/pg/lib/native/client.js Module not found: Error: Can't resolve 'pg-native'
The error above was thrown when attempting to run a Cypress test that required the npm package 'pg'.
Attempting to install the pg-native package was not successful and resulted in another error; namely ->
Call to 'pg_config --libdir' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
I found that executing pg_config --libdir in the VSCode cmd prompt resulted in that command failing.
However, I knew that it should work since running that command from the system command prompt resulted in this -> C:/PROGRA~1/POSTGR~1/9.3/lib
That is the path that contains a required dll.
So, instead of running npm install from the VSCode command prompt, I ran it from the command prompt as launched from windows.
The result...success!!! pg-native was installed successfully.
After, the Cypress test was able to run as well.
Errors in now way helped me to arrive at this solution. It was more just checking that things were installed that were required, etc.
You may have pg-native globally installed locally. Hence the packet manager does not include the pg-native in the lock file. That was a issue i experienced where it did run fine locally but every time i build in the cloud webpack complained about pg-native missing. I solved it by removing the lockfile in the files pushed to the cloud (In this case seed.run).

Angular2 and express node.d.ts conflict

I am trying to configure an angular2+express project. I understand the cause of the problem, but not the correct solution. Here are the relevant parts of my package.json dependencies:
"dependencies": {
"angular2": "2.0.0-beta.0",
"express": "^4.13.3",
"tsd": "^0.6.5",
"typescript": "^1.4.1",
<...lots of peer dependencies>
}
Node 5.2.0 is installed globally. When I run tsd install, I get ./typings/node.d.ts pulled in, for what the comments claim to be v0.12.0 API. But this conflicts with angular2/typings/node/node.d.ts (which also claims v0.12.0). The .d.ts files are different, for example:
./node_modules/angular2/typings/node/node.d.ts
---> declare var global: NodeJS.Global;
./typings/node/node.d.ts
---> declare var global: any;
The result is a mass of TS2300: Duplicate identifier errors. I can hack around this by manually deleting ./typings/node and editing ./typings/express/express.d.ts to have:
/// <reference path="../../node_modules/angular2/typings/node/node.d.ts" />
Now everything works, but obviously this is just plain 'wrong'. What is the standard way to pull in expres.d.ts so it plays nice with Angular 2?

Resources