I'm trying to bundle my node.js application with webpack or browserify, but I need some backend modules such as knex, bookshelf and others.
But webpack and browserify fail to package these modules since they do some strange stuff with require()...
I got these kind of errors :Cannot find module 'sqlite3' or Error: Cannot find module './dialects/postgres/index.js'
I can't ignore my node_modules directory since I need the modules in the package because I can't access the environment where my package will be used. (AWS Lambda)
I don't need webpack or browserify to optimize my node_modules but I need them, is there a way to tell webpack or browserify to just bundle the node_module folder and trust me if a require is missing ?
EDIT: I'm using serverless to test and deploy my lambdas and the only plugins allowing me to use ES2015 with babel force me to use webpack / browserify
Thank you :)
You can incorporate node_modules in the your Lambda package (the zip file containing your code that you upload in Amazon Lambda) and don't need to package it (I mean create a file with Webpack or Browserify).
Some node modules are not meant to be used in the browser and do not support packaging because they use dynamic requires. They can have good reasons to do that, specially ORMs like Bookshelf or Sequelize.
Webpack can manage simple dynamic requires, but it works only for rules easy to parse. If you really want to use webpack for whatever reason, you could have a look to the ContextReplacementPlugin but I don't think it worth the effort.
I had this exact problem and finally got it to work with webpack (don't know about browserify) by adding this to my webpack config:
{
plugins: [
new webpack.NormalModuleReplacementPlugin(/\.\.\/migrate/, '../util/noop.js'),
new webpack.NormalModuleReplacementPlugin(/\.\.\/seed/, '../util/noop.js'),
new webpack.IgnorePlugin(/mariasql/, /\/knex\//),
new webpack.IgnorePlugin(/mssql/, /\/knex\//),
new webpack.IgnorePlugin(/mysql/, /\/knex\//),
new webpack.IgnorePlugin(/mysql2/, /\/knex\//),
new webpack.IgnorePlugin(/oracle/, /\/knex\//),
new webpack.IgnorePlugin(/oracledb/, /\/knex\//),
new webpack.IgnorePlugin(/pg-query-stream/, /\/knex\//),
new webpack.IgnorePlugin(/sqlite3/, /\/knex\//),
new webpack.IgnorePlugin(/strong-oracle/, /\/knex\//),
new webpack.IgnorePlugin(/pg-native/, /\/pg\//)
]
}
If you're using serverless-webpack like me, you'll have to explicitly npm install the normal webpack module in your project and require it in your webpack config file.
This config is specifically for my setup where I use postgres without pg-native. Just ignore the modules you're not using.
The two top plugins aren't needed for webpack to build, but they get rid of a ton of annoying warnings. They're probably a little dangerous since they could match requires in other modules than knex. I couldn't find a better way without having to write my own plugin.
I was struggling to get webpack to bundle knex properly and want to share my configuration.
We could instead use ContextReplacementPlugin to avoid listing all drivers not needed for the project as when using webpack.IgnorePlugin. A side benefit is that it would avoid ignoring these packages in other modules (as mention by #Erik Frisk).
For example, I use a mysql database with mysql2 driver and my webpack config looks like this:
plugins: [
new webpack.ContextReplacementPlugin(/knex\/lib\/dialects/, /mysql2\/index.js/),
]
This will only bundle node_modules/knex/lig/dialects/mysql2/index.js excluding other dialects thus effectively ignoring the dependency in other packages like oracledb, mssql, etc.
To find more information about ContextReplacementPlugin have a look at Webpack’s ContextReplacementPlugin examples:
Related
Currently I am playing around with electron using vue-cli-plugin-electron-builder along side a simple vue project. This is the project https://github.com/nklayman/vue-cli-plugin-electron-builder .
vue create my-project
cd my-project
vue add electron-builder
npm run electron:serve
My goal is to add a simple plugin-like architecture. The app serves only base functionality but can be extended with "plugins". Those plugins therefore are not included in the built, but will be loaded at runtime by electron. I would prefere when those plugins just behave like node modules ( module.exports = ) with its own dependencies ( probably with a package.json file inside ). I would locate those plugins at app.getPath('userData') + '/Plugins.
I looked at a few approaches on how to tackle this problem :
1. Using Nodejs vm module
First, I tried using Nodejs vm module to read and execute a script from an external file, all at runtime. It works great so far, although I would not be able to use external dependencies inside those loaded scripts. If I want to use external dependencies inside the plugin scripts, those dependencies must have been included in the electron build beforehand. Somehow defeats the whole purpose of having plugins ... only vanilla js + nodejs base modules would be possible .
2. using global.require
I saw this solution in another SO answer.
Using node require with Electron and Webpack
Webpack/electron require dynamic module
They say to use global.require but it throws an error saying global.require is not a function. The solution looked promising first, but somehow I can't get it to work.
3. simply use require
Of course I had to try it. When I try to require an external module from a non-project location it won't find the module, even if the path is correct. Again, the path I am trying to locate the module should be at app.getPath("userData"), not in the projects root directory. When however, I locate the plugins inside the root directory of the project it gets included in the built. This again defeats the purpose of having plugins.
Goal
So far, I haven't found a viable solution to this. I simply want my electron app to be extendible with basic node modules at runtime ( following a pre-defined schema to simplify ) . Of course there is atom, made with electron, using their own apm manager to install and load plugins, but this seems way to overpowered. Its enough for me to only have plugin files located locally, to have a public "marketplace" is no goal. Also, it's ok if the app has to reload / restart to load plugins.
Any ideas ?
After more and more research I stumbled over 2 packages :
https://www.npmjs.com/package/live-plugin-manager
https://github.com/getstation/electron-package-manager
both integrating npm to programmatically handle package installation at runtime. I settled for live-plugin-manager for now since its better documented and even allow package installation from local file system.
Pro
I was able to integrate the system out-of-the-box into a vanilla electron app. Works like a charm.
Cons
.I was not able to use it inside a vue electron boilerplate (like the one I said I was using in OP), since webpack is interferring with the require environment. But there sure is a solution to this.
Update : I was able to get it to work eventually inside a webpack bundled electron vue boilerplate. I accidentally mixed import and require . The following code works for me using live-plugin-manager
// plugin-loader.js
const path = require('path');
const { PluginManager } = require('live-plugin-manager');
const pluginInstallFolder = path.resolve(app.getPath('userData'), '.plugins');
const pluginManager = new PluginManager();
module.exports = async (pkg) => {
// installs pkg from npm
await pluginManager.install(pkg);
const package = pluginManager.require(pkg);
return package
}
// main.js
const pluginLoader = require('./plugin-loader');
pluginLoader("moment").then((moment) => {
console.log(moment().format());
})
This will install "moment" package from npm during runtime into a local directory and load it into the app, without bundling it into the executable files.
I have been absent from nodejs development for a few years, and to my surprise when I did
$ npm install lodash
the size of my node_modules grew by 4.8MiB
how come a library that judging by its source line count should be around 260KiB takes so much space?
I am deploying my code as an AWS Lambda, and the ability to edit and debug in the console is lost due the increased size of my distribution file (partly because of this).
What's the 2019 way to deal with this?
You node_modules size is irrelevant.
If you build you application the size is much much smaller:
minified 69.2kb
minified + gzipped 24.3kB
see here
If you're using Webpack, ESBuild, or a similar bundler with tree-shaking capabilities, you can import lodash functions discretely.
import foo from 'lodash/foo';
Support for this syntax was added in Lodash v4.
Note the syntax should be 'lodash/foo'. Contrarily, import { foo } from 'lodash' will not be tree-shaken.
Like Norbert said, your node_modules size is irrelevant. You'll want to look at the size of your bundle after building your application. In the case of Webpack, you can use Webpack Bundle Analyzer to gain additional insights into your bundle's distribution. For Vite and other toolchains that use Rollup, consider rollup-plugin-visualizer.
Here's a few more tips:
As of lodash v4, tree-shaking works without additional configuration in Webpack >= v4 and ESBuild. If you use lodash-es and you have other dependencies that require lodash, both will end up in your bundle.
If you're using Babel, and you don't want to refactor your entire codebase to use the aforementioned lodash/foo syntax, consider using babel-plugin-lodash. This plugin rewrites lodash import statements to use the tree-shaking syntax for you.
If you're using Babel's preset-env plugin, set the modules option to false. Babel rewrites modules to use CommonJS by default, which won’t tree-shake.
If you're only using a single Lodash function, you can install it as a standalone module in lieu of installing the entire lodash package. e.g. npm install --save lodash.functionName
lodash-es introduces a few exceptions to this strategy. Refer to this answer for more insight into this.
Can someone help me run a node.js file that has ES6 features? This is my current workflow:
Save change in file
npm run build --- builds file to my src/dist folder
cd into my dist directory
node
var file = require('./index.js')
file.someMethod()
Then when I make a change to my method, I need to repeat all the steps above again.
Does anyone know of a more efficient way to do this?
Thanks in advance!
You dont need anything if you are not using import/export. Just Install node 8.x and then run node my_file.js. Nodejs supports most of the es6 features out of the box.
If you want to use import/export then I would recommend to use typescript: https://www.typescriptlang.org/. It has compiler TS (statically typed ES6) to ES5.
Oh, and of course both babel and typescript have watcher modules (auto recompiling on file change).
It really depends on what npm run build actually does in your app. Does it run gulp/grunt? Webpack? Etc. If it's webpack there is a webpack-dev-server that you should look into.
If it's grunt or gulp then you have a custom build chain that you're working with. If all you need is to support es6 features without having to build the project then you could use something simple like babel-cli and run your file from outside the dist directory.
$ npm i -g babel-cli
$ babel-node
> var file = require('./index.js');
> file.someMethod()
By running babel-node you're running the normal node repl but with babel required. Es6 features should now be supported in the repl without needing to be compiled to es5.
You may also want to try simply updating to the latest version of node because node 8 has pretty extensive es6 support.
I wrote a very basic express.js app. Then tried to make it one .js file. Browserify compiled the whole thing to a one file. But browserify-compiled code didn't work. As far as I know, browserify just replaces require statements with module codes. Error is:
C:\Users\HP\n\express\app.js:27025
__proto__: http.IncomingMessage.prototype
^
TypeError: Cannot read property 'prototype' of undefined
at Object.__dirname.173.accepts (C:\Users\HP\n\express\app.js:27025:34)
at s (C:\Users\HP\n\express\app.js:1:316)
at C:\Users\HP\n\express\app.js:1:367
at Object.__dirname.170../application (C:\Users\HP\n\express\app.js:26823:11)
at s (C:\Users\HP\n\express\app.js:1:316)
at C:\Users\HP\n\express\app.js:1:367
at Object.__dirname.168../lib/express (C:\Users\HP\n\express\app.js:26154:18)
at s (C:\Users\HP\n\express\app.js:1:316)
at C:\Users\HP\n\express\app.js:1:367
at Object.__dirname.153.express (C:\Users\HP\n\express\app.js:24010:15)
Browserify is designed specifically to package code for a browser.
Node.js supports a number of modules that a browser doesn't which have to be emulated by builtins. These modules will be replaced by a browser-specific shim. Some only supply a subset of the Node API that makes sense to have in a browser.
So you are running an app that has converted all the Node.js modules to support running what it can in a browser, back in Node where the modules are available but are no longer being used.
Try rollup or you could possibly configure babel to work like you need
I had this very same issue but like you said the compile code should work on server side. I solved it from this link:
https://www.linkedin.com/pulse/bundling-nodemodules-your-nodejs-app-one-single-file-xuan-son-nguyen/
Use browserify for bundling and terser for minifying. Starting by installing them globally:
npm install -g browserify
npm install -g terser
Next, we have to add a build script to package.json
...
"scripts": {
...
"build": "browserify --node --ignore-missing index.js | terser > bundle.js"
}
...
Each time you want to promote to production, you have to make a new bundle:
npm run build
A new file called "bundle.js" will be created.
Let there be peace, and there was peace. Happy coding.
So, I'm not using Node or WebPack serverside, but would still like to use modules from npm every now and then. My clientside uses requirejs, so I would need the modules in either AMD (preferred) or CommonJS.
What I want to archieve is a script that takes the module name + "external dependencies" as arguments and creates a module that contains all the other deps.
E.g.
sh npmtoamd.sh draft-js react react-dom
It creates an ES5 AMD module that contains draft-js and all of it's dependencies excluding react and react-dom. If it's not possible to eg. include css files and other non-js content in the module, providing them in eg. draft-js.css is tolerable.
While I don't use Node or Webpack serverside, we can use npm and webpack in the said script.
Fetching the module from npm is the trivial part, but I'm pretty lost in how to do the webpack parts. I know for a fact that it's possible though, as I managed to do it earlier with help, just don't have it down anywhere and no idea how it went.
I think as elmigranto commented, Browserify is what you're looking for. Unlike its name implies, it can be used in both the browser environment and the node environment. In a nutshell, it does this:
Browserify starts at the entry point files that you give it and searches for any require() calls it finds using static analysis of the source code's abstract syntax tree.
For every require() call with a string in it, browserify resolves those module strings to file paths and then searches those file paths for require() calls recursively until the entire dependency graph is visited.
Each file is concatenated into a single javascript file with a minimal require() definition that maps the statically-resolved names to internal IDs.
This means that the bundle you generate is completely self-contained and has everything your application needs to work with a pretty negligible overhead.
If you check out some of the demos you can see that all the dependencies (and their co-dependencies) are bundled into one file.
A simple example:
browserify main.js -o bundle.js
In regards to using AMD as well Browserify supports it by using deamdify.
Using AMD modules:
browserify -t deamdify main.js -o bundle.js
I ended up doing the npm fetch thingy in java instead of a batch script and finally got it working. Didn't get browserify to work however.
Heres what I do:
creating the following webpack.config.js
module.exports = {
entry: './main.js',
output: {
filename: 'bundle.js',
library:"<modulename>",
libraryTarget:"amd"
},
externals: {
react: "React",
"react-dom": "ReactDOM",
immutable:"Immutable",
moment:"Moment"
}
};
npm install <modulename>
Creating the following main.js
define('FOOBAR', ['<modulename>'], function(Foo) {
return Foo;
});
Running webpack main.js