This must be a commonly solved problem, but I cannot find a whole lot on Google/SO so far.
When we run npm install and fetch say 50+ packages including devDependencies as well as runtime dependencies, npm creates node_modules (if needed) and adds each of those packages inside that folder. This means we end up with thousands of extraneous files included under node_modules. Each of those packages contains their own package.json, README.md, minified files, source files, etc. Our application really only cares about jquery.js (for DEV) and jquery.min.js (for PROD), so it seems to be a waste to include all of these other files into our build and therefore our web server.
Is there a standard when it comes to handle these npm packages in a way so that we simply expose ONLY the necessary files to the user? I imagine many people have this kind of issue but I don't see any built in npm constructs that allow us to do this easily.
See below.. the yellow highlighted files are the only files we really care about in Production, but we get all these extra files as well including the source code.
The most common solution consist of bundling your application on a different machine and then expose the built artefacts on production server.
There are a lot of great JS bundlers out there. The ones I have personally used are Browserify, Webpack, and Rollup. All amazing tools.
The main idea consists of writing your application in a Node environment and then bundle it to make it readable to the browser.
For simpler projects I find Browserify a very good compromise between power and ease of configuration. But it's a matter of taste, at the end. :)
Base on what I read about npm install documentation I do not think there is a option to manipulate the installation in the way you want. The packages will install the way the package author decides to package it, sometimes minified sometimes not.
Having said that, you should look for third party solutions like modclean which does exactly what you want post package installation. Run this command in the root of your project directory
npm install modclean -g
modclean
As long as your test coverage is good, ModClean would be perfect for your need.
Edit the package.json file and remove all the unnecessary dependencies then do
npm install --save
By doing this, it will create a local node_modules folder and only download the necessary packages into it (not the global node_modules folder)
Keep in mind, by default, node checks for local node_modules folder. If it couldn't find it, it will use the global folder.
Also, you don't expose all the packages in the node_modules folder. In fact, they will not be used unless you require(); them in the node.js file
EDIT:
For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as jsdom. This can be useful for testing purposes. https://www.npmjs.com/package/jquery
require("jsdom").env("", function(err, window) {
if (err) {
console.error(err);
return;
}
var $ = require("jquery")(window);
});
So jquery module do things a bit differently behind the scene for node.js comparing to the regular front-end jquery.
It requires jsdom so you will have to download that as well from here https://github.com/tmpvar/jsdom
Related
I am trying to figure out an effective way to bundle and distribute various dependencies (node modules and/or "client"-side scripts and framework like Angular) with my Electron App.
Although the basic approach of npm install module-name --save works well for development, it is not so good in the end when it comes to minimizing the size of your app and using minified resources at runtime. For instance, virtually all npm packages (including node modules) come with a lot of "extra baggage" like readmes, various versions of components (minified, not minified, ES2015, no-ES2015, etc). While these are great for development, all these files have absolutely no need to be included in the version you will be distributing.
Currently there seem to be 2 ways to sort of address the problem:
Electron Builder recommends using 2-file package.json system.
Any dependency that is used during development only should be npm-installed using --save-dev and then prunning should be used when building the app for distribution.
In that regard I have several questions:
I am not quite sure why there is a need for 2-file package.json system if one can install dev-only modules/ dependencies with --save-dev and then use pruning during the actual app build/compilation?
Regardless of which method above is used, you still end up with full npm packages in your app, inclduying all the miscellaneous/duplicated files that are not used by your app. So how does one "prune" so to speak the npm packages themselves so that only the actual files that are being used at run-time (like minified scripts) get included?
Will using Bower for "client-side" packages (like AngularJS 2, Bootstrap, jQuery, etc.) and using npm for node modules (like fs-extra) be a better option in as far as separation of concerns and ease of bundling later?
Could WebPack be used to produce only the needed files, at least for the "cient-side", so that only real node modules will be included with the app, while the rest of it will be in the form of web-pack compiled set of files?
Any practical tips on how this bundling of dependencies and distribution should be accredited out in practice? Gulp-scripts? Web-pack scripts? Project structure?
Thank you.
I am still in the learning curve of adopting the best practices in code deployment. But here is my starting list of what is recommended.
Yes, npm install --save-dev is the first easiest thing to isolate dev and build specific packages. This includes gulp/grunt/webpack and its loaders or additional packages. These are used only for building and never in the code that actually is run. All packages used by the app should be installed with npm install --save so that it is project level available. So, in production, you would no npm install --production in machines which will not install dev packages at all. See What's the difference between dependencies, devDependencies and peerDependencies in npm package.json file? for more info.
While the original recommendation was to use bower for client side and npm for server side, both can be installed using npm too. After all, both does the same job of managing the packages and dependencies. However, if web pack is used, it is recommended that npm is used for client side dependencies also.
package.json should be thought of managing the dependent packages only and not for building. For building and picking only the required files, you need task runners like gulp/grunt or bundlers like web pack.
While gulp/grunt is very popular for build automation which includes bundling all dependent javascript in file and minifying them in to one file, webpack/browserify is a better option as it supports module import. Module import is intuitive way of require one module in another in node js type of coding
var util = require('./myapp/lib/utils.js') This is powerful way of mentioning the required dependencies in the code. The web pack builder runs like gulp as build process. But instead of looking through html file for all js files, looks at starting js file and and determines all dependent code mentioned by the require statements recursively and packages accordingly. It also minifies the code. It also loads css and image files in one bundle to reduce server trips. If needed, some modules can be configured to be loaded at runtime dynamically further reducing page load. NPM vs. Bower vs. Browserify vs. Gulp vs. Grunt vs. Webpack discusses this at length.
Webpack can be used to bundle client side app optimally while server side need not be bundled or minified as there is no download.
In web pack, though you can mention dependent modules with lib file path, the recommendation is to npm install all dependencies and mention the module name. For example if you have installed jquery, instead of giving path like /libs/jquery.min.js, you can mention as 'jquery'. Webpack will automatically pull the jquery lib and dependencies and minimise it. If they are common modules, it will be chunked too. So, it is better to npm install dependent packages instead of bower install.
ES2015 provides lot of benefits during coding time including type checking and modules. However all browsers do not yet support the spec natively. So you need to transpile the code to older version that browsers understand. This is done by transpilers like Babel that can be run with gulp. Webpack has in-built babel loader so web pack understands ES2015. It is recommended to use ES2015 module system as it will soon become the defacto way of coding and since there is transpiler, there is no worry of this not being supported in IE8/9.
For project structure you could have
server
client
src containing js files
dist containing html and build files generated
webpack.dev.config.js and webpack.prod.config.js can be at root level.
I have found that this area is an ocean and different schools of best practices.This is probably one set of best practices. Feel free to choose the set that works for your scenario. Look forward for more comments to add to this set.
Recently started working with Gulp and I can't figure out is it really necessary to have a copy of node_modules directly in folder with current project?
E.g. I have this structure:
mysite
└─builder
└──node_modules
└─work
└─work2
How can I access node_modules in folder 'builder' from folder 'work' or 'work2' without copying it? It is quite large, about 100mb, and seems to me it has no sense to have a copy of it for every new project.
I tried this line export NODE_PATH='D:\OpenServer\domains\mysite\build' in file package.json and then tried command gulp but it replied[10:24:27] Local gulp not found in d:\OpenServer\domains\mysite\work
[10:24:27] Try running: npm install gulp
Short answer
Don't do it. Let NPM work the way it's designed to. However, to save space, you can delete the node_modules folder on projects that are currently dormant, and recreate it with a single shot of npm install when you switch back to them.
Justification
Even if you share your node_modules, you'll probably have redundancies in it anyway. What will you do about them next ?
It is the essence of NPM to replicate modules per project. If you dig into the node_modules folder tree, you may notice that it can even contain several replications of a same library under one given dependencies tree. Say you requested two modules explicitely, and both these modules themselves pulled a dependency that takes care of a lot of things, and is therefore called lib_DADDYMUMMY :
node_modules
+ a_module_you_use v0.5
+ lib_DADDYMUMMY v0.1 (pulled as a dependency of this module)
+ another_module_that_you_requested v0.3
+ lib_DADDYMUMMY v0.1 (again ! pulled as a dependency of this other module)
This comes in handy when your two module start needing different versions of lib_DADDYMUMMY. This comes in handy when you maintain long-lived projects ! And hell knows that in the JavaScript world, with fast changing APIs, you can consider most any decent project as long-lived. :)
One could imagine having all dependencies being shared by everyone, living in a flat structure, with several versions of a library living next to each other and every one finding what he needs there. That repository could be called, say, .m2. But that's just not the way NPM works unfortunately.
NPM considers that storage space is cheap. That's its price for helping you manage versions in dependencies, dependencies of dependencies, and dependencies of dependencies of dependencies. I consider that it's an affordable price for taking care of the dirty jobs the day when work and work2, as their lives go on, take diverging maintenance paths. I wouldn't try getting in its way by forcing a half-Maven-like folder model.
Maybe you should put your package.json into your root directory(mysite/package.json),
then try to install node_modules on the root.
In addition, you write gulpfile on the same dir.
eg.
mysite
|- package.json
|- node_modules
|- gulpfile.js
└─builder
└─work
└─work2
However, I recommend that you write one single gulpfile for each project.
One problem why you shouldn't do this is because of versioning. If your modules require different versions of the same package, you're going to run into problems. One package is going to win, and it might break another package.
Further, you get into the problem of having to merge the dependency lists in some way - meaning, you'll have to get the dependencies from work/package.json, work2/package.json, etc. and then install all of them at once.
Merging node_modules/ won't solve your problem, either - believe me, don't try.
Paste the node_modules folder inside your mySite directory.
All npm packages such as gulp will work in your work or work2 directory.
But, now(your folder structure) work folders can't find node_modules in their parent directory.
Before start using node package manager, i was just downloading packages into one folder and giving references from html files. Now i have started to use node package manager and i want to done thing right.
I have downloaded jquery via npm install jquery --save command. Jquery is downloaded with minified, unminified, source, readme etc files, so i got more than 30 files downloaded.
How should i use those files in production? I mean all i need is one minified jquery file in production. Should i delete rest before deploying? I feel like npm can make my life easier but i am missing the point.
How should i approach to this?
When you use npm, the dependency is gonna be used only on server-side. I recommend you search for bower, it has the same purpose npm has, but for the client-side.
And about minified and full versions it's ok do maintain both versions, as long as you configure that, on production, to load the minified versions of your libraries.
I would recommend against deleting the entire package even if you need 1 file as there probably won't be any harm having a couple extra files. npm will allow you automatically update packages in the future... something you may not be able to do if you manually include a single file.
What would be the preferred way to handle internal modules within a node.js application?
I mean, currently we have a fairly big application and there are several dependencies, installed via npm. We have node_modules in .gitignore. Therefore it's not in the repository. We use npm shrinkwrap to freeze the modules versions, so, upon deployment, an npm install puts together eveything we need into node_modules.
The problem is, since out app is getting bigger, we want to split it to smaller modules. Now, if I create a foo module and put it in node_modues, I need to allow it in the repo, which does not seem so neat to have both ignored and checked out modules in node_modules.
We can not publish these on npm registry because they are not really "public".
Is there any obvious solution that I'm not aware of?
I agree with your aesthetic of not mixing 3rd-party non-repo stuff with your in-house revision-controlled contents.
The NODE_PATH search path variable can help you here, you can use a different directory with contents under revision control.
http://nodejs.org/api/modules.html
Depending on your appetite for flexibility vs complexity, you can consider a build step that actually copies your various node modules from wherever you keep them, into an output directory (probably "named node_modules" but not necessarily).
A common solution is to store the modules in a private Git repository somewhere, and then use a git:// URL as a dependency; npm knows how to clone repositories as dependencies.
npm allows us to specify bundledDependencies, but what are the advantages of doing so? I guess if we want to make absolutely sure we get the right version even if the module we reference gets deleted, or perhaps there is a speed benefit with bundling?
Anyone know the advantages of bundledDependencies over normal dependencies?
For the quick reader : this QA is about the package.json bundledDependencies field, not about the package.
What bundledDependencies do
"bundledDependencies" are exactly what their name implies. Dependencies that should be inside your project. So the functionality is basically the same as normal dependencies. They will also be packed when running npm pack.
When to use them
Normal dependencies are usually installed from the npm registry.
Thus bundled dependencies are useful when:
you want to re-use a third party library that doesn't come from the npm registry or that was modified
you want to re-use your own projects as modules
you want to distribute some files with your module
This way, you don't have to create (and maintain) your own npm repository, but get the same benefits that you get from npm packages.
When not to use bundled dependencies
When developing, I don't think that the main point is to prevent accidental updates though. We have better tools for that, namely code repositories (git, mercurial, svn...) or now lock files.
To pin your package versions, you can use:
Option1: Use the newer NPM version 5 that comes with node 8. It uses a package-lock.json file (see the node blog and the node 8 release)
Option2: use yarn instead of npm.
It is a package manager from facebook, faster than npm and it uses a yarn.lock file. It uses the same package.json otherwise.
This is comparable to lockfiles in other package managers like Bundler
or Cargo. It’s similar to npm’s npm-shrinkwrap.json, however it’s not
lossy and it creates reproducible results.
npm actually copied that feature from yarn, amongst other things.
Option3: this was the previously recommended approach, which I do not recommend anymore. The idea was to use npm shrinkwrap most of the time, and sometimes put the whole thing, including the node_module folder, into your code repository. Or possibly use shrinkpack. The best practices at the time were discussed on the node.js blog and on the joyent developer websites.
See also
This is a bit outside the scope of the question, but I'd like to mention the last kind of dependencies (that I know of): peer dependencies. Also see this related SO question and possibly the docs of yarn on bundledDependencies.
One of the biggest problems right now with Node is how fast it is changing. This means that production systems can be very fragile and an npm update can easily break things.
Using bundledDependencies is a way to get round this issue by ensuring, as you correctly surmise, that you will always deliver the correct dependencies no matter what else may be changing.
You can also use this to bundle up your own, private bundles and deliver them with the install.
Other advantage is that you can put your internal dependencies (application components) there and then just require them in your app as if they were independent modules instead of cluttering your lib/ and publishing them to npm.
If/when they are matured to the point they could live as separate modules, you can put them on npm easily, without modifying your code.
I'm surprised I didn't see this here already, but when carefully selected, bundledDependencies can be used to produce a distributable package from npm pack that will run on a system where npm is not configured. This is helpful if you have e.g. a system that's not networked / not on the internet: bring your package over on a thumb drive (or whatever) and unpack the tarball, then npm run or node index.js and it Just Works.
Maybe there's a better way to bundle up your application to run "offline", but if there is I haven't found it.
Operationally, I look at bundledDependencies as a module's private module store, where dependencies is more public, resolved among your module and its dependencies (and sub-dependencies). Your module may rely on an older version of, say, react, but a dependency requires latest-and-greatest. Your package/install will result in your pinned version in node_modules/$yourmodule/node_modules/react, while your dependency will get their version in node_modules/react (or node_modules/$dependency/node_modules/react if they're so inclined).
A caveat: I recently ran into a dependency that did not properly configure its dependency on react, and having react in bundledDependencies caused that dependent module to fail at runtime.