I am a beginner (~15 days into learning web-development) and I am currently learning React among other things and I am sorry if this sounds too trivial.
I am trying to understand the difference between devDependencies and dependencies and the correct usage of the same.
I have tried to figure it out from the docs and stackoverflow questions but I am not a 100% sure if I have this right. So kindly review my understanding as of now and let me know if I have this right so far.
Definition
dependencies: only the packages which are finally going to be used by the production build. These will be there in the final package.json file.
devDepndencies: the packages which eases my development efforts and finally will not be used by the product / application. These will not be included in the package.json folder of the final build.
Importance of correct usage
Fairly important as correctly excluding devDependencies from dependencies can make the app lighter. At the same time, incorrectly excluding required dependencies will cause my app to break.
Practical example
In the package.json file created during my tutorial, I had the following packages and I am mentioning the type of dependency that the package should have according to my current understanding. Please let me know if I am erring somewhere:
babel-cli : devDependency
babel-core: devDependency
babel-loader: devDependency
babel-plugin-transform-class-properties: devDependency
babel-preset-env: devDependency
babel-preset-react: devDependency
css-loader: devDependency
node-sass: dependency
react: dependency
react-dom: dependency
react-modal: dependency
sass-loader: dependency
style-loader: dependency
validator: dependency
webpack: dev-dependency
webpack-dev-server: dev-dependency
Please let me know if I have any of these wrong
devDependencies are dependencies only required within your development environment or that are required for you to build your UI, for example nodemon is a dev dependency because you'll never run your application with it.
One of the advantages of splitting up your devDependencies from your normal dependencies is a smaller docker image size when building your final layer.
For example, in my dockerfile I'll run a suite of tests and also build the UI which requires an npm install but when building the final image that is going to actually run I will simply copy over the built UI files via docker then I'll run an npm install --production so that my devDependencies won't install and bloat my node_modules folder.
Hope this helps.
devDependencies are the module dependencies which are only required during the active development of your web application. For example, when you're coding new features into your web application. A lot of devDependencies will make development easier on your end and can provide functionality such as linting, bundling, transpiling, etc...
In contrast, regular dependencies are the modules which are necessary during the runtime of your web application. I.e. these are the necessary dependencies for your web application to work correctly when other users want to interact with your web application.
Note: By module I mean the underlying code from the library which you're leveraging. A more complete definition can be found here.
Your concerns about including more code than necessary for your production bundle are valid, and I would recommend reading more about the Cost of JavaScript. However, in the beginning I would encourage one to first get a working code base, and to keep iterating and improving your code as your skillset grows. Improving performance along the way.
Lastly, some common type of devdependencies include libraries for testing your code base, building, minifying, bundling, transpiling, and linting your code as well.
Hopefully that helps!
Related
I am developing a library that I want to release as a node package and I am building the library using the webpack build system. I have a package.json and a package-lock.json that are commited to the repository. The webpack build is producing a set of compiled and bundled artifacts and assets into a dist folder that make up the library that I want to release.
My assumption is, that when I release the compiled and bundled library to an npm repository, that the developers who consume the package do not want to rebuild the library and thus do not need to download any of the dependencies or devDependencies of the library since I am shipping the compiled output of the library in the package that I release.
This means that during the npm publish step, I need to have a package.json that has removed the dependencies and devDependencies fields. Otherwise, developers who depend on my library will receive all these dependencies when they run npm install into their workspace resulting in extra overhead.
Is there a best-practice on how to generate a new package.json out of the checked-in version that removes these fields and places them into the dist folder before release?
I can think of many ways to do this such as:
Using the webpack build with the copy-webpack-plugin and a transform function to output a new package.json into the dist folder.
By adding a custom step to my build pipeline that generates a modified package.json into the dist folder.
By committing a separate package.json into a subfolder that use used specially for release and automatically copy that into the dist folder at release time.
I am wondering if there is some commonly accepted best-practice way on how to do this or if the npm tooling already has support for that use-case build-in?
Answer
I have not seen any standard way. All the ways you propose are aiming to generate a new and clean package.json and there is nothing wrong with that. I guess what you can see weird is the generation of a new package.json. Don't worry, it's quite common but, as far as I know, there are no rules about how to generate the modified copy. However, I would like to share some points of view about your worries.
About devDependencies
If installed from npm repo, the dev dependencies of a dependency will not be installed (or shouldn't) and this should be enough in most cases.
What about dependencies?
Should be included. If you consider that some dependency is not needed in the final dist and only in development, it's a development dependency by definition. Move it to devDependencies array in the package.json.
What about bundled dependencies?
Well, your dependencies should never be bundled or packed and excluded from dependencies. You can but this goes against the purpose of npm and similar package managers, losing the advantages related to dependencies modularity, caching and version control. If you want to force the distribution of some files because you are working with modified third party libraries or other reasons to have absolute distribution control, you maybe should at least take a look to bundledDependencies key in the package.json.
I'm using webpack to bundle my typescript nodejs code.
I use webpack-node-externals to avoid errors in node_modules during the compile time.
webpack-node-externals says that, allows you to define externals - modules that should not be bundled.
But why? Webpack should bundle everything that I need to start my bundle right? It can extract and remove module that I don't use. (tree-shake for example).
If I use webpack-node-externals, then I'll have to do npm i in my prod folder to get all the dependencies.
I think this is miss the point of webpack can do. right?
I think you are correct that in your case, bundling into a single file would make more sense. webpack-node-external appears to be designed for use of NodeJS libraries, not standalone apps. From their doc:
When writing a node library, for instance, you may want to split your code to several files, and use Webpack to bundle them. However - you wouldn't want to bundle your code with its entire node_modules dependencies, for two reasons:
It will bloat your library on npm.
It goes against the entire npm dependencies management. If you're using Lodash, and the consumer of your library also has the same Lodash dependency, npm makes sure that it will be added only once. But bundling Lodash in your library will actually make it included twice, since npm is no longer managing this dependency.
As a consumer of a library, I want the library code to include only its logic, and just state its dependencies so they could me merged/resolved with the rest of the dependencies in my project. Bundling your code with your dependencies makes it virtually impossible.
I disagree with the comments that suggest Webpack was not designed to bundle Node scripts, considering that Webpack has a specific setting for just that (target). Unfortunately, there are too many third-party libraries that do not play nice with Webpack (as I just discovered today), so pragmatically speaking you're better off installing modules in the distribution folder anyway.
This is because of the binary dependency in node_modules/ as explained in:
https://archive.jlongster.com/Backend-Apps-with-Webpack--Part-I
Webpack will load modules from the node_modules folder and bundle them
in. This is fine for frontend code, but backend modules typically
aren't prepared for this (i.e. using require in weird ways) or even
worse are binary dependencies.
I went through this explanation, you can see my studies here:
https://github.com/ApolloTang/wf-backend-with-webpack-explained/tree/main/steps
I've been playing around with webpack and npm registry. I created my own little project. which I published to npm registry. Everything seems to work fine, but there is one thing that worries me.
The file which is built by webpack (lib/index.js) contains almost 2k lines of code while my library is really tiny. I see that it includes two dependencies that I use which are prop-types and detect-browser. They make most of the code.
Hover after downloading these two libraries and using them in a test project I would expect them to be somehow deduped by tree-shaking or something like that, but the size of my package is still the same.
Basically, all I'm asking is: does my library includes more code than necessary? It seems like so to me.
I also tried to additionally install prop-types and detect-browser in the test project to see if bundle size will remain the same. Hower the size of my library is still the same and bundle size increased (I expected that redundant libraries will be removed or something).
How did you generate the dependency graph?
BTW your package looks fine except for the fact that your are including in you distributed code both the packages you are depending on (prop-types and detect-browser).
In order to avoid that you should mark as external those libraries in your webpack config file. Like:
externals: {
react: 'react',
prop-types: 'prop-types',
detect-browser: 'detect-browser',
}
Just a final (personal) note: give Rollup a try to bundle your libraries. :)
I've converted to npm for my build system: no gulp etc. Also no webpack, rollup etc, it's an es6 system based on modules & no bundling. Sure is simple!
Obviously I don't want to drag around a node_modules hierarchy for my run-time, front end modules. And don't want to import foo from './node_modules/god/awful/path.js'. So I'd like to have a top level directory for the run-time, front-end dependencies.
So how do I copy my "dependencies", not "devDependencies", to a top level directory for deployment?
I've got a run script that can do it but it's pretty messy and the location of the package under node_modules is not always obvious. Maybe there's a package for doing this automatically? Or a nifty npm trick of some sort?
OK, I'm starting to need this even more than before so thought I'd nag and be clearer about what I seem to be forced to do.
First of all, I use no workflow task managers, just npm run scripts in package.json.
My dependencies (npm --save .. not --save-dev) are:
"dependencies": {
"lzma": "^2.3.0",
"pako": "^1.0.0",
"three": "*"
},
.. and my scripts cli for hoisting the dependencies into a top level libs/ dir is simply a huge cp:
"build-deps": "cp
node_modules/lzma/src/lzma.js
node_modules/lzma/src/lzma_worker.js
node_modules/pako/dist/pako.min.js
node_modules/three/build/three.js
node_modules/three/build/three.min.js
node_modules/three/examples/js/controls/OrbitControls.js
node_modules/three/examples/js/controls/FlyControls.js
node_modules/three/examples/js/controls/FirstPersonControls.js
node_modules/three/examples/js/libs/stats.min.js
node_modules/three/examples/js/libs/dat.gui.min.js
libs/",
This is pretty primitive: I have to find the dependencies in node_modules (not always obvious) and add them to the list by hand. Sure makes me want fewer dependencies! :)
I know that bower is designed for "front end" dependencies (--save in npm speak). But it seems like npm would be perfect for dependencies, and yet it appears I need to be this primitive.
What am I missing here? What do you do?
This is something I wrestled with in my earliest forays into development with Node packages. I had the exact same idea you had: "I can use npm to pull in whatever I need for my project, great!" and then trying to manage how I reference each of my dependent libraries. I didn't properly grasp at the time that npm wasn't just there to help me fetch my dependencies, it's also there to help me manage them.
Here's the long & short of it: you do NOT want to be copying anything out of your node_modules folder to some other, more human-convenient location. There's a lot of reasons why, but the biggest is that you don't need to copy anything out of node_modules -- everything your project needs is right there.
When you're developing in ECMAScript 2015+ you should only ever have to do the following (apologies for the overly-simplistic code):
/* N.B. These all reside under node_modules, yet I don't
* have to spell out their paths under node_modules: */
import $ from 'jquery';
import _ from 'lodash';
import moment from 'moment';
import NiftyLibrary from 'niftywhatever';
// ... code ...
let $name = $('#name');
let now = moment();
// ... other code ...
In other words, your development environment setup should just handle the module resolution for you. You should never have to specify that the jQuery library you want to use is at "node_modules/jquery/dist/jquery.min.js". If you're doing this you should take a minute to figure out why you're doing this -- it's unnecessary, it's a time- and brain-suck and you'd rather be writing your application code than managing your dependencies ... NOT managing the node_modules tree.
You mentioned you're developing with ES6 modules, but not using webpack, gulp, Grunt, rollup or any other build or bundling tool. Is your project intended to run entirely in Node? I ask because the last I heard most browsers aren't quite ready to run ES6 modules natively. So how are your modules getting transpiled to ES5? Maybe you're approaching this in some novel way I haven't heard of yet, but in my experience a build or bundling tool is necessary. (Also, it's a heck of a lot of fun.)
I've used Grunt with RequireJS in the past, but am now using webpack 3 with Babel and a few additional loaders (depending on the type of project I'm working on). I use npm scripts to handle my top-level tasks (running a development server, building a finished distribution package, running tests, etc.), but I let webpack handle all the business of transpiling ES6 into ES5, translating Sass styles, precompiling Vue components, etc. It's a bit of work to wrap your mind around the webpack approach, but it's well worth the effort.
Maybe webpack doesn't fit your style -- fair enough. But there are a number of other tools you could use instead. They all require a little time to get acclimated to their approaches, but they should all be able to take care of module resolution for your dependencies. Once you get a build environment set up correctly it should cease being a visible part of your development workflow; you'll just reference your dependent libraries by their names, map them to a module-local variable and use them.
Is this helpful?
EDIT: This is webpack-specific, but there should be similar options available with other bundlers or build tools.
In webpack you can use the copy-webpack-plugin to copy npm-sourced dependencies to a separate folder. This can be useful within a service worker, for example, where the execution context is a little different.
What I would suggest is scoping your dependencies (#frontend/...) and symlinking the scoped folder to your top-level dir in a post install script (similar to how bower handled the transition to yarn: see https://github.com/sheerun/bower-away)
Example:
...
"dependencies": {
"#frontend/jquery": "npm:jquery#~3.4.1",
},
"engines": {
"npm": ">=6.9.0",
"node": ">=12.10.0"
},
"scripts": {
"postinstall": "node -e \"try { require('fs').symlinkSync(require('path').resolve('node_modules/#frontend'), 'static/bower', 'junction') } catch (e) { }\""
}
Requires NPM >= 6.9 for alias support.
FYI: aliases break audits
I am trying to figure out an effective way to bundle and distribute various dependencies (node modules and/or "client"-side scripts and framework like Angular) with my Electron App.
Although the basic approach of npm install module-name --save works well for development, it is not so good in the end when it comes to minimizing the size of your app and using minified resources at runtime. For instance, virtually all npm packages (including node modules) come with a lot of "extra baggage" like readmes, various versions of components (minified, not minified, ES2015, no-ES2015, etc). While these are great for development, all these files have absolutely no need to be included in the version you will be distributing.
Currently there seem to be 2 ways to sort of address the problem:
Electron Builder recommends using 2-file package.json system.
Any dependency that is used during development only should be npm-installed using --save-dev and then prunning should be used when building the app for distribution.
In that regard I have several questions:
I am not quite sure why there is a need for 2-file package.json system if one can install dev-only modules/ dependencies with --save-dev and then use pruning during the actual app build/compilation?
Regardless of which method above is used, you still end up with full npm packages in your app, inclduying all the miscellaneous/duplicated files that are not used by your app. So how does one "prune" so to speak the npm packages themselves so that only the actual files that are being used at run-time (like minified scripts) get included?
Will using Bower for "client-side" packages (like AngularJS 2, Bootstrap, jQuery, etc.) and using npm for node modules (like fs-extra) be a better option in as far as separation of concerns and ease of bundling later?
Could WebPack be used to produce only the needed files, at least for the "cient-side", so that only real node modules will be included with the app, while the rest of it will be in the form of web-pack compiled set of files?
Any practical tips on how this bundling of dependencies and distribution should be accredited out in practice? Gulp-scripts? Web-pack scripts? Project structure?
Thank you.
I am still in the learning curve of adopting the best practices in code deployment. But here is my starting list of what is recommended.
Yes, npm install --save-dev is the first easiest thing to isolate dev and build specific packages. This includes gulp/grunt/webpack and its loaders or additional packages. These are used only for building and never in the code that actually is run. All packages used by the app should be installed with npm install --save so that it is project level available. So, in production, you would no npm install --production in machines which will not install dev packages at all. See What's the difference between dependencies, devDependencies and peerDependencies in npm package.json file? for more info.
While the original recommendation was to use bower for client side and npm for server side, both can be installed using npm too. After all, both does the same job of managing the packages and dependencies. However, if web pack is used, it is recommended that npm is used for client side dependencies also.
package.json should be thought of managing the dependent packages only and not for building. For building and picking only the required files, you need task runners like gulp/grunt or bundlers like web pack.
While gulp/grunt is very popular for build automation which includes bundling all dependent javascript in file and minifying them in to one file, webpack/browserify is a better option as it supports module import. Module import is intuitive way of require one module in another in node js type of coding
var util = require('./myapp/lib/utils.js') This is powerful way of mentioning the required dependencies in the code. The web pack builder runs like gulp as build process. But instead of looking through html file for all js files, looks at starting js file and and determines all dependent code mentioned by the require statements recursively and packages accordingly. It also minifies the code. It also loads css and image files in one bundle to reduce server trips. If needed, some modules can be configured to be loaded at runtime dynamically further reducing page load. NPM vs. Bower vs. Browserify vs. Gulp vs. Grunt vs. Webpack discusses this at length.
Webpack can be used to bundle client side app optimally while server side need not be bundled or minified as there is no download.
In web pack, though you can mention dependent modules with lib file path, the recommendation is to npm install all dependencies and mention the module name. For example if you have installed jquery, instead of giving path like /libs/jquery.min.js, you can mention as 'jquery'. Webpack will automatically pull the jquery lib and dependencies and minimise it. If they are common modules, it will be chunked too. So, it is better to npm install dependent packages instead of bower install.
ES2015 provides lot of benefits during coding time including type checking and modules. However all browsers do not yet support the spec natively. So you need to transpile the code to older version that browsers understand. This is done by transpilers like Babel that can be run with gulp. Webpack has in-built babel loader so web pack understands ES2015. It is recommended to use ES2015 module system as it will soon become the defacto way of coding and since there is transpiler, there is no worry of this not being supported in IE8/9.
For project structure you could have
server
client
src containing js files
dist containing html and build files generated
webpack.dev.config.js and webpack.prod.config.js can be at root level.
I have found that this area is an ocean and different schools of best practices.This is probably one set of best practices. Feel free to choose the set that works for your scenario. Look forward for more comments to add to this set.