package.json: generating a package.json for release - node.js

I am developing a library that I want to release as a node package and I am building the library using the webpack build system. I have a package.json and a package-lock.json that are commited to the repository. The webpack build is producing a set of compiled and bundled artifacts and assets into a dist folder that make up the library that I want to release.
My assumption is, that when I release the compiled and bundled library to an npm repository, that the developers who consume the package do not want to rebuild the library and thus do not need to download any of the dependencies or devDependencies of the library since I am shipping the compiled output of the library in the package that I release.
This means that during the npm publish step, I need to have a package.json that has removed the dependencies and devDependencies fields. Otherwise, developers who depend on my library will receive all these dependencies when they run npm install into their workspace resulting in extra overhead.
Is there a best-practice on how to generate a new package.json out of the checked-in version that removes these fields and places them into the dist folder before release?
I can think of many ways to do this such as:
Using the webpack build with the copy-webpack-plugin and a transform function to output a new package.json into the dist folder.
By adding a custom step to my build pipeline that generates a modified package.json into the dist folder.
By committing a separate package.json into a subfolder that use used specially for release and automatically copy that into the dist folder at release time.
I am wondering if there is some commonly accepted best-practice way on how to do this or if the npm tooling already has support for that use-case build-in?

Answer
I have not seen any standard way. All the ways you propose are aiming to generate a new and clean package.json and there is nothing wrong with that. I guess what you can see weird is the generation of a new package.json. Don't worry, it's quite common but, as far as I know, there are no rules about how to generate the modified copy. However, I would like to share some points of view about your worries.
About devDependencies
If installed from npm repo, the dev dependencies of a dependency will not be installed (or shouldn't) and this should be enough in most cases.
What about dependencies?
Should be included. If you consider that some dependency is not needed in the final dist and only in development, it's a development dependency by definition. Move it to devDependencies array in the package.json.
What about bundled dependencies?
Well, your dependencies should never be bundled or packed and excluded from dependencies. You can but this goes against the purpose of npm and similar package managers, losing the advantages related to dependencies modularity, caching and version control. If you want to force the distribution of some files because you are working with modified third party libraries or other reasons to have absolute distribution control, you maybe should at least take a look to bundledDependencies key in the package.json.

Related

Why is my node_modules folder full of so many files I have not installed?

I'm creating a nodeJS application and struggling to upload it onto my server. I only have 6 dependencies yet for some reason my node_modules folder has 119 folder in it. Do I need the ones that I have not installed or have in my dependencies in my pack.json?
Thanks in advance!
According to Nodejs.org Tools section
Node.js is all about modularity, and with that comes the need for a quality package manager; for this purpose, npm was made. With npm comes the largest selection of community-created packages of any programming ecosystem, which makes building Node.js apps quick and easy
How dependencies are managed
Each library / dependency you add to your project must have a semantic version as 1.2.x and can either has other dependencies used in it or none.
So if each dependency you added to your project has its own dependencies with different versions this will make the package manager you are using add them to node_modules
Project dependencies with dependencies that are common and have no breaking API / semantic version are bubbled up to your node_modules directory and the rest of dependencies versions which aren't common live in their own node_modules directory in the libraries you added to your project
This is a general idea of how dependencies are managed and the reason for Node.js modularity.
There are also downsides to this workflow as said by Ryan Dahl the original developer of Node.js, because node_modules directory becomes bloated.
You should listen to one of his talks 10 Things I Regret About Node.js - Ryan Dahl - JSConf EU
The dependencies you've installed are reliant on other dependencies, just like your code is dependent on the ones you've installed.
That is why there are so many other folders.
You can see a list of all the other dependencies on which the library you installed is dependent, by running:
npm view (package name) dependencies
For visual learners, this concept is well demonstrated in this portion of this video.

Need validation of my understanding of dev-dependencies and dependencies

I am a beginner (~15 days into learning web-development) and I am currently learning React among other things and I am sorry if this sounds too trivial.
I am trying to understand the difference between devDependencies and dependencies and the correct usage of the same.
I have tried to figure it out from the docs and stackoverflow questions but I am not a 100% sure if I have this right. So kindly review my understanding as of now and let me know if I have this right so far.
Definition
dependencies: only the packages which are finally going to be used by the production build. These will be there in the final package.json file.
devDepndencies: the packages which eases my development efforts and finally will not be used by the product / application. These will not be included in the package.json folder of the final build.
Importance of correct usage
Fairly important as correctly excluding devDependencies from dependencies can make the app lighter. At the same time, incorrectly excluding required dependencies will cause my app to break.
Practical example
In the package.json file created during my tutorial, I had the following packages and I am mentioning the type of dependency that the package should have according to my current understanding. Please let me know if I am erring somewhere:
babel-cli : devDependency
babel-core: devDependency
babel-loader: devDependency
babel-plugin-transform-class-properties: devDependency
babel-preset-env: devDependency
babel-preset-react: devDependency
css-loader: devDependency
node-sass: dependency
react: dependency
react-dom: dependency
react-modal: dependency
sass-loader: dependency
style-loader: dependency
validator: dependency
webpack: dev-dependency
webpack-dev-server: dev-dependency
Please let me know if I have any of these wrong
devDependencies are dependencies only required within your development environment or that are required for you to build your UI, for example nodemon is a dev dependency because you'll never run your application with it.
One of the advantages of splitting up your devDependencies from your normal dependencies is a smaller docker image size when building your final layer.
For example, in my dockerfile I'll run a suite of tests and also build the UI which requires an npm install but when building the final image that is going to actually run I will simply copy over the built UI files via docker then I'll run an npm install --production so that my devDependencies won't install and bloat my node_modules folder.
Hope this helps.
devDependencies are the module dependencies which are only required during the active development of your web application. For example, when you're coding new features into your web application. A lot of devDependencies will make development easier on your end and can provide functionality such as linting, bundling, transpiling, etc...
In contrast, regular dependencies are the modules which are necessary during the runtime of your web application. I.e. these are the necessary dependencies for your web application to work correctly when other users want to interact with your web application.
Note: By module I mean the underlying code from the library which you're leveraging. A more complete definition can be found here.
Your concerns about including more code than necessary for your production bundle are valid, and I would recommend reading more about the Cost of JavaScript. However, in the beginning I would encourage one to first get a working code base, and to keep iterating and improving your code as your skillset grows. Improving performance along the way.
Lastly, some common type of devdependencies include libraries for testing your code base, building, minifying, bundling, transpiling, and linting your code as well.
Hopefully that helps!

How to modify an npm package built with TypeScript

I want to try and make some changes to a package published in npm? (I've suggest some changes as an issue but I think they are simple enough for me to attempt them).
https://www.npmjs.com/package/bt-presence#contributing--modifying
The author supplies some information on how to modify the package, but not really enough for someone doing it for the first time.
Where should I clone the GitHub repo to? The folder where the package is installed? I tried it in my home folder and that would not build (unmodified).
The command npm run build - where is this run from? The root folder of the package where the package.json is?
Will I need to modify the package.json?
In general what is the best way to develop something like this for npm? I've worked on packages before but they were simply Javascript.
If you want to work on the bt-presence package in isolation, you can put the cloned repository anywhere. If you want to use your modified version of bt-presence in combination with an application, my recommended approach is to register bt-presence as a dependency in the application's package.json file with the version set to a relative path to your bt-presence repository; then running npm install in the application will make a symlink from node_modules/bt-presence in the application to your bt-presence repository.
npm run build should indeed be run from the root folder that contains the package.json of bt-presence.
If you just want to change the code of bt-presence, you won't need to modify its package.json. You would only modify the package.json if you need to change any of the settings in there, e.g, if you need to add additional dependencies to your version of bt-presence.
None of the above is really specific to TypeScript. (Some JavaScript packages have build processes too if they need to transform or package the JavaScript files in some way.)

Best way to set up a node.js web project in a closed environment

We build a web application and our project uses various npm packages for development, testing and run-time.
The project is built as part of a large project in TFS. TFS runs ant to build the project. Our build.xml first runs npm install, then transpiles and minifies the TypeScript and Sass files (using Grunt tasks) and then builds the final war fie.
This all works OK, but our TFS is not allowed to access the Internet during the build, only our local network. Therefore, we have all the npm libraries we use copied to a file server in our network, and our package.json dependencies point to paths on that file server.
Does this seems like a reasonable solution?
The problem we have is that the npm install takes about 10 minutes to get all the >50 packages we use (which includes karma, grunt, sass, tslint, etc. – total is 170MB).
We are now looking for way to reduce the TFS build time. One option is to but the node_modules in our source control and skip the npm install step, but is seems wrong to put third-party code in our source control.
I’d love to hear other ideas to handle this and have shorter build time.
Note that on developers machine the project builds in no time, as all packages are already installed, but TFS builds start by getting a clean environment from source control, so nothing is installed.
Tough problem. You could have TFS check if your package.json checksum has changed in order to determine if a "clean" is necessary. You'd still have a 10 minute build whenever package.json is updated, but package.json changes are usually infrequent.
The lines become blurred when you host your own npm libraries since this is essentially taking a snapshot of only the dependencies you need. Therefore, if you added a dependency, colors, you'd have to update your npm repo. That could be viewed as updating the node_modules folder on your npm repo. It's a static list of available dependencies which essentially defeats the purpose of a package.json (unless of course other internal apps use the internal npm repo).
BUT, I digress, I'd argue that the best option is to have a package.json checksum for TFS to know if it should bother rebuilding node_modules.

Using NodeJS, Bower and GulpJS in a project, what should I exclude from my Git repository?

I recently started working on a web project. I'm using:
NodeJS as a server
Bower - to get all the dependencies
GulpJS - for build and other tasks
Git - to save my work
For simplicity, let's say that I'm only writing HTML/CSS/JavaScript.
It doesn't seem to make sense to save ALL the project files in Git including external JavaScript libraries, since I only want to save the core files (the files that I created and modified myself).
On the other hand, if I want to hand over the project to another developer and I only hand him my own files without all the dependencies and libraries, how will he know which dependencies to get? How will he be able to build and run the project?
(I'm still new to bower, gulp and node)
So, what files do I need to save in my repository (the minimum number of files) to be able to build and work on the project?
According to what you said, your .gitignore file should look like this :
node_modules
bower_components
dist
.tmp
On the other hand, if I want to hand over the project to another developer and I only hand him my own files without all the dependencies and libraries, how will he know which dependencies to get?
You don't include your dependencies such as the node_modules and bower_components but the package.json and bower.json are tracking those dependencies so that when a new user makes a clone, he only has to npm install and bower install.
This is, if you took care to use the --save or --save-dev flags when you npm or bower install new packages.
There's a quite active repository on github, containing predefined .gitignore files for different languages, platforms and buildtools.
https://github.com/github/gitignore
Although there's no bower- or gulp-specific configuration there (yet), I usually find it quite useful when trying new things.

Resources