node_modules polluting my codebase - node.js

Hello I'm new to Node and in particular the dependency management system. When installing a module I find that my codebase is being covered by lots of dependencies when the actual code I've written is not that lengthy. I've also noticed that sometimes when I do an npm install instead of packaging all of the dependencies under one folder representing the module I'm downloading sometimes the modules dependencies endup sitting in parallel polluting the main folder. For example, I created a module that had maybe 3 sub modules that all are used by the main module and fit well together.
index.js
node_modules
my_authentication_module
my_authorization_module
my_persistance_module
Then when I installed an AWS dependency the number of modules jumped considerable so my code base looks like
index.js
node_modules
my_authentication_module
my_authorization_module
my_persistance_module
aws_module_1
aws_module_2
.
.
.
.
.
aws_module_20
Problem
This is cluttering my code and making it look like theres a lot more going on than there is. Is there a more efficient way of managing a node project?
Secondary Question
How come running "npm install some-module --save" does not confine all of the modules dependencies to a single folder? Or is there a way of doing this so that if some package needs 50 packages I don't end up with 50 packages sitting in parallel with the package that needs them.
For example. Instead of:
node_modules
my_authentication_module
my_authorization_module
my_persistance_module
aws_module_1
aws_module_2
.
.
.
.
.
aws_module_20
It would be nice to get
node_modules
my_authentication_module
my_authorization_module
my_persistance_module
aws
node_modules
aws_module_1
aws_module_2
.
.
.
.
.
aws_module_20
So at least navigating to the top level you can easily see theres only really 3 modules of interest with a bunch of AWS dependencies crammed neatly into one folder. Is anything like this possible?

You seem to be misinterpreting the purpose of node_modules. It is exclusively the province of npm (or yarn etc.). It will never include your own code (except perhaps other separate packages of yours, which are brought in as dependencies). It is (usually) not version-controlled; that is, it is .gitignored'd. At any point, it should be able to erased entirely and repopulated with a simple npm install. Of course, as with anything, there are more nuances and divergent opinions here that are discussed in detail across the web.
There are many ways to manage and structure your own code and artifacts. In many cases, it will go under a src or possibly lib directory at the top of the project, parallel to node_modules. Within src, some people prefer to group code by function (eg, controllers, routers, services), while others prefer to group by area of concern--that's a matter for the project owner to decide.
In any case, since node_modules is fundamentally not your direct concern, it is of no real consequence whether npm organizes the dependencies in hierarchical fashion, as was the case with npm#2, or in flat fashion, as with npm#3. Yes, it can be a bit disconcerting to see a node_modules with 100 entries or 500, but that's really a matter for npm to worry about, and there are good reasons for the change made with npm#3.

Related

npm: refer to a peer dependency; how to align the version from a peer dependency

In abstract, I'm ok with the provided version of dependency-B, which is already installed thanks to dependency-A.
"dependencies": {
"dependency-A": "x.y.z",
}
$> npm ls --depth=1
├─┬ dependency-A#x.y.z
│ ├── dependency-B#x.y.z
So when I require('dependency-B'), I'll expect A's dependency.
I'm using the root function from that library and, in fact, if dependency-A bumps the version, I'd like to align with it and use the same version it uses.
If dependency-B is listed on the dependencies, a brand new package will be installed.
"dependencies": {
"dependency-A": "x.y.z",
"dependency-B": "a.b.c",
}
$> npm ls --depth=1
├─┬ dependency-A#x.y.z
│ ├── dependency-B#x.y.z
│ ├── ...
├─┬ dependency-B#a.b.c
I'm tempted to not list dependency-B on my dependencies. Should I avoid this practise? Isn't ok to rely on the peer version installed by my main dependency?
If this is a brad practise, how can I tell npm to give me the very same version it's installed by another package?
"dependencies": {
"dependency-A": "x.y.z",
"dependency-B": "~try the one that is installing dependency-A~",
}
tl;dr: You should always have all dependencies that you're using in your own dependencies object, as conformant implementations of package managers are not required to give you access to your dependencies' dependencies.
This is an interesting question, and I can think of two scenarios in which you might encounter this:
Both your package and dependency-A use dependency-B independently, for your own set of reasons, and you simply don't care which version to use.
You need to use dependency-B in order to interact with dependency-A, by creating objects of B or receiving objects of B created by A.
Scenario 1: Independent usage
If you and your dependency need the same package but don't need to share anything about it, Node gives you the amazing ability of using different versions of the same package in different places by specifying different versions in the package.json of your package and your library's. This is one of the strengths of the Node module system.
Your situation, however, is that you don't care about the actual version of the package (which makes me think this is not your scenario). In particular, you wonder if it's just better to not define anything in your own package.version and just let Node find your dependecy's dependency.
This last situation is only possible because you're using npm, and npm does one particular thing: it flattens the module tree in an effort to deduplicate packages, that is, so that multiple dependency specifications that can be satisfied by the same version are, in the end, using the exact same version. This reduces both the size and depth of the module tree, but creates the unintended consequence that you now have access to packages you havent specified as dependencies, just because they were installed in you node_modules directory for the purpose of deduplication.
This is not the only possible strategy though, and pnpm, another package manager, instead useds symlinks to achieve the same goals. I won't enter into much detail, but pnpm installs all dependencies in a different, system-wide (or user-specific) directory, and then symlinks from your node_modules (and from the dependencies' own node_modules) to the appropriate location in that folder. This achieves not only project-wise deduplication, but system-wide deduplication, as all of your projects using a specific package version will use the same installation. The consequence of this system, though, is that you "lose" the ability to use your dependencies' dependencies in your own package, because they're no longer physically in node_modules.
Apart from all that, is the idea that you don't care about the version they use. That's almost never the case, as the whole point of semantic versioning is to avoid or contain breakage due to dependency version upgrades. You don't care about the version you use now, but if that package gets upgraded in your dependency to a different major version, your package can break unexpectedly.
In conclusion, not defining a dependency that you are going to use anyway is a bad practice, both because it prevents other developers from using your package in a different package manager, and because it opens you to unexpected breakage that you won't be able to properly manage.
Scenario 2: Dependent usage
The more likely scenario given your description of the problem is that at some point in your usage of dependency-A, either it asks for something or returns something from dependency-B. In this situation it is desirable that both use the same, or at least compatible versions, so that all assumptions about the shape of the objects that are being exchanged hold.
The correct way of specifying this situation is to explicitly declare dependency-B as a peer dependency of dependency-A. If that's not the case, they're not being correct and you should most definitely bring that up in an issue if possible. As a workaround, you might just declare the same version as them and be wary o possible breakages due to version upgrades on their part. Not defining anything in your own package.json can have the same problems as in Scenario 1.
However, another possibility is that you don't even need to require that dependency. It might be the case that they expect you to pass data, functions, objects or anything that will be further passed to dependency-b, but in a way that shields you from ever having to interact with dependency-B directly. In this situation, they're essentially incorporating part of B's API into their own, and therefore any breaking change from dependency-B should also incur in a breaking change of dependency-A. This shields you from unexpected breakages, avoids you having to define anything in your package.json and means you're safe.

Best way to distribute modules used the same framework

I am creating my first open source project, and I am making some plugins for it. These plugins will be published as npm packages, and they will have identical dependencies.
My question is, what I the best way to deliver them and avoid code repetition? I know I can use something like Rollup.js to pack all dependencies used by that module in the final distribution js file, but if the user is using multiple modules, the inlined dependencies will be repeated and make the file bloat.
I know end user can use a bundler to remove those repeated codes, but is there anything more I can do to reduce the size of my distribution js files?

create scafolding generators like react/open-wc using node

I am trying to create a project structure to my team like how it implemented by open-wc or create-react-app
just say npm init #open-wc and it asks couple of questions and creates the folder with specified configurations.
I didn't find good articles on google except exploring the github projects.
Maintainer of open-wc here :)
So to get an npm init script all you need to do is define a bin in your package.json.
This is what we use for npm init #open-wc:
"name": "#open-wc/create",
"bin": {
"create-open-wc": "./dist/create.js"
},
So then for the name you have 2 choices:
create-foo will be available via npm init foo
#foo/create will be available via npm init #foo
The generators itself
That's a rather sad story... we looked around but we didn't find anything that really fit our use case well. There is http://yeoman.io/ which we used initially but it's huge and it meant we had a bootup time of ~30-40 seconds before the menu appeared. We felt we needed to do something so now we roll our own solution.
It covers what we need now with a fraction of the size (especially by being very careful with dependencies) which reduced our bootup time to about ~5-10 seconds.
We thought about promoting it as a separate standalone project but truth be told we don't have the manpower for it. It's just 4 files you can find here https://github.com/open-wc/open-wc/tree/master/packages/create/src - beware as there is no docu and quite some rough edges.
Still, if you don't find a better solution feel free to join us and with some help, we could make it a separate product.

Node installs loads of modules

I am trying to install some node modules for my application.
Now after entering this command: npm install laravel-elixir it creates a folder node_modulesand installes over a hundred modules!! this cannot be right.
How would I solve this problem?
How would I solve this problem?
Write your own code from scratch.
Really, there's very little that can be done. Large dependency trees are very common in Node.js. A lot of modules are built on the backs of other modules. The module in question is an especially large piece of software, trying to do what seems like a lot of different things, and relying on many other modules to do so.
You can try
$ npm install laravel-elixir --no-optional
to see if you can trim some optional dependencies from the tree. Another methood is to add optional=false to your .npmrc.
In my brief, and unscientific testing this seems to drop about six dependencies from the tree. Not much.
You should also make sure you've updated to npm 3.0 (3.8.6 being the latest), as it does a better job of flattening dependencies.
Sometimes there are needless dependencies in the middle of a tree, and in that event there is not much you can do other than reach out to the maintainers, and see if these dependencies can be removed, but then all the downstream packages will need to update.
This is generally called depedency hell, and it is an unfortunate symptom of certain modules that rely on too many submodules.
In reality though, if this module does what you need it to do, and there are no ill effects of having many dependencies installed, does it really matter? Other than the install time, when using the module, can you tell that it is pulling in a lot of other modules?

Notating Multiple Modules in the Same Package.json File

I am trying to package up some modules that I have been working on. I have five modules, split in to five files. Four of them are the actual outward-facing modules that I want the user to be able to install. The other one is a support module that they all need to function correctly. They are all stored in the same directory. I want to be able to specify each as a separate module in the same directory. But as far as I can tell, one can only define a single module in package.json.
Is there a way to specify multiple modules? If not, that means this must be a bad practice. How should I structure my module's exports to move it in to one main module?
Currently there's not a supported way of having a separate package.json file for each module you'll be publishing within the same directory. And really, this makes sense, as each package you deploy may have issues, feature requests, bugs, etc that need to be handled separately and don't force updates of the others. Separating these out will allow you to focus on the maintenance of each independently, and also allow the consumers of these modules to include them separately. A lot of larger scale projects who have started by creating something they think people will like, end up having the thing that everyone actually use be the random sub-project that was created separately.
So separate directories, and separate package.json files, then include dependencies within the package.json for each. If you haven't already seen there's a couple good writeups to help development of node packages here:
https://docs.npmjs.com/about-packages-and-modules
https://docs.npmjs.com/creating-a-package-json-file
https://docs.npmjs.com/using-npm-packages-in-your-projects

Resources