Access node_modules from another folder - node.js

Recently started working with Gulp and I can't figure out is it really necessary to have a copy of node_modules directly in folder with current project?
E.g. I have this structure:
mysite
└─builder
└──node_modules
└─work
└─work2
How can I access node_modules in folder 'builder' from folder 'work' or 'work2' without copying it? It is quite large, about 100mb, and seems to me it has no sense to have a copy of it for every new project.
I tried this line export NODE_PATH='D:\OpenServer\domains\mysite\build' in file package.json and then tried command gulp but it replied[10:24:27] Local gulp not found in d:\OpenServer\domains\mysite\work
[10:24:27] Try running: npm install gulp

Short answer
Don't do it. Let NPM work the way it's designed to. However, to save space, you can delete the node_modules folder on projects that are currently dormant, and recreate it with a single shot of npm install when you switch back to them.
Justification
Even if you share your node_modules, you'll probably have redundancies in it anyway. What will you do about them next ?
It is the essence of NPM to replicate modules per project. If you dig into the node_modules folder tree, you may notice that it can even contain several replications of a same library under one given dependencies tree. Say you requested two modules explicitely, and both these modules themselves pulled a dependency that takes care of a lot of things, and is therefore called lib_DADDYMUMMY :
node_modules
+ a_module_you_use v0.5
+ lib_DADDYMUMMY v0.1 (pulled as a dependency of this module)
+ another_module_that_you_requested v0.3
+ lib_DADDYMUMMY v0.1 (again ! pulled as a dependency of this other module)
This comes in handy when your two module start needing different versions of lib_DADDYMUMMY. This comes in handy when you maintain long-lived projects ! And hell knows that in the JavaScript world, with fast changing APIs, you can consider most any decent project as long-lived. :)
One could imagine having all dependencies being shared by everyone, living in a flat structure, with several versions of a library living next to each other and every one finding what he needs there. That repository could be called, say, .m2. But that's just not the way NPM works unfortunately.
NPM considers that storage space is cheap. That's its price for helping you manage versions in dependencies, dependencies of dependencies, and dependencies of dependencies of dependencies. I consider that it's an affordable price for taking care of the dirty jobs the day when work and work2, as their lives go on, take diverging maintenance paths. I wouldn't try getting in its way by forcing a half-Maven-like folder model.

Maybe you should put your package.json into your root directory(mysite/package.json),
then try to install node_modules on the root.
In addition, you write gulpfile on the same dir.
eg.
mysite
|- package.json
|- node_modules
|- gulpfile.js
└─builder
└─work
└─work2
However, I recommend that you write one single gulpfile for each project.

One problem why you shouldn't do this is because of versioning. If your modules require different versions of the same package, you're going to run into problems. One package is going to win, and it might break another package.
Further, you get into the problem of having to merge the dependency lists in some way - meaning, you'll have to get the dependencies from work/package.json, work2/package.json, etc. and then install all of them at once.
Merging node_modules/ won't solve your problem, either - believe me, don't try.

Paste the node_modules folder inside your mySite directory.
All npm packages such as gulp will work in your work or work2 directory.
But, now(your folder structure) work folders can't find node_modules in their parent directory.

Related

packages on a build machine without permanent access to internet

I am in the process of migrating a machine from an older Linux version to a newer one. We have some node projects that seem to be more difficult to transplant to the newly desired setup.
On the old machine the build script from package.json could be run without problems because node_modules is tracked by the source control.
We don't want to use that method anymore since it is not very scalable as we want to make sure we can easily upgrade all the packages for all the users at once without the hassle of putting hundreds of subfolders (from node_modules) together with various resources.
Did you encounter such issues and, if so, how did you manage it?
Can we use a global node_modules somehow to avoid multiple node_modules for each project?
Thank you!
Can we use a global node_modules somehow to avoid multiple node_modules for each project?
There are (at least) three possibilities. Here they are in the order that I would recommend using them:
If Node.js does not find a package in the application's/module's own node_modules directory, it will check in the parent directory (../node_modules) and then that directory's parent directory (../../node_modules) etc. So if all your projects have a shared parent directory, you can put the dependencies in a node_modules directory. See "Loading from node_modules folders" in the Node.js documentation.
A second option is to set the NODE_PATH environment variable to the directory where you would like your projects to search for modules not found elsewhere. See "Loading from the global folders" in the Node.js documentation.
Lastly, it's legacy behavior, but Node.js will also search in three other locations before giving up, so you can use one of those as well: $HOME/.node_modules, $HOME/.node_libraries, and $PREFIX/lib/node. See (again) "Loading from the global folders" for more information.
All that said, please note that the docs also correctly say "It is strongly encouraged to place dependencies in the local node_modules folder. These will be loaded faster, and more reliably."
If you must do this, I would use the first option above and place the node_modules folder as far down in the directory hierarchy as you can.

Build strategies for utilizing npm packages

This must be a commonly solved problem, but I cannot find a whole lot on Google/SO so far.
When we run npm install and fetch say 50+ packages including devDependencies as well as runtime dependencies, npm creates node_modules (if needed) and adds each of those packages inside that folder. This means we end up with thousands of extraneous files included under node_modules. Each of those packages contains their own package.json, README.md, minified files, source files, etc. Our application really only cares about jquery.js (for DEV) and jquery.min.js (for PROD), so it seems to be a waste to include all of these other files into our build and therefore our web server.
Is there a standard when it comes to handle these npm packages in a way so that we simply expose ONLY the necessary files to the user? I imagine many people have this kind of issue but I don't see any built in npm constructs that allow us to do this easily.
See below.. the yellow highlighted files are the only files we really care about in Production, but we get all these extra files as well including the source code.
The most common solution consist of bundling your application on a different machine and then expose the built artefacts on production server.
There are a lot of great JS bundlers out there. The ones I have personally used are Browserify, Webpack, and Rollup. All amazing tools.
The main idea consists of writing your application in a Node environment and then bundle it to make it readable to the browser.
For simpler projects I find Browserify a very good compromise between power and ease of configuration. But it's a matter of taste, at the end. :)
Base on what I read about npm install documentation I do not think there is a option to manipulate the installation in the way you want. The packages will install the way the package author decides to package it, sometimes minified sometimes not.
Having said that, you should look for third party solutions like modclean which does exactly what you want post package installation. Run this command in the root of your project directory
npm install modclean -g
modclean
As long as your test coverage is good, ModClean would be perfect for your need.
Edit the package.json file and remove all the unnecessary dependencies then do
npm install --save
By doing this, it will create a local node_modules folder and only download the necessary packages into it (not the global node_modules folder)
Keep in mind, by default, node checks for local node_modules folder. If it couldn't find it, it will use the global folder.
Also, you don't expose all the packages in the node_modules folder. In fact, they will not be used unless you require(); them in the node.js file
EDIT:
For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as jsdom. This can be useful for testing purposes. https://www.npmjs.com/package/jquery
require("jsdom").env("", function(err, window) {
if (err) {
console.error(err);
return;
}
var $ = require("jquery")(window);
});
So jquery module do things a bit differently behind the scene for node.js comparing to the regular front-end jquery.
It requires jsdom so you will have to download that as well from here https://github.com/tmpvar/jsdom

a lot of node_modules make my mac inodes exhausted

I ran out of inodes this morning. And I found out that quite a proportion of inodes were used by node_modules/. It's not the biggest trouble maker, but it still haunt me a lot.
I have a project that made up of hundreds of addons. For each addon I have a separate package.json and node_modules for it. Since the user can choose to install any combination of my addons, I think this is a good practice to manage the dependencies separately. But it leave me so many small files and most of which are identical. e.g. bootstrap, lodash, gulp.
/project
/addon_a
/node_modules
package.json
/addon_b
/node_modules
package.json
I can think of 2 possible solutions:
Install the most common libs globally, and manage the versions by not upgrading too often.
Share common libs between addons, but how to?
FYI: I think it was actually docker and its friends used up the inodes. And I've recovered by removing and managing some docker images. But I definitely believe it that node_modules will someday be the cause of the same problem.
have you tried npm dedupe?
it searches the local package tree and attempts to simplify the overall structure by moving dependencies further up the tree, where they can be more effectively shared by multiple dependent packages.
https://docs.npmjs.com/cli/dedupe

keep node_modules outside source tree in development (not production)

I prefer to keep all generated files and dependencies outside my source tree while I work on it.
npm and grunt make that difficult: npm will never allow moving local node_modules, so I have to use --global and --prefix. But grunt does not support such a scheme, apparently.
How can I achieve my objective given the above constraints?
So, if I have a project:
foo/
.git/
src/
gruntfile.js
package.json
I want no extra files in my tree, specifically, node_modules. (Also bower_components and build etc but this is about npm.) This directory should remain untouched while I am working on it and running it. That is all.
Apparently npm link is supposed to do this, but when I tried it still installed all the dependencies in ./node_modules. Any other invocation I cannot fathom; the docs are not clear.
A related suggestion was to use another directory with symlink to my gruntfile or package.json, but grunt just resolved the symlink and continued to work in my original directory!
So far the closest I have come is to link to e.g. ~/.cache/foo/node_modules from my project. Although it achieves keeping the deps out of my tree, I still have this link cluttering my workspace.
I want to know if there is a better way. Will some combination of npm install, npm link, ln, ~/.cache, NODE_PATH and PWD allow me to run my project, from my source tree, and keep it clean of all non-source artefacts?
Swimming against standards is a Very Bad Idea ®.
What you can (and should) do is add node_modules/ to your .gitignore (or whatever ignore file you have for your given source control system) so you don't version these files.
Also, you can use a directory like src/ to organize your code and "shelter" it from the mandatory configuration files (package.json, Gruntfile.coffee, etc).

Installation of gulp through node.js's npm breaks windows

I want to use gulp on my windows machine and it actually works pretty fine, unless I try to use the created files (like pushing to github or deleting). Then it breaks, because the filepaths are too long and it seems to be a fairly common problem. https://github.com/joyent/node/issues/6960#issuecomment-45569604
I understand that the problem arises through npm's nested directories, which extend the maximal char count for Windows directories, but in my understanding there is not any solution yet.
As I see it right now I have three options:
Try to reduce the chars of npm's directories, by changing the default from 'node_modules' to 'n_m' and hope that problem ist postponed. Like suggested here:
https://github.com/joyent/node/issues/6960#issuecomment-45569604
Then it is my question, how exactly do I change the default 'node_modules' directory name?
Change my development environment to Ubuntu, which is frankly a solution I dislike, because I've never used Ubuntu.
Stop using gulp overall.
So, how do I change the default 'node_modules' directory created through npm or what solution do you actually suggest?
There is one more tricky option.
Main problem is that gulp has a lot of nested dependencies and it creates very long nested file pathes.
But if you install some of npm modules that gulp requires in your main node_modules directory gulp will not download them as nested.
Currently you have something similar to (this may be not real path you have but idea is the same):
\node_modules\gulp\node_modules\lodash.bind\node_modules\lodash._createwrapper...
If you will add "lodash.bind" module to your project's package.json as dependency it will be installed in one level with gulp and not as gulp's dependency
\node_modules\gulp
\node_modules\lodash.bind\node_modules\lodash._createwrapper
And this will shorter all urls. You will need to fix only one or two with the longest pathes and it will work.
In my project it was enough to add this dependencies: “lodash.createcallback” and “lodash.bind” to package.json to fix everything.
Take in mind that befor doing this you probably would need to clear current node_modules folder. If you are not able to do that because off too long url you can create symbolic link to temporary short file path and delete it.

Resources