I've been installing a few node packages and what I noticed is that NPM creates a bunch of duplicates.
For example, I first installed mongoose, which installed a bunch of dependencies, naturally. Then I installed the mongodb package, which also came with bson as a dependency. Due to overlapping dependencies, I have the following anomaly:
Mongodb is present in the following directories:
/usr/local/lib/node_modules/mongodb/
/usr/local/lib/node_modules/mongoose/node_modules/mongodb/
Also, bson, a dependency of mongodb is present in both of these:
/usr/local/lib/node_modules/mongodb/
/usr/local/lib/node_modules/mongoose/node_modules/mongodb/
I realize these are only files of kilobytes, but I feel like this might create a lot of redundancy end I might end up with a very complex tree similar to the following:
/usr/local/lib/node_modules/[something1]/node_modules/[something2]/node_modules/[something3/.../.../node_modules/[somethingX]/
In this scenario a given [dependency] might be present on X levels under /usr/local/lib/node_modules.
My major concern is related to updating these modules. I do not find it hard to imagine to have concurrent modules of different versions installed at the same time.
Wouldn't it be easier to just put everything directly in /usr/local/lib/node_modules/ and then cross-reference dependencies?
The problem is when mongoose is only coded to work with say v1 of mongodb, and you've coded your app to work with v2 of mongodb - as such, it installs and loads up both versions so it all works. We can do this easily in node, as the require module way doesn't pollute the global namespace, unlike the browser - which makes managing and including the right dependencies a royal pain due the global namespace pollution.
Now if your package.json and mongoose's package.json allow the same mongodb version (your can specify a specific version or ranges), then doing a rm -Rf node_modules; npm install will only install one copy of mongodb, rather than two. However as said before, if multiple versions are specified, multiple versions will be installed and loaded.
Related
I understand that packages hold several modules, but I'm starting to get confused as to if packages and modules are dependencies. Also, libraries to me maybe seem like packages you install via NPM, Nuget, RubyGems, Bower, Homebrew, or Chocolatey. So are libraries packages? Dependencies are something you need to load within your application, to have a certain functionality, but aren't some libraries(jQuery) seen as a dependency? So yea, what are the differences between these concepts?
Libraries are just a bunch of code anyone can use. For example, React.js is a JavaScript library for building front end components.
If I decide to use this library in my app, then React will become one of the modules (aka an installed instance of the library) that my app depends on. So dependencies are pretty much all of the libraries your app depends on, in order to run the way you expect it to run.
I asked the same question you did about dependencies, and I learned that it's a matter of understanding how these terms relate to one another rather than finding isolated definitions for each of them.
Hope this helps!
Basically a package is a pack with some features which fullfills some functionality of your app.
Once you install any package using npm then the package is installed as a dependency in your app inside your package.json file along with its modules(aka libraries consist of classes) stored inside node_modules folder.
I hope its clear now.
I lately helped out on a project, where I added a really small dependency - in fact, it only contained a regular expression (https://www.npmjs.com/package/is-unc-path).
The feedback I got from the developer of the project was that he tries to minimize third-party dependencies if they can be implemented easily - whereby he - if I understand it correctly - asks me to just copy the code instead of adding another dependency.
To me, adding a new dependency looks just like putting some lines of code into an extra file in the repo. In addition, the developers will get informed by an update if the code needs a change.
Is it just a religious thought that drives a developer to do this? Are there maybe any costs (performance- or space-wise, etc) when adding a dependency?
I also had some disputes with my managers once concerning the third party libraries, the problem was even greater he got into believing that you should version the node_modules folder.
The source of any conflict usually is the ignorance.
His arguments were:
you should deliver to the client a working product not needing for him to do any other jobs like npm install
if github, npm is down in the moment when you run npm install on the server what you will do ?
if the library that you install has a bug who will be responsible
My arguments were:
versioning node_modules is not going to work due to how package dependencies work, each library will download his own node_modules dependencies and then your git repository will grow rapidly to hundreds of mb. Deploy will become more and more slow, downloading each time half a gb of code take time. NPM does use a module caching mechanism if there are no changes it will not download code uselessly.
the problem with left-pad was painfull but after that npm implemented a locking system and now for each package you just lock to a specific commit hash.
And Github, and npm does not have just a single instance service, they run in cloud.
When installing a dependency you always have some ideas behind and there are community best practices, usually they resume to: 1. Does the repo has unit tests. 2. The download number 3. When was the latest update.
Node.js ecosystem is built on modularity, it is not that node is so popular cause of some luck, but cause of how it was designed to create modules and reuse them. Sometimes working in node.js environment feels like putting lego pieces together and building your toy. This is the main cause of super fast development in node.js. People just reuse stuff.
Finally he stayed on his own ideas, and I left the project :D.
If I update node or a package, will it affect the applications I have that are currently dependent on the previous node/package version I had? If yes, how do I fix this? Maybe like a virtual environment :)
You should be looking at the update from 2 standpoints.
1. Node itself
2. Updating NPM Packages
For instance, there is the Node LTS which at this time is 6.1.10 and then there is v7.7.3. If you desire total stability then use the LTS. As Cihan said above, upgrading Node can be a long process if you have a system running on an older version.
However, if you want to test out the new async/await (async functions) which is already in 7.7.3 and should be released officially in 7.8 then 7.7.3 is the way to go. But keep in mind some things just may not work as you think they should or you may get some wonky results.
Also be aware depending on your server or system the Node update works differently and make sure you read the documentation for that specific system you need to upgrade.
NPM is a different ballgame. You are reliant upon many different programmers or groups of programmers. The package is depending upon their capabilities and desire to maintain the backwards compatibility. Most package creators are really good about this. Some are not.
Take a for instance. MongoDB issued a really new driver which is an incredible upgrade from its previous one. The new driver contains ES6 and in conjunction with the co package, it is basically operating with promises.
Updating this package for MongoDB was essential for me. But it does maintain my previous code as well, (even though lots of it may be superfluous now!)
So when you think about updating Node..it is not the same as updating NPM modules. But if you wish, you can go to our project root where package.json exists and just type npm update and all packages will be updated. You can update to only a specific version - take a look here.
Remember also, NPM itself also requires updating from time to time.
So in summary:
Node Version - decide which one based upon needs, requirements and your own servers.
NPM as NPM also needs updating from time to time
NPM packages can be updated constantly with npm update or just update to a specific version number based upon the url above.
Not as confusing as it first seems, once one gets it all down straight :)
Good luck
It's possible that you have a package that is being used by another package which a version change could break it. There are several options, if this is a personal project then you would have to either reinstall the correct version of the package or see if there are updates available otherwise for the packages that broke.
It should not, because npm was made to resolve this issue: each project has its own dependencies, and it is totally independent with other projects.
In a word, no matter what you change, you do npm install in your project, npm will fix the dependency issue by itself (by checking your project's package.json)
npm-packages sometimes change or deprecate certain functions & functionalities in newer versions. If something stops working when you update a package, you must figure out from the documentation of the updated package on how to make it work again. Many packages might also give runtime information/errors in the console about functionalities that have deprecated or that have been marked for deprecation in future versions.
npm will download specific versions of a packages' dependencies (and thus you can often have multiple version of a certain package in one project), so you shouldn't need to worry about the dependencies of the packages you are updating, only about the changes of the package itself.
Certain npm-package versions only support certain versions of Node, so updating the node version might require updating some packages. If you switch the node version, it will also switch the npm version, which will then install the correct version of the package per node version.
Some Node.js functionalities might stop working if you update Node.js. In these cases you must refer to the Node.js docs for help. Sometimes updating Node.js from a old version to a new version in a project is a large and tedious task.
Is there a way to install meteor packages globally?
So, having the once globally installed packages installable without internet connection for projects created later, avoid repetitive downloading, and other benefits one may imagine.
Like in Node.js, using npm command (of Node Package Manager) with -g flag, npm install -g, doing so npm installs node packages into a global directory and when wanted to be loaded from javascript programs, loading from there if available, as well as looking in and loading packages from project's node modules folder.
Meteor already downloads packages into a global repository that all your local apps benefit off of.
So if you meteor add iron:router#1.0.7 it is downloaded and added to your project. Next time another project requires the same version, it is used off that same spot.
Also, there is a PACKAGES_DIR environment variable, when set, allows you to keep your own local packages centrally, so that you can share them among projects. In fact, you can keep that on a network drive (NFS) which your whole team can mount and consume centrally.
Yet, there is an inherent problem. Meteor's version resolver looks up for updates unless you pin down your package dependency versions so that is exactly why meteor seems to be so desperate to be connected.
Even if you pin your dependencies, the packages you depend on may not have (which apparently is the case for most packages) so Meteor keeps looking for updates to the whole package tree and downloads those that it deems satisfying the version constraint resolver.
The good news is, they are constantly improving their tooling, requiring lower number of lookups, faster builds, better search etc.
All in all, in essence, there is not much you can do unless Meteor provides some way of hosting an entire mirror of its package repository for you to consume offline. And I guess it is very unlikely to happen.
Meteor is a tool for the connected world and it does assume your connectivity. Heck, the whole journey begins with a curl https://install.meteor.com/ | sh
And yes, it would be great if we could hack away on a remote beach, or the 12 hour flight to that beach.
Until then, happy coding online ;)
How should we deal with local packages that are a dependency in other local packages?
For simplicities sake, say we have the follow packages
api - express application
people - a package to deal with people
data-access - a package that deals with data access
And then the dependencies are
api depends on people
people depends on data-access
Currently we have these dependencies setup as file dependencies.
I.e. api package.json would have
"dependencies": {
"people": "file:../people"
}
Trouble with this is that we're finding it a PITA when we make updates to one package and want those changes in the other packages that depend on it.
The options we have thought of are:
npm install - but this won't overwrite previously installed packages if changes are made, so we have to delete the old one from the node_modules directory and re-run npm install... which can be niggly if the package dependency is deep.
npm link - we're not sold on the idea because it doesn't survive version control... Just thinking about it now, maybe we have some kind of local build script that would run the npm link commands for us... this way it could survive version control. Would that be a grunt job?
grunt - we haven't dived too deep into this one yet, but it feels like a good direction. A little bit of googling we came accross this: https://github.com/ahutchings/grunt-install-dependencies
So, what option would work best for our situation?
Are there other options that we haven't thought of yet?
Ps. we're a .NET shop doing a PoC in node, so assume we know nothing!
Pps. if you strongly believe we're setting up our project incorrectly and we shouldn't have smaller individual packages, let me know in the comments with a link to some reading on the subject.
So, I agree that going with 'many small packages' is usually a good idea. Check out 12factor.net if you haven't already.
That said, in specific answer to your question I'd say your best bet is to consider mainly how you want to maintain them.
If the 'subcomponents' are all just parts of your app (as, for example, data-access implies), then I'd keep them in the same folder structure, not map them in package.json at all, and just require them where you need them. In this case, everything versions together and is part of the same git repository.
If you really want to or need to keep them all in separate git repositories, then you can do npm link, but to be honest I've found it more useful to just use the URL syntax in package.json:
dependencies: {
"people" : "git://path.to.git:repo#version.number"
}
Then, when you want to explicitly update one of your dependencies, you just have to bump the version number in your package.json and run npm install again.