Let's say I've made a TDD tool called foo, and I want to use foo v1 to help me develop foo v2.
But when I npm install --save-dev foo#^1.0.0, npm says "Refusing to install foo as a dependency of itself".
Why does npm refuse to do this?
What can I do instead?
Workarounds I've got so far (and why they're not good enough):
Workaround 1: just require the relevant script directly using a relative require, e.g. require('../lib') (this is how mocha does it, and it's how I've been doing it so far).
but if you're working on a new version of your module, adding features, perhaps even removing old features, then you're constantly having to alter not just the content of your tests but also their format, because you're actually using the thing you're working on to test the thing you're working on. If it breaks, you have to fix it in the dark with no tests to guide you. It would be much better to use the settled v1 API for tdd-ing the new v2 API.
Workaround 2: publish a duplicate of your module to npm under a different name, like "foo-clone". (Then you can just install foo-clone as a devDependency of foo.)
but that seems messy and a misuse of npm. Anyway, if installing an exact clone would work, then what would be the harm in npm allowing me to install [an old version of] foo as a devDependency of foo?
Here is a better alternative to your Workaround 2.
Let's assume that you'll only need this dependency on early stage of development. So, before publishing first production-ready version you'll get rid of it, e.g. by adopting mocha solution (using current stable version to test itself).
In this case, instead of publishing duplicate package you could temporary rename your package (i.e. postfix it with -dev).
To guarantee that this renamed package will not be published, you could also add private flag.
So, your dev package.json will look something like:
{
"name": "mytdd-dev",
"version": "2.0.0-dev",
"private": true,
...
"devDependencies": {
"mytdd": "1.x.x",
...
},
...
}
Then, when your package will be ready for the first release, you'll remove all -dev postfixes, private flag and dev dependency on previous version.
The only problem with this solution is that you won't be able to publish early dev versions of your TDD tool to npm (as long as you'll depend on previous version).
if installing an exact clone would work, then what would be the harm in npm allowing me to install [an old version of] foo as a devDependency of foo
I think it's a safety precautions against circular dependencies.
If you believe that npm should make an exception for devDependencies here, which sounds reasonable to me, then you should post your suggestion to npm issues tracker.
Related
My Node/Electron project has a dependency graph sort of like this:
My Project
aaa#^4.0
bbb#^5.6
ccc#^1.0.0
ddd#^0.8
ccc#^1.0.1
eee#^4.4
eee#^4.3
npm install resolves this to the following node_modules:
aaa#4.0
bbb#5.6
ccc#1.0.1
ddd#0.8
eee#4.4
It figures that ccc#1.0.0 and 1.0.1 are compatible and deduplicates them, as it should. However, runtime errors indicate that they are, in fact, not compatible at all. That is, bbb#5.6 does not work with ccc#1.0.1 and requires 1.0.0.
Now, I could use --legacy-bundling so both versions would be installed, as nested modules in those that depend on them, but I don't want to do that for all dependencies. Specifically, there must be only one copy of eee at runtime.
How can I disable the deduplication for either aaa and all its dependencies or just for bbb? Preferably in NPM, but I'm willing to switch to Yarn or another tool if needed.
Edit: the question originally used ccc versions 1.0 and 2.0, but I was wrong, npm did in fact view those as not compatible and already installed them the way I wanted. My problem was with Webpack after all. So I've updated the question to the current version, which is still a possible scenario.
I've found that it can be achieved by
editing package-lock.json
and changing .["packages"]["node_modules/bbb"]["dependencies"]["bbb"]
from "^1.0.0"
to "1.0.0",
then running npm ci.
I have no idea if this is the proper way to do it, but at least running npm i doesn't overwrite the change.
Now I'm running into the problem that Webpack doesn't resolve bbb's import to the nested ccc, but that's outside of the scope of this question...
So in my project, I require multiple packages, including "dep1".
"dep1" requires another dependency, "dep2".
And "dep2" requires multiple packages, including "dep3".
The problem is, "dep3" is not compatible with the licence I want to use (well actually, with the licence my boss wants to use).
Fortunatly, only one function of "dep2" uses "dep3", and "dep1" does not use this function. So if I remove "dep3" from node_modules, all will go smoothly.
The problem I have is, how to modify package.json to take this into account when doing npm install, and not install this package ?
I am aware I could branch "dep2" to supress the function, and then "dep1" to use the modifies "dep2", but this seems overkill, and I would be dependant of the owner of the packages to accept my branch. I am looking for a solution like 'well, just add the line ignore : "dep3" into package.json', but can't find any.
Thanks for your help !
The obvious way is to remove unwanted dependency from node_modules in NPM postinstall hook.
Alternatively, a stub can be provided instead of dep3. A stub should contain package.json that will identify it as a substitute with matching version:
{
"name": "dep3",
"version": "VERSION THAT MATCHES DEP2 CONSTRAINT"
}
It can be specified as local dependency in dep:
...
"dependencies": {
"dep3": "./dep3-stub",
...
Or as Git dependency:
...
"dependencies": {
"dep3": "github-user-name/dep3-stub",
...
If version constraint matches, dep3 stub will be installed and used instead of real package, otherwise dep2 may install its own copy of dep3.
I am aware I could branch "dep2" to supress the function, and then "dep1" to use the modifies "dep2", but this seems overkill, and I would be dependant of the owner of the packages to accept my branch.
This is a reasonable approach. This is no concern to the owner. dep2 fork can be used instead of dep2 NPM package in the same way as shown for dep3.
I know that doing something like this in package.json :
....
...
"dependencies" : {
"some-node-module" : "*"
}
is a bad idea since you're basically telling node to always update this module to its latest version, even though your code might not be able to handle any other version other than the current one for this particular module.
So I should instead do something like this :
....
...
"dependencies" : {
"some-node-module" : "3.4.1"
}
Which basically tells node to always use the version of the module that my code was built around.
Question
I have an app which I've first tested locally. The app has now been built, and using the package.json dependencies, npm has installed all of the appropriate node modules locally under my app's root folder (as opposed to globally, in some obscure folder I don't have immediate access to and which is irrelevant to this app - I simply don't like global installations of node modules - I find them to.. "abstract").
Given that all of the node modules are now installed locally isn't the node modules dependencies part in my package.json now redundant ?
I mean, what if something happens and npm is not available or the specific version of a module can't be found?
Isn't it best to be independent of dynamic node module installations and just have everything installed locally the first time without having to use the package.json dependencies ?
npm install & update
"you're basically telling node to always update this module to its latest version"
Packages won't be automatically updated. The only time the "*" will be an issue is when you are installing the project for the first time via npm install or when you manually run an update via npm update.
I personally prefer to pick a specific version of a module rather than use any wildcards, but even then there are some gotchas...which is why npm shrinkwrap exists.
npm shrinkwrap
Next gotcha:
basically tells node to always use the version of the module that my
code was built around
Sorta true. Let's say you use version 1.2.3 of your favorite module and package.json reflects that, but in the module itself is a package.json dependency on another module and that uses "*"...so when you install, the new internal dependency and the wildcard can wind up breaking the module you thought was 'locked down'.
See the gotcha? Hard coding a version controls for the top level versions but does not enforce anything beneath that...and if a module author you depend upon (or a module they depend upon) uses wildcards, you can't be 100% sure things will be copacetic.
To strictly enforce a version, you'll want to use npm shrinkwrap. (The link there to the docs provides more background, which is good to understand if your project uses more than a few very simple modules.)
And now...your question.
You say:
I mean, what if something happens and npm is not available or the
specific version of a module can't be found?
Based on the first two parts of this answer, it should now be clear that it doesn't hurt to have the dependencies explicitly listed in the package.json because node isn't checking things every time the app runs. npm uses package.json when specific actions (install, update, etc) are called but even then, it is a manual trigger.
While situations vary, there are very few that I can imagine where omitting dependencies in package.json is a good idea. If you ever wind up having to rebuild the project, you'll be in trouble. If the project is so good you want to share it, you'll be in trouble. Heck, if this is something for work and you want to go on vacation and need to install it on another machine...you'll be in trouble.
So given the fact that after the initial install, dependencies have no negative impact...use --save or add the dependencies to your package.json. Your future self will thank you. :)
npm allows us to specify bundledDependencies, but what are the advantages of doing so? I guess if we want to make absolutely sure we get the right version even if the module we reference gets deleted, or perhaps there is a speed benefit with bundling?
Anyone know the advantages of bundledDependencies over normal dependencies?
For the quick reader : this QA is about the package.json bundledDependencies field, not about the package.
What bundledDependencies do
"bundledDependencies" are exactly what their name implies. Dependencies that should be inside your project. So the functionality is basically the same as normal dependencies. They will also be packed when running npm pack.
When to use them
Normal dependencies are usually installed from the npm registry.
Thus bundled dependencies are useful when:
you want to re-use a third party library that doesn't come from the npm registry or that was modified
you want to re-use your own projects as modules
you want to distribute some files with your module
This way, you don't have to create (and maintain) your own npm repository, but get the same benefits that you get from npm packages.
When not to use bundled dependencies
When developing, I don't think that the main point is to prevent accidental updates though. We have better tools for that, namely code repositories (git, mercurial, svn...) or now lock files.
To pin your package versions, you can use:
Option1: Use the newer NPM version 5 that comes with node 8. It uses a package-lock.json file (see the node blog and the node 8 release)
Option2: use yarn instead of npm.
It is a package manager from facebook, faster than npm and it uses a yarn.lock file. It uses the same package.json otherwise.
This is comparable to lockfiles in other package managers like Bundler
or Cargo. It’s similar to npm’s npm-shrinkwrap.json, however it’s not
lossy and it creates reproducible results.
npm actually copied that feature from yarn, amongst other things.
Option3: this was the previously recommended approach, which I do not recommend anymore. The idea was to use npm shrinkwrap most of the time, and sometimes put the whole thing, including the node_module folder, into your code repository. Or possibly use shrinkpack. The best practices at the time were discussed on the node.js blog and on the joyent developer websites.
See also
This is a bit outside the scope of the question, but I'd like to mention the last kind of dependencies (that I know of): peer dependencies. Also see this related SO question and possibly the docs of yarn on bundledDependencies.
One of the biggest problems right now with Node is how fast it is changing. This means that production systems can be very fragile and an npm update can easily break things.
Using bundledDependencies is a way to get round this issue by ensuring, as you correctly surmise, that you will always deliver the correct dependencies no matter what else may be changing.
You can also use this to bundle up your own, private bundles and deliver them with the install.
Other advantage is that you can put your internal dependencies (application components) there and then just require them in your app as if they were independent modules instead of cluttering your lib/ and publishing them to npm.
If/when they are matured to the point they could live as separate modules, you can put them on npm easily, without modifying your code.
I'm surprised I didn't see this here already, but when carefully selected, bundledDependencies can be used to produce a distributable package from npm pack that will run on a system where npm is not configured. This is helpful if you have e.g. a system that's not networked / not on the internet: bring your package over on a thumb drive (or whatever) and unpack the tarball, then npm run or node index.js and it Just Works.
Maybe there's a better way to bundle up your application to run "offline", but if there is I haven't found it.
Operationally, I look at bundledDependencies as a module's private module store, where dependencies is more public, resolved among your module and its dependencies (and sub-dependencies). Your module may rely on an older version of, say, react, but a dependency requires latest-and-greatest. Your package/install will result in your pinned version in node_modules/$yourmodule/node_modules/react, while your dependency will get their version in node_modules/react (or node_modules/$dependency/node_modules/react if they're so inclined).
A caveat: I recently ran into a dependency that did not properly configure its dependency on react, and having react in bundledDependencies caused that dependent module to fail at runtime.
I'm having problems following the 'official' recommendation to check in all external dependencies into git (article http://www.mikealrogers.com/posts/nodemodules-in-git.html linked fron FAQ)
How do you make sure that not only top-level dependencies are checked-in? Most npm modules do currently not follow the recommendation. They all have their node_modules in .gitignore . Just Deleting their .gitignore seems risky.
For compiled module the article recommends to check-in only the sources and run 'npm rebuild' and deploy time. Unfortunately 'npm rebuild' does not do a 'clean make' for all modules (despite bugfix https://github.com/isaacs/npm/issues/1872 being included in npm version 1.0.106 i'm using). This means that I have to prevent compile targets from being checked in (otherwise i would have object code compiled for the developer machine on the production machine without being overwritten by npm rebuild). But: how do i do this? Unfortunately the modules don't have a common compile output directory, so just git-ignoring "node_modules//build" and "/node_modules//out/" (as mentioned in this good article eng.yammer.com/blog/2012/1/4/managing-nodejs-dependencies-and-deployments-at-yammer.html won't help in every case.
Short version: how do you make sure that production servers use the exact same version of all dependent modules as you use during development?
UPDATE: there is now npm shrinkwrap which solves the problem of locking down exact dependency versions, even of dependencies' dependencies! More info here.
Checking in node_modules can be problematic, as the environment it's running on may differ from user to user - so what is compiled on some environment may not work on another. Plus it would fill up your changelogs and repositories with 3rd party code. Which I take it is the conslusion you've come to with your short version of the question, so let me address that.
Short version: how do you make sure that production servers use the exact same version of all dependent modules as you use during development?
Inside your package.json, there will be dependencies: {}, if it is not there, then add it. To accomplish what you want, add your dependencies as the key, and their exact versions as the value. E.g. dependencies: { docpad: '2.5.0', mocha: '1.1.0' }
However, generally (it depends on the author) upgrades to the revision number (the x.x.X number) are just bugfixes and safe. You can allow minor changes by doing dependencies: { docpad: '2.5.x', mocha: '1.1.x' } which saves you from having to update your package.json and do a release everytime there is a bugfix release. You can even do things like 2.x if you wish.
This is the solution I've come to use for all of my modules, as it ensures that even 6 months later or whatever the module will still work - whereas doing something like >= 2.0.0 means when v3 of a dependency comes out, your module will probably be unusable at that time. Ensuring you stick to specific versions "guarantees" stability over time.
For reference you can see how I've done it in my open-source node.js modules here
About the .gitignore of your dependencies (in "node_modules"), npm 1.1 ignores the .gitignore files, so that they are not installed;
npm 1.1 will exclude .gitignore files from the things it installs.
npm 1.0 did not have this feature, so you have to be careful about that.
Deleting them recursively is fine:
find node_modules -name .gitignore | xargs rm
But, in npm 1.1, you never have to do this, because it excludes them
from the install automatically.
That's coming from the chief himself (Isaac), and it's here and seems to cover pretty much everything. The "extraneous" problem I have must be something silly I've done, I'll try a clean setup.