NPM: version 2.2.0.
Operating System: Windows 8.1.
Over time, I see that many of my node dependencies have their package.json metadata changed, even though the version number of the dependency does not change. This happens most frequently with the homepage attribute within package.json.
As an example, this is currently happening with the "is-utf8#0.2.0" package. My npm cache was populated locally and on our CI server about 1 month ago. At that time, the package.json file of this module contained a homepage attribute. Today, if I do a "npm remove is-utf8 && npm clear cache is-utf8" then npm will download version 0.2.0 again and install it. However, this time the package.json is missing the homepage attribute.
This happens fairly consistently with 4-5 modules every two weeks. I know this because my nightly builds are always using an npm cache and they never complain about missing homepages. But my release builds (once a week) often complain about 4-5 modules missing homepages.
Why does this happen?
Is it possible that the .tgz file is being updated for so many projects?
UPDATE: In the last week, 12 of my dependencies had their homepage disappear from the package.json.
The homepage field is set from the github url when normalize-package-data is run. This is not consistently happening when you do the install with a clear vs. primed cache.
It looks like you have found a bug in npm -- thanks! I have created an issue on npm's github issue tracker https://github.com/npm/npm/issues/7243
Related
I'm using React (v17.0.2), with NodeJs (v16.14.0) and NPM (v8.5.1) and I've been using antd#4.18.8 in my project (it is listed as a dependency in package.json). The antd package has it's own dependencies as described in package-lock.json. One of these dependencies is copy-to-clipboard#3.3.1.
An AWS vulnerability scan marked some issues with our project. Running an "npm audit" in our project resulted in 0 vulnerabilities listed. However after cloning the copy-to-clipboard#3.3.1 project and running an "npm audit" from there the result was:
16 vulnerabilities (1 low, 4 moderate, 6 high, 5 critical)
It turns out that copy-to-clipboard#3.3.1 has dependencies and/or sub dependencies that have vulnerabilities associated to it. Mostly or possibly all to do with outdated versions of packages used. However copy-to-clipboard#3.3.1 is the latest version and the project doesn't appear to be particularly active (latest version was produced well over a year ago).
Solutions I thought to try were to use the "override" flag in package.json or npm-force-resolutions or "npm shrinkwrap". The problem is that the dependencies of copy-to-clipboard do not have their versions specified in package-lock.json. Instead these dependencies are defined in their own "lock" file located in node_modules/copy-to-clipboard/yarn.lock. These are not "lock" files that I have control over (i.e. they will be overwritten). Therefore I can't use any of those solutions.
I've created an issue with the copy-to-clipboard project to update their dependencies although it doesn't appear to be a particularly active project. So I don't know when that will happen if ever.
Swapping out antd for something else isn't really an option either.
What are my other options for trying to address this? Fork copy-to-clipboard to try and update the dependencies myself and hope I know what I'm doing? antd is widely used so I don't know how others aren't seeing this in their AWS (or whatever) vulnerability scans. Or perhaps they have solved this in a way that I haven't thought of.
I have a package.json that gives a load of security warnings. Looking at the first critical item I see its open#0.0.5 which hasn't been updated for five years. Looking at npm ll it is included by npm#6.5.0 where I am using the latest that was updated about two weeks ago.
I would like to remove the insecure dependencies. In the Java world the maven package manager lets you put exclude certain transitive dependencies. Ideally, with npm or another node package manager, I should be able to exclude dependencies with vulnerabilities. Then I can retest that my app works and not see any security errors. Is there a way to quickly exclude anything that has a security vulnerability from my package.json? If there isn't a way to do this what approaches can a take to ensure that no insecure packages are used by my application?
Update: Although "npm": "^6.5.0" is specified in the package.json I was building it with an older npm which was picking up the critical issue mentioned above. I fixed all the issues with ./node_modules/.bin/npm audit fix --force
By definition, you can't exclude a package that a dependency you are using relies on. In other words, if you require package A, and package A claims it is dependent on package B, then removing package B will cause A to either stop working altogether or begin behaving erratically.
Unfortunately this does happen, and your options include:
Ignoring the security warning.
Replacing package A with something else (applies in some cases and not others).
Asking the maintainer of package A to upgrade the version of package B they rely on, possibly opening a pull request yourself.
In your case, though, I'm not sure if your investigation is complete yet - I don't see open in npm's dependency list. Might be worth scrapping your node_modules and re-running npm install, then check again to see who is using open.
This specific warning is targeting at your lockfile, and can be easily fixed by removing the yarn.lock or package-lock.json and reinstall dependencies.
Tarn package manager has feature resulution by which you can set fixed libraries to insecure thirdparties.
See
How do I override nested dependencies with `yarn`?
NPM has something similar.
When trying to develop a library, npm link and npm link [libaryname] can be used to connect local packages for debugging.
For some reason when using it with the webpack dev server, it does not update the consumed package in the consumer project. The first time the project is run, it correctly imports the linked package. On subsequent runs the imported module stays the same as it was the first time it was run, despite the code of the package in node_modules being correct and up to date. It never changes from its initial state on the first run, even when removing and re-installing the referenced package.
Is there some caching mechanism at work that can be disabled? If not, where does webpack get the outdated sources from?
I know that doing something like this in package.json :
....
...
"dependencies" : {
"some-node-module" : "*"
}
is a bad idea since you're basically telling node to always update this module to its latest version, even though your code might not be able to handle any other version other than the current one for this particular module.
So I should instead do something like this :
....
...
"dependencies" : {
"some-node-module" : "3.4.1"
}
Which basically tells node to always use the version of the module that my code was built around.
Question
I have an app which I've first tested locally. The app has now been built, and using the package.json dependencies, npm has installed all of the appropriate node modules locally under my app's root folder (as opposed to globally, in some obscure folder I don't have immediate access to and which is irrelevant to this app - I simply don't like global installations of node modules - I find them to.. "abstract").
Given that all of the node modules are now installed locally isn't the node modules dependencies part in my package.json now redundant ?
I mean, what if something happens and npm is not available or the specific version of a module can't be found?
Isn't it best to be independent of dynamic node module installations and just have everything installed locally the first time without having to use the package.json dependencies ?
npm install & update
"you're basically telling node to always update this module to its latest version"
Packages won't be automatically updated. The only time the "*" will be an issue is when you are installing the project for the first time via npm install or when you manually run an update via npm update.
I personally prefer to pick a specific version of a module rather than use any wildcards, but even then there are some gotchas...which is why npm shrinkwrap exists.
npm shrinkwrap
Next gotcha:
basically tells node to always use the version of the module that my
code was built around
Sorta true. Let's say you use version 1.2.3 of your favorite module and package.json reflects that, but in the module itself is a package.json dependency on another module and that uses "*"...so when you install, the new internal dependency and the wildcard can wind up breaking the module you thought was 'locked down'.
See the gotcha? Hard coding a version controls for the top level versions but does not enforce anything beneath that...and if a module author you depend upon (or a module they depend upon) uses wildcards, you can't be 100% sure things will be copacetic.
To strictly enforce a version, you'll want to use npm shrinkwrap. (The link there to the docs provides more background, which is good to understand if your project uses more than a few very simple modules.)
And now...your question.
You say:
I mean, what if something happens and npm is not available or the
specific version of a module can't be found?
Based on the first two parts of this answer, it should now be clear that it doesn't hurt to have the dependencies explicitly listed in the package.json because node isn't checking things every time the app runs. npm uses package.json when specific actions (install, update, etc) are called but even then, it is a manual trigger.
While situations vary, there are very few that I can imagine where omitting dependencies in package.json is a good idea. If you ever wind up having to rebuild the project, you'll be in trouble. If the project is so good you want to share it, you'll be in trouble. Heck, if this is something for work and you want to go on vacation and need to install it on another machine...you'll be in trouble.
So given the fact that after the initial install, dependencies have no negative impact...use --save or add the dependencies to your package.json. Your future self will thank you. :)
I'm having problems following the 'official' recommendation to check in all external dependencies into git (article http://www.mikealrogers.com/posts/nodemodules-in-git.html linked fron FAQ)
How do you make sure that not only top-level dependencies are checked-in? Most npm modules do currently not follow the recommendation. They all have their node_modules in .gitignore . Just Deleting their .gitignore seems risky.
For compiled module the article recommends to check-in only the sources and run 'npm rebuild' and deploy time. Unfortunately 'npm rebuild' does not do a 'clean make' for all modules (despite bugfix https://github.com/isaacs/npm/issues/1872 being included in npm version 1.0.106 i'm using). This means that I have to prevent compile targets from being checked in (otherwise i would have object code compiled for the developer machine on the production machine without being overwritten by npm rebuild). But: how do i do this? Unfortunately the modules don't have a common compile output directory, so just git-ignoring "node_modules//build" and "/node_modules//out/" (as mentioned in this good article eng.yammer.com/blog/2012/1/4/managing-nodejs-dependencies-and-deployments-at-yammer.html won't help in every case.
Short version: how do you make sure that production servers use the exact same version of all dependent modules as you use during development?
UPDATE: there is now npm shrinkwrap which solves the problem of locking down exact dependency versions, even of dependencies' dependencies! More info here.
Checking in node_modules can be problematic, as the environment it's running on may differ from user to user - so what is compiled on some environment may not work on another. Plus it would fill up your changelogs and repositories with 3rd party code. Which I take it is the conslusion you've come to with your short version of the question, so let me address that.
Short version: how do you make sure that production servers use the exact same version of all dependent modules as you use during development?
Inside your package.json, there will be dependencies: {}, if it is not there, then add it. To accomplish what you want, add your dependencies as the key, and their exact versions as the value. E.g. dependencies: { docpad: '2.5.0', mocha: '1.1.0' }
However, generally (it depends on the author) upgrades to the revision number (the x.x.X number) are just bugfixes and safe. You can allow minor changes by doing dependencies: { docpad: '2.5.x', mocha: '1.1.x' } which saves you from having to update your package.json and do a release everytime there is a bugfix release. You can even do things like 2.x if you wish.
This is the solution I've come to use for all of my modules, as it ensures that even 6 months later or whatever the module will still work - whereas doing something like >= 2.0.0 means when v3 of a dependency comes out, your module will probably be unusable at that time. Ensuring you stick to specific versions "guarantees" stability over time.
For reference you can see how I've done it in my open-source node.js modules here
About the .gitignore of your dependencies (in "node_modules"), npm 1.1 ignores the .gitignore files, so that they are not installed;
npm 1.1 will exclude .gitignore files from the things it installs.
npm 1.0 did not have this feature, so you have to be careful about that.
Deleting them recursively is fine:
find node_modules -name .gitignore | xargs rm
But, in npm 1.1, you never have to do this, because it excludes them
from the install automatically.
That's coming from the chief himself (Isaac), and it's here and seems to cover pretty much everything. The "extraneous" problem I have must be something silly I've done, I'll try a clean setup.