npm audit versus yarn audit - node.js

I have a React Native project (0.61.4) that uses yarn as its package manager.
When I run yarn audit a huge number of vulnerabilities are reported:
18202 vulnerabilities found - Packages audited: 958823
Severity: 18202 High
✨ Done in 14.34s.
Most are in some very deep dependency paths. For instance:
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ high │ Prototype Pollution │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in │ >=4.17.12 │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ react-native │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ react-native > #react-native-community/cli > │
│ │ metro-react-native-babel-transformer > #babel/core > lodash │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://www.npmjs.com/advisories/1065 │
└───────────────┴──────────────────────────────────────────────────────────────┘
When I run npm audit, it first reports:
Neither npm-shrinkwrap.json nor package-lock.json found: Cannot audit a project without a lockfile
So I run:
npm i --package-lock-only
A package-lock.json file is generated. On inspection this file seems correct.
When I now run npm audit, the results are:
=== npm audit security report ===
found 0 vulnerabilities
I don't understand the discrepancy between these two package managers. Why does npm report 0 errors, and yarn 18.202?

It's very hard to estimate why is this happening without looking at both the lock files and comparing. But, as far as I can tell, it can happen only if both the lock files are resolving to different versions of same dependencies.
Your yarn.lock file was generated earlier, thus it contains vulnerable and old versions of dependencies and since the package-lock.json was generated afterwards, it would have resolved to latest/fixed versions of those dependencies.
Remember that npm i --package-lock-only would just create the package-lock.json file, not install anything, but it won't be at-par with the actual installed packages. I think you assumed that running that command would just derive the lock file from installed packages, but it actually generates the lock file as if you ran it without the flag.
So in conclusion, both the lock files are resolving to different (minor/patch)versions of same dependencies.

This is not an apples-to-apples comparison between Yarn and Npm. It's true that they don't report the exact same audit warnings, but you could get the same/similar result from Yarn if you did the following:
rm ./yarn.lock
yarn
yarn audit
Why is that? Because Yarn was working with knowledge from your previous dependency management work, while Npm started from scratch, having never been run before in your project. If you remove yarn.lock, Yarn will start from scratch as well. Either way, you will get the latest patch level versions of all your libraries, which you wouldn't have otherwise. This will cause many of your audit warnings to go away.
So what are yarn.lock and package-lock.json? After each successful install run, Yarn saves the latest state of your node_modules (your dependency tree) in yarn.lock. NPM does the same in package-lock.json. These two auto-generated files are like detailed versions of your package.json, listing every dependency und sub-dependency you have installed (with version numbers down to the x.x.x patch level) – and why they were installed. That way, your node_modules will look exactly the same on each run of yarn install or npm install.
However, these 2 lock files are exclusively used by the respective package manager and not compatible: Yarn and Npm would likely keep changing your node_modules if you use them in an alternating fashion. This will cause lots of subtle bugs. That's why you generally should stick with using either Yarn or Npm in one project.

Related

When npm installs the same dependencies of different versions for different root dependencies, where are they located and how are they tracked?

I had no good idea on how to word this, but let me try to show an example and explain.
taking a look at the json5 dep used in our app via npm ls json5 you will see 2 different verions of 2.2.1 and 1.0.1
├─┬ #babel/core#7.20.5
│ └── json5#2.2.1
├─┬ babel-loader#8.3.0
│ └─┬ loader-utils#2.0.4
│ └── json5#2.2.1 deduped
└─┬ eslint-plugin-import#2.26.0
└─┬ tsconfig-paths#3.14.1
└── json5#1.0.1
Now my question relates to how npm installs and knows which version to use for which dependency (no matter how deep) and where are the located.
For example on my where are the located. part of my question.
When i search my root node_modules folder, and look for eslint-plugin-import dep, i find a node-modules within it but it only 3 things, I thought i would find another folder that is labeled tsconfig-path and inside that, find another node_modules folder that contains json5#1.0.1. But i dont, the only thing i do find is my root node_modules folder contains all three, eslint-plugin-import, tsconfig-paths, and json5.
I also noticed, that even thought this dep tree shows version 1.0.1 and 2.2.1 for json5, my root node_modules folder is the 2.2.1 version.
So that makes me wonder, where is the 1.0.1 version located, and how does it keep track of these different verions for these dep trees to know which one to us since i couldnt even find the 1.0.1 version physically anywhere. I guess i need a better understanding on how npm works and how the npm ecosystem for dependencies.
Sorry for long explanation, its just hard for me to rely my thoughts in a shorter summary.

shell-quote 1.7.2 vulnerabilities -unable to fix it using force reslutions

I have a nextjs app which has "next": "^10.2.0". It in turn has shell-quote as a transitive dependency and the version installed in 1.7.2 which has some critical security vulnerabilities. I have to fix this for now, and shell-quote version 1.7.3 does not have these vulnerabilities. So I added this
"preinstall": "npx npm-force-resolutions"
and
"resolutions": {
"shell-quote": ">=1.7.3"
}
in package.json.
But it still gives me the error and when I check npm ls shell-quote, I see that
├─┬ #storybook/react#6.4.9
│ └─┬ react-dev-utils#11.0.4
│ └── shell-quote#1.7.2
└─┬ next#10.2.3
└─┬ #next/react-dev-overlay#10.2.3
└── shell-quote#1.7.2 deduped
Does this mean, next#10.2.3 cannot have shell quote of 1.7.2? Can this issue be fixed for now without a nextjs upgrade?
You don't need to use resolutions as you are not changing the package but only the version of it. Override is fine here. So, add following code block to the package.json
"overrides": {
"react-dev-utils#11.0.4": {
"shell-quote": "1.7.3"
},
"#next/react-dev-overlay#10.2.3": {
"shell-quote": "1.7.3"
}
},
Now remove node_modules with rm -rf node_modules. Then, you have two options:
Remove package-lock.json completely. This way, you'll lose locked versions for your other packages too.
Or, open package-lock.json and remove all entries with react-dev-utils, node_modules/react-dev-utils, #next/react-dev-overlay, node_modules/react-dev-overlay, shell-quote and node_modules/shell-quote. This way, you'll keep locked versions for other packages.
And run npm install, when you run npm list shell-quote, you'll see all packages uses it with v1.7.3.
Since npm install will edit your package-lock.json for this change, you will have complete packge-lock.json and won't have to edit it next time you run npm install.
I've been editing our repository to get rid of vulnerabilities in this way and it works fine. I've used this answer from another SO question.
Of course, you have to make sure that these versions are compatible with each other. Because with overrides, npm does not do any checking for the versions, you force it. For example, you have to make sure that react-dev-utils#11.0.4 can work with shell-quote#1.7.3. In minor version upgrades, libraries generally work as the same before but it doesn't always have to be this way.

npm dependencies pull in different versions of the same package

Wondering if someone can help me understand why this is happening. I have two npm packages that use tar-fs#2.1.0 but that package pulls in different versions of tar-stream. It is defined in the package.json as "tar-stream": "^2.0.0" but they resolve to different patch versions when running npm install. Reading about caret-ranges I'm not seeing anything to explain this.
I tried deleting the node_modules and lock file, running npm update tar-stream, and running npm install --package-lock-only to update the lock file based on 'package.json'. Any help is appreciated.
$ npm ls tar-stream
project#1.2.0 /Users/akerr/Documents/GitHub/project
├─┬ chrome-aws-lambda#5.3.0
│ └─┬ lambdafs#2.0.0
│ └─┬ tar-fs#2.1.0
│ └── tar-stream#2.1.3
└─┬ puppeteer-core#5.3.0
└─┬ tar-fs#2.1.0
└── tar-stream#2.1.4

Unmet Peer Dependency (WebPack)

Anybody knows why I am still having a missing dependency error, even though it clearly shows the correct version of webpack is already installed below??
When I ran npm start :
'''
There might be a problem with the project dependency tree.
It is likely not a bug in Create React App, but something you need to fix locally.
The react-scripts package provided by Create React App requires a dependency:
"webpack": "4.41.5"
Don't try to install it manually: your package manager does it automatically.
However, a different version of webpack was detected higher up in the tree:
When I run npm ls webpack, it gives me :
Chelseas-MacBook-Pro:website-expo-2018-master ipchelsea$ npm ls webpack
uwbce#0.1.0 /Users/ipchelsea/Desktop/website-expo-2018-master
├─┬ react-loading-screen#0.0.17
│ └── webpack#2.7.0
├─┬ react-scripts#3.4.0
│ └── webpack#4.41.5
└── webpack#4.41.6
You missed out the steps you took to get here. You did something, or missed something out in the steps you did to end up where you are now.
You should delete node_modules, then do npm i, and see if that correctly installs the packages.
Also, add the contents of your package.json file to the question. You need to have one in the root of this project.

How to get rid of the ‘hoek’ vulnerabilities

I recently pushed an Angular CLI 5 application to GitHub and it indicated the following:
We found a potential security vulnerability in one of your dependencies.
A dependency defined in net-incident/package-lock.json has known security vulnerabilities and should be updated.
Dependencies defined in net-incident/package-lock.json 816
hapijs / hoek Known security vulnerability in 2.16.3
I have gone through the output from ‘npm audit’ and executed the various updates, including the following (which was not suggested):
npm install --save-dev request#2.86.0
The ‘request’ package contains ‘hawk’ which contains ‘hoek’. When I look at the ‘request’ package in node_modules the version has changed. But the following two updates from ‘npm audit’ do not seem to do anything:
npm update fsevents --depth 4 npm update stringstream --depth 5
And I am left with the following:
[!] 33 vulnerabilities found [12201 packages audited]
Severity: 5 Low | 24 Moderate | 4 High
Run `npm audit` for more detail
And many of the vulnerabilities are like the following:
Moderate Prototype pollution
Package hoek
Patched in > 4.2.0 < 5.0.0 || >= 5.0.3
Dependency of karma
Path karma > log4js > loggly > request > hawk > boom > hoek
More info https://nodesecurity.io/advisories/566
In the end, the application would not compile, so I replaced the the package and lock files, and now I am back to the beginning. I really want to fix the security issues. How do I get rid of the pesky ‘hoek’ vulnerabilities?
You should runrm package-lock.json && npm update && npm install, if this still doesn't fix your issue, you can then continue by running npm ls hoek, which should gave you:
├─┬ fuse-box#3.3.0
│ └─┬ request#2.81.0
│ └─┬ hawk#3.1.3
│ ├─┬ boom#2.10.1
│ │ └── hoek#2.16.3
│ ├── hoek#2.16.3
│ └─┬ sntp#1.0.9
│ └── hoek#2.16.3
└── hoek#5.0.3
Check the version of hawk against the one on npm hawk, if it doesn't tally, run npm i hawk --save or npm i hoek#latest --save, then you should also run: npm i karma#latest --save, then npm audit After which I again ran my normal git commands:
git add .
git commit -m 'whatever_message'
git push
Then you can go back to Github, the security vulnerability should be fixed.
This answer addresses similar hoek problem, and this answer explains non-vulnerability audit reports in detail.
npm audit reports possible problems. It's unnecessary that they are real problems that should be solved.
A nested dependency like karma > log4js > loggly > request > hawk > boom > hoek may require to fork numerous packages in dependency chain in case it has to be fixed.
Prototype pollution diagnosis indicates code smell. The reason why prototype pollution smells is that it can cause security problems. This is the reason why it's labeled as Moderate. It's unlikely that it causes any security risks in hoek package due to how it works, regardless of how the package is used (that's important as well).
Additionally, karma > log4js > loggly > request > hawk > boom > hoek dependency chain means that the problem occurs in development dependency. Most security problems are primarily applicable to dependencies that are used in production. This problem is specific to tests and Karma. It's virtually impossible it is a threat.
TL;DR: this is not a vulnerability. It has to be ignored. Any npm audit report should pass sanity check before any efforts to fix it will be made.
I was patient and they fixed the problem:
npm update karma#latest
should work.

Resources