We have a node.js project, and we want to start managing its dependencies using npm's package.json with specified versions for each dependency.
However, we are afraid that one of the packages our project depends on might get unpublished. Should I worry about unpublishing or is it a rare occurrence? What is the most effective way to handle this kind of problems?
It is very rare occurence. Never happened to me.
Unpublish is mostly used to remove a published version in which a major bug is reported. Thus, automatic semantic versioning upgrade will not fetch this version until a new one is published.
Related
At work we're using a very old template (generated around April 2021, so node v14.19) which has an out of sync package-lock.json. This means that if you do:
rm package-lock.json && npm install
The install will fail due to conflicting dependencies.
For a couple of weeks me and my teammates tried to fix this, but we haven't succeed yet: when you fix the dependencies you break eslint, when you fix eslint you break deployment, when you fix deployment then the logger will stop working and so on. We have thousands of dependencies that are turning out to be an hell to maintain.
The pov of our CTO is that we simply shouldn't delete the package-lock.json, but this means we can't update node and we are stick with what I think is a huge technical risk.
Do you think it's fixable? Have you ever been in a similar situation?
Is not deleting the package-lock.json enough to avoid the problem?
If not, how could I produce an example where I can break the flow? maybe by installing a modern package that is incompatible with the old resolution?
This is one of the reasons why both modular code and teams are important. Chunking the code up to one big file or just a few files will cause this kind of mess.
Do you think it's fixable? Have you ever been in a similar situation?
Yes
Is not deleting the package-lock.json enough to avoid the problem?
No. As said in the comments, it is a big security risk not to keep your dependencies up to date. That's why you have package managers like npm. Plus, new vulnerabilities are openly discussed in different forums, so not only are scrupulous elements aware of them, good intentioned programmers are aware of them and will definitely judge your software to be of low quality. Also, you put your clients at a big risk of running into trouble with your software and put your company at risk of facing litigations.
If not, how could I produce an example where I can break the flow? maybe by installing a modern package that is incompatible with the old resolution?
My Suggested Solution:
Leave the code as is (Good thing it is still working even with new input data).
Modularize the code. Put chunks of related functionalities into separate files and import them into the main code (make sure everything is still working).
Assign teams to the seperate files (modules) to build new versions of the code (there also have to be modular testing implemented here so you can test each module independent of the main file).
For each test unit, make sure they have their own up-to-date package.json file.
Integrate everything into a new project software.
One advantage of doing things this way is that the main file code rarely changes. Also, each module can be updated independent of the main program and other modules. Only downside to this approach is that you have to manage the package.json file intentionally so that dependecies do not conflict or are not recursive during integration, thereby (sometimes) requiring a seperate team to manage integration.
Dependabot first reported and then retracted a security problem in a package. The basis of the retraction isn't given, just that the package "is no longer vulnerable." That makes no sense. The original CVE is still out there and the affected code is still referenced.
On investigation I find that the given package is in yarn.lock twice, once in a version that contains the vulnerability and later in a version that contains the patch:
ansi-regex#^2.0.0:
version "2.1.1"
ansi-regex#^5.0.0:
version "5.0.0"````
I'd be grateful for any way to make sense of this.
If you landed on this question due to your own "no longer vulnerable" error in a different package: you may still be vulnerable.
According to one of the Dependabot maintainers, the (suspected) most common cause of "no longer vulnerable" is when you are using multiple versions of the same dependency, and one of them is vulnerable, but not the lowest version. The maintainer's post says, in part:
Here's my current understanding:
The alerts are triggered by manifest parsing code that is a separate code path from dependabot-core's parsing code. So if the
alert manifest parsing code thinks the repo is still vulnerable, the
alert will persist.
npm allows multiple versions of a dependency in the dep tree... and these are used in the code at runtime... the newest is not
superseding the old one. So if those exist, you're still vulnerable.
dependabot-core has a known bug where it only updates the lowest version of a dependency... so Dependabot may try to create a PR, then
report that npm is no longer vulnerable, when in fact a later version
of the dependency is vulnerable and is still in the tree. That's
tracked in npm erroneously reports no longer vulnerable with multiple versions of dependency. #5741
The alerts are generated based on information in the GitHub advisory database. So if that database lists all versions > 0.16.5 are
vulnerable, but the maintaner just pushed v0.17.0 which fixes the
issue, then the alert will not disappear until the advisory database
is updated. Thankfully, you can submit a PR to the advisory DB to fix
an incorrect version specifier.
So I suspect that the majority of these cases are symptoms of #5741.
Dependabot issue #5741 says:
when there is a vulnerability that affects some versions of the dependency but not the lowest version. In this case Dependabot incorrectly reports that no security update is needed.
If this source is accurate, it's because the two ansi-regex versions you've listed do not contain the vulnerability:
Confirmed 4.1.0 and 3.0.0 as affected testing using the provided reproducer. 2.1.1 does not reproduce the issue.
3.0.0 is the first affected, as that's the first version that includes 69bebf6 that the problematic part of the regex.
Thanks to answerers for your expertise.
I think the most likely diagnosis is a Dependabot bug. Generally it is wishful thinking to blame your tools but in this case it is the simplest answer by far.
Which is the best practice while saving package.json dependencies?
For example, i see that lot's of dependencies are not fixed, like:
"tslint": "~5.11.0"
I would like to have fixed dependencies, so that will not change in the future when new developers join a team.
I have little knowledge about package-lock.json and shrinkwrap, but I'm not sure about the "best practice" on this.
On this case is an Angular app, but it can be everything. Keeping for example package-lock.json on the repo caused some issues in the past (i know! it is a best practice to push it!)
Any thoughts?
Short answer: Carets (^) and committing your package-lock.json is probably your best approach. This ensures developers always get the same dependencies, and is the least surprising.
Why package-lock.json?
npm specifically recommends you commit your package-lock.json.
It is highly recommended you commit the generated package lock to source control: this will allow anyone else on your team, your deployments, your CI/continuous integration, and anyone else who runs npm install in your package source to get the exact same dependency tree that you were developing on.
(from the npm documentation)
You mentioned pushing package-lock.json to your repository caused some issues in the past. I'm guessing this was due to this issue where the package lock was being ignored and rewritten every time anyone installed anything. This was not the correct behaviour and was fixed in npm#5.4.2, according to this answer.
What you should not do is leave out the package-lock.json and just specify exact versions in your package.json. If you do this, your top level dependencies will look nice and consistent, but their dependencies won't have locked down versions. This way, you're almost as likely to run into bugs, but they'll be harder to find.
Why not npm-shrinkwrap.json?
You also mention shrinkwrap files. Shrinkwrap files are meant for
applications deployed through the publishing process on the registry
(from the npm documentation)
You probably aren't npm publishing your angular webapp, so there's no reason to use npm-shrinkwrap.json.
Why use caret ranges?
I can't find any documentation saying caret (^) ranges are best practice, but I believe they are.
npm makes caret ranges the default option, so it's clear they think this is best practice, though I can't find any of their documentation to justify it.
Using the default is the least surprising approach. If I saw any other kind of version in a package.json, I'd assume it was changed for a good reason, and would be hesitant to update the package without knowing what that reason is, even if it really needed to be updated.
If you ever decide to update all your dependencies at once, caret ranges will serve you well. You're dependencies will normally be locked, but deleting your package-lock.json and rerunning npm install will automatically install the latest versions that are supposedly backwards compatible with the versions you specified (see the npm docs for details on the caret range).
In summary, it's standard to use caret ranges and a package-lock.json. This fulfills your requirement of fixed dependencies, and provides a few other benefits, so it's best to do what's standard, unless you find another reason to change.
If I update node or a package, will it affect the applications I have that are currently dependent on the previous node/package version I had? If yes, how do I fix this? Maybe like a virtual environment :)
You should be looking at the update from 2 standpoints.
1. Node itself
2. Updating NPM Packages
For instance, there is the Node LTS which at this time is 6.1.10 and then there is v7.7.3. If you desire total stability then use the LTS. As Cihan said above, upgrading Node can be a long process if you have a system running on an older version.
However, if you want to test out the new async/await (async functions) which is already in 7.7.3 and should be released officially in 7.8 then 7.7.3 is the way to go. But keep in mind some things just may not work as you think they should or you may get some wonky results.
Also be aware depending on your server or system the Node update works differently and make sure you read the documentation for that specific system you need to upgrade.
NPM is a different ballgame. You are reliant upon many different programmers or groups of programmers. The package is depending upon their capabilities and desire to maintain the backwards compatibility. Most package creators are really good about this. Some are not.
Take a for instance. MongoDB issued a really new driver which is an incredible upgrade from its previous one. The new driver contains ES6 and in conjunction with the co package, it is basically operating with promises.
Updating this package for MongoDB was essential for me. But it does maintain my previous code as well, (even though lots of it may be superfluous now!)
So when you think about updating Node..it is not the same as updating NPM modules. But if you wish, you can go to our project root where package.json exists and just type npm update and all packages will be updated. You can update to only a specific version - take a look here.
Remember also, NPM itself also requires updating from time to time.
So in summary:
Node Version - decide which one based upon needs, requirements and your own servers.
NPM as NPM also needs updating from time to time
NPM packages can be updated constantly with npm update or just update to a specific version number based upon the url above.
Not as confusing as it first seems, once one gets it all down straight :)
Good luck
It's possible that you have a package that is being used by another package which a version change could break it. There are several options, if this is a personal project then you would have to either reinstall the correct version of the package or see if there are updates available otherwise for the packages that broke.
It should not, because npm was made to resolve this issue: each project has its own dependencies, and it is totally independent with other projects.
In a word, no matter what you change, you do npm install in your project, npm will fix the dependency issue by itself (by checking your project's package.json)
npm-packages sometimes change or deprecate certain functions & functionalities in newer versions. If something stops working when you update a package, you must figure out from the documentation of the updated package on how to make it work again. Many packages might also give runtime information/errors in the console about functionalities that have deprecated or that have been marked for deprecation in future versions.
npm will download specific versions of a packages' dependencies (and thus you can often have multiple version of a certain package in one project), so you shouldn't need to worry about the dependencies of the packages you are updating, only about the changes of the package itself.
Certain npm-package versions only support certain versions of Node, so updating the node version might require updating some packages. If you switch the node version, it will also switch the npm version, which will then install the correct version of the package per node version.
Some Node.js functionalities might stop working if you update Node.js. In these cases you must refer to the Node.js docs for help. Sometimes updating Node.js from a old version to a new version in a project is a large and tedious task.
Do you know any tool that monitors breaking changes in github npm or bower projects?
I'd like to list all changes from commit history that are marked with breaking change and list them.
I am using npm-check-updates, that tells me what is new, but it doesn't tell me what has been changed since.
Recently, I have found greenkeeper.io, but as far as I know it doesn't list what is new, it just simply does upgrade and see if your tests are still running. If tests fails, you have to fix it yourself.
In ideal opensource world author marks every breaking change. Also he writes code without bugs. In the real world is not true. You have opensource packages for free, but with possible bugs and undocumented changes API.
There is only working answer, testing your application for each dependencies update. You should have tests and shrinkwrap file.
greenkeeper.io is amazing tool. It makes pull request with dependency changelog. Check example