Can anyone please let me know the exact use of package-lock.json file ?
Though many have mentioned that it is used for viewing the versioned dependency tree.
Looking for simple and easier explanation.
Thanks in advance.
npm install use this file to make sure the packages it is going to install, is the same as dictated in this file. It makes npm install operation consistent across machines. Thus, you will be more unlikely to nuke the node_modules folder.
On top of consistent packages view, GitHub also use package-lock.json to scan if your repository contains known security vulnerability.
You can use lock-walker to visually walk the dependency tree in package-lock.json, especially useful when checking out security vulnerability.
I think that npm documention is quite explanatory.
Its main purpose is to provide
single representation of a dependency tree such that teammates, deployments, and
continuous integration are guaranteed to install exactly the same dependencies.
so that for example on a different system and/or by different people, the same dependencies (and same versions) will be used.
For a better explanation see this
Hope this helps.
Related
At work we're using a very old template (generated around April 2021, so node v14.19) which has an out of sync package-lock.json. This means that if you do:
rm package-lock.json && npm install
The install will fail due to conflicting dependencies.
For a couple of weeks me and my teammates tried to fix this, but we haven't succeed yet: when you fix the dependencies you break eslint, when you fix eslint you break deployment, when you fix deployment then the logger will stop working and so on. We have thousands of dependencies that are turning out to be an hell to maintain.
The pov of our CTO is that we simply shouldn't delete the package-lock.json, but this means we can't update node and we are stick with what I think is a huge technical risk.
Do you think it's fixable? Have you ever been in a similar situation?
Is not deleting the package-lock.json enough to avoid the problem?
If not, how could I produce an example where I can break the flow? maybe by installing a modern package that is incompatible with the old resolution?
This is one of the reasons why both modular code and teams are important. Chunking the code up to one big file or just a few files will cause this kind of mess.
Do you think it's fixable? Have you ever been in a similar situation?
Yes
Is not deleting the package-lock.json enough to avoid the problem?
No. As said in the comments, it is a big security risk not to keep your dependencies up to date. That's why you have package managers like npm. Plus, new vulnerabilities are openly discussed in different forums, so not only are scrupulous elements aware of them, good intentioned programmers are aware of them and will definitely judge your software to be of low quality. Also, you put your clients at a big risk of running into trouble with your software and put your company at risk of facing litigations.
If not, how could I produce an example where I can break the flow? maybe by installing a modern package that is incompatible with the old resolution?
My Suggested Solution:
Leave the code as is (Good thing it is still working even with new input data).
Modularize the code. Put chunks of related functionalities into separate files and import them into the main code (make sure everything is still working).
Assign teams to the seperate files (modules) to build new versions of the code (there also have to be modular testing implemented here so you can test each module independent of the main file).
For each test unit, make sure they have their own up-to-date package.json file.
Integrate everything into a new project software.
One advantage of doing things this way is that the main file code rarely changes. Also, each module can be updated independent of the main program and other modules. Only downside to this approach is that you have to manage the package.json file intentionally so that dependecies do not conflict or are not recursive during integration, thereby (sometimes) requiring a seperate team to manage integration.
Which is the best practice while saving package.json dependencies?
For example, i see that lot's of dependencies are not fixed, like:
"tslint": "~5.11.0"
I would like to have fixed dependencies, so that will not change in the future when new developers join a team.
I have little knowledge about package-lock.json and shrinkwrap, but I'm not sure about the "best practice" on this.
On this case is an Angular app, but it can be everything. Keeping for example package-lock.json on the repo caused some issues in the past (i know! it is a best practice to push it!)
Any thoughts?
Short answer: Carets (^) and committing your package-lock.json is probably your best approach. This ensures developers always get the same dependencies, and is the least surprising.
Why package-lock.json?
npm specifically recommends you commit your package-lock.json.
It is highly recommended you commit the generated package lock to source control: this will allow anyone else on your team, your deployments, your CI/continuous integration, and anyone else who runs npm install in your package source to get the exact same dependency tree that you were developing on.
(from the npm documentation)
You mentioned pushing package-lock.json to your repository caused some issues in the past. I'm guessing this was due to this issue where the package lock was being ignored and rewritten every time anyone installed anything. This was not the correct behaviour and was fixed in npm#5.4.2, according to this answer.
What you should not do is leave out the package-lock.json and just specify exact versions in your package.json. If you do this, your top level dependencies will look nice and consistent, but their dependencies won't have locked down versions. This way, you're almost as likely to run into bugs, but they'll be harder to find.
Why not npm-shrinkwrap.json?
You also mention shrinkwrap files. Shrinkwrap files are meant for
applications deployed through the publishing process on the registry
(from the npm documentation)
You probably aren't npm publishing your angular webapp, so there's no reason to use npm-shrinkwrap.json.
Why use caret ranges?
I can't find any documentation saying caret (^) ranges are best practice, but I believe they are.
npm makes caret ranges the default option, so it's clear they think this is best practice, though I can't find any of their documentation to justify it.
Using the default is the least surprising approach. If I saw any other kind of version in a package.json, I'd assume it was changed for a good reason, and would be hesitant to update the package without knowing what that reason is, even if it really needed to be updated.
If you ever decide to update all your dependencies at once, caret ranges will serve you well. You're dependencies will normally be locked, but deleting your package-lock.json and rerunning npm install will automatically install the latest versions that are supposedly backwards compatible with the versions you specified (see the npm docs for details on the caret range).
In summary, it's standard to use caret ranges and a package-lock.json. This fulfills your requirement of fixed dependencies, and provides a few other benefits, so it's best to do what's standard, unless you find another reason to change.
How should we deal with local packages that are a dependency in other local packages?
For simplicities sake, say we have the follow packages
api - express application
people - a package to deal with people
data-access - a package that deals with data access
And then the dependencies are
api depends on people
people depends on data-access
Currently we have these dependencies setup as file dependencies.
I.e. api package.json would have
"dependencies": {
"people": "file:../people"
}
Trouble with this is that we're finding it a PITA when we make updates to one package and want those changes in the other packages that depend on it.
The options we have thought of are:
npm install - but this won't overwrite previously installed packages if changes are made, so we have to delete the old one from the node_modules directory and re-run npm install... which can be niggly if the package dependency is deep.
npm link - we're not sold on the idea because it doesn't survive version control... Just thinking about it now, maybe we have some kind of local build script that would run the npm link commands for us... this way it could survive version control. Would that be a grunt job?
grunt - we haven't dived too deep into this one yet, but it feels like a good direction. A little bit of googling we came accross this: https://github.com/ahutchings/grunt-install-dependencies
So, what option would work best for our situation?
Are there other options that we haven't thought of yet?
Ps. we're a .NET shop doing a PoC in node, so assume we know nothing!
Pps. if you strongly believe we're setting up our project incorrectly and we shouldn't have smaller individual packages, let me know in the comments with a link to some reading on the subject.
So, I agree that going with 'many small packages' is usually a good idea. Check out 12factor.net if you haven't already.
That said, in specific answer to your question I'd say your best bet is to consider mainly how you want to maintain them.
If the 'subcomponents' are all just parts of your app (as, for example, data-access implies), then I'd keep them in the same folder structure, not map them in package.json at all, and just require them where you need them. In this case, everything versions together and is part of the same git repository.
If you really want to or need to keep them all in separate git repositories, then you can do npm link, but to be honest I've found it more useful to just use the URL syntax in package.json:
dependencies: {
"people" : "git://path.to.git:repo#version.number"
}
Then, when you want to explicitly update one of your dependencies, you just have to bump the version number in your package.json and run npm install again.
What exactly should I put in .npmignore?
Tests? Stuff like .travis.yml, .jshintrc? Anything that isn't needed when running the module (except the readme)?
I can't find any guidance on this.
As you probably found, NPM doesn't really state specifically what should go in there, rather they have a list of ignored-by-default files. Many people don't even use it as everything in your .gitignore is ignored in npm by default if .npmignore doesn't exist. Additionally, many files are already ignored by default regardless of settings and some files are always excluded from being ignored, as outlined in the link above.
There is not much official on what always should be there because it is basically a subset of .gitignore, but from what I gather from using node for 5-ish years, here's what I've come up with.
Note: By production I mean any time where your module is used by someone and not to develop on the module itself.
Pre-release cross-compiled sources
Pros: If you are using a language that cross-compiles into JavaScript, you can precompile before release and not include .coffee files in your package but keep tracking them in your git repository.
Build file leftovers
Pros: People using things like node-gyp might have object files that get generated during a build that never should go into the package.
Cons: This should always go into the .gitignore anyway. You must place these things inside here if you are using a .npmignore file already as it overrides .gitignore from npm's point of view.
Tests
Pros: Less baggage in your production code.
Cons: You cannot run tests on live environments in the slim chance there is a system-specific failure, such as an out of date version of node running that causes a test to fail.
Continuous integration settings/Meta files
Pros: Again, less baggage. Things such as .travis.yml are not required for using, testing, or viewing the code.
Non-readme docs and code examples
Pros: Less baggage. Some people exist in the school-of-thought where if you cannot express at least minimum viable functionality in your Readme, your module is too big.
Cons: People cannot see exhaustive documentation and code examples on their own file system. They would have to visit the repository (which also requires an internet connection).
Github-pages objects
Pros: You certainly don't need to litter your releases with CNAME files or placeholder index.htmls if you use your module serves double-duty as a gh-pages repository as well.
bower.json and friends
Pros: If you decide to build in your dependencies prior to release, you don't need the end-user to install bower then install more things with that. I would, personally, keep that stuff in the package. When I do an npm install, I should only be relying on npm and no other external sources.
Basically, you should ever use it if there is something you wish to keep out of your npm package but checked-in to your module's repo. It's not a long list of items, but npm would rather build in the functionality than having people stuck with irrelevant objects in their package.
I agree with lante's short and syntetic answer and SamT's big answer:
You should not include your tests in your package.
Your package should only contains production runtime files.
That will make your package more straightforward and faster to be dowloaded.
My contribution to those answers:
.npmignore is the blacklist way to achieve package file selection. But in a more practical way, you can whitelist files you need to include in your package using the files field in your package.json:
{
"files": [
"lib/",
"index.js"
]
}
I think that's simpler, future proof and have better semantics ;)
Just to clarify, anytime someone do npm install your-library, npm will download all source files that the package includes. Those files that were included in the .npmignore file in the source code of the package your-library will be excluded when publishing the lib, so users of your-library won't download them.
Know that people installing your library will need just your library running, anything else will be not necessary.
For example, when someone installs a library, its probably that he/she doesn't care about your .travis.yml or your .jshintrc files, or even some images, Grunt files, documentation, etc.
.npmignore could let your npm package to have less files, and faster to be downloaded
Don't include your tests. Oftentimes tests are like 5x the size of the actual codebase. As long as your tests are on Github, etc, that's good enough.
But what you absolutely should do is test your NPM package in its published format. Create some smoke tests that reside in the actual codebase, but are not part of the test suite.
You can read about testing your package after tarballing it, here:
https://github.com/ORESoftware/r2g
How to test an `npm publish` result, without actually publishing to NPM?
In doing research on whether Node's node_modules should be checked into your version control repository, the most recent consensus seems to be that you should include it for applications you deploy.
Sources:
Check in node_modules vs. shrinkwrap
Should I check in node_modules to git when creating a node.js app on Heroku?
https://www.npmjs.org/doc/misc/npm-faq.html#Should-I-check-my-node_modules-folder-into-git
In reading these arguments, it lead me to question whether Composer's /vendors directory should also be checked into version control. Composer's documentation suggests that you don't:
Should I commit the dependencies in my vendor directory?
The general recommendation is no. The vendor directory [...] should be added to .gitignore/svn:ignore/etc.
The best practice is to then have all the developers use Composer to install the dependencies. Similarly, the build server, CI, deployment tools etc should be adapted to run Composer as part of their project bootstrapping.
While it can be tempting to commit it in some environment, it leads to a few problems:
Large VCS repository size and diffs when you update code.
Duplication of the history of all your dependencies in your own VCS.
[...]
Contrasting that argument is this one (source):
Doesn’t checking in node_modules create a lot of noise in the source tree that isn’t related to my app?
No, you’re wrong, this code is used by your app, it’s part of your app, pretending it’s not will get you in to trouble. You depend on other people’s code and they are just as likely to write new bugs as you are, probably more so. Checking all of that code in to source control gives you a way to audit every line that ever changed in your application. It allows you to use $ git bisect locally and be ensured that it’s the same as in production and that every machine in production is identical. No more tracking down unknown changes in dependencies, all the changes, in every line, are viewable in source control.
In summary, the question is this: Why would one gitignore (i.e. not version control) node_modules but not do the same for Composer's vendor/ directory?
The reason to commit external dependencies is
it's easier to deploy with git pull
the code used is directly included in the commit anyone checks out
Arguments against this are
Git is no deployment tool
all dependency managers do have a way to make exactly sure the code used can be fetched
I don't know anything about npm, but for Composer that last point is implemented by committing composer.lock.
I don't think the "audit code" argument is a valid one in every case. If you do write software that get's audited by itself, and subsequently needs all libraries to be audited, then probably all code changes need to be conserved. This isn't true for the general case.
git bisect works still as well with a committed composer.lock. It does require installing the dependencies with every bisect step, but this is just one simple step, probably already done in the automatic test suite run.
The last thing to worry about is when used packages go offline. This really is more of a problem with Composer, because there is no central hosting of the downloadable releases (npm probably does this). If this is a problem, either commit the code (and try to figure out how to update that missing package in the future), or setup an instance of Satis to create a local copy of the code you use.
Putting all your modules in you VCS makes it really heavy to download or upload. For example, I work on two node.js projects and the total node_modules directories size is between 250MB and 500MB whereas the whole code with assets is generally less than 40MB. Every Node.js developer likes Node lightness, so the code must stay easy to download and share.
For the second point, an alternative to avoid regressions is to be more restrictive in your package.json dependencies versions. You will find more information here.
Finally the best argument is to take a look on the famous modules everybody know :
express ignores
node_modules
mocha ignores
node_modules
q ignores node_modules
...
The more I research this the more I'm starting to think that there is no one correct answer to this, just different opinions as well as pros and cons of each method.
This blog, looking from the context of Bower, does a good job of weighing the pros and cons of each: http://addyosmani.com/blog/checking-in-front-end-dependencies/.
In short: At least for right now, weigh the pros and cons and decide what best fits your situation.