I installed express, mongodb, and mongoose.
This is the result of my npm ls:
/home/merc/Bookings
├─┬ connect-mongo#0.1.9
│ └─┬ mongodb#0.9.9-8
│ └── bson#0.0.4
├─┬ express#3.0.0rc2
│ ├── commander#0.6.1
│ ├─┬ connect#2.4.2
│ │ ├── bytes#0.1.0
│ │ ├── formidable#1.0.11
│ │ ├── pause#0.0.1
│ │ └── qs#0.4.2
│ ├── cookie#0.0.4
│ ├── crc#0.2.0
│ ├── debug#0.7.0
│ ├── fresh#0.1.0
│ ├── methods#0.0.1
│ ├── mkdirp#0.3.3
│ ├── range-parser#0.0.4
│ └─┬ send#0.0.3
│ └── mime#1.2.6
├─┬ jade#0.27.2
│ ├── commander#0.6.1
│ └── mkdirp#0.3.0
├─┬ mongodb#1.1.2
│ └── bson#0.1.1
└─┬ mongoose#3.0.0
├── hooks#0.2.1
├─┬ mongodb#1.1.2
│ └── bson#0.1.1
└── ms#0.1.0
You can clearly see that for some reason Jade is on the root directory (I assume this happened when I run "express". But then again, "mongodb" is available in different versions (0.9.9 and 1.1.2) and so is bson (0.1.1 and a worrying 0.0.4).
Hence my questions: how are dependencies managed with npm? Does every package simple install whatever they like, whichever version they pick?
I guess the question is: is this kind of duplication "normal", and "by design" so to speak?
Merc.
The short answer is, yes, this is by design. When you require a module from the node_modules directory, it uses the top-level directory--e.g., whichever one you specify in your package.json.
Other packages have their own package.json files, and are free to use whatever versions they want, and when they require them down in their own code, they will use their own node_modules folder.
Ideally, the modules you use have tests, etc. that ensure that versions (or even specify a range of versions, such as 0.9.x) of dependencies they specify work well, and seeing older versions of sub-dependencies in there doesn't necessarily mean danger, although new versions of these modules could of course potentially fix bugs and so forth. It may be worth finding a module you're concerned about on GitHub, downloading the repository, updating the package.json and dependencies yourself and running the tests to see if a new version works. If so, perhaps the author would be willing to accept a pull request with your update.
Related
I have monorepo setup and upgraded from yarn 2 to npm 7.
Was working perfectly fine till I tried to deploy my application to the standard app engine environment.
gcloud app deploy kicks off a build inside Cloud Build, which builds my project via buildpacks (nodejs14 buildpack).
My problem is, that this buildpack includes nodejs14 and npm#6 and without using the flex environment it doesn't seem that there is a way to use npm#7 in this standard app engine cloud build.
What I've tried by now:
Use same OS and install node_modules, tar them, upload them, untar them. (could not really do it)
Link packages locally before uploading (don't know if I did that right. App was not finding local packages afterwards)
Trying to use a cloudbuild.yaml, but it seems that this is not possible with standard environment.
in the custom build step "gcp-build": "npm i -g npm#7" (does not work, permission error, see full build log here)
I would really appreciate every pointer in any direction that could help me out.
This is my project setup:
[project]/
├── packages/
│ ├── app
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── app.yaml
│ │ ├── ...
│ │ └── package.json
│ ├── shared
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── shared-admin
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── base-server
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
├── package.json
│
└── node_modules/
Many people are using lerna and/or yarn workspace.
I guess either migrating from them to Bazel, or just using them with Bazel together is good to be guided with an example project.
For example, currently, I have a directory structure like this, where foo is an express server and bar is a library consumed by foo, both based on typescript.
<project root>
├── jest.config.js
├── lerna.json
├── package.json
├── packages
│ ├── bar
│ │ ├── jest.config.js
│ │ ├── package.json
│ │ ├── src
│ │ │ └── index.ts
│ │ ├── test
│ │ │ └── unit
│ │ │ └── index.test.ts
│ │ ├── tsconfig.build.json
│ │ └── tsconfig.json
│ └── foo
│ ├── jest.config.js
│ ├── package.json
│ ├── src
│ │ ├── hello.ts
│ │ └── index.ts
│ ├── test
│ │ ├── integration
│ │ │ └── index.test.ts
│ │ └── unit
│ │ └── index.test.ts
│ ├── tsconfig.build.json
│ └── tsconfig.json
├── tsconfig.build.json
├── tsconfig.json
└── yarn.lock
How should I align it with Bazel, like you know, WORKSPACE, BUILD, and their contents?
Any tips or examples?
Thanks!
There are some examples of repo structures somewhat similar to this in the rules_nodejs examples directory. This shows (in this case an Angular app) having shared libs and consuming them, but the principle is the same here.
Generally, you'd only have one WORKSPACE file in the root of your project. While it's possible to have multiple package.json files for different apps and libs, it adds some extra complexity to the ts_library rules which, for getting started, may be best avoided. This example repo shows multiple package.json files, but without Typescript.
For BUILD (or BUILD.bazel) files, the minimum you'll need here is one in foo and one in bar (and one at the root). The more BUILD files you have, the more you split up the compilation units for your source, therefore increasing incrementality.
Then add ts_library rules to those BUILD files, docs for which can be found here, they also show the differences between using tsc directly and ts_library. You can then define source dependencies between foo and bar, a quick example shown below:
packages/foo/BUILD:
ts_libaray(
name = "foo",
srcs = glob(["src/**/*.ts"]),
deps = [
"//packages/bar", <-- this is the source dep for bar
"#npm//some-package",
],
)
packages/bar/BUILD:
ts_libaray(
name = "bar",
srcs = glob(["src/**/*.ts"]),
deps = [
"#npm//some-other-package",
],
)
I have a packages.json file and I'm installing the needed node modules with npm install from the same directory where the file is located.
The problem is that I'm doing this on different machines and some of them might already have some dependencies installed globally.
This normally shouldn't represent a problem but in my case it is.
For example I need to install grunt-contrib-uglify and since some machine might already have some dependencies installed they won't try to fetch and get them. This lead to two slightly different versions of the dependencies tree.
Example:
npm list (truncated) produces:
# Machine 1
├─┬ grunt-contrib-uglify#0.2.7
│ ├─┬ grunt-lib-contrib#0.6.1
│ │ └── zlib-browserify#0.0.1
│ └─┬ uglify-js#2.4.21
│ ├── async#0.2.10
│ ├─┬ source-map#0.1.34
│ │ └── amdefine#0.1.0
│ ├── uglify-to-browserify#1.0.2
│ └─┬ yargs#3.5.4
│ ├── camelcase#1.0.2
│ ├── decamelize#1.0.0
│ ├── window-size#0.1.0
│ └── wordwrap#0.0.2
# Machine2
├─┬ grunt-contrib-uglify#0.2.7
│ ├─┬ grunt-lib-contrib#0.6.1
│ │ └── zlib-browserify#0.0.1
│ └─┬ uglify-js#2.4.23
│ ├── async#0.2.10
│ ├─┬ source-map#0.1.34
│ │ └── amdefine#0.1.0
│ ├── uglify-to-browserify#1.0.2
│ └─┬ yargs#3.5.4
│ ├── camelcase#1.1.0
│ ├── decamelize#1.0.0
│ ├── window-size#0.1.0
│ └── wordwrap#0.0.2
In this case camelcase and uglify-js are not exactly the same version.
When I use this in conjunction with grunt to minify the production js files I get minor differences between the compiled files. Of course the two files acts exactly the same but for git they are different (and I would like to avoid this)
Question: how can I tell npm that I want exactly the same modules but also exactly the same dependencies?
There is a file named package-lock.json. It contains the exact dependency tree of all installed packages including the registry where they were downloaded from. Whenever you add a new dependency using npm install <package-name> this file is updated automatically. It should be checked in into your version control.
To make sure that the same package versions listed in the file are installed in your node_modules folder you have to execute the command npm ci (ci = clean install). This is going to delete your node_modules folder and download the exact packages listed int the package-lock.json. This command should be used instead of npm install in any build script.
I've found the solution: npm-shrinkwrap
So, first I should install and test the modules as normally I would with npm install then run npm shrinkwrap to lock down all the installed modules and their deps into a file called npm-shrinkwrap.json. We could use the flag --dev if we want also to save dev deps.
Then we could for example track this file with git and from other machines retrieve the tracked file.
Then normally npm install => If the file npm-shrinkwrap.json is present it will take precedence over packages.json and npm will use it to install exactly all the deps specified in the file.
I had some issues with NPM so I decided to simply uninstall Node, NPM & NVM and then reinstall everything on Mac OS X 10.8. After following various guides and Stackoverflow questions I was pretty sure I had gotten rid of everything. But to my surprise after I first installed NVM and then Node (which installed NPM for me) I couldn't understand why running npm -g ls shows lots of different modules which, after looking them up on the NPM registry, look to be very basic helpers and boilerplate modules which other more advanced modules build on. Here is the entire list of modules it prints out:
unknownd8a25e8b001d:~ [username]$ npm -g ls
/Users/[username]/.nvm/v0.10.18/lib
└─┬ npm#1.3.8
├── abbrev#1.0.4
├── ansi#0.1.2
├── archy#0.0.2
├── block-stream#0.0.7
├── child-process-close#0.1.1
├── chmodr#0.1.0
├── chownr#0.0.1
├── cmd-shim#1.0.1
├── editor#0.0.4
├── fstream#0.1.24
├─┬ fstream-npm#0.1.5
│ └── fstream-ignore#0.0.7
├── github-url-from-git#1.1.1
├── glob#3.2.6
├── graceful-fs#2.0.0
├── inherits#2.0.0
├── ini#1.1.0
├─┬ init-package-json#0.0.11
│ └── promzard#0.2.0
├── lockfile#0.4.0
├── lru-cache#2.3.0
├─┬ minimatch#0.2.12
│ └── sigmund#1.0.0
├── mkdirp#0.3.5
├── node-gyp#0.10.9
├── nopt#2.1.2
├─┬ npm-registry-client#0.2.28
│ └── couch-login#0.1.18
├── npm-user-validate#0.0.3
├─┬ npmconf#0.1.2
│ └─┬ config-chain#1.1.7
│ └── proto-list#1.2.2
├── npmlog#0.0.4
├── once#1.1.1
├── opener#1.3.0
├── osenv#0.0.3
├─┬ read#1.0.5
│ └── mute-stream#0.0.4
├── read-installed#0.2.3
├─┬ read-package-json#1.1.1
│ └─┬ normalize-package-data#0.2.1
│ └── github-url-from-git#1.1.1
├─┬ request#2.25.0
│ ├── aws-sign#0.3.0
│ ├── cookie-jar#0.3.0
│ ├── forever-agent#0.5.0
│ ├─┬ form-data#0.1.0
│ │ ├── async#0.2.9
│ │ └─┬ combined-stream#0.0.4
│ │ └── delayed-stream#0.0.5
│ ├─┬ hawk#1.0.0
│ │ ├── boom#0.4.2
│ │ ├── cryptiles#0.2.2
│ │ ├── hoek#0.9.1
│ │ └── sntp#0.2.4
│ ├─┬ http-signature#0.10.0
│ │ ├── asn1#0.1.11
│ │ ├── assert-plus#0.1.2
│ │ └── ctype#0.5.2
│ ├── json-stringify-safe#5.0.0
│ ├── mime#1.2.10
│ ├── node-uuid#1.4.0
│ ├── oauth-sign#0.3.0
│ ├── qs#0.6.5
│ └── tunnel-agent#0.3.0
├── retry#0.6.0
├── rimraf#2.2.2
├── semver#2.1.0
├─┬ sha#1.2.1
│ └── readable-stream#1.0.2
├── slide#1.1.4
├── tar#0.1.18
├── uid-number#0.0.3
└── which#1.0.5
So my question is, does NPM install various global modules for its own use when it is installed? It does look, from the indenting, as though these are modules npm uses because they're nested underneath npm#1.3.8 and with Finder I can see how NPM looks to have these within its own node_modules folder.
The Node Package Manager (NPM) itself is a module, which yes, does have dependencies. Therefore the answer to your question is yes, except that the modules are installed as dependencies, not globally. The module NPM itself is the global module.
In some cases, that comes into conflict with the Windows 260 chr path limitation. I'm having a lot of problems with npm install and this limitation.
.
├── app
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── to_infinity_and_beyond...
│ └── It's a madness!
...
That structure produces paths like:
c:/path_to_my_app/
node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/.....
What i'm doing wrong? It's there a way to avoid it?
Perhaps renaming "node_module" to "nm", or something like that, may helps to save some characters...
That's in advance!
As Brandon Tilley said:
npm dedupe
works fine for me!