I'm working with NodeJS in a small standalone isolated environment, as such commands like npm install -g can't reach out to the internet for dependencies. To "workaround" this constraint, I'm doing installs on a regular system and just air-gaping the pieces over.
Any given module (and all its dependencies) lives in /usr/local/lib/node_modules/XXXXX, and if I'm actually in that directory and enter node cli.js, it works.
However, back on the main system invoking module XXXXX by name from the command line runs that particular module. The isolated system does not, and I'm trying to find that last little step of how npm is wiring things up.
What's the pedantic methodology once one's this far to teach npm (or node) of the new command?
I'm willing to resort to hacks or aliases if absolutely necessary, but I'd like to do it the npm way.
Unfortunately, I'm not at liberty to set up a CouchDB or such and run a mirrored repository — I'm working in a tight footprint that's trying to keep things small and sleek.
Related
I'm working on a Javascript project, and as it so happens one of my dependencies pulls in puppeteer, which in turn downloads a whole copy of Chromium into my node_modules. My larger project is split into multiple Javascript packages, so I end up with multiple identical copies of Chromium among other stuff.
Is there a way to deduplicate these packages system wide? Note, npm dedupe seems to do something completely different to what I want.
I imagine there would be a module repository in my home directory which contains every package I need (in every version needed), and then in the local node_modules directories would contain only symlinks to the repository. This seems like an incredibly obvious optimisation, but I can't find any way to do it in npm. If not in npm, is it maybe possible in yarn?
As an added complication, this should also work on Windows (where symbolic link support has historically been not so good).
It seems the following command does what I want:
npm config set link -g
Then delete node_modules, and do npm install again. It should be much smaller now.
The documentation says:
If true, then local installs will link if there is a suitable globally installed package.
Note that this means that local installs can cause things to be installed into the global space at the same time. The link is only done if one of the two conditions are met:
The package is not already installed globally, or
the globally installed version is identical to the version that is being installed locally.
I am not sure if this has any negative side-effects - for example clobbering the global namespace with commands I don't want. For now, it seems to work fine.
I am currently working on a project at a large company, and according to the project I am working on, every time I want to quickstart the app, I would need to first run the command npm install and then run all the additional compiling instructions, but the problem is that running npm install can take a long time, and that is why I am wondering if it is necessary to run this command every time I make a change to the code, and then want to compile and run it.
What exactly does npm install do? If you could explain to me in terms of how we compile and run java code i.e. javac bob.java && java bob and try to make analogies on that basis, that would greatly help me understand the concept. The way I am currently thinking about it right now is that npm install kind of runs like how javac runs, but I am not sure if that is correct. Thanks in advance!
NPM Install
npm install simply reads your package.json file, fetches the packages listed there from (usually) https://www.npmjs.com/, and sometimes engages in the build steps for those packages.
So you only have to run npm install when you change your package.json file, and need to fetch new dependencies.
Keep in mind that npm install --save <packagename> (or npm install -S <packagename>) will update your package.json and run npm install all in one line!
You can view the results of your npm install inside ./node_modules/.
To compare to java
This might be a helpful resource if you're trying to get stuff done: Getting Started with Node.js for the Java Developer
Javascript is not a compiled language, unlike java. When you invoke javac, the java compiler reads in all your .java files, compiles them to java bytecode, and then writes them to .class files, which can then be bundled together into a .jar for execution.
Javascript doesn't do any of this! When you invoke node foo.js, the node executable wakes up, reads foo.js, and gets to work invoking it line by line**. Node does other cool things, including maintaining an event loop (which allows it to operate "asynchronously", and allows it to be very efficient as a webserver-- it doesn't sit around waiting for requests to complete, it carries forward with the next event in the queue.
Node also performs JIT and optimization, these details allow it to improve the performance of sections code it notices are running "hot".
Note also that node.js uses the V8 javascript engine (also used in Google Chrome). Really everything I've said above is handled by V8.
(** Technically there is a syntax checker which is run first, before execution. But this is not a compile step!)
It is not necessary to do "npm install" each time you want to compile. You just need to do it when you change the dependencies of your project.
NPM basically is the package manager for node. It helps with installing various packages and resolving their various dependencies. It greatly helps with your Node development. NPM helps you install the various modules you need for your web development and not just given you a whole bunch of features you might never need.
When you start an app, it comes with a package.json file. That package contains the list of node_modules you are gonna need. Whenever you enter npm install, what you are doing is to download that list of node_modules. So yeah, you have to download the modules all over again.
#NOTE: In your project, you have a file called package.json. this file is responsible for holding track of your project's dependencies. That's why you have to install it every time#.
I only realised relatively recently that running npm install <foo> on my own machine is completely unsafe. Anyone can publish anything to NPM, and even if there was some way of verifying that an individual package was not nefarious, it would be extremely difficult to verify all of its dependencies.
NPM runs with my user permissions (not root), but in the context of my Macbook, that's pretty much irrelevant, as all of my personal files are also owned by the same user. (The only thing root would add is the ability to destroy the operating system itself, which is vastly less important.)
So: how do people safely run untrusted code from NPM? Obviously running everything inside a VM is safer, but I always seem to have difficulties with local networking.
Or I guess I could create a new user, and make everything NPM-related owned by that user, who doesn't have access to the rest of my system.
Are there any other good options?
Mitigations for this would include:
If you are a user who owns modules you should not stay logged into
npm. (Easily enough, npm logout and npm login)
Use npm shrinkwrap to lock down your dependencies
Use npm install someModule --ignore-scripts
Also see this link and this link.
Be aware that a malicious module could introduce backdoors or vulnerabilities into any code built with it, if not noticed by the community.
In a Node package.json file, you can map multiple executables to the PATH environmental variable on a global NPM install (npm install -g):
"bin": {
"foo": "./bin/foo.js",
"bar": "./bin/bar.js"
},
I have a unique project that requires mapping existing PATH variables on Operating Systems that do not have it. For example, I want to add a command named grep to PATH, if and only if it is being installed on a Windows computer. If the computer is running any other OS, the NPM installation will obviously fail.
Is there any way to run logic that pre-determines what bin options are available in the installation?
Oh snap - I just had an idea!
Would this work:
Parent module has npm (programmatic version) as a dependency.
On global installation, run a post-install script as declared in the package.json of parent module.
Post-install script does a check on the system to see which commands exist. This would be more mature than "Windows or not Windows" - it would try to exec a list of commands and see which ones fail.
For every command that doesn't exist, post-install script programmatically runs npm install -g on all sub-modules (one for each command, such as grep).
This would take a while and the npm module is huge, but it seems like it would work. No?
There doesn't seem to be a way to do this directly through package.json, but it might be possible (and desirable) to do something like:
Make a separate npm module for each executable you want to register (eg my-win-grep).
In the my-win-grep module, implement the executable code you want to run, and register the PATH/executable value in this module.
In the package.json for my-win-grep, include an os field that limits it to installing on windows.
In your original module, list my-win-grep as an optionalDependency.
In this way, if someone installs your original module on Windows, it would install my-win-grep, which would register an executable to run under the grep command.
For users on other systems, the my-win-grep module would not install because of the os requirement, but the main module would continue to install because it ignores failures under optionalDependencies. So the original grep executable would remain untouched.
Updated question
This approach does sound like it should work - as you say, the npm dependency is pretty large, but it does avoid having to preform additional symlinking, and still has the benefit outlined above of having each piece of OS specific functionality in a separate module.
The only thing to watch for, possibly, in the programmatic npm docs:
You cannot set configs individually for any single npm function at
this time. Since npm is a singleton, any call to npm.config.set will
change the value for all npm commands in that process
So this might just mean that you can't specify -g on your installs, but instead would have to set it globally before the first install. This shouldn't be a problem, but you'll probably need to test it out to find out exactly.
Lastly...
You might also want to have a look at https://github.com/lastboy/package-script - even if you don't use it, it might give you some inspiration for your implementation.
I've been using Node.js and npm for a few weeks with great success and have started to question the best practice for installing local modules. I understand the Global vs Local argument, however, my question has more to do with where to place a local install. Let's say that I have a project located at ~/ProjectA/ which is version controlled and worked on by multiple developers. When initially playing with Node.js and npm I wasn't aware of the default local installation paths and just simply installed the necessary modules in a default terminal which resulted in a installation path of ~/node_modules. What this ended up doing is requiring all the other developers working on the project to install the modules on their own machines in order to run the application. Having seen where some of the developers ran npm install I'm still really surprised that it worked on their machines at all (I guess it relates to how Node.js and require() looks for modules), but needless to say, it worked.
Now that the project is getting past the "toying around" stage, I would like to setup the project folder correctly. So, my question is, should the modules be installed at ~/ProjectA/node_modules and therefore be part of the version controlled project files, or should it continue to be located at a developer-machine specific location...or does it not really matter at all?
I'm just looking for a little "best-practice" guidance on this one and what others do when setting up your projects.
I think that the "best practice" here is to keep the dependencies within the project folder.
Almostly all Node projects I've seen so far (I'm a Node developer has about 8 months now) do that.
You don't need to version control the dependencies. That's how I manage my Node projects:
Keep the versions locked in the package.json file, so everyone gets the same working version, or use the npm shrinkwrap command in your project root.
Add the node_modules folder to your VCS ignore file (I use git, so mine is .gitignore)
Be happy, you're done!