How to use npm in an environment with restricted internet access - node.js

In a restricted environment where not every user has access I would like to be able to use npm offline where ever possible.
My idea is to point the global config at a shared cache directory so that power users can do installs and the dependencies will end up in the cache directory. Other users can then do npm offline installs for anything previously in the cache.
So 2 Questions:
Will this work?
Short of setting up my own local npm repository is there an easier way?

Per documentation:
npm install (with no args in a package dir)
npm install <tarball file>
npm install <tarball url>
npm install <folder>
npm install <name> [--save|--save-dev|--save-optional]
npm install <name>#<tag>
npm install <name>#<version>
npm install <name>#<version range>
npm i (with any of the previous argument usage)
As such, npm allows you to do:
npm install /path/to/folder/containing/ node_modules
For example: npm install ~/Downloads/http-proxy, provided that the node_modules folder resides within http-proxy.
You could set your repository up on an internal (accessible) server and direct people to download by the same name from there.

Thanks for the answers. What I've ended up doing is using https://github.com/rlidwka/sinopia
This acts as a mirror repository. I can give this process internet access and not other users. Then I set a environment variable for all users to point their npm repository at the sinopia instance.
Early days but this seem to be working well.

r3mus is right. Though, for each user it would result in some cognitive overhead and possibly management issues.
What might work better is to have a corporate hosted npm repository (as described here: http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository) and then just have the users change (once) their registry settings via npm set registry http://yourregistry.com

For build servers, a reasonable strategy is to symlink the node_modules directory against an existing directory where the modules have already been installed.
e.g. my powershell script might read something like this
If (-Not (Test-Path node_modules))
{
& cmd /c mklink /d /j node_modules D:\npm-cache\node_modules
Write-Verbose "Symlinked node_modules"
}

Related

Install multiple npm packages from private gitlab registry

I'm using gitlab to host my private npm packages. At the moment I've 2 projects I published to the gitlab package registry. Both packages are used by another project (let's say 3rd project). According to the gitlab documentation, I installed both packages in the 3rd project using the following commands:
npm config set #myscope:registry https://gitlab.com/api/v4/projects/<first-project-id>/packages/npm/
npm config set "//gitlab.com/api/v4/projects/<first-project-id>/packages/npm/:_authToken" "<auth-token>"
npm install #myscope/first-package
npm config set #myscope:registry https://gitlab.com/api/v4/projects/<second-project-id>/packages/npm/
npm config set "//gitlab.com/api/v4/projects/<second-project-id>/packages/npm/:_authToken" "<auth-token>"
npm install #myscope/second-package
My problem is now that I cannot run a simple "npm install" anymore because my 3rd project depends on the 2 private packages. Sure, I can call "npm config set" before running "npm install" in the 3rd project, but the problem is that when I call "npm config set" for the second project, it replaces the first config (because both are having the same scope). So I can only install 1 package, for the second I'm getting an error because it cannot be found (because it has another project-id in gitlab, thus another package registry url). I already tried the following without success:
npm config set #myscope/first-package:registry https://gitlab.com/api/v4/projects/<first-project-id>/packages/npm/
npm config set "//gitlab.com/api/v4/projects/<first-project-id>/packages/npm/:_authToken" "<auth-token>"
npm install #myscope/first-package
npm config set #myscope/second-package:registry https://gitlab.com/api/v4/projects/<second-project-id>/packages/npm/
npm config set "//gitlab.com/api/v4/projects/<second-project-id>/packages/npm/:_authToken" "<auth-token>"
npm install #myscope/second-package
But now I cannot install any of the packages. So, anyone know how I can set the registry-url for packages sharing the same scope but having a different url?
I finally solved the issue by using the instance-level installation. First, I thought it's not working for me because it still told me 404. But the problem was that gitlab instructs me to run npm config set '//gitlab.com/api/v4/packages/npm/:_authToken' "<my-token>". But this is not working on my Windows because of the apostrophes that surrounds the url. I replaced them with " and now everything works fine.

Issue cloning every repository

I have been cloning repositories with node and MongoDB but I always run into issues running npm install when I cd into the folder directory.
It has to be me because I happens every time. What am I missing?
Seems like npm install is the 3rd step after 1: git clone "repository link", 2: cd in clone folder.
Please guide me in the right direction.
Thanks a ton
You can run npm install from a directory where package.json file is located since this is the one which holds all the info regarding packages.
So just try to make it sure that the directory from where you are firing npm install command, have package.json file.
Apart from this, just check if you have npm installed. Run npm -v

Is there any way to configure multiple registries in a single npmrc file

Here is my problem. We have a private NPM registry which only works in VPN. I would like to have a fallback registry https://registry.npmjs.org so that when I am out of VPN it works seamlessly.
P.S. Currently I am using npmrc which does a good job in switching between .npmrc files as a workaround
You can have multiple registries for scoped packages in your .npmrc file. For example:
#polymer:registry=<url register A>
registry=http://localhost:4873/
Packages under #polymer scope will be received from https://registry.npmjs.org, but the rest will be received from your local NPM.
On version 4.4.1, if you can change package name, use:
npm config set #myco:registry http://reg.example.com
Where #myco is your package scope.
You can install package in this way:
npm install #myco/my-package
For more info: https://docs.npmjs.com/misc/scope
I believe the top-voted answer might be outdated. As of June 2021, there is a much easier way to do this using npmrc.
Refer to npm Docs.
1. Install npmrc
To install npmrc, on the command line, run
npm i npmrc -g
2. Create your first npm profile
After installing npmrc, you can create a profile to access your custom (maybe company's) registry.
To create an npm Enterprise profile, on the command line, run npmrc -c name-of-profile.
For example, to create a profile called "work", run the following command:
npmrc -c work
To set an npm Enterprise registry for the profile, run the following command, replacing your-company-registry with the name of your company's npm Enterprise registry:
npm config set registry https://registry.your-company-registry.npme.io/
3. Create a profile for the public npm registry
After you have created your npm Enterprise profile, you can create a second profile for a different registry, such as the public npm registry.
To create a profile for the public registry, on the command line, run npmrc -c name-of-profile. For example, to create a profile called "open-source", run npmrc -c open-source.
To set the public registry for your open source profile, run the following command:
npm config set registry https://registry.npmjs.org/
4. Switch profiles with npmrc
To switch profiles, on the command line, run the following command, replacing profile-name with the name of your profile:
npmrc profile-name
Not the best way but If you are using mac or linux even in windows you can set alias for different registries.
##############NPM ALIASES######################
alias npm-default='npm config set registry https://registry.npmjs.org'
alias npm-sinopia='npm config set registry http://localhost:4873/'
For anyone looking also for a solution for authentication, I would add on the scoped packages solution that you can have multiple lines in your .npmrc file:
//internal-npm.example.com:8080/:_authToken=xxxxxxxxxxxxxxx
//registry.npmjs.org/:_authToken=yyyyyyyyyy
Each line represents a different NPM registry
Since it has been a couple years and it doesn't seem possible to do this (using npm alone), a solution to this problem is to use the Nexus Repository Manager (from Sonatype). Nexus supports multiple repositories, lets you order them, and also proxies/caches to improve speed.
A free version and pro/paid version exist. The feature that supports this is described at:
https://help.sonatype.com/repomanager3/node-packaged-modules-and-npm-registries
The relevant information is duplicated below so if/when the above URL/link stops working the information is still here.
A repository group is the recommended way to expose all your npm registries repositories from the repository manager to your users, without needing any further client side configuration. A repository group allows you to expose the aggregated content of multiple proxy and hosted repositories with one URL to npm and other tools.
It lets you create private npm registries
A private npm registry can be used to upload your own packages as well as third-party packages.
And
To reduce duplicate downloads and improve download speeds for your developers and CI servers, you should proxy the registry hosted at https://registry.npmjs.org. By default npm accesses this registry directly. You can also proxy any other registries you require.
So a quick bulleted list of things you do to get this working is:
Install Nexus
Create a local/private repo (or point to your private repo on another server)
Create a GROUP that lists your private repo, and the public repo.
Configure your $HOME/.npmrc file to point to the "GROUP" just created.
Publish your private npm packages to the local repo.
Users now can run a one time setup.
npm config set registry https://nexus/content/groups/GROUP
Then users can install both public or private packages via npm install.
npm install my-private-package
npm install lodash any-other-public-package
And both your public and private packages can be installed via a simple npm install command. Nexus finds the package searching each repo configured in the group and returns the results. So npm still thinks there is just one registry but behind the curtain there are multiple repos being used.
IMPORTANT NOTE: When you publish your components, you'll need to specify the npm publish --registry https://nexus/content/repositories/private-repo my-private-package command so your package is published to the correct repo.
You can use multiple repositories syntax for the registry entry in your .npmrc file:
registry=http://serverA.url/repository-uri/
//serverB.url/repository-uri/
//serverC.url/repository-uri/:_authToken=00000000-0000-0000-0000-0000000000000
//registry.npmjs.org/
That would make your npm look for packages in different servers.
Some steps you can try. (its how we do it at my workplace)
Create a registry group with two (or more) repository source address. One would be your internal private and the other a proxy to npmjs giving priority to the internal one.
Make this group your registry in the .npmrc file. This way npm will always try to get it from the internal one, if not found get it from the proxy
Hope that helps.
I encounter the same problem when my company set up its own registry, so I heavily rework on proxy-registry into proxy-multi-registries to solve this problem. Hope it will also helps you.
As of 13 April 2020 there is no such functionality unless you are able to use different scopes, but you may use the postinstall script as a workaround. It is always executed, well, after each npm install:
Say you have your .npmrc configured to install #foo-org/foo-pack-private from your private github repo, but the #foo-org/foo-pack-public public package is on npm (under the same scope: foo-org).
Your postinstall might look like this:
"scripts": {
...
"postinstall": "mv .npmrc .npmrcc && npm i #foo-org/foo-pack --dry-run && mv .npmrcc .npmrc".
}
Don't forget to remove #foo-pack/foo-org from the dependencies array to make sure npm install does not try and get it from github and to add the --dry-run flag that makes sure package.json and package-lock.json stay unchanged after npm install.
My approach was to make a slight command line variant that adds the registry switch.
I created these files in the nodejs folder where the npm executable is found:
npm-.cmd:
#ECHO OFF
npm --registry https://registry.npmjs.org %*
npm-:
#!/bin/sh
"npm" --registry https://registry.npmjs.org "$#"
Now, if I want to do an operation against the normal npm registry (while I am not connected to the VPN), I just type npm- where I would usually type npm.
To test this command and see the registry for a package, use this example:
npm- view lodash
PS. I am in windows and have tested this in Bash, CMD, and Powershell. I also
I use Strongloop's cli tools for that; see https://strongloop.com/strongblog/switch-between-configure-public-and-private-npm-registry/ for more information
Switching between repositories is as easy as : slc registry use <name>
I had the same issue and I've tried many solutions that didn't work, now i encountered that by using different scoops for my npm private packages, in that way in can use multiple registries with same .npmrc file like that :
.npmrc : (You can put as many packages you want)
#scop1:registry=https://gitlab.example.com/api/v4/projects/<project1_id>/packages/npm/
#scop2:registry=https://gitlab.example.com/api/v4/projects/<project2_id>/packages/npm/
//gitlab.example.com/api/v4/projects/<project1_id>/packages/npm/:_authToken=${GITLAB_TOKEN}
//gitlab.example.com/api/v4/projects/<project2_id>/packages/npm/:_authToken=${GITLAB_TOKEN}
No, NPM does not support multiple registry except for scoped one.
Refer to: https://docs.npmjs.com/cli/v7/using-npm/scope

How can I change the cache path for npm (or completely disable the cache) on Windows?

I've installed Node.js on my Windows 7 x64 development machine, the manual way:
mkdir C:\Devel\nodejs
cd C:\Devel\nodejs
set NODE_PATH=%CD%
setx /M PATH "%PATH%;%NODE_PATH%"
setx /M NODE_PATH "%NODE_PATH%\node_modules"
I've placed the main node x64 binary along with npm package manager in C:\Devel\nodejs. Works like a charm and I can update the main binary without dealing with the installer.
The only problem I can't solve is moving the cache folder. When I install a local package:
npm install express
... cache is placed under %APP_DATA%\npm-cache folder. I'd like to change it to:
C:\Devel\nodejs\npm-cache
How can I change the npm cache folder, or disable it completely?
You can change npm cache folder using the npm command line. (see https://docs.npmjs.com/cli/v6/using-npm/config#cache)
So you might want to try this command :
> npm config set cache C:\Devel\nodejs\npm-cache --global
Then, run npm --global cache verify after running this command.
You can also set an environment variable with export npm_config_cache=/path/to/cache (Unix) or set npm_config_cache=C:\path\to\cache (Win) as an alternative to npm config set (this is true for all config options in npm).
For anyone using docker you can add the env var at runtime with:
docker run -e npm_config_cache=/path/to/cache mydockerimage:tag
You can also do following:
For having cache path as you wish, for a single package while installing it:
npm install packageName --cache path/to/some/folder
For having cache path as you wish, for all the packages in package.json:
Just be in the directory where package.json is as usual and do
npm install --cache path/to/some/folder
You may not find this in npm documentation but i have tried it with npm 6 and it works.
Looks like it works since npm 5 [Refer: How to specify cache folder in npm5 on install command?
In Windows you can simply cd to the desired cache folder and do npm set cache --global
Solution
Paste the following code into npmrc file.
Location of npmrc file: C:\Program Files\nodejs\node_modules\npm\npmrc
prefix=D:\nodejs\npm
cache=D:\nodejs\npm-cache
Notes:
There is no '.' in front of npmrc
Diagrams
NPMRC file folder look like this
NPMRC Content look like this
Hope it helps. Cheers
In addition, I found that running an update command works also - for example:
npm update npm
Lastly, one can check their npm-cache directory to see if is being filled or not.

How to npm install to a specified directory?

Is it possible to specify a target directory when running npm install <package>?
You can use the --prefix option:
mkdir -p ./install/here/node_modules
npm install --prefix ./install/here <package>
The package(s) will then be installed in ./install/here/node_modules. The mkdir is needed since npm might otherwise choose an already existing node_modules directory higher up in the hierarchy. (See npm documentation on folders.)
As of npm version 3.8.6, you can use
npm install --prefix ./install/here <package>
to install in the specified directory. NPM automatically creates node_modules folder even when a node_modules directory already exists in the higher up hierarchy.
You can also have a package.json in the current directory and then install it in the specified directory using --prefix option:
npm install --prefix ./install/here
As of npm 6.0.0, you can use
npm install --prefix ./install/here ./
to install the package.json in current directory to "./install/here" directory. There is one thing that I have noticed on Mac that it creates a symlink to parent folder inside the node_modules directory. But, it still works.
NOTE: NPM honours the path that you've specified through the --prefix option. It resolves as per npm documentation on folders, only when npm install is used without the --prefix option.
In the documentation it's stated:
Use the prefix option together with the global option:
The prefix config defaults to the location where node is installed. On
most systems, this is /usr/local. On windows, this is the exact
location of the node.exe binary. On Unix systems, it's one level up,
since node is typically installed at {prefix}/bin/node rather than
{prefix}/node.exe.
When the global flag is set, npm installs things into this prefix.
When it is not set, it uses the root of the current package, or the
current working directory if not in a package already.
(Emphasis by them)
So in your root directory you could install with
npm install --prefix <path/to/prefix_folder> -g
and it will install the node_modules folder into the folder
<path/to/prefix_folder>/lib/node_modules
There currently is no documented way to install a package in an arbitrary directory. You should change into the target directory, make sure it has a package.json, and then use the regular commands.
While there currently is an undocumented option called --prefix, this feature might be removed in a future release. At least, it it is not planned to ever document this as an official feature.
I am using a powershell build and couldn't get npm to run without changing the current directory.
Ended up using the start command and just specifying the working directory:
start "npm" -ArgumentList "install --warn" -wo $buildFolder

Resources