I have encountered an issue that I'm not sure how to resolve in the best way possible. Here it is:
We have recently started using private NPM packages, and are trying to figure out how to tie our local development loop with CI and a Deployment pipeline.
I've looked and started leveraging the NPM_TOKEN variable.
In CI, we are doing the following:
echo "//registry.yarnpkg.com/:_authToken=$NPM_TOKEN" >> ~/.npmrc
This works well, but during deployment on Heroku, we don't have access to home. So to make it work on Heroku we added an .npmrc file to the project directory. This worked well since npm uses environment variables to fill this in.
The problem is that locally, all yarn commands fail with a missing variable. The suggested way on the NPM website (https://blog.npmjs.org/post/118393368555/deploying-with-npm-private-modules) is to add the token to the environment in a .profile. This doesn't seem like the best solution, since the setting is now global, and should be kept per repository.
I've found a similar question here that uses npm but it does not seem to work with yarn.
Using auth tokens in .npmrc
A comment there also mentions that it does not work for npm and there is no documentation that mention a dotenv file.
Is there a better way to deal with this? Seems like a common issue that should be resolved a long time ago...
Use a .yarnrc file:
npmRegistryServer: "https://npm.pkg.github.com"
npmAuthToken: "secretAuthTokenValue"
Faced this issue today with Azure Artifacts.
Following their documented project setup, the auto-generated code on Azure DevOps encodes the # symbol to %40 in the <FEED_NAME> which caused yarn add artifacts_package_name to return a 401 Unauthorized error:
# .npmrc doc example
//pkgs.dev.azure.com/<ORGANIZATION_NAME>/_packaging/<FEED_NAME>/npm/registry/
# .npmrc auto generated code on AzureDevOps
//pkgs.dev.azure.com/<ORGANIZATION_NAME>/_packaging/<ORGANIZATION_NAME%40Local>/npm/registry/
Issue was fixed by replacing all occurrences of %40 with # like:
//pkgs.dev.azure.com/<ORGANIZATION_NAME>/_packaging/<ORGANIZATION_NAME#Local>/npm/registry/
source
In my experience this can be a bit tricky. It feels like the sources you provided should give the best guidance. There is a .yarnrc file that I guess is linkied to the npmrc file as well. Anyway I can at least share a set up that I used in a project that worked for me: (this is from an older project so it might not be relevant anymore. If it isn't, let me know and I can remove this answer)
.npmrc file:
_auth = token
registry = http://example.com
ca = null
email = email#example.com
always-auth = true
.yarnrc file:
registry "http://example.com"
Related
we probably have a really specific case. We are using
Private NPM registry on Azure DevOps
Yarn v1
Semantic-Release Monorepo (containing multiple packages. I think if it would only be one package, it would be ok)
Everything works fine and as expected, we now want to update to Yarn v3, but this is where the issue arises
semantic-release, per default, uses npm to create releases. It also uses .npmrc, if available to authenticate against the private registry.
However, since yarn v3, the .npmrc is not supported anymore, so it doesn't exist and also, while npm creates the same yarn.lock as yarn v1 it doesn't create the same yarn.lock as yarn v3
So, now we are facing the following issue
When running our pipeline our semantic-release fails, because it is changing the lockfile (we tested this locally, and the lockfile is indeed changed), because semantic-release internally uses npm with a plugin and this plugin which formats the yarn.lock to an outdated format and then fails to deploy multiple packages
To solve this, we found the following library https://github.com/dmeents/semantic-release-yarn/
however, this library requires that we set an NPM_TOKEN so yarn can authenticate with it. With the npm plugin, it checked for an .npmrc and used this one to authenticate, but since this doens't exist anymore, the new package searches for NPM_TOKEN.
However, I have not found any way how too get NPM_TOKEN from Microsoft Azure DevOps. In the .yarnrc.yml we are using the npmAuthIdent which you can see here has the wrong format
Defines the authentication credentials to use by default when accessing your registries (equivalent to _auth in the v1). This settings is strongly discouraged in favor of npmAuthToken.
npmAuthIdent: "username:password"
Defines the authentication credentials to use by default when accessing your registries (equivalent to _authToken in the v1). If you're using npmScopes to define multiple registries, the npmRegistries dictionary allows you to override these credentials on a per-registry basis.
npmAuthToken: "ffffffff-ffff-ffff-ffff-ffffffffffff"
https://yarnpkg.com/configuration/yarnrc#npmAuthIdent
As I see it, there are 3 possible solutions
Migrate back to yarn v1 (Least preferred solution)
Find out how to get such an npmAuthToken using Azure DevOps (Most preferred solution)
Create our own plugin for semantic-release to suppoort yarn in combination with Azure DevOps
Do you have any idea how to solve this or maybe someone already faced this issue?
I have created a private typings npm module that I am using for my firebase functions and app projects. When I went to deploy firebase functions, I get a big error for every function that basically says ERR! remote: Invalid username or password.
For what I have read, it looks like I need to create a .npmrc file and put it in the /functions directory. (https://cloud.google.com/functions/docs/writing/specifying-dependencies-nodejs#using_private_modules)
I cannot however find proper instructions on how to do this anywhere. From what I found, I have done the following:
ran npm login
ran npm token create --read-only
This then gave me a token that looks like this: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.
I then create a file called .npmrc in my functions directory, and placed //registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX in it.
Additionally, I saw that the error message was trying to use ssh to install my private repo package, I have not setup ssh and am using https instead, because of this I changed my package file to git+https://github.com/accounts-name/repo#commit-num so that it uses HTTPS.
However, I still get the same error message. What am I missing? The above is what I have strung together from lots of google searching.
It seems that you have made too many different changes while trying to make it work, so let's just go through the whole process step by step.
Check the package.json of your npm module and publish it:
Remove "private" property or set it to false because private modules in the npm are meant to be never published. That is not obvious but that is true.
Next step is to apply restricted access to the package. In order to do that, add such property in the package.json file:
"publishConfig": {
"access": "restricted"
},
Make sure that npm account you use for publishing supports private packages.
Now open the terminal in the root directory of your package, type npm login then sign in to npm. Check if you put the proper version in the package.json.
Run npm publish. The package should be published in few seconds. No worries, thanks to publishConfig property nobody can access it.
Now it is time to allow package installation in your project
Go to the directory of the project and open package.json file
Check that you have the name and version of your package in the dependencies list
Open browser, navigate to https://npmjs.com, login to your account, navigate to settings page of your account and open the tokens tab
Create new token and copy it
Now again go to the directory of your project, on same level where package.json file is situated (that is important!) and create .npmrc file there.
Put such string in the .npmrc file:
//registry.npmjs.org/:_authToken=TOKEN_HERE
You are done!
Deployment with remote CI/CD services
The easiest approach is not add .npmrc into .gitignore. In such case the file will be always in repository, so npm install will run smoothly on any machine where project was cloned
If you don't want to have token string in the repository, you can move it to the environment variable of your CI/CD service and then link .npmrc file to that variable. For example, you can put generated token into the NPM_TOKEN env variable (Just token from npmjs, not the whole string from .npmrc!)
And then change the .npmrc file in the next way:
//registry.npmjs.org/:_authToken=${NPM_TOKEN}.
So, with those steps you should be able to install your restricted packages without any issues. Good luck!
If you are trying to deploy you functions with firebase deploy from a CI and your .npmrc file looks like this.
#acmecorp:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${NPM_REGISTRY_TOKEN}
You will run into the problem even if you have the env var set.
Build failed: Error: Failed to replace env in config: ${NPM_REGISTRY_TOKEN}
Firebase for some reason needs access to that private repo. But the env var is not sent over to firebase.
Solution I've implemented was to replace ${NPM_REGISTRY_TOKEN} in the .npmrc file on every run of the CI pipeline.
sed -i.bak "s/\${NPM_REGISTRY_TOKEN}/${NPM_REGISTRY_TOKEN}/g" .npmrc
This breaks if you use Yarn. Took me a while to find a thread pointing to npm install in the firebase cli predeploy step. If there's no package-lock.json and you only use yarn, this will break. Remove yarn.lock and install using npm to resolve the issue.
When I hardcode my auth token into my .npmrc file, the private module installs as expected.
When I replace the hardcoded token with an environment variable, the private module fails to install.
I've tried multiple ways of writing the variable name, as well as the syntax of the variable in the .npmrc file, due to the following resources:
npm-config docs on env vars
a counter answer to the npm config docs above about the .npmrc variable syntax
npm blog post re: private modules that many folks link to as the source of truth on the matter
Example .npmrc files:
#fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=$NPM_TOKEN
#fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=${NPM_TOKEN}
#fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=${npm_token}
#fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=$npm_token
And, example .env files, both with and without strings around the value:
npm_config_npm_token=ABC123
npm_token=ABC123
NPM_CONFIG_NPM_TOKEN=ABC123
NPM_TOKEN=ABC123
Nothing has worked.
Any clues?
EDIT
I'm using npm v6.9.0.
I figured out the error in my assumption, and have come to a solution.
tl;dr - create shell-based persistant environment variables, and use the curly braces variable syntax in a .npmrc file.
The error was assuming npm reads project .env files. Apparently npm does not read .env files located in the a project root.
I wanted npm to read from a .env file so that I could keep all relevant data for a project contained within the project.
Instead, I created a shell-based environment variable that is always available. The following code blocks show how to add environment variables to your (oh-my-zsh) shell, even if you git watch and publish your dot files.
# ~/.oh-my-zsh/custom/env.zsh
# be sure this file is gitignored!
export TOKEN="ABC123"
# ~/.zshrc
source $ZSH/custom/env.zsh
# example .npmrc
#fortawesome:registry=https://npm.fontawesome.com/
//npm.fontawesome.com/:_authToken=${TOKEN}
Thanks to this npm forum answer that nudged me towards this shell-based solution.
ps - because the native OS X man pages file export and source under the builtin category, and don't actually provide any details on how each command works, here are links the man page for each:
export
source (or dot operator)
Assuming you're using macOS, try using the source command in the terminal to refresh the env file.
source .env
I had the same problem and looked and the same posts you did but none worked until I tried that. I got the hint from here. I also used curly brackets around my variable in the .npmrc file unlike suggested in the posts you linked. Good luck!
I have a private registry configured in my global .npmrc file. Now I want to configure a fallback registry also in the same npmrc file. When npm is not able to find module in my private repository, I want it to download it from npm global registry i.e. https://registry.npmjs.org
Below are the two repositories I want to configure:
http://devint:9999/nexus/content/groups/NPM-Release //npm should first look into this
https://registry.npmjs.org // fallback registry
I know there is a solution available on StackOverflow for this similar problem, but those solution is suggesting to used scoped package approach. I don't want to use the scoped package approach for configuring multiple repositories in a npmrc file.
I have already gone through the solution available on
Is there any way to configure multiple registries in a single npmrc file
But I am looking for a different approach, where I can define multiple repositories with its priority in npmrc file.
I request not to mark this question as closed without giving a satisfactory correct answer.
Thanks.
As far as I know you can not define multiple NPM registry URLs in .npmrc , on the CLI, or anywhere else, and have NPM check them based on priority.
Configure your NPM server to check for a requested package locally first and fall back to the public NPM registry if not found. This can be done with Nexus, and I believe Sinopia/Verdaccio do this out of the box.
I'm using npm link as described here
http://npmjs.org/doc/link.html
Locally everything works perfectly. When I deploy to Heroku I get the error message
Error: Cannot find module '...'
How can I get this working with Heroku?
I wish there were an elegant solution to this (it would make my life a hell of a lot easier). Your custom package is symlinked into node_modules by npm link, but git doesn't follow symbolic links nowadays. So when you git push to Heroku, there's no way to make your custom packages go along for the ride.
Note, however, that from my experiments, Heroku will honor any node_modules you do push in, instead of trying to install them from the network. It just runs npm install --production, essentially. Perhaps a hard link directly to the development source of your package would do the trick, but I'm not sure whether Git would play nicely with that. Use at your own risk!
EDIT: If you want to know exactly what Heroku does, it's all open source.
The ideal situation would be to get the packages, if they're open source, onto NPM itself. Which is pretty painless and automatic.
If you are hosting your private module on GitHub (or BitBucket), you can add the git repo as a dependency in your package.json.
"dependencies": {
// ... all your deps
"my_private_module": "git+ssh://git#github.com:my-username/my-private-module.git"
}
You will, however, need to grant privileges to Heroku to read your repo (assuming it's private -- which is the crux of the issue). Check out this answer for a detailed set of instructions showing how to do so with Github. This might help for Bitbucket.
I've found that the build time increases when doing this. Worth it for my needs.