I have a Node project in a private GitHub repo that includes an NPM dependency on another private GitHub repo, e.g.:
{
"dependencies": {
"my-secret-module": "git+ssh://github.com/user/repo-name.git#sha"
}
}
The project is being run on Heroku, where the npm install step fails during deployment while attempting to install from the private repo.
I'm aware of the possibility of generating OAuth tokens on GitHub and using them in the above URL like so: git+ssh://<token>#github.com/user/repo-name.git, but I don't want to store the token in version control.
The most ideal situation would be to simply store the token in an environment variable, so my question is: is it possible to authenticate with GitHub for this situation using environment variables? Or something else similarly bound to the Heroku app?
You could take a look at this buildpack that allows you to add your ssh key to be used as environment variable. Also, take a look at this SO answer that will do basically the same but using pre-build scripts.
Related
I'm trying to deploy a microservice to GCF, which has a dependency of a private GitHub-hosted package. To gain access to the package, I added a .npmrc file to the function as described here, the file looks like this:
registry=https://npm.pkg.github.com/OWNER
//npm.pkg.github.com/:_authToken=PERSONAL-ACCESS-TOKEN
Also tried using a NPM_TOKEN env variable in the Cloud Function
NPM_TOKEN=PERSONAL-ACCESS-TOKEN
Both result in the following error:
OperationError: code=3, message=Build failed: { error: { canonicalCode: "INVALID_ARGUMENT" }}
npm ERR! 404 Not Found: #packagescope/packagename
Installing locally works fine, so does deploying on Zeit Now.
I just ran into this problem so I'm sharing the fix that works with node v8 and v10 Cloud Functions.
The following is required to use the .npmrc to install packages from a private Github packages registry:
The .npmrc needs to be located in the functions folder at the same level as the package.json file
A registry entry for the your account/org scope is required, including the url like so, assuming Anchorman uses Github: #ronburgundy:registry=https://npm.pkg.github.com/ronburgundy
A Github personal access token is required for authentication, like so //npm.pkg.github.com/:_authToken=ronburgundypersonalaccesstoken
Assuming you follow best practice and do not commit the .npmrc with secrets you will need to ensure that the file is transformed to include the personal access token prior to deploying via firebase cli, as there's no other way to inject the value at runtime.
So the original example would work if it looked like this:
#OWNER:registry=https://npm.pkg.github.com/OWNER
//npm.pkg.github.com/:_authToken=PERSONAL-ACCESS-TOKEN
While Github's docs seem to suggest that you should redirect ALL scoped and unscoped package installs to their registry it appears that Google Cloud Functions does not allow us to redirect all package installs to a private registry, only those that we configure based on scope.
I have created a private typings npm module that I am using for my firebase functions and app projects. When I went to deploy firebase functions, I get a big error for every function that basically says ERR! remote: Invalid username or password.
For what I have read, it looks like I need to create a .npmrc file and put it in the /functions directory. (https://cloud.google.com/functions/docs/writing/specifying-dependencies-nodejs#using_private_modules)
I cannot however find proper instructions on how to do this anywhere. From what I found, I have done the following:
ran npm login
ran npm token create --read-only
This then gave me a token that looks like this: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.
I then create a file called .npmrc in my functions directory, and placed //registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX in it.
Additionally, I saw that the error message was trying to use ssh to install my private repo package, I have not setup ssh and am using https instead, because of this I changed my package file to git+https://github.com/accounts-name/repo#commit-num so that it uses HTTPS.
However, I still get the same error message. What am I missing? The above is what I have strung together from lots of google searching.
It seems that you have made too many different changes while trying to make it work, so let's just go through the whole process step by step.
Check the package.json of your npm module and publish it:
Remove "private" property or set it to false because private modules in the npm are meant to be never published. That is not obvious but that is true.
Next step is to apply restricted access to the package. In order to do that, add such property in the package.json file:
"publishConfig": {
"access": "restricted"
},
Make sure that npm account you use for publishing supports private packages.
Now open the terminal in the root directory of your package, type npm login then sign in to npm. Check if you put the proper version in the package.json.
Run npm publish. The package should be published in few seconds. No worries, thanks to publishConfig property nobody can access it.
Now it is time to allow package installation in your project
Go to the directory of the project and open package.json file
Check that you have the name and version of your package in the dependencies list
Open browser, navigate to https://npmjs.com, login to your account, navigate to settings page of your account and open the tokens tab
Create new token and copy it
Now again go to the directory of your project, on same level where package.json file is situated (that is important!) and create .npmrc file there.
Put such string in the .npmrc file:
//registry.npmjs.org/:_authToken=TOKEN_HERE
You are done!
Deployment with remote CI/CD services
The easiest approach is not add .npmrc into .gitignore. In such case the file will be always in repository, so npm install will run smoothly on any machine where project was cloned
If you don't want to have token string in the repository, you can move it to the environment variable of your CI/CD service and then link .npmrc file to that variable. For example, you can put generated token into the NPM_TOKEN env variable (Just token from npmjs, not the whole string from .npmrc!)
And then change the .npmrc file in the next way:
//registry.npmjs.org/:_authToken=${NPM_TOKEN}.
So, with those steps you should be able to install your restricted packages without any issues. Good luck!
If you are trying to deploy you functions with firebase deploy from a CI and your .npmrc file looks like this.
#acmecorp:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${NPM_REGISTRY_TOKEN}
You will run into the problem even if you have the env var set.
Build failed: Error: Failed to replace env in config: ${NPM_REGISTRY_TOKEN}
Firebase for some reason needs access to that private repo. But the env var is not sent over to firebase.
Solution I've implemented was to replace ${NPM_REGISTRY_TOKEN} in the .npmrc file on every run of the CI pipeline.
sed -i.bak "s/\${NPM_REGISTRY_TOKEN}/${NPM_REGISTRY_TOKEN}/g" .npmrc
This breaks if you use Yarn. Took me a while to find a thread pointing to npm install in the firebase cli predeploy step. If there's no package-lock.json and you only use yarn, this will break. Remove yarn.lock and install using npm to resolve the issue.
I have an NPM project that uses a git dependency
{
"repository": {
"type": "git",
"url": "https://bitbucket.org/my-private-group"
},
"dependencies": {
"my-dependency": "bitbucket:group/lib#version",
},
}
Now I want to build this project in CI using Docker with node installed.
Problem: node install tries to call git and fails because git is not there. But even if I install git it still requires authentication because it is a private repository.
At the moment I see the following solutions:
I would have to install git in docker and add an SSH key to be able to download the source code.
I may pack the related repository into the Docker image and use npm link. But this option still requires knowing dependencies set up in package.json which makes it complicated.
Setup an own npm repository to post artifact and do not use git dependencies. This option is unfortunately not achievable in my case.
Question: What is the best way of handling git dependencies in CI? Are there any other options a part from the listed options. What is the best practice?
Pulling from git without git installed is kinda hard. And installing git is easy. Just list is as a dependency for your project. This project requires windows/linux/mac os, node js, git.
You're allowing people to pull people from a private repo... that moment they have access to your source code... so all use of having the repo private is lost anyway. Anyone who wants to duplicate your code can do easily the moment it's on their computer, even if it's obfuscated.
So, I would go back a step and ask you to start asking yourself why is the repo private? Is it code that is only distributed when an NDA is present? If so, you could consider working with ssh key files to log in.
Or, you could host your files on a privately hosted gogs server, where you whitelist IP's in the firewall/nginx router that can pull from the gogs repository on your server.
If you want anyone to be able to use your repository in the final distribution of your project, you're better off lifting the private setting of your repository. You might even get some free help fixing some bugs.
I believe bitbucket has something called deployment keys which gives read only access to repositories. I am using deployment keys to build my private projects and its private dependencies.
The private key is stored in the CI server (Jenkins) and is injected into the appropriate project during its build process.
Another way is to use deployment keys with the private key stored in the project itself which can then be used during the build process.
Update
Assuming Jenkins Pipeline the following is an example of how to access ssh keys set in Jenkins using the Credentials Binding Plugin
stage('Sample') {
agent {
docker {
image 'node:12'
}
}
steps {
withCredentials([
sshUserPrivateKey(
credentialsId: 'ssh-key-name-here',
keyFileVariable: 'GIT_DEPLOY_KEY_FILE'
)
]) {
sh "cat $GIT_DEPLOY_KEY_FILE"
}
}
}
Update Sept 16th 2019
I recently came across the Build Enhancements made in Docker 18.09.
I have not yet to explore this but I think it can be used to solve the credential problem.
I'm setting up some CI builds of NodeJS projects in VS Team Services. I currently have four projects being cloned from private Github repositories, running npm install, and running unit tests. Because I'm giving the VS Team Services build an access token for the repositories it's cloning, these builds all pass.
The fifth project relies on one of the other four projects as a dependency in package.json. The build for this project fails on npm install when attempting to clone the private repository because the build does not have permission to clone the repository. How can I give this build access to clone our private repositories during npm install?
I have one build definition for each of the five projects, and each project is a separate repository in my-org on Github. In each build definition, the connection to Github is managed by personal access token, and the repository points to my-org/project-name. The package.json file of the project in the fifth, failing, build has a dependencies configuration that looks like this:
"dependencies": {
"project-name": "git+ssh://git#github.com/my-org/project-name.git#master",
"jquery": "^2.2.4",
"react": "^15.0.1",
"react-dom": "^15.0.1"
}
The first image shows the npm install step of my build definition.
The second image shows the npm test step of my build definition.
The problem is that the build agent will not able to authenticate because of lack of SSH keys on the build agent and because the host verification will fail anyway.
Instead you should create a Personal Access Token on GitHub with 'repo' only scope, then you should use it on your packages.json file (notice that ssh is replaced with https protocol):
"project-name": "git+https://<user>:<token>#github.com/my-org/project-name.git#master",
Recently i needed to do this in Azure Devops on a Windows based agent, Instead of changing dependencies to include user and token in the packages.json, it was easier to create a build step to add the authentication information from environment variables to windows credentials store and instructing git to use that.
git config --global credential.helper wincred
cmdkey /generic:LegacyGeneric:target=git:https://our-private-domain.com /user:git-readonly-use#our-private-domain.com /pass:"Personal Access Token With Read Permissions"
After adding these configurations, Npm installs that we were doing using a script in source code went through without any issue
"dependencies": {
"example-1": "git+https://our-private-domain.com/proj/example-1.git#v0.1.9",
"example-2": "git+https://our-private-domain.com/proj/example-2.git#master"
}
I need to have my common npm module as git dependency and use it in many modules.
I'm trying to use access token for authorize in git. Bit I don't want to store the token in package.json (at least, it's hard to change it in few modules if it change).
Can I do something like this:
package.json
...
"scripts": {
"preinstall": "read-credentials.sh"
}
...
"dependencies": {
...
"my-module": "git+https://$$$token-from-variable$$$:x-oauth-basic#some-repo-url"
...
}
I know about options with private npm modules or make ssh keys for git access, but it isn't a good fit for me. The main point of the question is to figure out if there is a way to set variables from some preinstall script (where I can have access to access tokens) and use them in package.json.
You can have private modules on npm. It costs $7/month and is created for exactly the use case that you have here.
If you don't want to do that for some reason, then you can rely on git-specific environment variables to resolve your dependencies. For example if you use git+ssh://git#SERVER:USER/MODULE.git as a dependency then npm will try to get it with git using the default credentials used by git (which in turn uses the credentials used by ssh). You can even change GIT_SSH environment variable to completely control the SSH session however you want.
For more options see Install node module from own gitlab server.
As I understood, there is no way to create such kind of variables.
My solution for my problem:
Creating other module, which can take a token from external place and which I can put in public git or npm repo.
Add this module in dependency.
Call it in postinstall script.
Get token, build url and install my private-git-module from JS using npm.commands.install.