I'm setting up some CI builds of NodeJS projects in VS Team Services. I currently have four projects being cloned from private Github repositories, running npm install, and running unit tests. Because I'm giving the VS Team Services build an access token for the repositories it's cloning, these builds all pass.
The fifth project relies on one of the other four projects as a dependency in package.json. The build for this project fails on npm install when attempting to clone the private repository because the build does not have permission to clone the repository. How can I give this build access to clone our private repositories during npm install?
I have one build definition for each of the five projects, and each project is a separate repository in my-org on Github. In each build definition, the connection to Github is managed by personal access token, and the repository points to my-org/project-name. The package.json file of the project in the fifth, failing, build has a dependencies configuration that looks like this:
"dependencies": {
"project-name": "git+ssh://git#github.com/my-org/project-name.git#master",
"jquery": "^2.2.4",
"react": "^15.0.1",
"react-dom": "^15.0.1"
}
The first image shows the npm install step of my build definition.
The second image shows the npm test step of my build definition.
The problem is that the build agent will not able to authenticate because of lack of SSH keys on the build agent and because the host verification will fail anyway.
Instead you should create a Personal Access Token on GitHub with 'repo' only scope, then you should use it on your packages.json file (notice that ssh is replaced with https protocol):
"project-name": "git+https://<user>:<token>#github.com/my-org/project-name.git#master",
Recently i needed to do this in Azure Devops on a Windows based agent, Instead of changing dependencies to include user and token in the packages.json, it was easier to create a build step to add the authentication information from environment variables to windows credentials store and instructing git to use that.
git config --global credential.helper wincred
cmdkey /generic:LegacyGeneric:target=git:https://our-private-domain.com /user:git-readonly-use#our-private-domain.com /pass:"Personal Access Token With Read Permissions"
After adding these configurations, Npm installs that we were doing using a script in source code went through without any issue
"dependencies": {
"example-1": "git+https://our-private-domain.com/proj/example-1.git#v0.1.9",
"example-2": "git+https://our-private-domain.com/proj/example-2.git#master"
}
Related
I am using GitHub packages to publish my private NPM packages. I need to download the published package of specific version for carrying out automation work. How can I download the package as a zip bundle using GitHub REST API or equivalent? Additionally, since it is a private package, it needs to be authenticated.
I know that equivalent functionality exists but it works for GitHub releases and assets. I could not find anything yet for GitHub packages!
After a few days of intermittent research regarding this question, here is what I have found. There seem to be two methods of retrieving and/or consuming privately published NPM packages from the GitHub Package Registry. And neither of them is an exact match to your desired method, so, here goes...
OPTION 1.
You can consume the npm package directly within your application provided you have a locally configured .nprmc file on your machine in your user directory (check first #~/.npmrc),
AND
you have created a GitHub personal access token with the following scopes.
repo: full (this is how you will be authenticated.)
workflow
write: packages
adminOrg: read
user: email
NOTE: You may not require all of these, but these are the permissions I used and had no issues. Once you generate the token, create or add it to your .npmrc file like so, replacing TOKEN with the actual token value.
//npm.pkg.github.com/:_authToken=TOKEN
Be sure you additionally add the following snippet into the project or container itself within another .npmrc file in the root directory of the codebase.
#YOUR_GITHUB_USERNAME:registry=https://npm.pkg.github.com
OPTION 2:
You can connect your package to a private GitHub repository, which will allow you to access the tar.gz artifact for each version of your package and directly download it to your local machine from the web.
github.com > Your profile > Packages > Connect repository > Select and link.
REFERENCE: How to publish packages to the GitHub Package Registry
I have created a private typings npm module that I am using for my firebase functions and app projects. When I went to deploy firebase functions, I get a big error for every function that basically says ERR! remote: Invalid username or password.
For what I have read, it looks like I need to create a .npmrc file and put it in the /functions directory. (https://cloud.google.com/functions/docs/writing/specifying-dependencies-nodejs#using_private_modules)
I cannot however find proper instructions on how to do this anywhere. From what I found, I have done the following:
ran npm login
ran npm token create --read-only
This then gave me a token that looks like this: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX.
I then create a file called .npmrc in my functions directory, and placed //registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX in it.
Additionally, I saw that the error message was trying to use ssh to install my private repo package, I have not setup ssh and am using https instead, because of this I changed my package file to git+https://github.com/accounts-name/repo#commit-num so that it uses HTTPS.
However, I still get the same error message. What am I missing? The above is what I have strung together from lots of google searching.
It seems that you have made too many different changes while trying to make it work, so let's just go through the whole process step by step.
Check the package.json of your npm module and publish it:
Remove "private" property or set it to false because private modules in the npm are meant to be never published. That is not obvious but that is true.
Next step is to apply restricted access to the package. In order to do that, add such property in the package.json file:
"publishConfig": {
"access": "restricted"
},
Make sure that npm account you use for publishing supports private packages.
Now open the terminal in the root directory of your package, type npm login then sign in to npm. Check if you put the proper version in the package.json.
Run npm publish. The package should be published in few seconds. No worries, thanks to publishConfig property nobody can access it.
Now it is time to allow package installation in your project
Go to the directory of the project and open package.json file
Check that you have the name and version of your package in the dependencies list
Open browser, navigate to https://npmjs.com, login to your account, navigate to settings page of your account and open the tokens tab
Create new token and copy it
Now again go to the directory of your project, on same level where package.json file is situated (that is important!) and create .npmrc file there.
Put such string in the .npmrc file:
//registry.npmjs.org/:_authToken=TOKEN_HERE
You are done!
Deployment with remote CI/CD services
The easiest approach is not add .npmrc into .gitignore. In such case the file will be always in repository, so npm install will run smoothly on any machine where project was cloned
If you don't want to have token string in the repository, you can move it to the environment variable of your CI/CD service and then link .npmrc file to that variable. For example, you can put generated token into the NPM_TOKEN env variable (Just token from npmjs, not the whole string from .npmrc!)
And then change the .npmrc file in the next way:
//registry.npmjs.org/:_authToken=${NPM_TOKEN}.
So, with those steps you should be able to install your restricted packages without any issues. Good luck!
If you are trying to deploy you functions with firebase deploy from a CI and your .npmrc file looks like this.
#acmecorp:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=${NPM_REGISTRY_TOKEN}
You will run into the problem even if you have the env var set.
Build failed: Error: Failed to replace env in config: ${NPM_REGISTRY_TOKEN}
Firebase for some reason needs access to that private repo. But the env var is not sent over to firebase.
Solution I've implemented was to replace ${NPM_REGISTRY_TOKEN} in the .npmrc file on every run of the CI pipeline.
sed -i.bak "s/\${NPM_REGISTRY_TOKEN}/${NPM_REGISTRY_TOKEN}/g" .npmrc
This breaks if you use Yarn. Took me a while to find a thread pointing to npm install in the firebase cli predeploy step. If there's no package-lock.json and you only use yarn, this will break. Remove yarn.lock and install using npm to resolve the issue.
I have an NPM project that uses a git dependency
{
"repository": {
"type": "git",
"url": "https://bitbucket.org/my-private-group"
},
"dependencies": {
"my-dependency": "bitbucket:group/lib#version",
},
}
Now I want to build this project in CI using Docker with node installed.
Problem: node install tries to call git and fails because git is not there. But even if I install git it still requires authentication because it is a private repository.
At the moment I see the following solutions:
I would have to install git in docker and add an SSH key to be able to download the source code.
I may pack the related repository into the Docker image and use npm link. But this option still requires knowing dependencies set up in package.json which makes it complicated.
Setup an own npm repository to post artifact and do not use git dependencies. This option is unfortunately not achievable in my case.
Question: What is the best way of handling git dependencies in CI? Are there any other options a part from the listed options. What is the best practice?
Pulling from git without git installed is kinda hard. And installing git is easy. Just list is as a dependency for your project. This project requires windows/linux/mac os, node js, git.
You're allowing people to pull people from a private repo... that moment they have access to your source code... so all use of having the repo private is lost anyway. Anyone who wants to duplicate your code can do easily the moment it's on their computer, even if it's obfuscated.
So, I would go back a step and ask you to start asking yourself why is the repo private? Is it code that is only distributed when an NDA is present? If so, you could consider working with ssh key files to log in.
Or, you could host your files on a privately hosted gogs server, where you whitelist IP's in the firewall/nginx router that can pull from the gogs repository on your server.
If you want anyone to be able to use your repository in the final distribution of your project, you're better off lifting the private setting of your repository. You might even get some free help fixing some bugs.
I believe bitbucket has something called deployment keys which gives read only access to repositories. I am using deployment keys to build my private projects and its private dependencies.
The private key is stored in the CI server (Jenkins) and is injected into the appropriate project during its build process.
Another way is to use deployment keys with the private key stored in the project itself which can then be used during the build process.
Update
Assuming Jenkins Pipeline the following is an example of how to access ssh keys set in Jenkins using the Credentials Binding Plugin
stage('Sample') {
agent {
docker {
image 'node:12'
}
}
steps {
withCredentials([
sshUserPrivateKey(
credentialsId: 'ssh-key-name-here',
keyFileVariable: 'GIT_DEPLOY_KEY_FILE'
)
]) {
sh "cat $GIT_DEPLOY_KEY_FILE"
}
}
}
Update Sept 16th 2019
I recently came across the Build Enhancements made in Docker 18.09.
I have not yet to explore this but I think it can be used to solve the credential problem.
bug #1) local packages
Created a test project "azure_test"
In folder "azure_test" ran the command "express --view=hbs app".
/azure_test/app as all the structure of an npm package, but is in a directory.
Essentially a local package.
in Azure_test/package.json is declare a dependency to this local package
"dependencies": {
"app": "./app"
}
a local package as per https://docs.npmjs.com/getting-started/packages
on my windows 10 machine, in folder /azure_test, I run "npm install", "npm start" and all works and expected.
save into github
In Azure, create an "App Service" and configure Azure to get source directly out of github
this fails. Log file at
https://purple01test.scm.azurewebsites.net/DebugConsole/?shell=powershell
contains the line
26 verbose git clone git://github.com/./app fatal: remote error:
If I remove all the dependencies out of /azure_test/app/package.json and copy these dependencies into /azure_test/package.json (ie: not using local package). Then all works on Azure, but I lose modularity of local packages (manually copy dependencies to the top level).
So am I using local packages incorrectly and it just happens to work in windows 10 or does Azure have a bug?
I made this test project public. It can be found at https://github.com/johngrabner/azure_test/
bug #2) private packages hosted in github
"dependencies": {
"purple_shared_enums_pic_pi": "git+https://github.com/johngrabner/purple_shared_enums_PIC_PI.git"
}
This also causes Azure to fail.
If the package hosted in github is public, Azure passed.
So bug #2 appears to be Azure fails to use the github password for dependencies.
I did some research and found that Docker / Kubernetes is probably a better choice for deployment. A number of courses and online video listed "no more, it works on my machine, but not in production".
You package your app in Docker, where you specify the exact OS, middleware, etc. Test on your machine. Then deploy this image to Azure, AWS, or Google. Kubernetes manages the spaning of these images.
I have a Node project in a private GitHub repo that includes an NPM dependency on another private GitHub repo, e.g.:
{
"dependencies": {
"my-secret-module": "git+ssh://github.com/user/repo-name.git#sha"
}
}
The project is being run on Heroku, where the npm install step fails during deployment while attempting to install from the private repo.
I'm aware of the possibility of generating OAuth tokens on GitHub and using them in the above URL like so: git+ssh://<token>#github.com/user/repo-name.git, but I don't want to store the token in version control.
The most ideal situation would be to simply store the token in an environment variable, so my question is: is it possible to authenticate with GitHub for this situation using environment variables? Or something else similarly bound to the Heroku app?
You could take a look at this buildpack that allows you to add your ssh key to be used as environment variable. Also, take a look at this SO answer that will do basically the same but using pre-build scripts.