I have an NPM project that uses a git dependency
{
"repository": {
"type": "git",
"url": "https://bitbucket.org/my-private-group"
},
"dependencies": {
"my-dependency": "bitbucket:group/lib#version",
},
}
Now I want to build this project in CI using Docker with node installed.
Problem: node install tries to call git and fails because git is not there. But even if I install git it still requires authentication because it is a private repository.
At the moment I see the following solutions:
I would have to install git in docker and add an SSH key to be able to download the source code.
I may pack the related repository into the Docker image and use npm link. But this option still requires knowing dependencies set up in package.json which makes it complicated.
Setup an own npm repository to post artifact and do not use git dependencies. This option is unfortunately not achievable in my case.
Question: What is the best way of handling git dependencies in CI? Are there any other options a part from the listed options. What is the best practice?
Pulling from git without git installed is kinda hard. And installing git is easy. Just list is as a dependency for your project. This project requires windows/linux/mac os, node js, git.
You're allowing people to pull people from a private repo... that moment they have access to your source code... so all use of having the repo private is lost anyway. Anyone who wants to duplicate your code can do easily the moment it's on their computer, even if it's obfuscated.
So, I would go back a step and ask you to start asking yourself why is the repo private? Is it code that is only distributed when an NDA is present? If so, you could consider working with ssh key files to log in.
Or, you could host your files on a privately hosted gogs server, where you whitelist IP's in the firewall/nginx router that can pull from the gogs repository on your server.
If you want anyone to be able to use your repository in the final distribution of your project, you're better off lifting the private setting of your repository. You might even get some free help fixing some bugs.
I believe bitbucket has something called deployment keys which gives read only access to repositories. I am using deployment keys to build my private projects and its private dependencies.
The private key is stored in the CI server (Jenkins) and is injected into the appropriate project during its build process.
Another way is to use deployment keys with the private key stored in the project itself which can then be used during the build process.
Update
Assuming Jenkins Pipeline the following is an example of how to access ssh keys set in Jenkins using the Credentials Binding Plugin
stage('Sample') {
agent {
docker {
image 'node:12'
}
}
steps {
withCredentials([
sshUserPrivateKey(
credentialsId: 'ssh-key-name-here',
keyFileVariable: 'GIT_DEPLOY_KEY_FILE'
)
]) {
sh "cat $GIT_DEPLOY_KEY_FILE"
}
}
}
Update Sept 16th 2019
I recently came across the Build Enhancements made in Docker 18.09.
I have not yet to explore this but I think it can be used to solve the credential problem.
Related
I am using GitHub packages to publish my private NPM packages. I need to download the published package of specific version for carrying out automation work. How can I download the package as a zip bundle using GitHub REST API or equivalent? Additionally, since it is a private package, it needs to be authenticated.
I know that equivalent functionality exists but it works for GitHub releases and assets. I could not find anything yet for GitHub packages!
After a few days of intermittent research regarding this question, here is what I have found. There seem to be two methods of retrieving and/or consuming privately published NPM packages from the GitHub Package Registry. And neither of them is an exact match to your desired method, so, here goes...
OPTION 1.
You can consume the npm package directly within your application provided you have a locally configured .nprmc file on your machine in your user directory (check first #~/.npmrc),
AND
you have created a GitHub personal access token with the following scopes.
repo: full (this is how you will be authenticated.)
workflow
write: packages
adminOrg: read
user: email
NOTE: You may not require all of these, but these are the permissions I used and had no issues. Once you generate the token, create or add it to your .npmrc file like so, replacing TOKEN with the actual token value.
//npm.pkg.github.com/:_authToken=TOKEN
Be sure you additionally add the following snippet into the project or container itself within another .npmrc file in the root directory of the codebase.
#YOUR_GITHUB_USERNAME:registry=https://npm.pkg.github.com
OPTION 2:
You can connect your package to a private GitHub repository, which will allow you to access the tar.gz artifact for each version of your package and directly download it to your local machine from the web.
github.com > Your profile > Packages > Connect repository > Select and link.
REFERENCE: How to publish packages to the GitHub Package Registry
I'm looking for a way to securely clone private npm modules from a proxy repository inside a Docker container that is spun up by a Jenkins that runs on Ubuntu. The Docker image will be thrown away, but it is supposed to compile the project and run the unit tests.
The Jenkinsfile used for the build looks, simplified, like this:
node('master') {
stage('Checkout from version control') {
checkout scm
}
stage('Build within Docker') {
docker.build("intermediate-image", ".")
}
}
The Dockerfile at the moment:
FROM node:10-alpine
COPY package.json package-lock.json .npmrc ./
RUN npm ci && \
rm -f .npmrc
COPY . .
RUN npm run build && \
npm run test
The .npmrc file (anonymized):
#domain:registry=https://npm.domain.com/
//npm.domain.com/:_authToken=abcdefg
The problem is that the COPY command creates a layer with the .npmrc file. Should I build outside of my own Jenkins server, the layer would be cached by the build provider.
Building manually, I could specify the token as a docker environment variable. Is there a way to set the environment variable on Ubuntu and have Jenkins pass it through to Docker?
(Maybe) I could inject environment variables into Jenkins and then into the pipeline? The user claims that the plugin is not fully compatible with the pipeline plugin though.
Should I use the fact that Docker and Jenkins run on the same machine and mount something into the container?
Or do I worry too much, considering that the image will not be published and the Jenkins is private too?
What I want to achieve is that a build can use an arbitrary node version that is independent of that of the build server's.
I have decided that, because the docker host is the same (virtual) machine as the Jenkins host, it is no problem if I bake the .npmrc file into a docker layer.
Anyone with access to the Docker host can, currently, steal the local .npmrc token anyway.
Furthermore, the group that has access to our private npm modules is a complete subgroup of people with access to the source control repository. Therefore, exposing the npm token to the build machine, Jenkins, Docker intermediate image, Docker image layer and/or repository poses no additional authentication problems as of now. Revoking access should then go hand in hand with rotating the npmrc token (so that removed developers do not use the build token), but that is a small attack surface, in any case waay smaller than people copying the code to a hard drive.
We will have to re-evaluate our options should this setup change. Hopefully, we will find a solution then, but it is not worth the trouble now. One possible solution could be requesting the token from a different docker container with the sole purpose of answering these (local) calls.
I'm setting up some CI builds of NodeJS projects in VS Team Services. I currently have four projects being cloned from private Github repositories, running npm install, and running unit tests. Because I'm giving the VS Team Services build an access token for the repositories it's cloning, these builds all pass.
The fifth project relies on one of the other four projects as a dependency in package.json. The build for this project fails on npm install when attempting to clone the private repository because the build does not have permission to clone the repository. How can I give this build access to clone our private repositories during npm install?
I have one build definition for each of the five projects, and each project is a separate repository in my-org on Github. In each build definition, the connection to Github is managed by personal access token, and the repository points to my-org/project-name. The package.json file of the project in the fifth, failing, build has a dependencies configuration that looks like this:
"dependencies": {
"project-name": "git+ssh://git#github.com/my-org/project-name.git#master",
"jquery": "^2.2.4",
"react": "^15.0.1",
"react-dom": "^15.0.1"
}
The first image shows the npm install step of my build definition.
The second image shows the npm test step of my build definition.
The problem is that the build agent will not able to authenticate because of lack of SSH keys on the build agent and because the host verification will fail anyway.
Instead you should create a Personal Access Token on GitHub with 'repo' only scope, then you should use it on your packages.json file (notice that ssh is replaced with https protocol):
"project-name": "git+https://<user>:<token>#github.com/my-org/project-name.git#master",
Recently i needed to do this in Azure Devops on a Windows based agent, Instead of changing dependencies to include user and token in the packages.json, it was easier to create a build step to add the authentication information from environment variables to windows credentials store and instructing git to use that.
git config --global credential.helper wincred
cmdkey /generic:LegacyGeneric:target=git:https://our-private-domain.com /user:git-readonly-use#our-private-domain.com /pass:"Personal Access Token With Read Permissions"
After adding these configurations, Npm installs that we were doing using a script in source code went through without any issue
"dependencies": {
"example-1": "git+https://our-private-domain.com/proj/example-1.git#v0.1.9",
"example-2": "git+https://our-private-domain.com/proj/example-2.git#master"
}
I have a Node project in a private GitHub repo that includes an NPM dependency on another private GitHub repo, e.g.:
{
"dependencies": {
"my-secret-module": "git+ssh://github.com/user/repo-name.git#sha"
}
}
The project is being run on Heroku, where the npm install step fails during deployment while attempting to install from the private repo.
I'm aware of the possibility of generating OAuth tokens on GitHub and using them in the above URL like so: git+ssh://<token>#github.com/user/repo-name.git, but I don't want to store the token in version control.
The most ideal situation would be to simply store the token in an environment variable, so my question is: is it possible to authenticate with GitHub for this situation using environment variables? Or something else similarly bound to the Heroku app?
You could take a look at this buildpack that allows you to add your ssh key to be used as environment variable. Also, take a look at this SO answer that will do basically the same but using pre-build scripts.
I'm using npm link as described here
http://npmjs.org/doc/link.html
Locally everything works perfectly. When I deploy to Heroku I get the error message
Error: Cannot find module '...'
How can I get this working with Heroku?
I wish there were an elegant solution to this (it would make my life a hell of a lot easier). Your custom package is symlinked into node_modules by npm link, but git doesn't follow symbolic links nowadays. So when you git push to Heroku, there's no way to make your custom packages go along for the ride.
Note, however, that from my experiments, Heroku will honor any node_modules you do push in, instead of trying to install them from the network. It just runs npm install --production, essentially. Perhaps a hard link directly to the development source of your package would do the trick, but I'm not sure whether Git would play nicely with that. Use at your own risk!
EDIT: If you want to know exactly what Heroku does, it's all open source.
The ideal situation would be to get the packages, if they're open source, onto NPM itself. Which is pretty painless and automatic.
If you are hosting your private module on GitHub (or BitBucket), you can add the git repo as a dependency in your package.json.
"dependencies": {
// ... all your deps
"my_private_module": "git+ssh://git#github.com:my-username/my-private-module.git"
}
You will, however, need to grant privileges to Heroku to read your repo (assuming it's private -- which is the crux of the issue). Check out this answer for a detailed set of instructions showing how to do so with Github. This might help for Bitbucket.
I've found that the build time increases when doing this. Worth it for my needs.