Docker + Nodejs + Private Repo + Private NPM Module - Access Problems - node.js

I am in process of setting up the the deployment of a Node.js Service with Docker.
The Dockerfile I have is pieced together from various examples from around the net.
The directory for the Dockerfile includes:
Dockerfile
id_rsa
start.sh
This is the Dockerfile:
FROM ubuntu:13.10
# make sure apt is up to date
RUN apt-get update
# install npm, git, ssh, curl
RUN apt-get install -y npm git git-core ssh curl
RUN mkdir /nodejs && curl http://nodejs.org/dist/v0.10.31/node-v0.10.31-linux-x64.tar.gz | tar xvzf - -C /nodejs --strip-components=1
# Fixes empty home
ENV PATH $PATH:/nodejs/bin
ENV HOME /root
# SSH SETUP
RUN mkdir -p /root/.ssh
ADD id_rsa /root/.ssh/id_rsa
RUN chmod 700 /root/.ssh/id_rsa
RUN echo "IdentityFile /root/.ssh/id_rsa" >> /root/.ssh/ssh_config
RUN ssh-keyscan github.com >> /root/.ssh/known_hosts
ADD start.sh /tmp/
RUN chmod +x /tmp/start.sh
CMD ./tmp/start.sh
After the set-up is complete, start.sh runs and I experience problems with a private NPM dependency that the private Node.js service has. This is what start.sh is doing:
cd /tmp
# try to remove the repo if it already exists
rm -rf MediaFX; true
git clone https://<username>:<password>#github.com/company/ExampleRepo.git
cd RepoName
node --version
ls
npm install
NODE_ENV=test DEBUG=* PORT=3000 node server.js
In package.json for ExampleRepo, there is one private module that we import like this:
"dependencies": {
"scribe": "git+ssh://git#github.com:Company/PrivateDep.git"
},
When npm install gets to this repo, it outputs these logs:
npm ERR! git clone git#github.com:InboxAppCo/scribe.git Cloning into bare repository '/root/.npm/_git-remotes/git-github-com-InboxAppCo-scribe-git-abae334a'...
npm ERR! git clone git#github.com:InboxAppCo/scribe.git
npm ERR! git clone git#github.com:InboxAppCo/scribe.git Warning: Permanently added the RSA host key for IP address '192.30.252.130' to the list of known hosts.
npm ERR! git clone git#github.com:InboxAppCo/scribe.git Permission denied (publickey).
npm ERR! git clone git#github.com:InboxAppCo/scribe.git fatal: Could not read from remote repository.
npm ERR! git clone git#github.com:InboxAppCo/scribe.git
npm ERR! git clone git#github.com:InboxAppCo/scribe.git Please make sure you have the correct access rights
npm ERR! git clone git#github.com:InboxAppCo/scribe.git and the repository exists.
npm ERR! Error: `git "clone" "--mirror" "git#github.com:InboxAppCo/scribe.git" "/root/.npm/_git-remotes/git-github-com-InboxAppCo-scribe-git-abae334a"` failed with 128
npm ERR! at ChildProcess.cpclosed (/usr/share/npm/lib/utils/exec.js:59:20)
npm ERR! at ChildProcess.EventEmitter.emit (events.js:98:17)
npm ERR! at Process.ChildProcess._handle.onexit (child_process.js:789:12)
npm ERR! If you need help, you may report this log at:
npm ERR! <http://bugs.debian.org/npm>
npm ERR! or use
npm ERR! reportbug --attach /tmp/MediaFX/npm-debug.log npm
npm ERR! System Linux 3.16.4-tinycore64
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /tmp/MediaFX
npm ERR! node -v v0.10.15
npm ERR! npm -v 1.2.18
I thought that since the git clone of the private Node service works fine, any of its private NPM dependencies would install smoothly.
I am fairly positive that my SSH set up is flawed (and that it didn't manifest its self while git cloning the private parents repo) because I added username and password to the link. However, I am unsure and would appreciate some guidance on how to do this correctly.

git clone https://<username>:<password>#github.com/company/ExampleRepo.git
Works, because you are passing the username and password and doing it over https
"dependencies": {
"scribe": "git+ssh://git#github.com:Company/PrivateDep.git"
},
Fails, because you are connecting directly over ssh and Docker doesn't do any ssh agent forwarding from the host machine.
Unfortunatly it dosen't look like npm supports any url format to send username and password like your clone line: https://docs.npmjs.com/files/package.json#git-urls-as-dependencies
You'd have to add your ssh keys to the docker container ( Not Reccomended )
Or do something funky like share you SSH_SOCKET from the host like:
https://gist.github.com/d11wtq/8699521

Here's an approach I'm going to try implementing this evening:
docker create --build-arg TOKEN <my priv token> <dockerFile>
maybe declare the arg in the docker file?
ARG TOKEN
then in the script have npm install use that TOKEN in dependencies
"privModule": "git+https://${TOKEN}:x-oauth-basic#github.com/<githubID>/<privateModule>.git"
and if that doesn't work, somehow replace that var in the package.json (with sed) or have npm use an environment var.

Related

Using SSH key for install npm module from private repository inside docker

I make container for nodejs project. Inside the project I am using private repository. I need an access to it. For that I am using next Dockerfile
FROM node:15
RUN echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
COPY keys/ssh_config /root/.ssh/config
COPY keys/bitbucket /root/.ssh/bitbucket
RUN chmod 600 /root/.ssh/bitbucket
RUN npm install -g typescript ts-node
WORKDIR /var/www
EXPOSE 3000
ssh_config file is
Host bitbucket.org
IdentityFile /root/.ssh/bitbucket
StrictHostKeyChecking no
in my package.json I added next line
"my-interfaces": "git+ssh://bitbucket.org:MY_USER_NAME/my-interfaces.git#master",
After building container using docker-compose, I login inside container and run
npm i
but in the end I see next error
npm ERR! code 128
npm ERR! command failed
npm ERR! command git ls-remote ssh://git#bitbucket.org/MY_USER_NAME/my-interfaces.git
npm ERR! Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-04-30T08_19_38_999Z-debug.log
I tried another way, I generate SSH key inside container and added public key to bitbucket, but I see the same error message.
You can use the host system’s ssh access.
FROM node:15
RUN apt install openssh-client git
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN --mount=type=ssh npm install -g typescript ts-node
WORKDIR /var/www
EXPOSE 3000
The image must be built as follows:
DOCKER_BUILDKIT=1 docker build --ssh default .
https://docs.docker.com/develop/develop-images/build_enhancements/#to-enable-buildkit-builds

Jenkins build issue - npm ERR! Your cache folder contains root-owned files

I am trying to build a small node app on my Jenkins pipeline, which is running in a virtual machine. cross this error:
+ npm install
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /.npm
npm ERR! errno EACCES
npm ERR!
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR!
npm ERR! To permanently fix this problem, please run:
npm ERR! sudo chown -R 111:120 "/.npm"
Running sudo chown -R 111:120 "/.npm" doesn`t help since it says:
chown: cannot access '/.npm': No such file or directory
And, as per my understanding, runs in a local context, when the problem is actually from the container perspective. I`ve tried to add the command above on my Docker and Jenkinsfile as well, to no avail. Below is my public repo:
Node app deploy on github
npm install --cache=".YourCustomCacheDirectoryName"
works perfectly fine, reason for this is your docker user isn't allowed to write in / ( root directory )
its not that a directory already exist at /.npm its that, your script is trying to create a directory at / which is not accessible for your user
you can either put
agent {
docker {
image 'node:latest'
args '-u root:root'
}
}
or just tell npm to use your custom cache directory
I had the same issue and fixed it by setting the npm cache directory to ENV variable in Dockerfile.
Add this to Dockerfile:
ENV npm_config_cache /home/node/app/.npm
As far as I can remember ,just updating npm version and deleting the whole project did the trick.

Docker container that pulls from private gilab repository

I'm building a Docker container for my Node.js + Vue application.
Since I have a global css library in another repository I have added this line in my package.json file:
"lib-css": "git+ssh://git#git.lib.com:9922/username/lib-css.git#development",
That way when I run npm install I install also my CSS library. The problem is that on my local env it asks for my password and I can insert it, but in the Docker build the process fails with the following error:
Step 7/10 : RUN npm install
---> Running in db10ca83586d
npm WARN deprecated babel-preset-es2015#6.24.1: 🙌 Thanks for using Babel: we recommend using babel-preset-env now: please read babeljs.io/env to update!
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git#git.lib.com:9922/username/lib-css.git
npm ERR!
npm ERR! Host key verification failed.
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR!
npm ERR! exited with error code: 128
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2017-12-11T08_49_11_152Z-debug.log
This is my current Dockerfile:
FROM node:carbon
WORKDIR /usr/src/app
RUN mkdir -p /root/.ssh
COPY .secrets /root/.ssh/id_rsa
RUN chmod 700 /root/.ssh && chmod 600 /root/.ssh/*
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm#5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm install --only=production
# Bundle app source
COPY . .
EXPOSE 8081
CMD [ "npm", "run dev" ]
~
My .secrets file contains my private key associated to the repository.
How can I make this works?

Performing a npm install via Docker on a windows host

I'm trying to create a docker dev tools container for a devlopement environment on a windows host via docker toolbox but I have some trouble running the npm install command.
It worked fine on a linux host but on the windows host I got the following error :
npm ERR! Linux 4.1.13-boot2docker
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.5.0
npm ERR! npm v3.3.12
npm ERR! path /var/www/site/.npm/gulp/3.9.0/package.tgz.e87c24357cd6065ee71ce44c6f23673b
npm ERR! code ETXTBSY
npm ERR! errno -26
npm ERR! syscall rename
npm ERR! ETXTBSY: text file is busy, rename '/var/www/site/.npm/gulp/3.9.0/package.tgz.e87c24357cd6065ee71ce44c6f23673b' -> '/var/www/site/.npm/gulp/3.9.0/package.tgz'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! <https://github.com/npm/npm/issues>
npm ERR! Linux 4.1.13-boot2docker
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.5.0
npm ERR! npm v3.3.12
npm ERR! path npm-debug.log.39d944b679d410e5293d6721cbc8287a
npm ERR! code ETXTBSY
npm ERR! errno -26
npm ERR! syscall rename
npm ERR! ETXTBSY: text file is busy, rename 'npm-debug.log.39d944b679d410e5293d6721cbc8287a' -> 'npm-debug.log'
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! <https://github.com/npm/npm/issues>
npm ERR! Please include the following file with any support request:
npm ERR! /var/www/site/npm-debug.log
Here is my Dockerfile :
FROM node:latest
RUN apt-get update
RUN apt-get install vim -y
RUN useradd -ms /bin/bash node
RUN echo "fs.inotify.max_user_watches=100000" > /etc/sysctl.conf
ADD . /var/www/site
RUN chown -R node:node /var/www/site
RUN chown -R node:node /usr/local/lib/node_modules
RUN chown -R node:node /usr/local/bin
USER node
ENV HOME /var/www/site
WORKDIR /var/www/site
RUN npm install -g bower
RUN npm install --global gulp -y
EXPOSE 80 8080 35729
In Docker quickstart terminal, I use the following commands :
Building the image (works fine)
docker build -t dev_tools .
Building the container (works fine)
docker run --name=dev_tools_container -t --rm -v "//c/Users/Public/site:/var/www/site" --net=host dev_tools
Trying to install npm dependencies (shoots the error):
docker exec -it dev_tools_container npm install
Thank you for your time !
Instead of
RUN npm install --global gulp -y
use
RUN sudo npm install --global gulp -y
You try to install gulp as a global package from user node (not superuser).
Or install gulp before switch user to node.
USER node
RUN npm install --global gulp -y
EDIT:
boot2docker is based on VirtualBox. Virtualbox does not allow symlinks on shared folders for security reasons.
To enable symlinks You must set VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_NAME to 1. (Here is link to description how to do it on Vargrant: Symbolic links and synced folders in Vagrant)
VBoxManage setextradata VM_NAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_NAME 1
Replace VM_NAME and SHARE_NAME and restart VirtualBox.
Another solution is add --no-bin-link to npm:
RUN npm install -g bower --no-bin-link
RUN npm install --global gulp -y --no-bin-link
EDIT 2
By default Windows 7 security policy does not allow creating symlinks as it's a potential security threat. If user is not in Administrators group run secpol.msc and navigate to Local Policies-User Rights Assignments and add your user to Create symbolic links.
If your user belongs to Administrators group then start VirtualBox with Run as Administrator.
You can mount node_modules as a volume, so it will be a Linux filesystem inside the Docker container. Add this to your Dockerfile:
VOLUME /var/www/site/node_modules
You will see the directory at C:Users/Public/site/node_modules because it is necessary for a mount point, but you will not see any contents unless you are inside the container.

Dockerfile build error on RUN npm install

I run a Docker build, using a Dockerfile to build an image. But I get an error on npm install:
Dockerfile :
# Download Runnable-web Repo
RUN eval $(ssh-agent) > /dev/null && ssh-add /.ssh/id_rsa && git clone git#github.com:CodeNow/runnable-web.git
WORKDIR runnable-web
RUN npm install
RUN bower install --allow-root
Error:
Step 5 : RUN npm install
---> Running in 3fefdf5af71d
npm ERR! install Couldn't read dependencies
npm ERR! Error: ENOENT, open '/runnable-web/package.json'
If you need help, you may report this log at:
<http://github.com/isaacs/npm/issues>
or email it to:
<npm-#googlegroups.com>
Linux 3.13.0-24-generic
"/usr/local/bin/node" "/usr/local/bin/npm" "install"
I tried it in Directory C:\ which is a shared driver.It throws the same error with yours.
Then I move Dockerfile to a sub directory like C:\Intel and cd it to run docker build ., it works
So I guess Dockerfile should not be placed in shared dirver root directory.

Resources