Gitlab-ci.yml, SSH and NVM: connect to server via ssh and run nvm use - node.js

I have a Node app that uses a specific version of node, so I use nvm to trigger the correct version in the .npmrc file.
The app is deployed with Gitlab. The process is simple: when I push from local to gitlab repo, gitlab will connect to the production server via ssh and pull the latest version of the repo in Gitlab
image: node:latest
before_script:
- apt-get update -qq
- apt-get install -qq git
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$K8S_SECRET_SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
stages:
- deploy_production
deploy_production:
stage: deploy_production
only:
- master
script:
- ssh myuser#example.com 'export NPM_TOKEN=${NPM_TOKEN} && cd www/myproject && rm -rf node_modules dist/* && git pull && export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh" --no-use && eval "[ -f .nvmrc ] && nvm install || nvm install stable" && nvm use --delete-prefix && npm ci && npm run prod'
- ssh myuser#example.com "cd www/myproject && nvm use --delete-prefix"
The problem is in the last script. I get this response:
Webpack Bundle Analyzer saved report to /usr/home/myuser/www/myproject/dist/client-report.html
$ ssh myuser#example.com "cd www/myproject && nvm use --delete-prefix"
bash: nvm: command not found
ERROR: Job failed: exit code 1
If I enter directly from my local terminal to the deployment server and ask for the NVM_DIR variable I can get it as expected:
ssh myuser#example.com
cd www/myproject
echo $NVM_DIR
$~/home/myuser/.nvm
But if I add a script to my gitlab-ci.yml with this same action:
image: node:latest
before_script:
- apt-get update -qq
- apt-get install -qq git
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$K8S_SECRET_SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
stages:
- deploy_production
deploy_production:
stage: deploy_production
only:
- master
script:
- ssh myuser#example.com 'export NPM_TOKEN=${NPM_TOKEN} && cd www/myproject && rm -rf node_modules dist/* && git pull && export NVM_DIR="$HOME/.nvm" && . "$NVM_DIR/nvm.sh" --no-use && eval "[ -f .nvmrc ] && nvm install || nvm install stable" && nvm use --delete-prefix && npm ci && npm run prod'
- ssh myuser#example.com "cd www/myproject && echo $NVM_DIR"
- ssh myuser#example.com "cd www/myproject && nvm use --delete-prefix"
The result is empty
[…]
Webpack Bundle Analyzer saved report to /usr/home/myuser/www/myproject/dist/client-report.html
$ ssh myuser#example.com "cd www/myproject && echo $NVM_DIR"
$ ssh myuser#example.com "cd www/myproject && nvm use --delete-prefix"
bash: nvm: command not found
ERROR: Job failed: exit code 1
What can I do to trigger NVM in example.com via ssh in gitlab-ci.yml?

When you log in, your environment (in particular, the variable NVM_PATH but also PATH itself which causes "bash: nvm: command not found") is different than when executing commands with the SSH command. You can check this like so:
ssh myuser#example.com
env
# vs:
ssh myuser#example.com env
The first will execute—depending on what shell and OS you use—files such as .bashrc which execute after an interactive login, the second will (often) not. An excerpt of what NVM adds:
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
The first defines the variable you're missing, the second executes a script that among other things, alters your PATH so that the nvm executable can be found.
You can execute this file manually or add the snippet that NVM adds to your CI pipeline before calling nvm use.

Related

dockerfile : npm not found at entrypoint?

I have this dockerfile :
ARG DEBIAN_FRONTEND=noninteractive
FROM ubuntu:latest
WORKDIR .
COPY . /service/api/
RUN apt-get update -y && apt-get install -y openssl python3 curl
ENV NODE_VERSION 16.19.1
# install nvm
RUN cd / && curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash && cd / && \
export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" && nvm install $NODE_VERSION && \
cd /service/api/ && rm -rf node_modules && npm install
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH
ADD start.sh /start.sh
#ENTRYPOINT ["tail", "-f", "/dev/null"]
CMD cd /service/api/ && npm start
The image compiles correctly but it can't start because 'npm not found'.
When I uncomment the other debug entrypoint above to exec in the container, I see that npm -v and node -v are available in it.
Why aren't they available in my entrypoint ?
The problem is that RUN ... export NVM_DIR="$HOME/.nvm" does not allow you to use $NVM_DIR later in the Dockerfile to benefit from Docker variables substitutions. Only variables defined by a line starting with ENV can be used later (except when using the variable in a line starting with RUN, but this variable replacement is not implemented by Docker but by a shell). Therefore, your Dockerfile line ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules does not resolve $NVM_DIR to the correct path ($NVM_DIR is simply replaced by an empty string, at this step).
Simply add ENV NVM_DIR /root/.nvm just before ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules and this will correct your Dockerfile.
The correct Dockerfile should be:
ARG DEBIAN_FRONTEND=noninteractive
FROM ubuntu:latest
WORKDIR .
COPY . /service/api/
RUN apt-get update -y && apt-get install -y openssl python3 curl
ENV NODE_VERSION 16.19.1
# install nvm
RUN cd / && curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash && cd / && \
export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" && nvm install $NODE_VERSION && \
cd /service/api/ && rm -rf node_modules && npm install
ENV NVM_DIR /root/.nvm
ENV NODE_PATH $NVM_DIR/v$NODE_VERSION/lib/node_modules
ENV PATH $NVM_DIR/versions/node/v$NODE_VERSION/bin:$PATH
ADD start.sh /start.sh
#ENTRYPOINT ["tail", "-f", "/dev/null"]
CMD cd /service/api/ && npm start

build pipeline is not working please check

[tag:```
stages: # List of stages for jobs, and their order of execution
build
build:
only:
refs:
- development
stage: build
when: manual
image: node
allow_failure: false
before_script:
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/load-secure-files/-/raw/main/installer" | bash
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- 'mkdir -p ~/.ssh'
- 'mv .secure_files/KEY0.pem ~/.ssh/id_rsa && chmod 400 ~/.ssh/id_rsa'
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- export NODE_OPTIONS=--max_old_space_size=8048
- apt update
- npm install -g #angular/cli
- npm install
- npm build --prod
- echo "BUILD SUCCESSFUL"
after_script:
#- scp -rq ./build/* $SSH_TARGET_DEMO:/var/www/html/
stages: # List of stages for jobs, and their order of execution
build
build:
only:
refs:
- development
stage: build
when: manual
image: node
allow_failure: false
before_script:
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/load-secure-files/-/raw/main/installer" | bash
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- 'mkdir -p ~/.ssh'
- 'mv .secure_files/KEY0.pem ~/.ssh/id_rsa && chmod 400 ~/.ssh/id_rsa'
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- export NODE_OPTIONS=--max_old_space_size=8048
- apt update
- npm install -g #angular/cli
- npm install
- npm build --prod
- echo "BUILD SUCCESSFUL"
after_script:
#- scp -rq ./build/* $SSH_TARGET_DEMO:/var/www/html/
]

Error loading key "/dev/fd/63: Error when trigger the CI/CD in Gitlab

Gitlab CI/CD failed while connecting to the Digital Ocean Droplet, via ssh:
This is my CI file
before_script:
- apt-get update -qq
- apt-get install -qq git
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "${SSH_PRIVATE_KEY}" | base64 --decode | tr -d "\r")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy:
type: deploy
environment:
name: production
script:
- ssh root#xxx.xxx.xxx.xxx "cd /var/www/html/customer-web && git checkout master && git pull origin master && npm install && npm run build && exit"
only:
- master
when I trigger this, I'm getting the following error
$ eval $(ssh-agent -s)
Agent pid 267
$ ssh-add <(echo "${SSH_PRIVATE_KEY}" | base64 --decode | tr -d "\r")
Error loading key "/dev/fd/63": invalid format
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
and I am saving the ~/.ssh/id_rsa in CI/CD variable too. any one have idea why this error comes and failed.
The key i generated with OPENSSH, not in RSA after i generate the key in RSA the issue has been fixed

Gitlab-ci alpine image : Host key verification failed

In gitlab I build my job with a maven image, then copy the jar to the ssh server -> it works fine.
For a php project, I try to use alpine image. But I get rejected with 'Host key verification failed'.
The server and the key are the same.
Not working:
image: alpine:latest
stages:
- deploy
deploy:
before_script:
- apk add --update openssh-client bash
- eval $(ssh-agent -s)
- bash -c 'ssh-add <(echo "$SSH_PRIVATE_KEY")'
stage: deploy
script:
- ssh root#devsb01 "ls"
Working:
image: maven:3.6.0-jdk-10-slim
stages:
- deploy
deploy:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- '[[ -f /.dockerenv ]] && mkdir -p ~/.ssh && echo "$KNOWN_HOST" > ~/.ssh/known_hosts'
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
stage: deploy
script:
- ssh root#devsb01 "ls"
I think this has to do with the way the ssh key is add.
try adding theses two lines:
- mkdir ~/.ssh
- ssh-keyscan -t rsa devsb01 >> ~/.ssh/known_hosts
It works for me!
Your file will look like that:
image: alpine:latest
stages:
- deploy
deploy:
before_script:
- apk add --update openssh-client bash
- eval $(ssh-agent -s)
- bash -c 'ssh-add <(echo "$SSH_PRIVATE_KEY")'
- mkdir ~/.ssh
- ssh-keyscan -t rsa devsb01 >> ~/.ssh/known_hosts
stage: deploy
script:
- ssh root#devsb01 "ls"

SSH example does not work with docker:git / dind image

I'm reading the documentation: https://gitlab.ida.liu.se/help/ci/ssh_keys/README.md
I'm using the current gitlab-ci.yml:
image: docker:git
services:
- docker:dind
stages:
- node_test
- docker_one
variables:
DOCKER_DRIVER: overlay
before_script:
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# For Docker builds disable host key checking. Be aware that by adding that
# you are suspectible to man-in-the-middle attacks.
# WARNING: Use this only with the Docker executor, if you use it with shell
# you will overwrite your user's SSH config.
- mkdir -p ~/.ssh
- ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
- ssh-keyscan gitlab.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_hosts
- '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
node_test:
image: node:7.3.0
stage: node_test
script:
- ls
docker_one:
stage: docker_one
script:
- docker info
node_test passes:
Running with gitlab-ci-multi-runner 1.10.4 (b32125f)
Using Docker executor with image node:7.3.0 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Waiting for services to be up and running...
Pulling docker image node:7.3.0 ...
Running on runner-4e4528ca-project-2641294-concurrent-0 via runner-4e4528ca-machine-1487244621-a8ff6b11-digital-ocean-4gb...
Cloning repository...
Cloning into '/builds/instanty/test'...
Checking out 0b261283 as master...
Skipping Git submodules setup
Identity added: /dev/fd/63 (rsa w/o comment)
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 11
$ ssh-add <(echo "$SSH_PRIVATE_KEY")
$ mkdir -p ~/.ssh
$ ssh-keyscan -H 'gitlab.com' >> ~/.ssh/known_hosts
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
$ ssh-keyscan gitlab.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_hosts
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
# gitlab.com SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
$ [[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ ls
Build succeeded
It fails when I use the image docker:git and service dind:
Running with gitlab-ci-multi-runner 1.10.4 (b32125f)
Using Docker executor with image docker:git ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:git ...
Running on runner-4e4528ca-project-2641294-concurrent-0 via runner-4e4528ca-machine-1487244733-2b616928-digital-ocean-4gb...
Cloning repository...
Cloning into '/builds/instanty/test'...
Checking out 0b261283 as master...
Skipping Git submodules setup
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 14
$ ssh-add <(echo "$SSH_PRIVATE_KEY")
/bin/sh: eval: line 50: syntax error: unexpected "("
ERROR: Build failed: exit code 2
Why does this image fail?
The syntax you are using is not recognized by the shell used by your image :
ssh-add <(echo "$SSH_PRIVATE_KEY")
ssh-add takes a file as an argument. So to fix this, you can replace this line with :
echo "$SSH_PRIVATE_KEY" > ssh.priv
ssh-add ssh.priv

Resources