GitLab AutoDevops Environment Issues - gitlab

So I am new to Gitlab Autodevops having switched from Travis and Github. The issue I am currently facing is that when I make a push and the pipeline kicks in, it doesn't see any of my list environment variables. I set production, and testing environment variables for mongodb and redis, but during the pipeline, it's trying to connect to localhost for both, totally ignoring the environment variables set in CI/CD settings. See pictures below:
Dockerfile
WORKDIR /app
COPY package*.json ./
RUN apk add --update alpine-sdk nodejs npm python
RUN LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH && export LD_LIBRARY_PATH && npm i
COPY . .
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
docker-compose.yml
version: "3.7"
services:
backend:
container_name: dash-loan
environment:
MONGODB_PRODUCTION_URI: ${MONGODB_PRODUCTION_URI}
MONGODB_TEST_URI: ${MONGODB_TEST_URI}
REDIS_PRODUCTION_URL: ${REDIS_PRODUCTION_URL}
REDIS_TEST_URL: ${REDIS_TEST_URL}
PM2_SECRET_KEY: ${PM2_SECRET_KEY}
PM2_PUBLIC_KEY: ${PM2_PUBLIC_KEY}
PM2_MACHINE_NAME: ${PM2_MACHINE_NAME}
PORT: ${PORT}
MODE_ENV: ${NODE_ENV}
restart: always
build: .
ports:
- "8080:3000"
links:
- mongodb
- redis
mongodb:
container_name: mongo
environment:
MONGO_INITDB_DATABASE: dashloan
MONGO_INITDB_ROOT_USERNAME: sampleUser
MONGO_INITDB_ROOT_PASSWORD: samplePassword
restart: always
image: mongo
ports:
- "27017-27019:27017-27019"
volumes:
- ./src/database/init-mongo.js:/docker-entrypoint-point.initdb.d/init-mongo.js:ro
- ./mongo-volume:/data/db
redis:
container_name: redis
restart: always
image: redis:5.0
ports:
- "6379:6379"
volumes:
mongo-volume:
.gitlab-ci.yml
image: node:latest
services:
- mongo:latest
- redis:latest
cache:
paths:
- node_modules/
job:
script:
- npm i
- npm test
I need help on how to make sure the test pipeline is using the environment variables I set; and not trying to connect to localhost which fails.
Error on gitlab pipeline
Variables in Gitlab
GKE which is running fine

You could use shell runner instead of docker runner and then just call docker-compose in before script.
cache:
paths:
- node_modules/
job:
before_script:
- docker-compose up -d
script:
- npm i
- npm test
after_script:
- docker-compose down

Related

Docker run exit wit code 0 - Node.js & Postgres

I've build a small Node.js app with a Postgres database. I wanted to Dockerize it, and everything worked fine when I was using docker-compose, now I'm trying to put it on AWS EC2, and I'm using docker run, but it doesn't work.
Docker Run command:
docker run -d -p 80:4000 --name appname hubname/appname
the container starts and stops immediatly
I'm fairly new to Docker so I guess I'm missing something.
Please find my Dockerfile:
FROM node:14 as base
WORKDIR /home/node/app
COPY package*.json ./
RUN npm install
COPY . .
FROM base as production
ENV NODE_PATH=./build
RUN npm start
My docker-compose:
version: '3.7'
services:
appname-db:
image: postgres
container_name: appname-db
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
ports:
- '5432:5432'
volumes:
- appname-db:/var/lib/postgresql/data
ts-node-docker:
build:
context: .
dockerfile: Dockerfile
target: production
volumes:
- ./src:/home/node/app/src
- ./nodemon.json:/home/node/app/nodemon.json
container_name: ts-node-docker
environment:
DB_HOST: ${DB_HOST}
DB_SCHEMA: ${DB_SCHEMA}
DB_USER: ${DB_USER}
DB_PASSWORD: ${DB_PASSWORD}
depends_on:
- appname-db:
condition: service_healthy
expose:
- '4000'
ports:
- '80:4000'
stdin_open: true
command: npm run start
volumes:
appname-db:
When I do:
docker ps -a I see that the container started and exited right after.
I can't find any logs whatsoever it's very frustrating.
Any help/pointer would be really appreciated.
Edit 1 - Some additional information:
When I'm using docker-compose we can see 2 containers:
However this is what I have with docker run:
You should use CMD and not RUN when you execute the npm run start command. CMD runs every time a container is started, while RUN is just executed once, at build time.
This means you start the container, but nothing else is started inside of it. So it exits because there's nothing more to execute. RUN was already executed last time you built the container.
Use RUN for npm install, otherwise it will try every time to reinstall node_modules you already have. But for the start command – do this instead: CMD ["npm", "run", "start"]
In case it could help you I show you my working docker-compose.yml and Dockerfile. Express API with a Postgres DB.
This is tested and works both on an Ubuntu server and on a dev machine (with docker-compose up for both environments). Note that there are two Dockerfiles (dev/prod).
You don't need to add password/user stuff in your docker-compose, you can just include your env_file and Docker/docker-compose sets everything up correctly for you.
version: "3.8"
services:
# D A T A B A S E #
db:
env_file:
- ./.env
image: postgres
container_name: "db"
restart: always
ports:
- "${POSTGRES_PORT}:${POSTGRES_PORT}"
volumes:
- ./db-data/:/var/lib/postgresql/data/
# A P I #
api:
container_name: "api"
env_file:
- ./.env
build:
context: .
dockerfile: "DOCKERFILE.${NODE_ENV}"
ports:
- ${API_PORT}:${API_PORT}
depends_on:
- db
restart: always
volumes:
- /app/node_modules
- .:/usr/src/service
working_dir: /usr/src/service
# A D M I N E R #
adminer:
env_file:
- ./.env
container_name: "adminer"
image: adminer:latest
restart: always
ports:
- ${ADMINER_PORT}:${ADMINER_PORT}
depends_on:
- db
Dockerfile.dev:
FROM node:14.17.3
ENV NODE_ENV=dev
WORKDIR /
COPY ./package*.json ./
RUN npm install
COPY . .
EXPOSE 8080
CMD ["npm", "run", "start"]
Dockerfile.prod is pretty much the same, just change the environment to prod.
I see is that you include the full path from your home directory when copying, try without that. Just copy from the location relative to your Dockerfile at /.

Nodemon not restarting in docker image

I am very new to docker and trying to get a working environment for local development. My issue is I can't seem to get nodemon to trigger correctly when there are changes. The nodemon config works outside of docker so I know that isn't the issue.
Here is what I have in my node app folder.
# Dockerfile.local
FROM node:16
WORKDIR /app
COPY package*.json .
RUN yarn install
COPY . .
RUN yarn build
CMD ["yarn", "watch"]
# docker-compose.yml
version: "3"
services:
db:
image: postgres:12.3
restart: always
volumes:
- db_data:/var/lib/postgresql/data
environment:
# ... my config
actions:
build:
context: ./action-handlers
dockerfile: .docker/${DOCKERFILE}
depends_on:
- "hasura"
volumes:
- actions:/./app
environment:
# ...my confg
hasura:
ports:
- 8080:8080
- 9691:9691
build:
context: ./hasura
dockerfile: .docker/${DOCKERFILE}
depends_on:
- "db"
environment:
# ...my config
volumes:
db_data:
actions:
The docker image works perfectly when I run docker-compose build to build the image, then to run it using docker-compose up. Nodemon runs, the other two services run as expect so there isn't any issues there. It just doesn't restart when I make code changes. I have a feeling I am not using volumes correctly.
I have a feeling that - "./actions:/app/action-handers" is incorrect.

How to solve docker compose cannot find module '/dockerize'

I have a docker-compose setup with three containers - mysql db, serverless app and test (to test the serverless app with mocha). On executing with docker-compose up --build, its seems like the order of execution is messed up and not in correct sequence. Both mysql db and serverless app need to be in working state in order for test to run correctly (depends_on barely works).
Hence I tried using dockerize module to set a timeout and listen to tcp ports on 3306 and 4200 before starting the test container. But I'm getting an error saying Cannot find module '/dockerize'.
Is there anything wrong with the way my docker-compose.yml and Dockerfile are setup? I'm very new to docker so any help would be welcomed.
Dockerfile.yml
FROM node:12
# Create app directory
RUN mkdir -p /app
WORKDIR /app
# Dockerize is needed to sync containers startup
ENV DOCKERIZE_VERSION v0.6.0
RUN wget --no-check-certificate https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
&& tar -C /usr/local/bin -xzvf dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
&& rm dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz
# Install app dependencies
COPY package.json /app/
COPY package-lock.json /app/
RUN npm install -g serverless#2.41.2
RUN npm install
# Bundle app source
COPY . /app
docker-compose.yml
version: '3'
services:
mysql-coredb:
image: mysql:5.6
container_name: mysql-coredb
expose:
- "3306"
ports:
- "3306:3306"
serverless-app:
build: .
image: serverless-app
container_name: serverless-app
depends_on:
- mysql-coredb
command: sls offline --stage local --host 0.0.0.0
expose:
- "4200"
ports:
- "4200:4200"
links:
- mysql-coredb
volumes:
- /app/node_modules
- .:/app
test:
image: node:12
working_dir: /app
container_name: test
volumes:
- /app/node_modules
- .:/app
command: dockerize
-wait tcp://serverless-app:4200 -wait tcp://mysql-coredb:3306 -timeout 15s
bash -c "npm test"
depends_on:
- mysql-coredb
- serverless-app
links:
- serverless-app
Your final test container is running a bare node image. This doesn't see or use any of the packages you install in the Dockerfile. You can set that container to also build: .; Compose will run through the build sequence a second time, but since all of the inputs are the same as the main serverless-app container, Docker will use the build cache for everything, the build will run very quickly, and you'll have two names for the same physical image.
You can also safely remove most of the options you have in the Dockerfile. You only need to specify image: with build: if you're planning to push the image to a registry; you don't need to override container_name:; the default command: should come from the Dockerfile CMD; expose: and links: are related to first-generation Docker networking and aren't necessary; overwriting the image code with volumes: results in ignoring the image contents entirely and getting host-specific behavior (just using Node on the host will be much easier than trying to convince Docker to act like a local Node development environment). So I'd trim this down to:
version: '3.8'
services:
mysql-coredb:
image: mysql:5.6
ports:
- "3306:3306"
serverless-app:
build: .
depends_on:
- mysql-coredb
ports:
- "4200:4200"
test:
build: .
command: >-
dockerize
-wait tcp://serverless-app:4200
-wait tcp://mysql-coredb:3306
-timeout 15s
npm test
depends_on:
- mysql-coredb
- serverless-app

Environment variables not loading inside docker container

I have two services which are dockerized. The first service is the api service which depends on the mongo service. However, for some reason, the api service can't connect to the mongo service because the environment variables inside the container are not been passed around well. I have two variables in my .env namely
JWT_SECRET=somesecret
DB_URL_LOCAL=mongodb://mongo:27017/apii
I tried setting the environment variables in the docker-compose file, when I run docker-compose config I see the variables and their values displayed, but my application can't detect these variables. MongoDB keeps throwing UnhandledPromiseRejectionWarning: MongoParseError: URI malformed, cannot be parsed error.
I also tried to set the environment variables in the main Dockerfile but it still didn't work. Please I need some assistance. Here is a copy of my Dockerfile and docker-compose.yml file.
This is the Dockerfile
FROM node:12-alpine AS base
WORKDIR /usr/src/app
FROM base AS build
COPY package*.json .babelrc ./
RUN npm install
COPY ./src ./src
RUN npm run build
RUN npm prune --production
FROM base AS release
COPY --from=build /usr/src/app/node_modules ./node_modules
COPY --from=build /usr/src/app/dist ./dist
USER node
EXPOSE 4000
CMD [ "node", "./dist/server/server.js" ]
This is the docker-compose.yml file
version: '3.8'
services:
api:
build: .
image: api
working_dir: /usr/src/app
expose:
- 4000
ports:
- '4000:4000'
depends_on:
- mongo
mongo:
container_name: mongo
env_file:
- .env
environment:
- DB_URL_LOCAL=$DB_URL_LOCAL
- JWT_SECRET=$JWT_SECRET
image: mongo
volumes:
- './data:/data/db'
ports:
- 27017:27017
restart: always
Thank you.
In docker-compose.yml, environment variables should be declared in api service not mongo service with something like this MONGO_INITDB_DATABASE=${DATABASE}
Use this, env_file is an alternative, read more:
version: '3.8'
services:
api:
build: .
image: api
working_dir: /usr/src/app
expose:
- 4000
ports:
- '4000:4000'
env_file:
- ./secret.env
environment:
- DB_URL_LOCAL=${DB_URL_LOCAL}
- JWT_SECRET=${JWT_SECRET}
depends_on:
- mongo
mongo:
container_name: mongo
image: mongo
volumes:
- './data:/data/db'
ports:
- 27017:27017
restart: always

docker-compose network error cant connect to other host

I'm new to docker and I'm having issues with connecting to my managed database cluster on the cloud services which was separated from the docker machine and network.
So recently I attempted to use docker-compose because manually writing docker run command every update is a hassle so I configure the yml file.
Whenever I use docker compose, I'm having issues connecting to the database with this error
Unhandled error event: Error: connect ENOENT %22rediss://default:password#test.ondigitalocean.com:25061%22
But if I run it on the actual docker run command with the ENV in dockerfile, then everything will work fine.
docker run -d -p 4000:4000 --restart always test
But I don't want to expose all the confidential data to the code repository with all the details on the dockerfile.
Here is my dockerfile and docker-compose
dockerfile
FROM node:14.3.0
WORKDIR /kpb
COPY package.json /kpb
RUN npm install
COPY . /kpb
CMD ["npm", "start"]
docker-compose
version: '3.8'
services:
app:
container_name: kibblepaw-graphql
restart: always
build: .
ports:
- '4000:4000'
environment:
- PRODUCTION="${PRODUCTION}"
- DB_SSL="${DB_SSL}"
- DB_CERT="${DB_CERT}"
- DB_URL="${DB_URL}"
- REDIS_URL="${REDIS_URL}"
- SESSION_KEY="${SESSION_KEY}"
- AWS_BUCKET_REGION="${AWS_BUCKET_REGION}"
- AWS_BUCKET="${AWS_BUCKET}"
- AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"
- AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"
You should not include the " for the values of your environment variables in your docker-compose.
This should work:
version: '3.8'
services:
app:
container_name: kibblepaw-graphql
restart: always
build: .
ports:
- '4000:4000'
environment:
- PRODUCTION=${PRODUCTION}
- DB_SSL=${DB_SSL}
- DB_CERT=${DB_CERT}
- DB_URL=${DB_URL}
- REDIS_URL=${REDIS_URL}
- SESSION_KEY=${SESSION_KEY}
- AWS_BUCKET_REGION=${AWS_BUCKET_REGION}
- AWS_BUCKET=${AWS_BUCKET}
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

Resources