Docker-compose setup mongodb with data and a nodejs server - node.js

i want to spin up a whole nodejs/mongodb environment with docker compose.i created a mongodump into radcupDevSample.json which i want to mongo restore i a new mongodb container. After this i want a new node container with my api linked to the mondodb container with the sample data.
i have the following files:
1. docker-compose.yml:
db:
image: mongo
ports:
- "27017:27017"`
mongo-importer:
build: .
web:
build: web
links:
- db
ports:
- "3000:3000"
volumes:
- ./src:/home/env
environment:
NODE_ENV: development
2. web Dockerfile:
FROM node
RUN apt-get update -y
RUN apt-get install -y git
RUN apt-get install -y vim
RUN git clone https://sdsds-oauth-basic#github.com/jdklfj /home/app
WORKDIR /home/app/src
RUN npm install
CMD "npm start"
EXPOSE 3000
3. "." dockerfile
FROM mongo
COPY radcupDevSample.json /radcupDevSample.json
CMD mongorestore -h db /radcupDevSample.json
I used this answer mentioned here:
How do I seed a mongo database using docker-compose?
My problem is: if i try docker-compose up i get this error:
docker-compose up
radcupbackend_db_1 is up-to-date
Building web
Traceback (most recent call last):
File "<string>", line 3, in <module>
File "/compose/compose/cli/main.py", line 54, in main
File "/compose/compose/cli/docopt_command.py", line 23, in sys_dispatch
File "/compose/compose/cli/docopt_command.py", line 26, in dispatch
File "/compose/compose/cli/main.py", line 171, in perform_command
File "/compose/compose/cli/main.py", line 587, in up
File "/compose/compose/project.py", line 313, in up
File "/compose/compose/service.py", line 404, in execute_convergence_plan
File "/compose/compose/service.py", line 303, in create_container
File "/compose/compose/service.py", line 326, in ensure_image_exists
File "/compose/compose/service.py", line 723, in build
File "/compose/venv/lib/python2.7/site-packages/docker/api/build.py", line 41, in build
TypeError: You must specify a directory to build in path
docker-compose returned -1
Have someone a "hint" why i get this errors ? Or maybe could someone help me out to set up this environment ?
Thank you!
PS. building the 2 Dockerfiles separately don't throw any error...

Looks like the docker is unable to find web/ directory from your current directory where you are running docker-compose up
Does your structure look like this?
anovil#ubuntu-anovil:~/tmp/docker-compose-mongo$ tree .
.
├── docker-compose.yml
├── Dockerfile
├── radcupDevSample.json
├── src
└── web
└── Dockerfile
2 directories, 4 files
anovil#ubuntu-anovil:~/tmp/docker-compose-mongo$
I created this kind of structure and then when I ran my build,
anovil#ubuntu-anovil:~/docker-compose-mongo$ docker-compose up
dockercomposemongo_db_1 is up-to-date
Starting dockercomposemongo_web_1
Building mongo-importer
Step 1 : FROM mongo
---> 94a166215fe3
...
In order to make sure everything is in order, I recommend you to run a docker-compose build --no-cache first from the same directory, this will ensure that you start from a clean slate.
anovil#ubuntu-anovil:~/docker-compose-mongo$ docker-compose build --no-cache
db uses an image, skipping
Building web
Step 1 : FROM node
---> ac9b478bfbbd
...
And then run a docker-compose up
Please let me know how it went.

Related

Cannot dockerize app with docker compose with secrets says file not found but file exists

I am trying to dockerize an API which uses firebase, the credentials file is proving to be difficult to dockerize, I'll be deploying using docker-compose, my files are:
docker-compose:
version: "3.7"
services:
api:
restart: always
build: .
secrets:
- source: google_creds
target: auth_file
env_file: auth.env
ports:
- 1234:8990
secrets:
google_creds:
file: key.json
the key.json is the private key file
The Dockerfile looks like:
FROM alpine
# Install the required packages
RUN apk add --update git go musl-dev
# Install the required dependencies
RUN go get github.com/gorilla/mux
RUN go get golang.org/x/crypto/sha3
RUN go get github.com/lib/pq
RUN go get firebase.google.com/go
# Setup the proper workdir
WORKDIR /root/go/src/secure-notes-api
# Copy indivisual files at the end to leverage caching
COPY ./LICENSE ./
COPY ./README.md ./
COPY ./*.go ./
COPY db db
RUN go build
#Executable command needs to be static
CMD ["/root/go/src/secure-notes-api/secure-notes-api"]
I've set the GOOGLE_APPLICATION_CREDENTIALS env from my auth.env to: /run/secrets/auth_file
The program panics with:
panic: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open "/run/secrets/auth_file": no such file or directory
I've tried:
Mounting a volume to a path and setting the env var to that, results in the same
Copying the key to docker image (out of desperation), resulted in the same
Overriding start command to cat the secret file - this worked, i could see the entire file being outputted
Curiously enough, if I mount a volume, shell into it and execute the binary manually, it works perfectly well.

Handling node modules with docker-compose

I am building a set of connected node services using docker-compose and can't figure out the best way to handle node modules. Here's what should happen in a perfect world:
Full install of node_modules in each container happens on initial build via each service's Dockerfile
Node modules are cached after the initial load -- i.e. functionality so that npm only installs when package.json has changed
There is a clear method for installing npm modules -- whether it needs to be rebuilt or there is an easier way
Right now, whenever I npm install --save some-module and subsequently run docker-compose build or docker-compose up --build, I end up with the module not actually being installed.
Here is one of the Dockerfiles
FROM node:latest
# Create app directory
WORKDIR /home/app/api-gateway
# Intall app dependencies (and cache if package.json is unchanged)
COPY package.json .
RUN npm install
# Bundle app source
COPY . .
# Run the start command
CMD [ "npm", "dev" ]
and here is the docker-compose.myl
version: '3'
services:
users-db:
container_name: users-db
build: ./users-db
ports:
- '27018:27017'
healthcheck:
test: exit 0'
api-gateway:
container_name: api-gateway
build: ./api-gateway
command: npm run dev
volumes:
- './api-gateway:/home/app/api-gateway'
- /home/app/api-gateway/node_modules
ports:
- '3000:3000'
depends_on:
- users-db
links:
- users-db
It looks like this line might be overwriting your node_modules directory:
# Bundle app source
COPY . .
If you ran npm install on your host machine before running docker build to create the image, you have a node_modules directory on your host machine that is being copied into your container.
What I like to do to address this problem is copy the individual code directories and files only, eg:
# Copy each directory and file
COPY ./src ./src
COPY ./index.js ./index.js
If you have a lot of files and directories this can get cumbersome, so another method would be to add node_modules to your .dockerignore file. This way it gets ignored by Docker during the build.

Docker compose VSTS task failder

I'working on Asp.Bet Core 2 app with MSSQL on Linux. I would like to configure CI and CD to Azure.
I defined docker-compose task in Visual Studio Team Services, but I'm getting an error.
2017-11-09T20:24:09.4725450Z ##[section]Starting: Create images
2017-11-09T20:24:09.4801610Z ==============================================================================
2017-11-09T20:24:09.4817660Z Task : Docker Compose
2017-11-09T20:24:09.4835680Z Description : Build, push or run multi-container Docker applications. Task can be used with Docker or Azure Container registry.
2017-11-09T20:24:09.4852370Z Version : 0.4.7
2017-11-09T20:24:09.4867100Z Author : Microsoft Corporation
2017-11-09T20:24:09.4881820Z Help : [More Information](https://go.microsoft.com/fwlink/?linkid=848006)
2017-11-09T20:24:09.4897520Z ==============================================================================
2017-11-09T20:24:10.5040990Z [command]/usr/local/bin/docker-compose -f /opt/vsts/work/1/s/docker-compose.yml -f /opt/vsts/work/1/s/docker-compose.ci.build.yml -f /home/vsts/agent/.docker-compose.1510259050468.yml -p Travelingowe build
2017-11-09T20:24:10.8628630Z db uses an image, skipping
2017-11-09T20:24:10.8646610Z Building api
2017-11-09T20:24:10.9373040Z Step 1/7 : FROM microsoft/aspnetcore:2.0
2017-11-09T20:24:12.0041790Z 2.0: Pulling from microsoft/aspnetcore
2017-11-09T20:25:03.6328690Z Digest: sha256:e36cb8d1edcd1bfd7aea0412349482a9c1a601089d76d1a294067f5f7f1098a9
2017-11-09T20:25:03.6497680Z Status: Downloaded newer image for microsoft/aspnetcore:2.0
2017-11-09T20:25:03.6524910Z ---> e0e49def2506
2017-11-09T20:25:03.6551560Z Step 2/7 : MAINTAINER Maciej Skuratowski <maciejskuratowski#gmail.com>
2017-11-09T20:25:03.7496070Z ---> Running in e7667861d293
2017-11-09T20:25:03.9222120Z ---> 24ba03484562
2017-11-09T20:25:03.9436840Z Removing intermediate container e7667861d293
2017-11-09T20:25:03.9458600Z Step 3/7 : ARG source
2017-11-09T20:25:04.0562980Z ---> Running in 7b57f9fc515f
2017-11-09T20:25:04.2292420Z ---> 4d07188edf18
2017-11-09T20:25:04.2503550Z Removing intermediate container 7b57f9fc515f
2017-11-09T20:25:04.2581270Z Step 4/7 : WORKDIR /app
2017-11-09T20:25:04.5347440Z ---> 4546b8dc771f
2017-11-09T20:25:04.5527690Z Removing intermediate container 0a595d10e668
2017-11-09T20:25:04.5558170Z Step 5/7 : EXPOSE 80
2017-11-09T20:25:04.6765650Z ---> Running in aa273cc9ca2c
2017-11-09T20:25:04.8330020Z ---> 6dd874db617e
2017-11-09T20:25:04.8619290Z Removing intermediate container aa273cc9ca2c
2017-11-09T20:25:04.8650100Z Step 6/7 : COPY ${source:-obj/Docker/publish} .
2017-11-09T20:25:04.8686070Z Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
2017-11-09T20:25:04.9064960Z ##[error]db uses an image, skipping
2017-11-09T20:25:04.9127660Z ##[error]Building api
2017-11-09T20:25:04.9175860Z ##[error]Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
2017-11-09T20:25:04.9258880Z ##[error]/usr/local/bin/docker-compose failed with return code: 1
2017-11-09T20:25:04.9325830Z ##[section]Finishing: Create images
Here's my docker-compose:
version: '3'
services:
api:
image: api
container_name: api
build:
context: ./Api
dockerfile: Dockerfile
ports:
- "8000:80"
depends_on:
- db
db:
image: "microsoft/mssql-server-linux"
container_name: mssql
environment:
SA_PASSWORD: "testtest3030!"
ACCEPT_EULA: "Y"
MSSQL_PID: "Developer"
ports:
- "127.0.0.1:8001:1433"
and docker-compose.ci.build.yml file:
version: '3'
services:
ci-build:
image: microsoft/aspnetcore-build:1.0-1.1
volumes:
- .:/src
working_dir: /src
command: /bin/bash -c "dotnet restore ./Travelingowe.sln && dotnet publish ./Travelingowe.sln -c Release -o ./obj/Docker/publish"
Also I attached my VSTS docker-compose taks:
Do you have any idea for I'm doing wrong?
Aha, I faced the same issue a few days ago. Your issue is inside your DockerFile. For example, let's look at my file:
FROM microsoft/aspnetcore:2.0
ARG source
WORKDIR /app
EXPOSE 80
COPY ${source:-obj/Docker/publish} .
ENTRYPOINT [ "dotnet","MyWeb.Api.dll" ]
If docker cannot find the path that you specify in your COPY command, it will produce an error:
Service 'api' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder211771018/obj/Docker/publish: no such file or directory
So, basically, what you need to do is to fix your build release path (you need to be sure that files after the build in the right folder).
Also, there is a problem, if you execute:
docker-compose build
It doesn't build the solution. That is why DockerFile cannot find the needed folder. If you execute the following command before your docker-compose, it will work.
dotnet publish ./YourSolution.sln -c Release -o ./obj/Docker/publish
Regarding the error, the reason is that there isn’t the obj/Docker/publish folder or file. (check the answer of Leonid)
The docker-compose.ci.build.yml file is for building the project and the docker-compose.yml file is used for running the project, so you need to build the project through docker-compose.ci.build.yml file with Dokcer-compose task, then build/run image with docker-compose.yml file with Docker-compose task.
There is a thread about docker-compose.ci.build.yml, docker-compose.yml and Dockerfile: Asp.net core with linux docker container
Update:
Refer to these steps to deploy project to Azure:
Create a new build definition
Add Docker Compose task (Docker Compose File: **/docker-compose.ci.build.yml; Action: Run a Docker Compose command; Command:run ci-build)
Add Copy Files task (Source Folder: $(System.DefaultWorkingDirectory)/[your project path]/obj/Docker/publish; Contents: **; Target Folder: $(Build.ArtifactStagingDirectory))
Add Publish Build Artifacts task (Path to publish: $(Build.ArtifactStagingDirectory))
Create a new release definitioin
Add Azure App Service Deploy task (Package or folder: $(System.DefaultWorkingDirectory)/[artifact build definition name]/[artifact name]; Uncheck Publish using Web Deploy option)

Docker with node bcrypt — invalid ELF header

I've tried every solution from this post and this post
I'm not finding a solution to get rid of the following error when running docker-compose up:
module.js:598
return process.dlopen(module, path._makeLong(filename));
^
Error: /code/node_modules/bcrypt/lib/binding/bcrypt_lib.node: invalid ELF header
Here's my latest attempt docker-compose.yml
version: "2"
services:
app:
build: ./client
ports:
- "3000:3000"
links:
- auth
volumes:
- ./client:/code
auth:
build: ./auth-service
ports:
- "3002:3002"
links:
- db
volumes:
- ./auth-service:/code
db:
...
And my auth service Dockerfile:
FROM node:7.7.1
EXPOSE 3002
WORKDIR /code
COPY package.json /code
RUN npm install
COPY . /code
CMD npm start
After trying each of the solution from the above two links, I rebuild the containers and it always results in the same error.
Also worth noting, the service runs fine locally, when I don't use docker.
How do I get docker to work with bcrypt?
Update
I was able to get it working by doing the following:
finding the id of the container: docker ps
accessing the container: docker exec -t -i containerId /bin/bash
installing bcrypt: npm install bcrypt
This isn't ideal for portability
I spent a few hours trying to solve this and in the end I came up with the following solution.
My compose file looks like this.....
version: "3"
services:
server:
build:
context: ./server
volumes:
- ./server:/usr/src/app
- /usr/src/app/node_modules/
ports:
- 3050:3050
depends_on:
- db
command: ["nodemon", "./bin/www"]
The second volume mount there is the important one as this gets around the local node_module issue.
Just for reference my dockerfile is like this:
FROM node
RUN npm install -g nodemon
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app
RUN npm install
EXPOSE 3050
CMD ["nodemon", "./bin/www"]
was struggling with this one few times, the ".dockerignore" solution wont work if you use volumes sadly, since its only related to the "Copy" command and only when you build the container.
the only solution i found which i think makes the most sense, is to volume only what you need, what does it mean - divide your source code and the "configurations" files (such as package.json):
- src
-- morecode
-- morecode2
-- server.js
- index.js
- package.json
- jslint.json
- tsconfig.json
- .env
- .dockerignore
- ... etc
and put the volume only on the "src" folder, that way your builds will also be much faster plus your node modules will be built and installed on the correct operation system, d'ont forget to add .dockerignore to node_modules to prevent builds form taking unnecessary longer time
do note that doing so will require re-build of the application every time your adding new package, but if you use npm best practice and you divide the npm installation in your docker file to be cached, it will be faster
The reason this error occurs is that the node module bcrypt is first compiled on your original machine (specific for your OS) and when an image is built on docker it cannot run since the OS is no longer the same. solution create a .dockerignore file in your root folder and add node_modules to this file.

Docker compose not finding my index.js

I'm following this guide and use my low docker knowledge to get a dev environment up and running. I've hit a wall I cannot solve. This is my docker-compose.yml:
version: '2'
services:
redis:
image: redis:3.2
mongo:
image: mongo:3.2
app:
build: .
ports:
- '3000:3000'
command: './node_modules/.bin/nodemon ./index.js'
environment:
NODE_ENV: development
volumes:
- .:/home/app/cardcreator
- /home/app/cardcreator/node_modules
depends_on:
- redis
- mongo
links:
- redis
- mongo
and this is my Dockerfile:
FROM node:6.3.1
RUN useradd --user-group --create-home --shell /bin/false app
ENV HOME=/home/app
COPY package.json npm-shrinkwrap.json $HOME/cardcreator/
RUN chown -R app:app $HOME/*
USER app
WORKDIR $HOME/cardcreator
RUN npm install
USER root
COPY . $HOME/cardcreator/
RUN chown -R app:app $HOME/*
USER app
CMD ["node", "index.js"]
When I try to start the app via docker-compose up, I get the error
app_1 | Usage: nodemon [nodemon options] [script.js] [args]
app_1 | See "nodemon --help" for more.
I then removed the command line of my docker-compose.yml, only leaving node index.js to start. I get an error saying index.js cannot be found.
The file is in my project folder, it is there and it has content. I can't figure out why this setup doesn't work, I did similar setups for tails and it worked fine.
Can anyone tell me what I'm doing wrong here?
Whatever you are mounting in your compose file here:
- .:/home/app/cardcreator
Is going to mount on top of whatever you built in $HOME/cardcreator/ in your Dockerfile.
So basically you seem to have conflicting volumes -- it's an order of operations issue -- the build is going to happen first and the volume mount happens later when the container runs, so your container will no longer have access to the files built in the Dockerfile.
You could try to use
docker exec -it app_1 bash
to go into the container, trying to execute the
node index.js
command manually and see what's going on. Not 100% sure if the 'node' docker images have bash installed though..

Resources