I'm trying to provision a project locally that's using NodeJs with NPM.
I'm running npm install on my host machine (MacBook Pro Retina, 15-inch, Mid 2015) using nvm with node version 10.19:
added 2335 packages from 985 contributors and audited 916010 packages in 61.736s
When I run the same setup in Docker, the result is much slower. This is my docker-compose.yml file:
version: '3.4'
services:
node:
image: node:10.19-alpine
container_name: node
volumes:
- .:/app/
- npm-cache:/root/.npm
working_dir: /app
command: ["tail", "-f", "/dev/null"]
volumes:
npm-cache:
external: false
Then I execute:
docker-compose up -d node; docker exec -t node npm install
And the result is:
added 2265 packages from 975 contributors and audited 916010 packages in 259.895s
(I'm assuming the number of resulting packages is different due to a different platform).
I thought the speedy installation was achieved by having a local cache (that's why there is an extra volume for caching in the docker-compose) but then I ran:
$ npm cache clean --force && rm -rf ~/.npm && rm -rf node_modules
and the result for installation on the host machine is still consistently ~60 seconds.
When it comes to resources allocated to the Docker VM, it shouldn't be a problem, here's my Docker VM configuration:
I don't know where else to look, any help would be greatly appreciated.
Thanks
This slowdown is caused by sharing files between a container and your host machine.
In order to cope with it, you can give a try to docker-sync.
This tool supports different strategies for automatical syncing between a host machine and containers (including rsync).
However, beware that it has own issues like occasional sync freezing.
Here is how I got around the issue.
Create a base docker image using a similar Dockerfile.
FROM node:latest
RUN mkdir -p /node/app
COPY ./package.json /node/app/package.json
WORKDIR "/node/app"
RUN yarn install --network-timeout 100000
Then in your container, make this script the entry point:
#!/bin/bash
mkdir -p /node/app
cp /srv/package.json /node/app
cd /node/app
yarn install
sleep 1
rm -f /srv/node_modules
ln -s /node/app/node_modules /srv/node_modules
cd /srv
sleep 1
yarn serve
This installs npm modules in another directory that is not synced between container and host, and links the directory for the app. This seems to work just fine until this is properly resolved by Docker.
Related
I would like to dockerize my react application and I have one question on doing so. I would like to install node_modules on the containter then have them synced to the host, so that I can run the npm commands on the container not the host machine. I achieved this, but the node_modules folder that is synced to my computer is empty, but is filled in the container. This is an issue since I am getting not installed warnings in the IDE, because the node_modules folder in the host machine is empty.
docker-compose.yml:
version: '3.9'
services:
frontend:
build:
dockerfile: Dockerfile
context: ./frontend
volumes:
- /usr/src/app/node_modules
- ./frontend:/usr/src/app
Dockerfile:
FROM node:18-alpine3.15
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install && \
mkdir -p node_modules/.cache && \
chmod -R 777 node_modules/.cache
COPY ./ ./
CMD npm run start
I would appreciate any tips and/or help.
You can't really "share" node_modules like this because there certain OS-specific steps which happen during installation. Some modules have compilation steps which need to target the host machine. Other modules have bin declarations which are symlinked, and symlinks cannot be "mounted" or shared between a host and container. Even different versions of node cannot share node_modules without rebuilding.
If you are wanting to develop within docker, you have two options:
Editing inside a container with VSCode (maybe other editors do this too?). I've tried this before and it's not very fun and is kind of tedious - it doesn't quite work the way you want.
Edit files on your host machine which are mounted inside docker. Nodemon/webpack/etc will see the changes and rebuild accordingly.
I recommend #2 - I've seen it used at many companies and is a very "standard" way to do development. This does require that you do an npm install on the host machine - don't get bogged down by trying to avoid an extra npm install.
If you want to make installs and builds faster, your best bet is to mount your npm cache directory into your docker container. You will need to find the npm cache location on both your host and your docker container by running npm get cache in both places. You can do this on your docker container by doing:
docker run --rm -it <your_image> npm get cache
You would mount the cache folder like you would any other volume. You can run a separate install in both docker and on your host machine - the files will only be downloaded once, and all compilations and symlinking will happen correctly.
I have a meteor application.This app works well on the Centos7 VM.
I need to create docker container of this app and install or import this container on other virtual machines.
What do ِdocker file need to save and load container on another VM?
NodeJs?
Mongodb?
MeteorJs?
Shouldn't I store Mongodb file in Docker container?
this is my docker file:
# Pull base image.
FROM node:8.11.4
# Install build tools to compile native npm modules
RUN npm install -g node-gyp
RUN apt-get install curl -y
RUN curl https://install.meteor.com/ | sh
# Create app directory
RUN mkdir -p /usr/app
COPY . /usr/app
RUN cd /usr/app/programs/server
RUN npm install
WORKDIR /usr/app
CMD ["node", "main.js"]
EXPOSE 3000
There are many ways to skin this cat ... lets assume you have researched the alternatives on how to execute a meteor app using containers by using tools which automates the below setup - meteor calls their version of this automation Galaxy
I suggest you run the meteor commands outside the container intended to run your app from since a meteor install is huge, slow to install and some of the libraries you may pull in, or the libraries your libraries pull in, may need c or c++ compilers so meteor and its friends do not need to get installed into your app container everytime you want to recompile your app ... your app container only needs nodejs and your bundle ... when you execute a meteor app it does not use meteor instead the app is executed using nodejs directly since at this point your code has been compiled into a bundle which is pure nodejs
Yes you would do well to put mongodb into its own container
No, no need to put MeteorJs inside your app container instead just like meteor itself those compile time tools are not needed during execution time so install MeteorJs as well as all other tools needed for a successful meteor build on your host machine which is where you execute your meteor build command
In your above Dockerfile the last statement EXPOSE 3000 will never get reached so put it before your CMD node
So outside your container get meteor installed then issue
cd /your/webapp/src
meteor build --server https://example.com --verbose --directory /webapp --server-only
above will compile your meteor project into a bundle dir living at
ls -la /webapp/bundle/
then copy into that freshly cut bundle dir your Dockerfile etc :
.bashrc
Dockerfile
bundle/
then create your container
docker build --tag localhost:5000/hygge/loudweb-admin --no-cache .
docker push localhost:5000/hygge/loudweb-admin
here is a stripped down Dockerfile
cat Dockerfile
# normal mode - raw ubuntu run has finished and base image exists so run in epoc mode
FROM ubuntu:18.04
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux
ENV NODE_VER=v8.11.4
ENV NODE_NAME=node-${NODE_VER}
ENV OS_ARCH=linux-x64
ENV COMSUFFIX=tar.gz
ENV NODE_PARENT=/${NODE_NAME}-${OS_ARCH}
ENV PATH=${NODE_PARENT}/bin:${PATH}
ENV NODE_PATH=${NODE_PARENT}/lib/node_modules
RUN apt-get update && apt-get install -y wget && \
wget -q https://nodejs.org/download/release/${NODE_VER}/${NODE_NAME}-${OS_ARCH}.${COMSUFFIX} && \
tar -xf ${NODE_NAME}-${OS_ARCH}.${COMSUFFIX}
ENV MONGO_URL='mongodb://$MONGO_SERVICE_HOST:$MONGO_SERVICE_PORT/meteor'
ENV ROOT_URL=https://example.com
ENV PORT 3000
EXPOSE 3000
RUN which node
WORKDIR /tmp
# CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf" ]
# I strongly suggest you wrap below using supervisord
CMD ["node", "main.js"]
to launch your container issue
docker-compose -f /devopsmicro/docker-compose.yml pull loudmail loud-devops nodejs-enduser
docker-compose -f /devopsmicro/docker-compose.yml up -d
here is a stripped down docker compose yaml file
version: '3'
services:
nodejs-enduser:
image: ${GKE_APP_IMAGE_ENDUSER}
container_name: loud_enduser
restart: always
depends_on:
- nodejs-admin
- loudmongo
- loudmail
volumes:
- /cryptdata6/var/log/loudlog-enduser:/loudlog-enduser
- ${TMPDIR_GRAND_PARENT}/curr/loud-build/${PROJECT_ID}/webapp/enduser/bundle:/tmp
environment:
- MONGO_SERVICE_HOST=loudmongo
- MONGO_SERVICE_PORT=$GKE_MONGO_PORT
- MONGO_URL=mongodb://loudmongo:$GKE_MONGO_PORT/test
- METEOR_SETTINGS=${METEOR_SETTINGS}
- MAIL_URL=smtp://support#${GKE_DOMAIN_NAME}:blah#loudmail:587/
links:
- loudmongo
- loudmail
ports:
- 127.0.0.1:3000:3000
working_dir: /tmp
command: /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf
Once you have your app executing using containers you can work to stop using ubuntu as your container base and use a smaller, simpler docker base image like nodejs, busybox, etc however using ubuntu is easier initially since it has ability to let you install packages from inside a running container which is nice during development
the machinations surrounding above are vast ... above is a quick copy N paste plucked from the devops side of the house with hundreds of helper binaries + scripts, config templates, tls certs ... this is a tiny glimpse into the world of getting an app to execute
#Scott Stensland answer is good, in that it explains how to manually create a docker container for Meteor.
There is a simpler way use Meteor-up (mup) http://meteor-up.com/
EASILY DEPLOY YOUR APP
Meteor Up is a production quality Meteor app deployment tool.
Install with one command:
$ npm install --global mup
You set up a simple config file, and it looks after creating the container, doing npm install, setting up ssl certs etc. Much less work than doing it by hand
Everytime I change a file in the nodejs app I have to rebuild the docker image.
This feels redundant and slows my workflow. Is there a proper way to sync the nodejs app files without rebuilding the whole image again, or is this a normal usage?
It sounds like you want to speed up the development process. In that case I would recommend to mount your directory in your container using the docker run -v option: https://docs.docker.com/engine/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume
Once you are done developing your program build the image and now start docker without the -v option.
What I ended up doing was:
1) Using volumes with the docker run command - so I could change the code without rebuilding the docker image every time.
2) I had an issue with node_modules being overwritten because a volume acts like a mount - fixed it with node's PATH traversal.
Dockerfile:
FROM node:5.2
# Create our app directories
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
RUN npm install -g nodemon
# This will cache npm install
# And presist the node_modules
# Even after we are using the volume (overwrites)
COPY package.json /usr/src/
RUN cd /usr/src && npm install
#Expose node's port
EXPOSE 3000
# Run the app
CMD nodemon server.js
Command-line:
to build:
docker build -t web-image
to run:
docker run --rm -v $(pwd):/usr/src/app -p 3000:3000 --name web web-image
You could have also done something like change the instruction and it says look in the directory specified by the build context argument of docker build and find the package.json file and then copy that into the current working directory of the container and then RUN npm install and afterwards we will COPY over everything else like so:
# Specify base image
FROM node:alpine
WORKDIR /usr/app
# Install some dependencies
COPY ./package.json ./
RUN npm install
# Setup default command
CMD ["npm", "start"]
You can make as many changes as you want and it will not invalidate the cache for any of these steps here.
The only time that npm install will be executed again is if we make a change to that step or any step above it.
So unless you make a change to the package.json file, the npm install will not be executed again.
So we can test this by running the docker build -t <tagname>/<project-name> .
Now I have made a change to the Dockerfile so you will see some steps re run and eventually our successfully tagged and built image.
Docker detected the change to the step and every step after it, but not the npm install step.
The lesson here is that yes it does make a difference the order in which all these instructions are placed in a Dockerfile.
Its nice to segment out these operations to ensure you are only copying the bare minimum.
I'm really struggling to figure out where i should put my grunt build step when building my docker image and deploying to dockerhub.
My workflow at the moment is as follows:
Push branch to github
CircleCI installs all dependencies, builds project, and runs tests on branch.
Merge branch branch to staging branch
CircleCI installs all dependencies, builds project, and runs tests on branch.
If tests pass, package the built files into the docker image with the source and also run npm install --production. CircleCI then deploys this staging image to dockerhub
Tutum is linked to dockerhub and deploys my image to DigitalOcean whenever a new image is pushed.
I do the same workflow as above, when merging to master, and a production image is created instead.
It feels a bit weird that i'm created 2 separate docker images. Is this standard practice?
I've seen quite a lot of people including the grunt/gulp build step in their dockerfiles, but that doesn't feel right either as all the devDependencies, and bower_components will then be in the image along with the built code.
What's the best practice for running build steps and building docker images? Is it better to have CI do it, or dockerhub do it from the dockerfile? I'm also after the most efficient way to create my docker image for staging and production.
Below is my circleCI.yml file, followed by my Dockerfile.
circle.yml:
machine:
node:
version: 4.2.1
# Set the timezeone - any value from /usr/share/zoneinfo/ is valid here
timezone:
Europe/London
services:
- docker
pre:
- sudo curl -L -o /usr/bin/docker 'http://s3-external-1.amazonaws.com/circle-downloads/docker-1.8.2-circleci'; sudo chmod 0755 /usr/bin/docker; true
dependencies:
pre:
- docker --version
- sudo pip install -U docker-compose==1.4.2
- sudo pip install tutum
override:
- npm install:
pwd: node
post:
- npm run bower_install:
pwd: node
- npm run grunt_build:
pwd: node
test:
override:
- cd node && npm run test
deployment:
staging:
branch: staging
commands:
- docker-compose -f docker-compose.production.yml build node
# - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS
- tutum login -u $DOCKER_USER -p $DOCKER_PASS -e $DOCKER_EMAIL
- docker tag dh_node:latest tutum.co/${DOCKER_USER}/dh_stage:latest
- docker push tutum.co/${DOCKER_USER}/dh_stage:latest
master:
branch: master
commands:
- docker-compose -f docker-compose.production.yml build node
# - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS
- tutum login -u $DOCKER_USER -p $DOCKER_PASS -e $DOCKER_EMAIL
- docker tag dh_node:latest tutum.co/${DOCKER_USER}/dh_prod:latest
- docker push tutum.co/${DOCKER_USER}/dh_prod:latest
Dockerfile:
FROM node:4.2
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app/
RUN npm install --production
COPY . /usr/src/app
#
#
# Commented the following steps out, as these
# now run on CircleCI before the image is built.
# (Whether that's right, or not, i'm not sure.)
#
# Install bower
# RUN npm install -g bower # grunt-cli
#
# WORKDIR src/app
# RUN bower install --allow-root
#
# Expose port
EXPOSE 3000
# Run app using nodemon
CMD ["npm", "start"]
What's the best practice for running build steps and building docker images? Is it better to have CI do it, or dockerhub do it from the dockerfile?
It's better to run the build steps themselves outside of docker. Thus the same steps work for local development, non-docker deployment, etc. Keep your coupling to docker itself loose when you can. Thus build your artifacts with regular build tools and scripts and simply ADD built files to your docker image via your Dockerfile.
It feels a bit weird that i'm created 2 separate docker images. Is this standard practice?
I would recommend instead using exactly the image you have already built and tested on stage in production. Once you rebuild the image, you become vulnerable to discrepancies breaking your production image even though your stage image worked OK. At this point neither docker nor npm can deliver strictly reproducible builds across time, thus once it's built and tested gold, it's gold and goes to production bit-for-bit identical.
Your circle ci should download all the dependencies and then create docker image from that downloaded packages. All testing is passed with the specified dependencies and should be carry forwarded to production. Once. The image is pushed to docker hub with all dependencies and tumtum will deploy the same to your production and as the dependencies are already downloaded it will take seconds to create containers.
Answering to your second query of building the same image. I would suggest to deploy the same image to production. This will guarantee you that what worked great on staging is also working the same on production.
I am having some trouble mounting a directory on my machine into my Docker container. I would like to mount a directory containing files necessary to run a node server. So far, I have successfully been able to run and access my server in browser using the Dockerfile below:
# Use an ubuntu base image
FROM ubuntu:14.04
# Install Node.js and npm (this will install the latest version for ubuntu)
RUN apt-get update
RUN apt-get -y install curl
RUN curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
RUN apt-get -y install nodejs
RUN apt-get -y install git
# Bundle app source (note: all of my configuration files/folders are in the current directory along with the Dockerfile)
COPY . /src
Install app dependencies
#WORKDIR /src
RUN npm install
RUN npm install -g bower
RUN bower install --allow-root
RUN npm install -g grunt
RUN npm install -g grunt-cli
#What port to expose?
EXPOSE 1234
#Run grunt on container start
CMD grunt
And these commands:
docker build -t test/test .
docker run -p 1234:1234 -d test/test
However, I figured that I would like the configuration files and whatnot to persist, and thought to do this by mounting the directory (with the files and Dockerfile) as a volume. I used other solutions on StackOverflow to get this command:
docker run -p 1234:1234 -v //c/Users/username/directory:/src -d test/test
My node server seems to start up fine (no errors in the log), but it takes significantly longer to do so, and when I try to access my webpage in browser I just get a blank page.
Am I doing something incorrectly?
EDIT: I have gotten my server to run--seems to have been a weird error in my configuration. However, it still takes a long time (around a minute or two) for my server to start when I mount a volume from my host machine. Does anyone have some insight as to why this is?