I am using Docker, so this case might look weird. But I want my whole /data directory to be mounted inside my docker container when developing.
My /data folder container my package.json file, an app directory and a bunch of other stuff.
The problem is that I want my node_modules folder to NOT be persistent, only the package.json file.
I have tried a couple of things, but package.json and npm is giving me a hard time here...
Mounting the package.json file directly will break npm. npm tries to rename the file on save, which is not possible when its a mounted file.
Mounting the parent folder (/data) will mount the node_modules folder.
I cant find any configuration option to put node_modules in another folder outside /data, example /dist
Putting package.json in /data/conf mounting the /data/conf as a volume instead wont work. I cant find any way to specify the package.json path in npmrc.
Putting package.json in /data/conf and symlinking it to /data/package.json wont work. npm breaks the symlink and replaces it with a file.
Copying data back and forth to/from inside the docker container is how I am doing it now.. A little tedious.. I also want a clean solution..
As you have already answered, I think that might be the only solution right now.
When you are building your Docker image, do something like:
COPY data/package.json /data/
RUN mkdir /dist/node_modules && ln -s /dist/node_modules /data/node_modules && cd /data && npm install
And for other stuff (like bower, do the same thing)
COPY data/.bowerrc /data/
COPY data/bower.json /data/
RUN mkdir /dist/vendor && ln -s /dist/vendor /data/vendor && cd /data && bower install --allow-root
And COPY data/ /data at the end (so you are able to use Dockers caching and not having to do npm/docker installation when there is a change to data.
You will also need to create the symlinks you need and store them in your git-repo. They will be invalid on the outside, but will happely work on the inside of your container.
Using this solution, you are able to mount your $PWD/data:/data without getting the npm/bower "junk" outside your container. And you will still be able to build your image as a standalone deployment of your service..
A similar and alternative way is to use NODE_ENV variable instead of creating a symlink.
RUN mkdir -p /dist/node_modules
RUN cp -r node_modules/* /dist/node_modules/
ENV NODE_PATH /dist/node_modules
Here you first create a new directory for node_modules, copy all modules there, and have Node read the modules from there.
I've been having this problem for some time now, and the accepted solution didn't work for me*
I found this link, which had an edit pointing here and this indeed worked for me:
volumes:
- ./:/data
- /data/node_modules
In this case the Engine creates a volume (see Compose reference on volumes) which is not mounted to your source directory. This was the easiest solution and didn't require me to do any symlinking, setting paths, etc.
For reference, my simple Dockerfile just looks like this:
# install node requirements
WORKDIR /data
COPY ./package.json ./package.json
RUN npm install -qq
# add source code
COPY ./ ./
# run watch script
CMD npm run watch
(The watch script is just webpack --watch -d)
Hope this is able to help someone and save hours of time like it did for me!
'*' = I couldn't get webpack to work from my package.json scripts and installing anything while inside the container created the node_modules folder with whatever I just installed (I run npm i --save [packages] from inside the container to get the package update the package.json until the next rebuild)
The solution I went with was placing the node_modules folder in /dist/node_modules, and making a symlink to it from /data/node_modules. I can do this both in my Dockerfile so it will use it when building, and I can submit my symlinks to my git-repo. Everything worked out nicely..
Maybe you can save your container, and then rebuild it regularly with a minimal dockerfile
FROM my_container
and a .dockerignore file containing
/data/node_modules
See the doc
http://docs.docker.com/reference/builder/#the-dockerignore-file
Related
I'm interested in running a NodeJS script inside a Docker container, because that seems to be the easiest way to run stuff in unRAID (small scripts at least).
My current Dockerfile looks like this:
FROM node:12.9.1
COPY app.js /home/node/
COPY package*.json /home/node/
RUN mkdir /home/node/saves
WORKDIR /home/node
RUN npm install
CMD ["node", "app.js"]
Not perfect, but works well enough. What my script does, is that it scrapes certain websites for data, then places it in a folder called saves.
But because I do COPY app.js /home/node/, every time I want to make a tiny change to my app.js file, I need to rebuild the whole image, delete the container, and start a new one. Kind of irritating, but it has worked for me.. for now.
When I start my container, I want that volume to stay persistent, so I do this:
docker run --net=bridge -h scraper --name scraper -d -v /mnt/user/scripts/scraper/saves:/home/node/saves scraper
This works, but as I said, if I want to change my app.js (like add a new site to scrape), I have to rebuild the image and run the above command again. Every single time.
What's a better approach than this? I could solve this by not copying the files, but instead run npm install and then node app.js every time, but this script runs every 3 minutes, so that would be a huge waste of resources.
I could also store the appropriate data inside my /saves/ folder, then read that in the NodeJS script every time, but I feel like that's kind of a hack.
Use Environment variables.
If the type of changes to app.js file already known, you could
use environment variables to supply such changes to docker run, and
an entrypoint script file to make such modification using
environment variable.
The content of entrypoint script will depend on what you want to do.
# Add to your Dockerfile
ENV SITE_TO_SCRAP=example.com
COPY docker-entrypoint.sh /
RUN chmod +x /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
# Run docker with
-e SITE_TO_SCRAP=abc.com
Entry Point script may looks like
#!/bin/bash
# Modify app.js
# Assuming changing the line SCRAP_URL="someurl"
sed -i -e "s#SCRAP_URL=\"someurl\"#SCRAP_URL=\"${SITE_TO_SCRAP}\"#" /home/node/app.js
# This will execute the CMD
exec "$#"
Make node cache folder persistent
To run npm install do it in entry point script not in docker file. make node.js cache folder persistent by mounting -v host/path/to/cache:/root/.npm. this way node install use cached files whenever possible. use docker run --rm node:{version} npm config get cache to get container cache directory.
Manually mount modified files.
# To your docker run command add required file/directory mount(s)
-v /path/to/modified/app.js:/home/node/app.js
I'm trying to build a nodejs container for my project which requires a local module. On my package.json i got a relative link to a folder above, since there is where the local module is located. Everything seems to work correctly except that inside the container, the local module is added as symlink to the host machine (windows).
This behavior only happens when i build using the dockerfile, if i do npm install manually inside the container, the module is copied into the node_module as expected.
package.json entry:
"app-lib": "file:../app_lib"
docker file:
FROM node:8.9-alpine
ENV NODE_ENV production
WORKDIR /usr/src/app
COPY ["./Provider/package.json", "./Provider/package-lock.json*", "./Provider/npm-shrinkwrap.json*", "./"]
COPY ["./app_lib/package.json", "./app_lib/package-lock.json*", "./app_lib/npm-shrinkwrap.json*", "../app_lib/"]
RUN cd ../app_lib && npm install
COPY ./app_lib .
RUN cd ../app && npm install
COPY ./Provider .
EXPOSE 3001
Annoying symlink:
app-lib -> E:\work\app_server\app_lib\
Anyone got any suggestion on how to make it work right on build or why might be the underlying cause?
Make sure you have node_modules in .dockerignore, else COPY ./app_lib . will overwrite the same and you will get the behaviour you see
I have an angularjs application, I'm running using docker.
The docker file looks like this:-
FROM node:6.2.2
RUN npm install --global gulp-cli && \
npm install --global bower
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app/
COPY bower.json /usr/src/app/
RUN npm install && \
bower install -F --allow-root --config.interactive=false
COPY . /usr/src/app
ENV GULP_COMMAND serve:dist
ENTRYPOINT ["sh", "-c"]
CMD ["gulp $GULP_COMMAND"]
Now when I make any changes in say any html file, It doesn't dynamically loads up on the web page. I have to stop the container, remove it, build the image again, remove the earlier image and then restart the container from new image. Do I have to do this every time? (I'm new to docker, and I guess this issue is coz my source code is not put into volume, but I don't know how to do it using docker file)
You are correct, you should use volumes for stuff like this. During development, give it the same volumes as the COPY directories. It'll override it with whatever is on your machine, no need to rebuild the image, or even restart the container. Perfect for development.
When actually baking your images for production, you remove the volumes, leave the COPY in, and you'll get a deterministic container. I would recommend you read through this article here: https://docs.docker.com/storage/volumes/.
In general, there are 3 ways to do volumes.
Define them in your dockerfile using VOLUME.
Personally, I've never done this. I don't really see the benefits of this against the other two methods. I believe it would be more common to do this when your volume is meant to act as a permanent data-store. Not so much when you're just trying to use your live dev environment.
Define them when calling docker run.
docker run ... -v $(pwd)/src:/usr/src/app ...
This is great, cause if your COPY in your dockerfile is ./src /usr/src/app then it temporarily overrides the directory while running the image, but it's still there for deployment when you don't use -v.
Use docker-compose.
My personal recommendation. Docker compose massively simplifies running containers. For sake of simplicity just calls docker run ... but automates the arguments based on a given docker-compose.yml config.
Create a dev service specifying the volumes you want to mount, other containers you want it linked to, etc. Then bring it up using docker-compose up ... or docker-compose run ... depending on what you need.
Smart use of volumes will DRAMATICALLY reduce your development cycle. Would really recommend looking into them.
Yes, you need to rebuild every time the files change, since you only modify the files that are outside of the container. In order to apply the changes to the files IN the container, you need to rebuild the container.
Depending on the use case, you could either make the Docker Container dynamically load the files from another repository, or you could mount an external volume to use in the container, but there are some pitfalls associated with either solution.
If you want to keep your container running as you add your files you could also use a variation.
Mount a volume to any other location e.g. /usr/src/staging.
While the container is running, if you need to copy new files into the container, copy them into the location of the mounted volume.
Run docker exec -it <container-name> bash to open a bash shell inside the running container.
Run a cp /usr/src/staging/* /usr/src/app command to copy all new files into the target folder.
Kind of a longshot, but has anyone had any problems using bcrypt in a linux container (specifically docker) and know of an automated workaround? I have the same issue as these two:
Invalid ELF header with node bcrypt on AWSBox
bcrypt invalid elf header when running node app
My Dockerfile
# Pull base image
FROM node:0.12
# Expose port 8080
EXPOSE 8080
# Add current directory into path /data in image
ADD . /data
# Set working directory to /data
WORKDIR /data
# Install dependencies from package.json
RUN npm install --production
# Run index.js
CMD ["npm", "start"]
I get the previously mentioned invalid ELF header error if I have bcrypt already installed in my node_modules, but if I remove it (either just itself or all my packages), it isn't installed for some reason when I build the container. I have to manually enter the container after the build and install it inside.
Is there an automated workaround?
Or maybe, just, what would be a good alternative to bcrypt with a Node stack?
Liam's comment is on the money, just expanding on it for future travellers on the internets.
The issue is that you've copied your node_modules folder into your container. The reason that this is a problem is that bcrypt is a native module. It's not just javascript, but also a bunch of C code that gets compiled at the time of installation.
The binaries that come out of that compilation get stored in the node_modules folder and they're customised to the place they were built. Transplanting them out of their OSX home into a strange Linux land causes them to misbehave and complain about ELF headers and fairy feet.
The solution is to echo node_modules >> .dockerignore and run npm install as part of your Dockerfile. This means that the native modules will be compiled inside the container rather than outside it on your laptop.
With this in place, there is no need to run npm install before your start CMD. Just having it in the build phase of the Dockerfile is fine.
protip: the official node images set NODE_ENV=production by default, which npm treats the same as the --production flag. Most of the time this is a good thing. It is not a good thing when your Dockerfile also contains some build steps that rely on dev dependencies (webpack, etc). In that case you want NODE_ENV=null npm install
pro protip: you can take better advantage of Docker's caching by copying in your package.json separately to the rest of your code. Make your Dockerfile look like this:
# Pull base image
FROM node:0.12
# Expose port 8080
EXPOSE 8080
# Set working directory to /data
WORKDIR /data
# Set working directory to /data
COPY package.json /data
# Install dependencies from package.json
RUN npm install
# Add current directory into path /data in image
ADD . /data
# Run index.js
CMD npm start
And that way Docker will only re-run npm install when you change your package.json, not every time you change a line of code.
Okay, so I have a working automated workaround:
Call npm install --production in the CMD instruction. I'm going to wave my hands at figuring out why I have to install bcrypt at the time of executing the container, but it works.
Updated Dockerfile
# Pull base image
FROM node:0.12
# Expose port 8080
EXPOSE 8080
# Add current directory into path /data in image
ADD . /data
# Set working directory to /data
WORKDIR /data
# Install dependencies from package.json
RUN npm install --production
# Run index.js
CMD npm install --production; npm start
Add this command before RUN npm install in your Dockerfile
RUN apk --no-cache add --virtual builds-deps build-base python3
It worked for me. Maybe it will work for you :)
I am now working on a nodejs project. There are too many small files in the node_modules folder. I want to move a docker-based development environment, so that the node_modules folder can be kept in the docker image (update when necessary). At the same time, I need the source folder of the app remain on the hosting environment. The following is the Docker file, I hoped to work:
FROM node
MAINTAINER MrCoder
// To simplify the process node_modules is installed and cached
ADD package.json /opt/app/
WORKDIR /opt/app
RUN npm install
// will be mapped to the local app source folder
VOLUME /opt/app
// This is to test whether local node_modules or the folder in the image is used
CMD ls node_modules
Apparently, when I change add any file in the local node_modules folder it is listed.
No, you can't skip some files when mounting a volume. This is a general Linux (and every other system I know of) limitation, not Docker-related.
However, it's pretty easy to work around it: have a root-level index.js (or whatever) that does nothing but require('./src');. Then have /opt/app be provided by your container and mount /opt/app/src as an external volume.
You could potentially use a bind mount i.e.
mkdir -p /node_modules
mount --bind /node_modules /usr/src/app/node_modules
after the volume declaration but before the npm install.