I have created my Dockerfile with attention to use the docker cache system. I'm adding the package.json and package-lock.json files into a clean container, and then run npm ci command. I expect it to be used from the cache if the package.json and package-lock.json were not changed, but it keeps running.
Am I missing something? Is there something wrong with my Dockerfile?
FROM node:13.10.1-stretch as base
ADD package.json /app/package.json
ADD package-lock.json /app/package-lock.json
WORKDIR /app
RUN npm ci --unsafe-perm
The output I get:
Step 1/12 : FROM node:13.10.1-stretch as base
---> 7aef30ae6655
Step 2/12 : ADD package.json /app/package.json
---> ce655a3453f2
Step 3/12 : ADD package-lock.json /app/package-lock.json
---> 797cda1e10b2
Step 4/12 : WORKDIR /app
Removing intermediate container 5b6929b80ad6
---> 80c2aac903c5
Step 5/12 : RUN npm ci --unsafe-perm
---> Running in 7732a8aca146
> fsevents#1.2.12 install /app/node_modules/webpack-dev-server/node_modules/fsevents
> node-gyp rebuild
make: Entering directory '/app/node_modules/webpack-dev-server/node_modules/fsevents/build'
SOLINK_MODULE(target) Release/obj.target/.node
COPY Release/.node
make: Leaving directory '/app/node_modules/webpack-dev-server/node_modules/fsevents/build'
[...]
Assuming that you don't run other commands before package.json was added, your Dockerfile is fine and the layer adding your package file is cached. For example, we build a simple Dockerfile that just adds your package.json config file:
FROM node:13.10.1-stretch as base
ADD package.json /app/package.json
First run:
$ docker build -t so-example .
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM node:13.10.1-stretch as base
13.10.1-stretch: Pulling from library/node
[...]
Status: Downloaded newer image for node:13.10.1-stretch
---> 7aef30ae6655
Step 2/2 : ADD package.json /app/package.json
---> a7bb80c06ecb
Successfully built a7bb80c06ecb
Successfully tagged so-example:latest
Second run
$ docker build -t so-example .
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM node:13.10.1-stretch as base
---> 7aef30ae6655
Step 2/2 : ADD package.json /app/package.json
---> Using cache
---> a7bb80c06ecb
Successfully built a7bb80c06ecb
Successfully tagged so-example:latest
As you can see, the caching works. Could you please verify this with such a minium example? Most of the time caching breaks because of a suboptimal ordner. Please check the following:
Execute the build command twice. Caching could only work after the first run
Make sure, that no other steps you may havent posted here were executed in your Dockerfile that invalidates the cache
Are there any cleanup commands running? Something like docker prune or docker image prune (or manual list/delete images on older versions) would delete your image
Check/post your calls how you build the image
Related
I have created a nestJS rest api. below is the docker file that I'm using to build the service.
FROM node:alpine3.16 AS development
WORKDIR /usr/src/app
COPY --chown=node:node package*.json ./
RUN npm ci
COPY --chown=node:node . .
RUN npm run build
ENV NODE_ENV production
RUN npm ci --only=production --omit=dev && npm cache clean --force
FROM node:alpine3.16 AS production
WORKDIR /app
COPY --chown=node:node --from=build /usr/src/node_modules ./node_modules
COPY --chown=node:node --from=build /usr/src/app/dist ./dist
USER node
# start the server using the production build
CMD [ "node", "dist/main.js" ]
On running the docker build this is the output that I receive. as you can see it fails to copy node_modules folder. I'm not sure if that is the real problem or something else is wrong in the dockerfile
Sending build context to Docker daemon 320.4MB
Step 1/16 : FROM node:alpine3.16 AS development
---> fbf8faa0b327
Step 2/16 : WORKDIR /usr/src/app
---> Using cache
---> 9586fc2253da
Step 3/16 : COPY --chown=node:node package*.json ./
---> Using cache
---> 170ae630fdff
Step 4/16 : RUN npm ci
---> Using cache
---> 1fd94efbfebb
Step 5/16 : COPY --chown=node:node . .
---> 0b8a645d4548
Step 6/16 : USER root
---> Running in e7882157586e
Removing intermediate container e7882157586e
---> b3ecda575e47
Step 7/16 : RUN npm run build
---> Running in 020d3ecc7f2a
> service#0.0.1 prebuild
> rimraf dist
> service#0.0.1 build
> nest build
webpack 5.72.1 compiled successfully in 6222 ms
Removing intermediate container 020d3ecc7f2a
---> abe9a1016a92
Step 8/16 : ENV NODE_ENV production
---> Running in eda90aa82fd5
Removing intermediate container eda90aa82fd5
---> 69070cf4854e
Step 9/16 : RUN npm ci --only=production --omit=dev && npm cache clean --force
---> Running in bcbceed9f0b0
npm WARN config only Use `--omit=dev` to omit dev dependencies from the install.
npm WARN deprecated multer#1.4.4: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10.
added 260 packages, and audited 261 packages in 19s
23 packages are looking for funding
run `npm fund` for details
6 high severity vulnerabilities
To address all issues (including breaking changes), run:
npm audit fix --force
Run `npm audit` for details.
npm WARN using --force Recommended protections disabled.
Removing intermediate container bcbceed9f0b0
---> f42e890665f0
Step 10/16 : FROM node:alpine3.16 AS production
---> fbf8faa0b327
Step 11/16 : WORKDIR /app
---> Using cache
---> bcd15d86cb7e
Step 12/16 : USER root
---> Running in 9d97f5cc54dd
Removing intermediate container 9d97f5cc54dd
---> cd1f956a6c91
Step 13/16 : COPY --chown=node:node --from=build /usr/src/app/node_modules ./node_modules
invalid from flag value build: pull access denied for build, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
What is causing the problem and how can I resolve it?
There is no build stage in your docker steps. You probably mean --from=development instead of --from=build
I am trying to move my tests to my docker image build stage but it seems to ignore my test build at all and just skip it when I build the image.
What can be the problem?
# Base build
FROM node:16.13 AS build
RUN mkdir /app
WORKDIR /app
ADD package*.json ./
RUN npm i -g npm#6.14.15
RUN npm i --production
COPY . .
RUN npm run build
# Clean
RUN npm prune --production
RUN npm install -g node-prune
RUN node-prune
# Test
FROM build AS test
RUN npm i -g npm#6.14.15
RUN npm i -D jest typescript
RUN npm i -D ts-jest #types/jest
RUN npm i -D #shelf/jest-mongodb
RUN npx jest
# Release
FROM node:16.13-alpine
RUN mkdir /app
WORKDIR /app
COPY --from=build /app/dist ./dist/
COPY --from=build /app/node_modules ./node_modules/
COPY --from=build /app/package.json ./
CMD npm start
Above you can see my Dockerfile where I am preparing the build, then plan to have tests and after that, I am making my release image.
I've already been playing around with that for hours; I cleared cache, made tweaks with the order in the file) but it didn't help. It keeps ignoring my test build.
Any hint of that and in general on my Dockerfile is welcomed
Docker internally has two different systems to build images. Newer Dockers default to a newer system called BuildKit. One of the major differences is the way the two systems handle multi-stage builds: the older system just runs all of the build stages until the end, but BuildKit figures out that the final stage COPY --from=build but doesn't use anything from the test stage, and just skips it.
I wouldn't run these tests in a Dockerfile (since it doesn't produce a runnable artifact) or in Docker at all. I'd probably run them in my host-based development environment, before building a Docker image:
# On the host
npm install # devDependencies include all test dependencies
npm run tests # locally
# repeat until the tests pass (without involving a container)
# Build the image _after_ the tests pass
docker build -t myapp .
docker build also has a --target option to name a specific stage so you could tell Docker to "build the test image", but you'd just immediately delete it. This is where I don't think Docker quite makes sense as a test runner.
docker build -t delete-me --target test
docker build -t my-app .
docker rmi delete-me
Your Dockerfile also installs a MongoDB test helper. If the test depends on a live database rather than a mock implementation, the other thing to be aware of here is that code in a Dockerfile can never connect to other containers. You have to run this test from somewhere else in this case.
I try to build an image for a client app (nextjs app), but the build keeps failing.
This is the docker file:
FROM node:12.18.3
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/
COPY package-lock.json /app/
RUN npm install
COPY . /app
RUN npm build
# start app
CMD [ "npm", "start" ]
It fails on the first step with this error:
Step 1/9 : FROM node:12.18.3
operating system is not supported
I followed this post https://stackoverflow.com/a/51071057/9608006 , changed the experimental settings to true, and it did pass the failing step.
but now it fails on the npm i step
npm notice
The command '/bin/sh -c npm install' returned a non-zero code: 4294967295: failed to shutdown container: container c425947f7f17ed39ed51ac0a67231f78ba7239ad199c7df979b3b442969a0a57 encountered an error during hcsshim::System::waitBackground: failure in a Windows system call: The virtual machine or container with the specified identifier is not running. (0xc0370110): subsequent terminate failed container c425947f7f17ed39ed51ac0a67231f78ba7239ad199c7df979b3b442969a0a57 encountered an error during hcsshim::System::waitBackground: failure in a Windows system call: The virtual machine or container with the specified identifier is not running. (0xc0370110)
I also get this warning in the start of this step:
Step 6/9 : RUN npm install
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (windows/amd64) and no specific platform was requested
I use windows 10,
docker v20.10.5
What is the issue ?
EDIT 1 - Folder structure
the following is the base folders layer of the client app
.next
.vercel
components
enums
hooks
node_modules
pages
pubilc
store
styles
utils
.dockerIgnore
.env.local
next.config.js
package.json
server.js
You are trying to build Linux based image under Windows.
It seems there is a problem in multiarch images of nodejs with tags version 12.
Try the answer under the post that you have tried:
Click on the docker icon in the tray and switch into Linux containers.
https://stackoverflow.com/a/57548944/3040844
If you are using docker desktop.. just change docker desktop option for windows containers bydefault to linux containers and run your dockerfile again.
I think that the problem related to your base image , I used this Dockerfile for nextjs app in my side and it's working correctly :
# Dockerfile
# base image
FROM node:alpine
# create & set working directory
RUN mkdir -p /app
WORKDIR /app
# copy source files
COPY . /app
# install dependencies
RUN npm install
# start app
RUN npm run build
EXPOSE 3000
CMD npm run start
I hope that can help you to resolve your issue .
According to your dockerfile
FROM node:12.18.3
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/
COPY package-lock.json /app/
RUN npm install
COPY . /app
RUN npm build
# start app
CMD [ "npm", "start" ]
You missed the right image FROM node:12.18.3
Correct way to do this FROM node:alpine3.12 or FROM ubuntu:18.04
FROM: FROM directive is probably the most crucial amongst all others for Dockerfiles. It defines the base image to use to start the build process. It can be any image, including the ones you have created previously. If a FROM image is not found on the host, Docker will try to find it (and download) from the Docker Hub or other container repository. It needs to be the first command declared inside a Dockerfile
Simplest Dockerfile with Node Image
FROM node:alpine3.12
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY ..
RUN npm run build
EXPOSE 3000
CMD npm run start
I am experiencing long build times for nodejs applications when building image with docker build command.
There is one big hang that takes couple of minutes
08:03:15 Step 1/11 : FROM node:14.1.0-alpine AS build
08:03:15 ---> 0854fcfc1637
08:03:15 Step 2/11 : COPY server/package*.json /nodejs/server/
08:03:15 ---> Using cache
08:03:15 ---> 4996283ff991
08:03:15 Step 3/11 : WORKDIR /nodejs/server
08:03:15 ---> Using cache
08:03:15 ---> 93e5b63fa81d
08:03:15 Step 4/11 : RUN npm ci
08:03:15 ---> Using cache
08:03:15 ---> 2c825e02ea01
08:03:15 Step 5/11 : COPY server ./
08:03:15 ---> Using cache
08:03:15 ---> 69c024cde79f
08:03:15 Step 6/11 : WORKDIR /nodejs
08:03:15 ---> Using cache
08:03:15 ---> 49d7f8bd9514
08:03:15 Step 7/11 : COPY package*.json ./
08:03:16 ---> e82bee625c3e
08:03:16 Step 8/11 : RUN npm ci
08:03:16 ---> Running in ecfd57702906
...
08:03:49 added 1483 packages in 26.419s
08:09:40 Removing intermediate container ecfd57702906
...
08:09:40 ---> 7c6b67d85b0b
08:09:40 Step 9/11 : COPY *.json ./
08:09:43 ---> 0165efd1c97d
08:09:43 Step 10/11 : COPY src ./src/
08:09:51 ---> 42e54cee6b91
08:09:51 Step 11/11 : RUN npm run build:prod
08:09:51 ---> Running in af6f9b013d27
This does not happen when building Java images.
My Dockerfile
FROM node:14.1.0-alpine AS build
COPY server/package*.json /nodejs/server/
WORKDIR /nodejs/server
RUN npm ci
COPY server ./
WORKDIR /nodejs
COPY package*.json ./
RUN npm ci
COPY *.json ./
COPY src ./src/
RUN npm run build:prod
...
I tried using buildkit but it has the same behaviour
08:37:20 #17 exporting to image
08:37:20 #17 exporting layers
08:50:12 #17 exporting layers 766.8s done
I also added node_modules to .dockerignore file but with no change.
Docker version 19.03.6, build 369ce74a3c
What could be the problem?
What is happening between "added 1483 packages..." and "Removing intermediate container"?
I have found the cause of the problem and wanted to share if someone encounters similar issue.
The long "export layers" time was caused by node_modules folder that was present in one of the layers. Removing it solved the problem but it's really a workaround.
I merged npm ci and npm run build... steps together and removed node_modules in the same step.
RUN npm ci && npm run build:prod && rm -rf node_modules
So the final build stage in Dockerfile looks like this
FROM node:14.1.0-alpine AS build
COPY server/package*.json /nodejs/server/
WORKDIR /nodejs/server
RUN npm ci
COPY server ./
WORKDIR /nodejs
COPY package*.json ./
COPY *.json ./
COPY src ./src/
RUN npm ci && npm run build:prod && rm -rf node_modules
...
As I said it's just a workaround and I think the root cause is that Docker is having problems with exporting layers with a lot of small files such as the ones in node_modules.
My Dockerfile is as shown below:
From ubuntu:14.04
WORKDIR /app
#COPY package.json /app/package.json
COPY . /app
RUN npm install
EXPOSE 3000
CMD ["npm","start"]
Now, when I run command sudo docker -t my-app .. It gives me the following error:
Sending build context to Docker daemon 453.6 kB Sending build context
to Docker daemon Step 0 : FROM ubuntu:14.04 ---> 37a9c4a8276c Step 1
: WORKDIR /app ---> Using cache ---> a83d4ef27948 Step 2 : COPY .
/app ---> 1029f5d7d8a3 Removing intermediate container eb9e7ea7f7e6
Step 3 : RUN npm install ---> Running in 5d4f2c05d2d8 /bin/sh: 1:
npm: not found INFO[0000] The command [/bin/sh -c npm install]
returned a non-zero code: 127
Is there anything missing in my Dockerfile?
This is expected. Your image doesn't have node installed, since the base image is ubuntu. You should use the node image as a base image.
From node
WORKDIR /app
#COPY package.json /app/package.json
COPY . /app
RUN npm install
EXPOSE 3000
CMD ["npm","start"]
Your Dockerfile is building vanilla ubuntu FROM ubuntu:14.04 so if you want to use npm/node in your container you'll need to setup node yourself by adding RUN commands to install node, following the install instructions for ubuntu.
Instead of this, you probably want to simply use the official node image found at:
https://hub.docker.com/_/node/
FROM node
or use a specific version/distro like
FROM node:8.4.0-wheezy
(other tags/versions/distros are listed on the docker hub page)