Docker build takes long time for nodejs application - node.js

I am experiencing long build times for nodejs applications when building image with docker build command.
There is one big hang that takes couple of minutes
08:03:15 Step 1/11 : FROM node:14.1.0-alpine AS build
08:03:15 ---> 0854fcfc1637
08:03:15 Step 2/11 : COPY server/package*.json /nodejs/server/
08:03:15 ---> Using cache
08:03:15 ---> 4996283ff991
08:03:15 Step 3/11 : WORKDIR /nodejs/server
08:03:15 ---> Using cache
08:03:15 ---> 93e5b63fa81d
08:03:15 Step 4/11 : RUN npm ci
08:03:15 ---> Using cache
08:03:15 ---> 2c825e02ea01
08:03:15 Step 5/11 : COPY server ./
08:03:15 ---> Using cache
08:03:15 ---> 69c024cde79f
08:03:15 Step 6/11 : WORKDIR /nodejs
08:03:15 ---> Using cache
08:03:15 ---> 49d7f8bd9514
08:03:15 Step 7/11 : COPY package*.json ./
08:03:16 ---> e82bee625c3e
08:03:16 Step 8/11 : RUN npm ci
08:03:16 ---> Running in ecfd57702906
...
08:03:49 added 1483 packages in 26.419s
08:09:40 Removing intermediate container ecfd57702906
...
08:09:40 ---> 7c6b67d85b0b
08:09:40 Step 9/11 : COPY *.json ./
08:09:43 ---> 0165efd1c97d
08:09:43 Step 10/11 : COPY src ./src/
08:09:51 ---> 42e54cee6b91
08:09:51 Step 11/11 : RUN npm run build:prod
08:09:51 ---> Running in af6f9b013d27
This does not happen when building Java images.
My Dockerfile
FROM node:14.1.0-alpine AS build
COPY server/package*.json /nodejs/server/
WORKDIR /nodejs/server
RUN npm ci
COPY server ./
WORKDIR /nodejs
COPY package*.json ./
RUN npm ci
COPY *.json ./
COPY src ./src/
RUN npm run build:prod
...
I tried using buildkit but it has the same behaviour
08:37:20 #17 exporting to image
08:37:20 #17 exporting layers
08:50:12 #17 exporting layers 766.8s done
I also added node_modules to .dockerignore file but with no change.
Docker version 19.03.6, build 369ce74a3c
What could be the problem?
What is happening between "added 1483 packages..." and "Removing intermediate container"?

I have found the cause of the problem and wanted to share if someone encounters similar issue.
The long "export layers" time was caused by node_modules folder that was present in one of the layers. Removing it solved the problem but it's really a workaround.
I merged npm ci and npm run build... steps together and removed node_modules in the same step.
RUN npm ci && npm run build:prod && rm -rf node_modules
So the final build stage in Dockerfile looks like this
FROM node:14.1.0-alpine AS build
COPY server/package*.json /nodejs/server/
WORKDIR /nodejs/server
RUN npm ci
COPY server ./
WORKDIR /nodejs
COPY package*.json ./
COPY *.json ./
COPY src ./src/
RUN npm ci && npm run build:prod && rm -rf node_modules
...
As I said it's just a workaround and I think the root cause is that Docker is having problems with exporting layers with a lot of small files such as the ones in node_modules.

Related

Docker build fails for NestJs service

I have created a nestJS rest api. below is the docker file that I'm using to build the service.
FROM node:alpine3.16 AS development
WORKDIR /usr/src/app
COPY --chown=node:node package*.json ./
RUN npm ci
COPY --chown=node:node . .
RUN npm run build
ENV NODE_ENV production
RUN npm ci --only=production --omit=dev && npm cache clean --force
FROM node:alpine3.16 AS production
WORKDIR /app
COPY --chown=node:node --from=build /usr/src/node_modules ./node_modules
COPY --chown=node:node --from=build /usr/src/app/dist ./dist
USER node
# start the server using the production build
CMD [ "node", "dist/main.js" ]
On running the docker build this is the output that I receive. as you can see it fails to copy node_modules folder. I'm not sure if that is the real problem or something else is wrong in the dockerfile
Sending build context to Docker daemon 320.4MB
Step 1/16 : FROM node:alpine3.16 AS development
---> fbf8faa0b327
Step 2/16 : WORKDIR /usr/src/app
---> Using cache
---> 9586fc2253da
Step 3/16 : COPY --chown=node:node package*.json ./
---> Using cache
---> 170ae630fdff
Step 4/16 : RUN npm ci
---> Using cache
---> 1fd94efbfebb
Step 5/16 : COPY --chown=node:node . .
---> 0b8a645d4548
Step 6/16 : USER root
---> Running in e7882157586e
Removing intermediate container e7882157586e
---> b3ecda575e47
Step 7/16 : RUN npm run build
---> Running in 020d3ecc7f2a
> service#0.0.1 prebuild
> rimraf dist
> service#0.0.1 build
> nest build
webpack 5.72.1 compiled successfully in 6222 ms
Removing intermediate container 020d3ecc7f2a
---> abe9a1016a92
Step 8/16 : ENV NODE_ENV production
---> Running in eda90aa82fd5
Removing intermediate container eda90aa82fd5
---> 69070cf4854e
Step 9/16 : RUN npm ci --only=production --omit=dev && npm cache clean --force
---> Running in bcbceed9f0b0
npm WARN config only Use `--omit=dev` to omit dev dependencies from the install.
npm WARN deprecated multer#1.4.4: Multer 1.x is affected by CVE-2022-24434. This is fixed in v1.4.4-lts.1 which drops support for versions of Node.js before 6. Please upgrade to at least Node.js 6 and version 1.4.4-lts.1 of Multer. If you need support for older versions of Node.js, we are open to accepting patches that would fix the CVE on the main 1.x release line, whilst maintaining compatibility with Node.js 0.10.
added 260 packages, and audited 261 packages in 19s
23 packages are looking for funding
run `npm fund` for details
6 high severity vulnerabilities
To address all issues (including breaking changes), run:
npm audit fix --force
Run `npm audit` for details.
npm WARN using --force Recommended protections disabled.
Removing intermediate container bcbceed9f0b0
---> f42e890665f0
Step 10/16 : FROM node:alpine3.16 AS production
---> fbf8faa0b327
Step 11/16 : WORKDIR /app
---> Using cache
---> bcd15d86cb7e
Step 12/16 : USER root
---> Running in 9d97f5cc54dd
Removing intermediate container 9d97f5cc54dd
---> cd1f956a6c91
Step 13/16 : COPY --chown=node:node --from=build /usr/src/app/node_modules ./node_modules
invalid from flag value build: pull access denied for build, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
What is causing the problem and how can I resolve it?
There is no build stage in your docker steps. You probably mean --from=development instead of --from=build

Why is "npm install" really slow and doesn't terminate in docker build?

I want to build a node js dockerfile. npm install doesn't do anything after printing this:
Sending build context to Docker daemon 14.5MB
Step 1/7 : FROM node:lts-alpine3.13
---> 186a89db3aeb
Step 2/7 : EXPOSE 3000
---> Using cache
---> 353c5796d929
Step 3/7 : RUN mkdir -p /usr/src/app
---> Using cache
---> 2515a1ce4ac9
Step 4/7 : WORKDIR /usr/src/app
---> Using cache
---> 47926cacc786
Step 5/7 : COPY package*.json /usr/src/app/
---> Using cache
---> c79428f26810
Step 6/7 : RUN npm install
---> Running in b3d3d97650b5
> sharp#0.26.3 install /usr/src/app/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.10.0/libvips-8.10.0-linuxmusl-x64.tar.br
I've been waiting for hours. But it hasn't downloaded https://github.com/lovell/sharp-libvips/releases/download/v8.10.0/libvips-8.10.0-linuxmusl-x64.tar.br. (which is only 7.1 MB!)
What can I do?
Thanks in advance.
UPDATE
I can download the file manually using an node:lts-alpine3.13 image without any problem:
docker run -ti --rm --entrypoint /bin/sh node:lts-alpine3.13
# apk add --no-cache wget
# wget https://github.com/lovell/sharp-libvips/releases/download/v8.10.0/libvips-8.10.0-linuxmusl-x64.tar.br

Cannot run npm install in docker on Raspberry Pi

I am trying to build a dockerfile on my Raspberry Pi 4, running arm64 Ubuntu 20.04. The same dockerfile runs fine on my main computer. The dockerfile is as follows:
FROM node:12 as builder
WORKDIR /app
COPY .client/package.json .
COPY .package-lock.json .
RUN npm install
...
Upon building this file I get the following error:
Sending build context to Docker daemon 29.35MB
Step 1/12 : FROM node:12 as builder
---> 70b24a2d7b95
Step 2/12 : WORKDIR /app
---> Using cache
---> 2ab48332f35b
Step 3/12 : COPY .client/package.json .
---> Using cache
---> d7009f4cfa0b
Step 4/12 : COPY .client/package-lock.json .
---> Using cache
---> 59d4f21519ed
Step 5/12 : RUN npm install
---> Running in 84f62d84dcbe
Aborted (core dumped)
The command '/bin/sh -c npm install' returned a non-zero code: 134
The error code of 134 refers to running out of memory on the host, but I have checked both memory and storage and there seems to be plenty of space left. What could be the issue? Thank you!

Docker build does not use cache from npm ci

I have created my Dockerfile with attention to use the docker cache system. I'm adding the package.json and package-lock.json files into a clean container, and then run npm ci command. I expect it to be used from the cache if the package.json and package-lock.json were not changed, but it keeps running.
Am I missing something? Is there something wrong with my Dockerfile?
FROM node:13.10.1-stretch as base
ADD package.json /app/package.json
ADD package-lock.json /app/package-lock.json
WORKDIR /app
RUN npm ci --unsafe-perm
The output I get:
Step 1/12 : FROM node:13.10.1-stretch as base
---> 7aef30ae6655
Step 2/12 : ADD package.json /app/package.json
---> ce655a3453f2
Step 3/12 : ADD package-lock.json /app/package-lock.json
---> 797cda1e10b2
Step 4/12 : WORKDIR /app
Removing intermediate container 5b6929b80ad6
---> 80c2aac903c5
Step 5/12 : RUN npm ci --unsafe-perm
---> Running in 7732a8aca146
> fsevents#1.2.12 install /app/node_modules/webpack-dev-server/node_modules/fsevents
> node-gyp rebuild
make: Entering directory '/app/node_modules/webpack-dev-server/node_modules/fsevents/build'
SOLINK_MODULE(target) Release/obj.target/.node
COPY Release/.node
make: Leaving directory '/app/node_modules/webpack-dev-server/node_modules/fsevents/build'
[...]
Assuming that you don't run other commands before package.json was added, your Dockerfile is fine and the layer adding your package file is cached. For example, we build a simple Dockerfile that just adds your package.json config file:
FROM node:13.10.1-stretch as base
ADD package.json /app/package.json
First run:
$ docker build -t so-example .
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM node:13.10.1-stretch as base
13.10.1-stretch: Pulling from library/node
[...]
Status: Downloaded newer image for node:13.10.1-stretch
---> 7aef30ae6655
Step 2/2 : ADD package.json /app/package.json
---> a7bb80c06ecb
Successfully built a7bb80c06ecb
Successfully tagged so-example:latest
Second run
$ docker build -t so-example .
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM node:13.10.1-stretch as base
---> 7aef30ae6655
Step 2/2 : ADD package.json /app/package.json
---> Using cache
---> a7bb80c06ecb
Successfully built a7bb80c06ecb
Successfully tagged so-example:latest
As you can see, the caching works. Could you please verify this with such a minium example? Most of the time caching breaks because of a suboptimal ordner. Please check the following:
Execute the build command twice. Caching could only work after the first run
Make sure, that no other steps you may havent posted here were executed in your Dockerfile that invalidates the cache
Are there any cleanup commands running? Something like docker prune or docker image prune (or manual list/delete images on older versions) would delete your image
Check/post your calls how you build the image

npm ERR! code SELF_SIGNED_CERT_IN_CHAIN AWS Ubuntu - Corporate Firewall

Had to move our AWS server from open internet access to behind a firewall. I set up the new server from scratch. I have a script to build and start a docker image. It works great on the original server. However, firewalled server, no luck.
I've tried "npm set strict-ssl false" and "npm install npm -g --ca=null" + "npm config set ca="""
I've read as much documentation I can find. Nothing has worked and I'm now stuck... Here is the output from the firewalled server.
$ docker build . -t lab-server
Sending build context to Docker daemon 23.68 MB
Step 1 : FROM node:latest
---> 14989a19b2c6
Step 2 : RUN apt-get update -y
---> Using cache
---> bf64a1065712
Step 3 : RUN mkdir -p /usr/src/app
---> Using cache
---> 785f66509c34
Step 4 : WORKDIR /usr/src/app
---> Using cache
---> 621dfc200257
Step 5 : COPY package.json tsconfig.json /usr/src/app/
---> Using cache
---> 1dc06475df7f
Step 6 : RUN npm install
---> Running in 7cf8b7b036d6
npm ERR! code SELF_SIGNED_CERT_IN_CHAIN
npm ERR! errno SELF_SIGNED_CERT_IN_CHAIN
npm ERR! request to https://registry.npmjs.org/body-parser failed, reason: self signed certificate in certificate chain
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2017-10-25T23_34_34_267Z-debug.log
The command '/bin/sh -c npm install' returned a non-zero code: 1
Output from the original non-firewalled server.
Sending build context to Docker daemon 24.2 MB
Step 1 : FROM node:latest
---> ae884087f2da
Step 2 : RUN apt-get update -y
---> Using cache
---> 09c7c4a2471c
Step 3 : RUN mkdir -p /usr/src/app
---> Using cache
---> 67ed0cc97232
Step 4 : WORKDIR /usr/src/app
---> Using cache
---> 144e8824e6c1
Step 5 : COPY package.json tsconfig.json /usr/src/app/
---> Using cache
---> 2fe8ebeed67d
Step 6 : RUN npm install
---> Using cache
---> c69c300495f1
Step 7 : RUN npm install -g typescript ts-node
---> Using cache
---> 7c6c6a52eb39
Step 8 : COPY ./src /usr/src/app/src
---> Using cache
---> 90e5adf015dd
Step 9 : EXPOSE 3001
---> Using cache
---> f0689a44228c
Step 10 : CMD npm start
---> Using cache
---> 112ff47fc584
Successfully built 112ff47fc584
f84120db5167603fa2df12503b0ac331c65daf325d25c71203ca3bfb99dc0d1a
Server Started

Resources