A very strange "JavaScript heap out of memory" issue - node.js

Question

I'm attempting to build a Next.js project on CircleCI using an ARM-based machine (specifically, arm.xlarge) which has 32GB of RAM. However, it seems to very quickly exit with an error about lack of memory. I've tried to increase the memory of the Node process all the way up to that of the machine, but with no luck - I receive the same error.
What's particularly strange is that the GCs have a tiny amount of memory utilization (3.5MB), and that it exits in sub-second time!
This build does work, if I swap out the arm resource type for an x86_64 resource type, and build a Docker image for the x86_64 platform instead. Obviously this doesn't cover my needs, but verifies the build works.
What might cause this very strange behaviour? What else can I do to further debug this issue?
What I've tried
Changing the Node version, ranging from 14 to 18.8.0
Changing the machine resource size
Changing the options of Docker, including adding a --memory flag to the extra-build-args field of the build job
Removing the yarn.lock
Purging the Node cache
Deleting the node_modules folder
Relevant resources:
Here's the trace:
#16 [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
#16 sha256:da0777711cb35943386ef4812f2df1ff4932fb5bee717be330a217104e5240c7
#16 0.178
#16 0.178 <--- Last few GCs --->
#16 0.178
#16 0.178 [1:0x570e320] 100 ms: Mark-sweep 1.2 (3.5) -> 1.2 (3.5) MB, 2.2 / 0.0 ms (average mu = 0.818, current mu = 0.007) allocation failure scavenge might not succeed
#16 0.178 [1:0x570e320] 102 ms: Mark-sweep (reduce) 1.2 (3.5) -> 1.2 (3.5) MB, 2.3 / 0.0 ms (average mu = 0.704, current mu = 0.006) last resort GC in old space requested
#16 0.178 [1:0x570e320] 104 ms: Mark-sweep (reduce) 1.2 (2.5) -> 1.1 (3.5) MB, 2.3 / 0.0 ms (average mu = 0.549, current mu = 0.010) last resort GC in old space requested
#16 0.178
#16 0.178
#16 0.178 <--- JS stacktrace --->
#16 0.178
#16 0.178 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
#16 ERROR: process "/bin/sh -c NODE_OPTIONS=\"--max_old_space_size=8192\" yarn build" did not complete successfully: exit code: 132
------
> [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build:
------
Dockerfile:32
--------------------
31 |
32 | >>> RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
34 |
--------------------
error: failed to solve: rpc error: code = Unknown desc = process "/bin/sh -c NODE_OPTIONS=\"--max_old_space_size=8192\" CI_BUILD=${CI_BUILD} DEPLOYMENT_VERSION=${DEPLOYMENT_VERSION} yarn build" did not complete successfully: exit code: 132
Here's the relevant parts of the Dockerfile:
FROM node:14.18.0-alpine
WORKDIR /app
RUN apk update
RUN apk add python3
RUN apk add git gnupg jq g++ make py3-pip
COPY package.json yarn.lock ./
RUN yarn
RUN yarn add pm2#5.2.0 --global
COPY . .
RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
EXPOSE 3001
CMD [ "npx", ... ]
and here's the job from CircleCI config.yml:
build_and_push_image:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.xlarge
steps:
- checkout
- docker/pull:
images: 'node:16.14.2'
- aws-ecr/build-and-push-image:
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
aws-cli-version: latest
create-repo: true
skip-when-tags-exist: true
dockerfile: Dockerfile
extra-build-args: >
--build-arg GPG_ENCRYPT_PASSPHRASE
--build-arg BUILD_STAGE
--build-arg CI_BUILD=true
platform: linux/arm64/v7
push-image: true
registry-id: AWS_ACCOUNT_ID
region: $AWS_REGION
repo: $PROJECT_NAME
tag: $version_tag

Answer 1

After a good few hours of debugging with trial and error, I fixed with the following changes to config.yml:
build_and_push_image:
machine:
image: ubuntu-2004:202101-01
resource_class: arm.xlarge
steps:
- checkout
- docker/pull:
images: 'node:16.14.2'
- aws-ecr/build-and-push-image:
aws-access-key-id: AWS_ACCESS_KEY_ID
aws-secret-access-key: AWS_SECRET_ACCESS_KEY
aws-cli-version: latest
create-repo: true
skip-when-tags-exist: true
dockerfile: Dockerfile
extra-build-args: >
--build-arg GPG_ENCRYPT_PASSPHRASE
--build-arg BUILD_STAGE
--build-arg CI_BUILD=true
- platform: linux/arm64/v7
+ platform: linux/arm64
push-image: true
registry-id: AWS_ACCOUNT_ID
region: $AWS_REGION
repo: $PROJECT_NAME
tag: $version_tag
I'm not sure why this works, but here's my best thoughts for others who are blocked by this in the future:
Some miconfiguration of the docker buildx command from the CircleCI orb. I'm also noticing that a lot of people are having problems with Docker image caching using the orb, and maybe there's some issue with the command itself (see https://github.com/CircleCI-Public/aws-ecr-orb/issues/202). It looks something like this:
docker context create builder
# This here is interesting - it is using qemu to emulate for builds
# See https://github.com/tonistiigi/binfmt
docker run --privileged --rm tonistiigi/binfmt --install all
docker --context builder buildx create --use
docker --context builder buildx build \
-f "${PARAM_PATH}"/"${PARAM_DOCKERFILE}" \
${docker_tag_args} \
--platform "${PARAM_PLATFORM}" \
--progress plain \
"$#" \
"${PARAM_PATH}"
https://github.com/docker/buildx/issues/1036 missing qemu dependencies
https://github.com/moby/buildkit/issues/1946#issuecomment-767374876 libseccomp blocking unknown (new) syscalls

Answer 2

Have you checked the versions of OS and Node? The Versions should match. (x64=x64)