Bumping package.json version without invalidating docker cache - node.js

Question

I'm using a pretty standard Dockerfile to containerize a Node.js application:
# Simplified version
FROM node:alpine
# Copy package.json first for docker build's layer caching
COPY package.json package-lock.json foo/
RUN npm install
COPY src/ foo/
RUN npm run build
Breaking up my COPY into two parts was advantageous because it allowed Docker to cache the (long) npm install step.
Recently, however, I started bumping my package.json version using semver. This had the side effect of invalidating the Docker cache for the npm install step, lengthening my build times significantly.
Is there an alternative caching strategy I can use so that npm install only runs when my dependencies change?

Answer 1

Here's my take on this, based off other answers, but shorter and with usage of jq:
Dockerfile:
FROM endeveit/docker-jq AS deps
# https://stackoverflow.com/a/58487433
# To prevent cache invalidation from changes in fields other than dependencies
COPY package.json /tmp
RUN jq '{ dependencies, devDependencies }' < /tmp/package.json > /tmp/deps.json
FROM node:12-alpine
WORKDIR /app
COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .
RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description
COPY . .
RUN npm run build
LABEL maintainer="Alexey Vishnyakov <n3tn0de#gmail.com>"
I extract dependencies and devDependencies fields to a separate file, then on next build step I copy it from the previous step as package.json (COPY --from=deps /tmp/deps.json ./package.json).
After RUN npm ci, COPY . . will overwrite gutted package.json with the original one (you can test it by adding RUN cat package.json after COPY . . command.
Note that npm-scripts commands like postinstall won't run since they're not present in the file during npm ci and also if npm ci is running from root and without --unsafe-perm
Either run commands after COPY . . or/and (if needed) include them via jq (changing command will invalidate cache layer) or add --unsafe-perm
Dockerfile:
FROM endeveit/docker-jq AS deps
COPY package.json /tmp
RUN jq '{ dependencies, devDependencies, peerDependencies, scripts: (.scripts | { postinstall }) }' < /tmp/package.json > /tmp/deps.json
# keep postinstall script
FROM node:12-alpine
WORKDIR /app
COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .
# RUN npm ci --unsafe-perm
# allow postinstall to run from root (security risk)
RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description
RUN npm run postinstall
...

Answer 2

You can add an additional "preparation" step in your Dockerfile that creates a temporary package.json where the "version" field is fixed. This file is then used while installing dependencies and afterwards replaced by the "real" package.json.
As all of this happens during the Docker build process, your actual source repository is not touched (so you can use the environment variable npm_package_version both during your build and when running the docker script, e.g. to tag) and the solution is portable:
Dockerfile:
# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json ./
# Create temporary package.json where version is set to 0.0.0
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN ["node", "-e", "\
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));\
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));\
fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));\
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));\
"]
# BUILD
FROM node:lts-alpine as build
# Install deps, using temporary package.json from preparation step
COPY --from=preparation package.json package-lock.json ./
RUN npm ci
# Copy source files (including "real" package.json) and build app
COPY . .
RUN npm run build
If you think inlining the Node script is iffy (I like it, because this way the entire Docker build process can be found in the Dockerfile), you can of course extract it to a separate JS file:
create-tmp-pkg.js:
const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));
fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));
and change your preparation step to:
# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json create-tmp-pkg.js ./
# Create temporary package.json where version is set to "0.0.0"
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN node create-tmp-pkg.js

Answer 3

I spent some time thinking about this. Fundamentally, I'm cheating because the package.json file is, in fact, changed, which means anything that circumvents the cache invalidation technically makes the build not reproducible.
For my purposes, however, I care more about build time than strict cache correctness. Here's what I came up with:
build-artifacts.js
/*
Used to keep docker cache fresh despite package.json version bumps.
In this script
- copy package.json to package-artifact.json
- zero package.json version
In Docker
- copy package.json
- run npm install normal
- copy package-artifact.json to package.json (undo-build-artifacts.js accomplishes this with a conditional check that package-artifact exists)
*/
const fs = require('fs');
const package = fs.readFileSync('package.json', 'utf8');
fs.writeFileSync('package-artifact.json', package);
const modifiedPackage = { ...JSON.parse(package), version: '0.0.0' };
fs.writeFileSync('package.json', JSON.stringify(modifiedPackage));
const packageLock = fs.readFileSync('package-lock.json', 'utf8');
fs.writeFileSync('package-lock-artifact.json', packageLock);
const modifiedPackageLock = { ...JSON.parse(packageLock), version: '0.0.0' };
fs.writeFileSync('package-lock.json', JSON.stringify(modifiedPackageLock));
undo-build-artifacts.js
const fs = require('fs');
const hasBuildArtifacts = fs.existsSync('package-artifact.json');
if (hasBuildArtifacts) {
const package = fs.readFileSync('package-artifact.json', 'utf8');
const packageLock = fs.readFileSync('package-lock-artifact.json', 'utf8');
fs.writeFileSync('package.json', package);
fs.writeFileSync('package-lock.json', packageLock);
fs.unlinkSync('package-artifact.json');
fs.unlinkSync('package-lock-artifact.json');
}
These two files serve to relocate package.json and package-lock.json, replacing them with artifacts that have zeroed-out versions. These artifacts will be used in the docker build, and will be replaced with the original versions upon npm install completion.
I run build-artifacts.js in a Travis CI before_script, and undo-build-artifacts.js in the Dockerfile itself (after I npm install). undo-build-artifacts.js incorporates a check for the build artifacts, meaning the Docker container can still build if build-artifacts.js hasn't run. That keeps the container portable enough in my books. :)

Answer 4

I went about this a bit different. I just ignore the version in package.json and leave it set to 1.0.0. Instead I add a file version.json then I use a script like the one below for deploying.
This approach won't work if you need to publish to npm, since the version will never change
version.json
{"version":"1.2.3"}
deploy.sh
#!/bin/sh
VERSION=`node -p "require('./version.json').version"`
#docker build
docker pull node:10
docker build . -t mycompany/myapp:v$VERSION
#commit version tag
git add version.json
git commit -m "version $VERSION"
git tag v$VERSION
git push origin
git push origin v$VERSION
#push Docker image to repo
docker push mycompany/myapp:v$VERSION
I normally just update the version file manually but if you want something that works like npm version you can use a script like this that uses the semvar package.
patch.js
var semver = require('semver')
var fs = require('fs')
var version = require('./version.json').version
var patch = semver.inc(version, 'patch')
fs.writeFile('./version.json', JSON.stringify({'version': patch}), (err) => {
if (err) {
console.error(err)
} else {
console.log(version + ' -> ' + patch)
}
})

Answer 5

Based on n3tn0de answer
I changed the Dockerfile to be
######## Preperation
FROM node:12-alpine AS deps
COPY package.json package-lock.json ./
RUN npm version --allow-same-version 1.0.0
######## Building
FROM node:12-alpine
WORKDIR /app
COPY --from=deps package.json package-lock.json ./
RUN npm ci
COPY . .
EXPOSE 80
CMD ["npm", "start"]
This approach will avoid using 2 different Docker images -less download and less storage- and fix/avoid any issues in package.json

Answer 6

Another option pnpm now has pnpm fetch which only uses the lockfile so you are free to make other changes to package.json
This requires switching from npm/yarn to using pnpm
Example from: https://pnpm.io/cli/fetch
FROM node:14
RUN curl -f https://get.pnpm.io/v6.16.js | node - add --global pnpm
# pnpm fetch does require only lockfile
COPY pnpm-lock.yaml ./
RUN pnpm fetch --prod
ADD . ./
RUN pnpm install -r --offline --prod
EXPOSE 8080
CMD [ "node", "server.js" ]

Answer 7

Patching the version can be done without jq, using basic sed:
FROM alpine AS temp
COPY package.json /tmp
RUN sed -e 's/"version": "[0-9]\+\.[0-9]\+\.[0-9]\+",/"version": "0.0.0",/'
< /tmp/package.json > /tmp/package-v0.json
FROM node:14.5.0-alpine
....
COPY --from=temp /tmp/package-v0.json package.json
...
The sed regex assumes that the version value follows the semver scheme (e.g. 1.23.456)
The other assumption is that the "version": "xx.xx.xx," string is not found elsewhere in the file. The "," at the end of the pattern can help to lower the probability of "false positives". Check it before with your package.json file by security of course.

Answer 8

Steps:
remove version from package json
install packages for production
Copy to production image
Benefits:
Can freely patch package json without invalidating docker
If dependencies were not changed will not do unecessary npm install for production (packages don't change that frequently)
in practice:
# prepare package
FROM node:14-alpine AS package
COPY ./package.json ./package-lock.json ./
RUN node -e "['./package.json','./package-lock.json'].forEach(n => { \
let p = require(n); \
p.version = '0.0.0'; \
fs.writeFileSync(n, JSON.stringify(p)); \
});"
# install deps
FROM node:14-alpine AS build
COPY --from=package package*.json ./
RUN npm ci --only=production
# production
FROM node:14-alpine
...
COPY . .
COPY --from=build ./node_modules ./node_modules
...