I have a Jenkins server on which I observe a private git repository for changes, which then triggers a pipeline script (the repository contains a nodejs app). In this pipeline script I need to do the following steps:
Install dependencies (npm install)
Build my application (npm run build, which creates a dist folder)
Build a docker container (docker build) and run the container (which runs a script in the dist folder)
Which of the following two options would be the recommended way to do this, and why?
Option A: Run npm install and npm run build in the jenkins pipeline and copy the dist folder to the docker container during the docker build. This would allow me to only install runtime dependencies in the docker container using npm install --only=production, therefore reducing the image size significantly.
Option B: Run npm install and npm run build during docker build (In the Dockerfile). This would allow me to run the docker container outside the CI server if I have to (I don't have a use case for it now, but it seems cleaner because it is more independent). However, the image size would significantly increase and I am not sure if this is the recommended way.
Any suggestions?
I would choose option B.
The reason behind it would be that there are some npm packages that runs a node-gyp, gcc, and other platform-dependent builds.
Look at the popular bcrypt package as an example.
Going with option A would mean that your docker and Jenkins machine need to hold the same infra for such builds which is not common, to say the least.
Related
I've been working on a fullstack project that's end-to-end JavaScript, and have a Dockerfile that simplifies to this
FROM node:18-slim AS backend-builder
WORKDIR /backend-staging
COPY ./backend/package.* .
RUN npm install
# bring in other files
FROM node:18-slim AS frontend-builder
WORKDIR /frontend-staging
COPY ./frontend/package.* .
RUN npm install
# bring in other files, run webpack, etc
FROM node:18-slim AS final
WORKDIR /core
COPY ./package.* # root file w/ some non-webpack'd externals and scripts to access after the contianer builds
RUN npm install
COPY --from=backend-builder /backend-staging/build/. .
COPY --from-frontend-builder /frontend-staging/build ./webapp
# setup some postinstall things, set start command, fin
When I run docker build, after pulling the node:18-slim image, I see that buildkit parallelization takes place across my stages, with all 3 npm install commands showing up in the output at the same time. However, it seems that the actual installation goes one stage at a time, with the output first appearing from the final stage and running to completion, then the backend, then the frontend, before the copying and whatnot resumes. It almost seems like there's a mutex on access to the actual npm program, so while the command can be parallelized, the execution is first-come-first-serve single threaded.
I was reading this question, but I seem to have a different problem, as the build stages are definitely being parallelized; their timers all start simultaneously. It's the commands within the build stages that are serialized across stages, for seemingly no reason.
I'm accessing the Docker engine/daemon via Docker Desktop integration w/ WSL2, with buildkit enabled through my Docker Desktop config. The exact command I'm running to build is docker build . -t 'some-image-tag'.
Say I have a node.js application (some web server for the manner of sake). So I have a directory structure with a src which contains my code and a package.json which includes some metadata about my project and all the packages needed to run it. So I run npm install to get all the packages and run node server.js to run the application.
I have a CI pipeline for my application that upon PR to master runs tests, and if succeeds, merges to master. The next step I want is to deploy the application on a server.
This means that eventually I need my source code plus all the dependencies on the server, and run node server.js.
Is it correct to publish my application as a package (as part of the CI pipeline), and then on the server run npm install to fetch it? Or is npm installing packages only makes sense for packages that serve as some functionality for other applications?
The reason I doubt this is that when you run npm install (at least on a directory with a package.json) you get all the packages in the node_modules directory, which makes me believe that the second option I stated is true.
NOTE
The application is running on a Windows server, no Dockers.
I use vuetify (vue)
Is it mandatory for each deployment to production for remove node modules and run npm install? Or just run npm run build?
I have two option :
Option 1 : Every deployment, I run the npm run build directly
Option 2 :
Delete the contents of dist folder
Delete node_modules folder
npm install
npm run build
Which is the best option?
npm install
This command installs a package, and any packages that it depends on. If the package has a package-lock or shrinkwrap file, the installation of dependencies will be driven by that, with an npm-shrinkwrap.json taking precedence if both files exist. See package-lock.json and npm-shrinkwrap.
If you did not install or update the package before releasing the project, you do not need to execute npm install, otherwise, you need to execute it to ensure that dependent packages on the production environment is consistent with your local dependent package version.
If you are using an automatic build deployment tool like jenkins, for convenience you can execute the install command before each build. It's okay.
Imagine more environments, not just a production:
development
testing1
staging
uat
production
Can we upload the npm run build result (compressed js) or node_modules to our git repository? ANSWER IS NOT!!. So if you need to have a version of your app running in any of these environments, you must to execute npm run build. And this command needs the classic npm run install. I think this last sentence, answer your question.
(ADVICE) Docker to the rescue
assumption 1 your client-side app (vue) is not complex(no login, no session, no logout, etc ), you could publish it using a basic nginx, apache, basic-nodejs.
assumption 2 you are able to have one more server for docker private repository. Also if you are in google, amazon o azure, this service is ready to use, of course a payment is required
In one line, with docker you must execute just one time npm install and npm run build. Complete flow is:
developer push some changes to the git repository
manually or automatically a docker build in launched.
inside Dockerfile, npm install and npm run build is executed. Also a minimal server with nodejs (example) is configured pointing to your builded assets
your new docker image is uploaded to your docker private repository
that's all
If your quality assurance team needs to perform some tests to your new app, just a docker image download is required. If everything is ok, you pass to the next stage (staging or uat) or production. Steps will be the same: just download the docker image.
Optimizations
Use docker stages to split build and start steps
If your app does not have complex flows(no login, no session, no logout, etc ), replace node basic server with a simple nginx
I need login and logout
In this case, nginx or apache does not helps you because they are a simple static servers.
You could use a minimal nodejs code like this:
https://github.com/jrichardsz/nodejs-static-pages/blob/master/server.js
Adding /login , /logout, etc
Or use my server:
https://github.com/utec/geofrontend-server
which has a /login, /logout and other cool features for example: How are you planning to pass your backend api urls to your vue app in any of your environments?.
I have a Node project that uses MongoDB. To for automated testing, we use Mongo Memory Server
For Mongo Memory Server, Alpine is not supported my Mongo, so it can't run on Alpine images
From the docs:
There isn't currently an official MongoDB release for alpine linux. This means that we can't pull binaries for Alpine (or any other platform that isn't officially supported by MongoDB), but you can use a Docker image that already has mongod built in and then set the MONGOMS_SYSTEM_BINARY variable to point at that binary. This should allow you to use mongodb-memory-server on any system on which you can install mongod.
I can run all of my tests in a Docker container using a Node base image, but for production I want to use an Alpine image to save on memory.
so my Dockerfile looks something like this.
FROM node:x.x.x as test
WORKDIR /app
COPY . /app
npm install
npm run build # we use Typescript, this runs the transpilation
npm test # runs our automated tests
FROM node:x.x.x-alpine
WORKDIR /app
COPY --from=test /app/src /app/src
COPY --from=test /app/package.json /app/package.json
COPY --from=test /app/package-lock.json /app/package-lock.json
COPY --from=test /app/config /app/config
COPY --from=test /app/scripts /app/scripts
RUN npm install --production
RUN npm run build
Doing smoke testing, the resulting Alpine image seems to work okay. I assume it is safe because I install the modules in the alpine image itself.
I am wondering, is this the best practice? Is there a better way to do something like this? That is, for Node specifically, have a larger test container and a small production container safely.
Few points
If you are building twice, what is the point of the multistage build. I don't do much node stuff. But the reason you would want a multistage build is that you build you application with npm build that take those artifacts and copy them to the image and serve/run that in some way. In go world it would be something like building in a builder stage then just running the binary.
You always want to have the most changing things on the top of the union file system. What it means is that instead of copying the entire application code and running npm install, you should copy just package.json and run npm install on it. That way docker can cache the result of npm install and save on downloading the node files if nothing has changed on top. You application code changes way more than the package.json
On the second stage same idea. If you have to - copy package.json first and run npm install then copy the rest of the stuff.
You can have more stages if you want. The name of the game is to get the leanest and cleanest final stage image. Thats the one that goes on registry. Everything else can be and should be removed.
Hope it helps.
I've got a nodejs project that references a module I wrote and hosted by private github repo. Dependencies in package.json look something like this:
"dependencies": {
... other stuff ...
"my_module": "git+https://github.com/me/mymodule.git",
}
That's fine, but I'd like to create a Docker container for the application, but I don't want git inside the container. I know I can host via private npm repos, but I'd love to find a way to have the build process pull the source (including that module) and then copy it to the container.
I'm fine with doing an npm install in the container, but it will not like the git dependency. Alternatively, I don't want to do an npm install on the build machine because I want the freedom to choose any container I want... I don't want the build machine to snag windows binaries to a mongo module, for example, and copy those to my debian container.
One option I considered was putting the dependency to "my_module" in devDependencies, then within the Docker container do "npm install --production", then copy the one module over. That's just inconsistent with the intent of devDependencies.
Any better/recommended solutions? I'm open to not hosting the module in github if there's a better way (but I use it on a few projects that only make sense for this client).
Theres a pretty easy solution to this. Build the node application
npm install etc
Then in your dockerfile include the COPY command, telling it where the node projects install directory is, and where you want it to copy to.
Edit:
To address the issue brought up by #angelok you should use npm rebuild once it's copied into the docker image so that it builds with the correct dependencies relative to the OS of the Docker image instead of the OS in which the node packages were initially installed. See docs for rebuild here.