In my project we are using nodejs with typescript for google cloud app engine app development. We have our own build mechanism to compile ts files into javascript ,then collect them into a complete runable package, so that we don't want to relay on google cloud to install dependencies, instead we want to upload all node packages inside the node_modules to google cloud.
But it seems google cloud will always ignore the node_modules folder and run npm install during the deployment. Even I tried to remove 'skip_files: - ^node_modules$' from app.yaml, it doesn't work, google cloud will always install packages by itself.
Does anyone have ideas of this of deploy node app with node_modules together? Thank you.
I observed the same issue.
My workaround was to rename node_modules/ to node_modules_hack/ before deploying. This prevents AppEngine from removing it.
I restore it to the original name on installation, with the following (partial) package.json file:
"__comments": [
"TODO: Remove node_modules_hack once AppEngine stops stripping node_modules/"
],
"scripts": {
"install": "mv -fn node_modules_hack node_modules",
"start": "node server.js"
},
You can confirm that AppEngine strips your node_modules/ by looking at the Docker image it generates. You can find it on the Images page. They give you a commandline that you can run on the cloud console to fetch it. Then you can run docker run <image_name> ls to see your directory structure. The image is created after npm install, so once you use the workaround above, you'll see your node_modules/ there.
The newest solution is to allow node_modules in .gcloudignore.
Below's the default .gcloudignore (one that initial execution of gcloud app deploy generates if you don't have one already) with the change you need:
# This file specifies files that are *not* uploaded to Google Cloud Platform
# using gcloud. It follows the same syntax as .gitignore, with the addition of
# "#!include" directives (which insert the entries of the given .gitignore-style
# file at that point).
#
# For more information, run:
# $ gcloud topic gcloudignore
#
.gcloudignore
# If you would like to upload your .git directory, .gitignore file or files
# from your .gitignore file, remove the corresponding line
# below:
.git
.gitignore
# Node.js dependencies:
# node_modules/ # COMMENT OR REMOVE THIS LINE
Allowing node_modules in .gcloudignore no longer works.
App Engine deployment is switched to buildpacks since Oct/Nov 2020. Cloud Build step triggered by it will always remove uploaded node_modules folder and reinstall dependencies using yarn or npm.
Here is the related buildpack code:
https://github.com/GoogleCloudPlatform/buildpacks/blob/89f4a6ba669437a47b482f4928f974d8b3ee666d/cmd/nodejs/yarn/main.go#L60
This is a desirable behaviour since uploaded node_modules could come from a different platform and could break compatibility with Linux runner used to run your app in App Engine environment.
So, in order to skip npm/yarn dependencies installation in Cloud Build, I would suggest to:
Use Linux runner CI with the same Node version you are using in the App Engine environment.
Create tar archive with your node_modules, to not upload multitude of files on each gcloud app deploy.
Keep node_modules dir ignored in .gcloudignore.
Unpack node_modules.tar.gz archive in preinstall script. Don't forget to keep backward compatibility in case tar archive is missing (local development, etc.):
{
"scripts": {
"preinstall": "test -f node_modules.tar.gz && tar -xzf node_modules.tar.gz && rm -f node_modules.tar.gz || true"
}
}
Note ... || true thing. This will ensure preinstall script returns zero exit code no matter what, and yarn/npm install will continue.
Github Actions workflow to pack and upload your dependencies for App Engine deployment could look like this:
deploy-gae:
name: App Engine Deployment
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
# Preferable to use the same version as in GAE environment
- name: Set Node.js version
uses: actions/setup-node#v2
with:
node-version: '14.15.4'
- name: Save prod dependencies for GAE upload
run: |
yarn install --production=true --frozen-lockfile --non-interactive
tar -czf node_modules.tar.gz node_modules
ls -lah node_modules.tar.gz | awk '{print $5,$9}'
- name: Deploy
run: |
gcloud --quiet app deploy app.yaml --no-promote --version "${GITHUB_ACTOR//[\[\]]/}-${GITHUB_SHA:0:7}"
This is just an expanded version of the initially suggested hack.
Note: In case you have a gcp-build script in your package.json you will need to create two archives (one for production dependencies and one for dev) and modify preinstall script to unpack the one currently needed (depending on the NODE_ENV set by buildpack).
Related
I have a monorepo with yarn workspaces and have the following structure
packages
server
... #my-repo/server files
shared
translations
... #my-repo/translations files
validations
... #my-repo/validations files
and I use #my-repo/translations and #my-repo/validations are dependencies of in #my-repo/server.
To build the server app, I use the following buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
nodejs: 14
commands:
- echo Installing Yarn...
- npm install -g yarn
pre_build:
commands:
- echo Installing source NPM dependencies...
- yarn install
- echo Building Shared Packages ...
- yarn buildShared
- echo Listing directories ...
- ls node_modules/#my-repo/*
- echo Installing source NPM dependencies...
- yarn install
build:
commands:
- echo Build started on `date`
- yarn workspace #my-repo/server build
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- '**/*'
discard-paths: no
The yarn buildShared command is a command I created to build all packages that are in the shared folder (simply run yarn workspace #my-repo/foo build for all shared packages in one command)
And one of the steps is to print what is inside node_modules/#my-repo directory and this prints everything as expected (I'm expecting all of my packages to be inside and correctly build, generating a dist folder).
And this CodeBuild create a Build Artifact in S3, but when I download and open the latest created BuildArtifact, I don't see any folder node_modules/#my-repo but I see a node_modules folder with the "normal" (not from yarn workspace) packages.
After try everything I could, I decided to create a temporary directory with every package I need from #my-repo in a temporary folder that I would copy from that and put back in node_modules (I know this is wrong, but I only did this to debug what was happening).
So I added this to my buildspec.yml
- cp -r node_modules/#happyr-health/ ./temp_modules
- ls
And apparently It worked, because the ls listed the temp_modules folder, but again, when downloading the latest build from S3, there was no temp_modules.
I can't figure out why some files are generated in the artifact and others aren't.
This is the second full day I'm trying to figure out "Why aren't the files from node_modules are correctly generated in the S3 bucket?"
Using gradle and java, I can build and publish (into a local repository) various versions. If a deployment has issues, I can reroll by deploying a prior version.
I can't find anything comparable for NPM applications, e.g. vue.
I foun npm pack which seems to build something that I can later install via npm install.
But I could not find out I then "run" the application. Are the any good documentations?
Thanks in advance.
Unline Jave Javascript doesn't compile into anything (because its interpreted). This means that all you need it the source code.
A basic node app source code will look something like this
main.js # entrypoint
packege.json # dependencies
src/ # application codebase
So if you like to create an artifact that can be deployed to a server you need only to download the dependencies (that's what the npm install is for) and that's it basically.
Now a node_module directory will be created
main.js # entrypoint
packege.json # dependencies
src/ # application codebase
node_module/ # dependencies codebase
You can run the application with node run main.js. But if you are creating an artifact you should bundle (zip) all the files and upload then to a file server (this will be the closest thing to .war in this flow)
You can now unzip and run the node application on a server that has NodeJS installed.
I am generating the production version of an API I made using the NESTJS framework and would like to know which files I should upload to the server. When I run the "npm run start: prod" compile it generates the "dist" folder but I tried to run only with it but it is not enough to run my application. Do I need to upload all files to the server? I did several tests removing the folders I used during development but only managed to run in production mode when I was all the same in dev mode.
I looked in the documentation for something about this but found nothing. can anybody help me?
Thank you
Honestly, you should only really need the dist folder as that's the JS 'complied' files. To run your application, commonly you'd use this command node dist/main.js. As to what files you upload it's up to you. Me personally, I use a lot of continuous integration so I would just clone to repo into my container/server and use yarn start:prod. This is so everytime I deploy I'm generating the required files to run in a production environment.
Like #Kim Kern mentioned, some node modules are native built using node-gyro; so it's also always best to build your node_modules on the server/container when deploying. Your deployment script should look something like this
git clone git#github.com:myuser/myrepo.git /var/www/
cd /var/www/
node -v && \
yarn && \
yarn build && \
yarn start:prod
The above script should
1) pull the required repo into a 'hosted' directory
2) check the node version
3) install node_modules and build native scripts etc
4) build the production distribution
5) run the production JS scripts
If you look in your package.json file you'll notice the different scripts that are run when you use yarn start, yarn start:dev and yarn start:prod. When in dev you'll notice the use of ts-node which is a typescript node runner type thing (can't remember the correct phrase). Also the start:dev script uses nodemode to restart the ts-node script. You'll also see the start:prod script uses node dist/main.js and that the prestart:prod script runs rm -rf dist && tsc which removes the dist folder and 'compiles' the javascript required for a production environment.
However, the drawback of a typescript application on your server without continuous integration is that there is the possibility of typescript compilation errors which you wouldn't see or know about until running the prod scripts. I would recommend putting a procedure in place to compile the javascipt from typescript before making a deployment as you don't want to delete the current dist build before knowing the next release will build and run!
For me this approach worked and all you need is the dist folder for this:
Create a prod build of your application using npm run start:prod, this would create a dist folder within your application source
Copy the dist folder to your server.
For getting all the node_modules dependencies on your server just copy your package.json file into the dist folder (that you have copied onto the server) and then run npm install from there.
If you are using pm2 to run your node applications just run pm2 start main.js from within the dist folder
Mostly, you will only need the dependencies in node_modules. You should build the libraries on your server though instead of copying them from your dev machine. Libraries like bcrypt have machine specific code and probably won't run on a different machine. (30% of the npm libraries have native bindings.)
So for your deployment I would recommend to checkout your git repository on your server and then just run npm run start:prod (which builds the project every time) directly there.
Just use the Nest-CLI and build with
nest build
Afterwards you get a dist folder with the compiled Code.
You can then place it on a server an run e.g. with PM2 proccess manager:
production=true pm2 start dist/main.js
In former command the environment variable production is set to true. That could e.g. be usefull when running the Nest.js server over HTTPS.
If you want to run a HTTPS secured server you also have to include the certificates in the starting process of the server. When the environment variable production is set and true the certificates get included in the starting proccess of the Nest.js application in main.ts like following:
async function bootstrap() {
let appConfig = {}
if (process.env.production) {
console.log('process env production: ', process.env.production)
const httpsOptions = {
key: fs.readFileSync('/etc/certs/letsencrypt/live/testtest.de/privkey.pem'),
cert: fs.readFileSync('/etc/certs/letsencrypt/live/testtest.de/fullchain.pem'),
}
// prod config
appConfig = {
httpsOptions,
}
}
const app = await NestFactory.create<NestExpressApplication>(
AppModule,
appConfig,
)
app.enableCors()
app.setGlobalPrefix('v1')
await app.listen(3300)
}
bootstrap()
We don't build our application on production, but instead build it when creating our docker container.
The steps for us roughly are:
Run npm install and whatever tooling you need to build the application.
Create docker container and copy dist/, node_modules and package.json
Inside the docker container run npm rebuild bcrypt --update-binary
We are using NX for monorepo where we hold our API's. And we use docker for our images and containers. When we have to create docker image, only run: npx nx build <project> and this generate build on dist/apps/<project>. This folder goes to the docker image, with the package.json and that's it. You don't need to add node_modules, because they are on the package.json. Just be sure to include npm install on your Dockerfile.
I am new to node.js, I was trying to deploy node.Js project via gitlab ci. But after spotting the build error in pipeline I realized I added node_modules folder to .gitignore, and I am not pushing node_modules to gitlab. And node_modules folder is 889MB locally there is no way I will push it, so what approach should I use to use node_module s folder from somewhere else.
i.e. node_modules path is always present and accessible on remote server! Do I need to include that path in package. Json
Can node_modules be maintained by using docker ? then how would I maintain to stay update specific to every project.
You are right not to check in the node_modules folder, they are automatically populated at the time you run npm install
This should be part of your build pipeline in the gitlab ci. The pipeline allows multiple steps and the ability to pass artefacts through to the next stage. In your case you want to save the node_modules folder that is created by running npm install you can then use the dependencies for tests or deployment.
Since npm v5 there is a lockfile to make sure what you are running locally will be the same as what you are running on the server
Also you can use something like rennovate to automatically update your dependancies if you want to fix them and automatically manage security updates. (rennovate is open source so can be ran on gitlab)
A really simple gitlab CI pipeline could be:
// .gitlab-ci.yml
stages:
- build
- deploy
build:
stage: build
script:
- npm install
artifacts:
name: "${CI_BUILD_REF}"
expire_in: 10 mins
paths:
- node_modules
deploy:
stage: deploy
script:
- some deploy command
Does GAE standard for Node support a way to have build scripts? I tried using postinstall within package.json but that did not work.
My codebase has subdirectories with package.json within the subdirectories. In my root package.json there is
scripts: {
postinstall: cd vendor && npm install
....
}
However I'm not seeing any vendor packages installed so I'm inclined to believe the postinstall does not get triggered on GAE Node standard.
Is there any way for me to install subdirectory dependencies without having to copy and paste all my vendor/package.json dependencies to the root?
Note: I've also tried putting an "install" within the package.json scripts but that didn't seem to get triggered either.
In GAE standard, installation of dependencies are automatically managed. You should add them in your package.json.
As Google documentation mentioned :
When you deploy your app, the Node.js runtime automatically installs all dependencies declared in your package.json file using the npm install command.
{
"dependencies": {
"lodash": "^4.0.1"
}
}
Installation will be done during app deployment via :
gcloud app deploy
To add a build step, run the following:
gcloud beta app gen-config --custom
This will generate the default dockerfile and config that is run. In your .dockerfile, add your build step:
RUN npm run build --unsafe-perm || \
((if [ -f npm-debug.log ]; then \
cat npm-debug.log; \
fi) && false)
"prestart": "if [ ! -d build ]; then npm run build; fi",
" -d build" here is the build process generated folder, replace it to whatever you actually use.
Not sure if this will work for your case, but seems like GAE standard has added the ability to run a custom build step.
However it does state:
After executing your custom build step, App Engine removes and regenerates the node_modules folder by only installing the production dependencies declared in the dependencies field of your package.json file.
Maybe since the node_modules are in your vendor/ directory, GAE may not detect and remove them, thus accomplishing your goal. This is a pre-install step, unlike postinstall specified in your script. Not sure if it matters.