I have an application in NodeJS which I am building with AWS CodeBuild and then deploying using SAM to AWS lambda. I want to remove all devDependencies from the project after the build phase. In build phase I run all tests which require the devDependencies but I don't want them to be zipped with other modules while pushing it to S3 as artifacts.
My buildspec.yml
version: 0.2
phases:
install:
commands:
# Update libs
- echo Executing the install phase.
runtime-versions:
nodejs: 10
pre_build:
commands:
- npm install
build:
commands:
- echo Executing the build phase.
- npm run test
- export BUCKET=alexa-v1
- aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
post_build:
commands:
- echo Build complete
artifacts:
type: zip
files:
- template.yml
- outputtemplate.yml
I am not sure if adding npm prune --production in post_build, is the right way to do it.
Try use the AWS SAM CLI instead of the regular AWS CLI. In particular, there is the sam package command that can be used when you package your application, e.g.
sam package \
--template-file template.yml \
--s3-bucket $BUCKET \
--output-template-file outputtemplate.yml
You could insert "npm prune --production" before the last line in the build phase ("aws cloudformation package...").
This would ensure that all dev dependencies are removed before deploying the code to lambda.
I had a problem with the same issue.
And in case you think, when you look for NODE_ENV=production
I found out that devDependency is not installing.
Related
I have a project that requires npm and gradle for build, and docker for building and pushing the image.
At first I thought that I should create my own ubuntu image with gradle and npm setup, but I found out that is not what docker images are for.
So I hoped to run official Gradle and node images as a service so that my script can call those commands, but that is not happening for some reason.
My .gitlab-ci.yml:
variables:
IMAGE_NAME: my.registry.production/project
IMAGE_TAG: $CI_COMMIT_BRANCH
GIT_SUBMODULE_STRATEGY: recursive
stages:
- build
- deploy
build_project:
stage: build
image: ubuntu:jammy
services:
- name: node:12.20
alias: npm
- name: gradle:6.3.0-jre8
alias: gradle
before_script:
- git submodule init && git submodule update --remote --recursive
script:
- cd project-server && npm install && gradle clean build -Pprod -Pwar -x test -x integrationTest
deploy_image:
stage: deploy
image: docker:20.10.17
services:
- name: docker:20.10.17-dind
alias: docker
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
script:
- docker login -u $REGISTRY_USER -p $REGISTRY_PASSWORD my.registry.production
- docker build -t $IMAGE_NAME:$IMAGE_TAG .
- docker push $IMAGE_NAME:$IMAGE_TAG
If anyone has any info on how to solve this I would greatly appreciate it, since I’m a novice DevOps.
Edit 1:
My Dockerfile for custom image with Gradle and Node installed.
FROM ubuntu:jammy
LABEL key=DevOps
SHELL ["/bin/bash", "--login", "-i", "-c"]
RUN apt update && apt upgrade -y && apt install curl -y
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
RUN source /root/.bashrc && nvm install 12.14.1
RUN nvm install 12.20.0
RUN apt install zip unzip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
RUN sdk install java 8.0.302-open
RUN sdk install gradle 3.4.1
SHELL ["/bin/bash", "--login", "-c"]
CMD [ "bin/bash" ]
After I run it, it says that npm is not found in $PATH, I tried Java, Gradle as well but they weren't found in the path as well.
I don't know why since I installed them as you can tell from the Dockerfile.
As I know, a docker image is equal to one build. So if you have multiple services you need to build each one into docker image then you can encapsulate all images into docker-compose.yml file.
I think you can do the following:
Build the npm project into a docker image
Build the Gradle project into a docker image
Write the docker-compose.yml file and put both images.
Once you have done it, the pipeline calls the docker-compose.yml file.
I hope this will be helpful.
Consider a few suggestions based on the fundamental concepts about the deployment in your CI/CD pipeline:
Remove the services keyword. Reference GitLab's official documents on what the services keyword inside gitlab-ci.yaml file is not for. The feature is used
to provide network accessable services to your job runtime (like
a database): https://docs.gitlab.com/ee/ci/services/index.html
Your project uses npm as a dependency management system, Gradle is
a build tool. Both of these pieces of software are more than
appropriate to run on the host operating system of the container
runtime inside GitLab's Pipeline job. You need these tools to assemble some build artifact as a result of the job on the same host your code has been downloaded on in the Runner.
Think about the overall size of the base image in your build_project job and consider how time to download the image over the network on to the Runner will impact your job and overall pipeline duration. If performance can be improved by baking build dependencies into a custom Dockerfile do this. If your image is too large, instead use shell commands inside the script keyword block to download them at the runtime of the job. There can be pros and cons for both.
Break shell scripts to one command per line for easier troubleshooting of failures in your scripts. You will be able to see the line number of the command which returned a non-zero exit code in your job logs:
...
script:
- cd project-server
- npm install
- gradle clean build -Pprod -Pwar -x test -x integrationTest
...
It's recommended to use the Gradle wrapper (gradlew) most of the time instead of the gradle executable directly. Configure this within your project and check the configuration files for the wrapper into your version control system and this will simplify your build dependency: https://docs.gradle.org/current/userguide/gradle_wrapper.html
Description problem
I have a problem with serverless-offline because I try to install this local and global, and it doesn't work.
serverless-offline image error
When executing the command I get this information
sls offline
Serverless command "offline" not found. Run "serverless help" for a list of all available commands.
This is my configuration in the serverless file.
provider:
name: aws
runtime: nodejs14.x
environment:
NODE_ENV: ${env:NODE_ENV}
plugins:
- serverless-plugin-typescript
- serverless-offline
This commands I used to install
In the machine environment
npm i -g serverless-offline
In the file project:
yarn add serverless-offline -D
Issue in git
Serverles environment configurated image
Try in your project with the npx prefix. So npx sls offline or npx serverless offline.
I am running my buildspec.yml where after npm run build command it should create the build directory in the root path, however, codebuild is unable to find the build directory.I have tried all possible ways from the resources, still I am unable to resolve "no matching base directory path found for build"
PS: I am using codecommit as the source, codebuild & codepipeline to run the deployment steps and S3 bucket to deploy the build directory.
My buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
nodejs: 10
pre_build:
commands:
- echo Installing dependencies...
- npm cache clean --force
- npm install
- npm --version
build:
commands:
- aws s3 rm s3://bucketname --recursive
post_build:
commands:
- pwd
- cd src
- npm run build
- ls -la
- aws s3 sync build s3://bucketname
artifacts:
files:
- "**/*"
I had to remove cd src from the post_build stage because my build and then it worked with pipeline, without any error.
I'm sure this is a common issue but I can't seem to find a definitive answer.
I have a node application which in order to build requires some devDependencies such as babel. In order to run my tests also requires devDependencies such as jest. But when CI runs in production environment, it obviously doesn't install any devDependencies so I get errors where the package isn't found.
What is the best practice for running builds and tests in prod without devDependencies?
If it helps, I am running my build in GitLab Pipelines:
image: node:8.11.2
stages:
- prepare
- test
- deploy
install_and_build:
stage: prepare
script:
- npm install yarn
- yarn
- yarn build
only:
- master
test:
stage: test
script:
- yarn test
only:
- master
deploy_production:
type: deploy
stage: deploy
image: ruby:latest
script:
- apt-get update -qy
- apt-get install -y ruby-dev
- gem install dpl
- dpl --provider=heroku --app=app-name --api-key=$HEROKU_API_KEY
only:
- master
From this answer,
You will want to follow have your process to the effect of the
following:
First, you need to "install with all dependencies".
npm install
Then do your tests.
npm test
Then "prune" your dev dependencies as below, as detailed in the
docs doing this
"will remove the packages specified in your devDependencies".
npm prune --production
I am trying to configure my gitlab-ci to use yarn install instead of npm install
My current gitlab-ci.yml looks like:
image: node:6.9.4
cache:
paths:
- node_modules/
- .yarn
before_script:
- apt-get update -qq && apt-get install -qy libelf1
stages:
- test
test_core:
stage: test
script:
- yarn config set cache-folder .yarn
- yarn install
- npm run build
- npm run test
tags:
- 2gb
But the build fails with the error:
/bin/bash: line 48: yarn: command not found
is there something I am missing?
I tried installing yarn with:
curl -o- -L https://yarnpkg.com/install.sh | bash
this gave me same error, probably because I need to reload the bash environment for the yarn command to become available.
The above config works perfect with npm install.
Please help me resolve this. If there is something missing in my config file or there is something wrong with the gitlab-ci.
Thanks.
Solved it by using the latest official node docker image.
Since image: 6.10.0, yarn is installed by default in the image.
But if you need node-gyp to build any package, it needs to be installed by adding a line to the script:
yarn global add node-gyp
Add the following to your ci script after yarn got installed:
export PATH=$HOME/.yarn/bin:$PATH
I use image:node:latest and sometimes it prompts the error. Clear Runner Caches do the job for me. Maybe the runner did not recover to the correct state after doing other jobs.
I solved it by using npx (package runner). It's better then extending docker-image only for this purpose
npx yarn someCommand