I am running my buildspec.yml where after npm run build command it should create the build directory in the root path, however, codebuild is unable to find the build directory.I have tried all possible ways from the resources, still I am unable to resolve "no matching base directory path found for build"
PS: I am using codecommit as the source, codebuild & codepipeline to run the deployment steps and S3 bucket to deploy the build directory.
My buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
nodejs: 10
pre_build:
commands:
- echo Installing dependencies...
- npm cache clean --force
- npm install
- npm --version
build:
commands:
- aws s3 rm s3://bucketname --recursive
post_build:
commands:
- pwd
- cd src
- npm run build
- ls -la
- aws s3 sync build s3://bucketname
artifacts:
files:
- "**/*"
I had to remove cd src from the post_build stage because my build and then it worked with pipeline, without any error.
Related
I have a project that requires npm and gradle for build, and docker for building and pushing the image.
At first I thought that I should create my own ubuntu image with gradle and npm setup, but I found out that is not what docker images are for.
So I hoped to run official Gradle and node images as a service so that my script can call those commands, but that is not happening for some reason.
My .gitlab-ci.yml:
variables:
IMAGE_NAME: my.registry.production/project
IMAGE_TAG: $CI_COMMIT_BRANCH
GIT_SUBMODULE_STRATEGY: recursive
stages:
- build
- deploy
build_project:
stage: build
image: ubuntu:jammy
services:
- name: node:12.20
alias: npm
- name: gradle:6.3.0-jre8
alias: gradle
before_script:
- git submodule init && git submodule update --remote --recursive
script:
- cd project-server && npm install && gradle clean build -Pprod -Pwar -x test -x integrationTest
deploy_image:
stage: deploy
image: docker:20.10.17
services:
- name: docker:20.10.17-dind
alias: docker
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
script:
- docker login -u $REGISTRY_USER -p $REGISTRY_PASSWORD my.registry.production
- docker build -t $IMAGE_NAME:$IMAGE_TAG .
- docker push $IMAGE_NAME:$IMAGE_TAG
If anyone has any info on how to solve this I would greatly appreciate it, since I’m a novice DevOps.
Edit 1:
My Dockerfile for custom image with Gradle and Node installed.
FROM ubuntu:jammy
LABEL key=DevOps
SHELL ["/bin/bash", "--login", "-i", "-c"]
RUN apt update && apt upgrade -y && apt install curl -y
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
RUN source /root/.bashrc && nvm install 12.14.1
RUN nvm install 12.20.0
RUN apt install zip unzip
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh"
RUN sdk install java 8.0.302-open
RUN sdk install gradle 3.4.1
SHELL ["/bin/bash", "--login", "-c"]
CMD [ "bin/bash" ]
After I run it, it says that npm is not found in $PATH, I tried Java, Gradle as well but they weren't found in the path as well.
I don't know why since I installed them as you can tell from the Dockerfile.
As I know, a docker image is equal to one build. So if you have multiple services you need to build each one into docker image then you can encapsulate all images into docker-compose.yml file.
I think you can do the following:
Build the npm project into a docker image
Build the Gradle project into a docker image
Write the docker-compose.yml file and put both images.
Once you have done it, the pipeline calls the docker-compose.yml file.
I hope this will be helpful.
Consider a few suggestions based on the fundamental concepts about the deployment in your CI/CD pipeline:
Remove the services keyword. Reference GitLab's official documents on what the services keyword inside gitlab-ci.yaml file is not for. The feature is used
to provide network accessable services to your job runtime (like
a database): https://docs.gitlab.com/ee/ci/services/index.html
Your project uses npm as a dependency management system, Gradle is
a build tool. Both of these pieces of software are more than
appropriate to run on the host operating system of the container
runtime inside GitLab's Pipeline job. You need these tools to assemble some build artifact as a result of the job on the same host your code has been downloaded on in the Runner.
Think about the overall size of the base image in your build_project job and consider how time to download the image over the network on to the Runner will impact your job and overall pipeline duration. If performance can be improved by baking build dependencies into a custom Dockerfile do this. If your image is too large, instead use shell commands inside the script keyword block to download them at the runtime of the job. There can be pros and cons for both.
Break shell scripts to one command per line for easier troubleshooting of failures in your scripts. You will be able to see the line number of the command which returned a non-zero exit code in your job logs:
...
script:
- cd project-server
- npm install
- gradle clean build -Pprod -Pwar -x test -x integrationTest
...
It's recommended to use the Gradle wrapper (gradlew) most of the time instead of the gradle executable directly. Configure this within your project and check the configuration files for the wrapper into your version control system and this will simplify your build dependency: https://docs.gradle.org/current/userguide/gradle_wrapper.html
I have a monorepo with yarn workspaces and have the following structure
packages
server
... #my-repo/server files
shared
translations
... #my-repo/translations files
validations
... #my-repo/validations files
and I use #my-repo/translations and #my-repo/validations are dependencies of in #my-repo/server.
To build the server app, I use the following buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
nodejs: 14
commands:
- echo Installing Yarn...
- npm install -g yarn
pre_build:
commands:
- echo Installing source NPM dependencies...
- yarn install
- echo Building Shared Packages ...
- yarn buildShared
- echo Listing directories ...
- ls node_modules/#my-repo/*
- echo Installing source NPM dependencies...
- yarn install
build:
commands:
- echo Build started on `date`
- yarn workspace #my-repo/server build
post_build:
commands:
- echo Build completed on `date`
artifacts:
files:
- '**/*'
discard-paths: no
The yarn buildShared command is a command I created to build all packages that are in the shared folder (simply run yarn workspace #my-repo/foo build for all shared packages in one command)
And one of the steps is to print what is inside node_modules/#my-repo directory and this prints everything as expected (I'm expecting all of my packages to be inside and correctly build, generating a dist folder).
And this CodeBuild create a Build Artifact in S3, but when I download and open the latest created BuildArtifact, I don't see any folder node_modules/#my-repo but I see a node_modules folder with the "normal" (not from yarn workspace) packages.
After try everything I could, I decided to create a temporary directory with every package I need from #my-repo in a temporary folder that I would copy from that and put back in node_modules (I know this is wrong, but I only did this to debug what was happening).
So I added this to my buildspec.yml
- cp -r node_modules/#happyr-health/ ./temp_modules
- ls
And apparently It worked, because the ls listed the temp_modules folder, but again, when downloading the latest build from S3, there was no temp_modules.
I can't figure out why some files are generated in the artifact and others aren't.
This is the second full day I'm trying to figure out "Why aren't the files from node_modules are correctly generated in the S3 bucket?"
I am trying to run a CI in gitlab
image: node:latest
stages:
- deploy
production:
stage: deploy
before_script:
- npm config set prefix /usr/local
- npm install -g serverless
script:
- serverless deploy
I am using the docker image like they suggest but it cannot find npm (or node)
How can I get this working?
Well, this is a bit weird, as your ci is correct.
If you are just using gitlab.com and their shared runners then this .gitlab-ci.yml will work.
One possible reason could be you have runners added as ssh/shell executors in the project repo. If so then the image tag you specified will be simply ignored.
So error like command not found could occur because of the server where you have added the runner doesn't have nodejs installed, and this error will occur for the npm config... command in before script with exit code 127 and pipeline will fail just there and stop.
If you have multiple runners then tag them and tag your jobs in ci.yml as well.
And if you are trying to run the job on your own server then you got to install docker first.
BTW for docker image node:latest you don't need npm config set prefix /usr/local as it already is /usr/local
I have an application in NodeJS which I am building with AWS CodeBuild and then deploying using SAM to AWS lambda. I want to remove all devDependencies from the project after the build phase. In build phase I run all tests which require the devDependencies but I don't want them to be zipped with other modules while pushing it to S3 as artifacts.
My buildspec.yml
version: 0.2
phases:
install:
commands:
# Update libs
- echo Executing the install phase.
runtime-versions:
nodejs: 10
pre_build:
commands:
- npm install
build:
commands:
- echo Executing the build phase.
- npm run test
- export BUCKET=alexa-v1
- aws cloudformation package --template-file template.yml --s3-bucket $BUCKET --output-template-file outputtemplate.yml
post_build:
commands:
- echo Build complete
artifacts:
type: zip
files:
- template.yml
- outputtemplate.yml
I am not sure if adding npm prune --production in post_build, is the right way to do it.
Try use the AWS SAM CLI instead of the regular AWS CLI. In particular, there is the sam package command that can be used when you package your application, e.g.
sam package \
--template-file template.yml \
--s3-bucket $BUCKET \
--output-template-file outputtemplate.yml
You could insert "npm prune --production" before the last line in the build phase ("aws cloudformation package...").
This would ensure that all dev dependencies are removed before deploying the code to lambda.
I had a problem with the same issue.
And in case you think, when you look for NODE_ENV=production
I found out that devDependency is not installing.
I am trying to configure my gitlab-ci to use yarn install instead of npm install
My current gitlab-ci.yml looks like:
image: node:6.9.4
cache:
paths:
- node_modules/
- .yarn
before_script:
- apt-get update -qq && apt-get install -qy libelf1
stages:
- test
test_core:
stage: test
script:
- yarn config set cache-folder .yarn
- yarn install
- npm run build
- npm run test
tags:
- 2gb
But the build fails with the error:
/bin/bash: line 48: yarn: command not found
is there something I am missing?
I tried installing yarn with:
curl -o- -L https://yarnpkg.com/install.sh | bash
this gave me same error, probably because I need to reload the bash environment for the yarn command to become available.
The above config works perfect with npm install.
Please help me resolve this. If there is something missing in my config file or there is something wrong with the gitlab-ci.
Thanks.
Solved it by using the latest official node docker image.
Since image: 6.10.0, yarn is installed by default in the image.
But if you need node-gyp to build any package, it needs to be installed by adding a line to the script:
yarn global add node-gyp
Add the following to your ci script after yarn got installed:
export PATH=$HOME/.yarn/bin:$PATH
I use image:node:latest and sometimes it prompts the error. Clear Runner Caches do the job for me. Maybe the runner did not recover to the correct state after doing other jobs.
I solved it by using npx (package runner). It's better then extending docker-image only for this purpose
npx yarn someCommand