I'm setting up a CI/CD pipeline using Google cloud build, I'm doing 3 simple steps : install, test and deploy, here is my CloudBuild.yaml file :
steps:
- name: "gcr.io/cloud-builders/npm"
args: ["install"]
- name: "gcr.io/cloud-builders/npm"
args: ["run", "test"]
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
And here is my package.json where I need to specify the hole path of the jest so it can identify it, otherwise I set it : "test": "jest" the step 2 of the build will fail and it will not recognize it.
"scripts": {
"start": "node index.js",
"test": "sh node_modules/.bin/jest",
"test:watch": "jest --watch"
},
In this case I need to always push the node_modules to the github repo to find the module jest, but it's not a good practise.
Any ideas how can I fix this ?
Thanks
Related
I am deploying a Google Cloud function and it fails to deploy because it sometimes doesn't find the module #google-cloud/pubsub and when it finds it, it instead fails on a transitive dependency from the pubsub package. I have had this with #google-cloud/storage as well and never managed to solve it.
See the logs screenshot below in this post for an example of the issue. It shows two consecutive deployments.
In the first deployment, the #google-cloud/pubsub is not found although it is listed in package.json
In the second deployment, the pubsub package is found(!), but then instead it fails on the transitive dependency google-gax.
Things I tried
Deploy using Google Cloud Build, hooked up to my github repo (deplyoment fails)
Deploy from local command line. See script target deployAnalyzer in package.json below. (deployment fails with same errors a Google Cloud Builds)
Deploy the function locally to the functions framework. See script target devFunctionAnalyzer below. (Deplyoment works and the function can be invoked)
Build locally using pack to repeat the same build procedure as Google Cloud Build. Result: Image builds and runs fine!
In short: It work fine locally, both in the Functions Framework and running the docker image built using the google nodejs buildpack.
The Full Code
Work in progress: https://github.com/aweijnitz/ornitho-de-monitor
Minimalistic repo with one dependency, that fails with the same error: https://github.com/aweijnitz/cloud-functions-dependency-test-01/tree/main
EDIT
Added link to stripped down repo that illustrates the problem without all the application code. Just a single dependency now.
package.json
{
"name": "ornitho-de-monitor",
"version": "0.0.1",
"description": "Collect latest interesting observations from ornitho.de",
"main": "src/index.js",
"engines": {
"node": ">=12.13"
},
"scripts": {
"devFunctionAnalyzer": "npx functions-framework --target=analyzeObservations --signature-type=event",
"devFunctionNotifier": "npx functions-framework --target=notifyAll --signature-type=event",
"buildCloudRunContainer": "gcloud builds submit --tag gcr.io/ornitho-de-monitor/ornitho-de-scraper",
"deployCloudRunContainer": "gcloud run deploy ornitho-de-scraper --region=europe-west6 --image gcr.io/ornitho-de-monitor/ornitho-de-scraper --platform managed",
"deployAnalyzer": "gcloud functions deploy analyzeObservations --source=./src --entry-point=analyzeObservations --runtime nodejs12 --memory=256MB --max-instances=3 --trigger-topic=ornitho-bus",
"deployNotifier": "gcloud functions deploy notifyAll --source=./src --entry-point=notifyAll --runtime nodejs12 --memory=256MB --max-instances=3 --trigger-topic=ornitho-bus",
"invoke": "gcloud functions call ornitho-de-monitor --data '{\"name\":\"Keyboard Cat\"}'",
"invokeEncoded": "DATA=$(printf 'Hello!'|base64) && gcloud functions call helloPubSub --data '{\"data\":\"'$DATA'\"}'",
"viewFunctionLogs": "gcloud functions logs read ornitho-de-monitor",
"serveTestFile": "cd testdata && python -m SimpleHTTPServer 8000",
"test": "echo \"No test specified\" && exit 0"
},
"author": "Anders Weijnitz",
"license": "ISC",
"dependencies": {
"#google-cloud/pubsub": "^2.7.0",
"#google-cloud/storage": "^5.7.0",
"express": "^4.17.1",
"express-async-handler": "^1.1.4",
"express-rate-limit": "^5.2.3",
"knex": "^0.21.15",
"pg": "^8.5.1"
},
"devDependencies": {
"#google-cloud/functions-framework": "^1.7.1"
}
}
These are the relevant steps from cloudbuild.yaml
steps:
# Retrieve credentials from Fort Knox
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: [ '-c', "gcloud secrets versions access latest --secret=ornitho-pubsub --format='get(payload.data)' | tr '_-' '/+' | base64 -d > pubsubkey.json" ]
# Install dependencies
- name: 'gcr.io/cloud-builders/npm'
args: [ 'install' ]
dir: '.'
# Run tests
- name: 'gcr.io/cloud-builders/npm'
args: [ 'test' ]
dir: '.'
# Deploy the notifier function
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'functions'
- 'deploy'
- 'notifyAll'
- '--source=./src'
- '--entry-point=notifyAll'
- '--runtime=nodejs12'
- '--memory=128MB'
- '--max-instances=3'
- '--trigger-topic=ornitho-bus'
# Deploy the analyzer function
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'functions'
- 'deploy'
- 'analyzeObservations'
- '--source=./src'
- '--entry-point=analyzeObservations'
- '--runtime=nodejs12'
- '--memory=128MB'
- '--max-instances=3'
- '--trigger-topic=ornitho-bus'
SOLVED!
In turns out that the --source parameter should not point to the location of the application source, but to the location of the project root, where package.json resides. Bad: --source=./src Good: --source=. .
For more details, see github.com/aweijnitz/cloud-functions-dependency-test-01 –
I need to setup build pipeline in AWS amplify for create react app.I couldn't find code snippet to add unit test that uses Jest with React testing library and typescript. Can you please share the amplify.yml that have jest unit test. I have tried below
version: 1
frontend:
phases:
preBuild:
commands: ['npm ci']
build:
commands: 'npm run test' && 'npm run build'
artifacts:
baseDirectory: build
files:
- '**/*'
cache:
paths:
- 'node_modules/**/*'
To add Jest you should first make sure it's installed and listed in your package.json file.
In the root of your project:
npm install jest --save-dev
You should also add this to package.json:
"scripts": {
"test": "jest"
},
Then run Jest locally by running
npm run test
If that works you just need to ensure that jest is available in your build environment.
Package.json:
{
"name": "Your-app-name",
"author": "",
"license": "",
"homepage": "",
"version": "1.0.0",
"description": "xxx",
"scripts": {
"test": "jest"
},
"devDependencies": {
"jest": "^26.6.3",
"uuid": "^8.3.2"
}
}
Hope it helps you some!
WARNING: Your package.json does not specify a supported Node.js version. Please pin your application to a major version of the Node.js runtime.
Application detection failed: Error: node.js checker: Neither "start" in the "scripts" section of "package.json" nor the "server.js" file were found.
I added below in package.json
"engines": {
"node": "^13.8.0",
"npm":"^6.13.4"
},
app.yaml file
# [START runtime]
runtime: nodejs
env: flex
service: dev
handlers:
- url: /.*
secure: always
script: auto
redirect_http_response_code: 301
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
cool_down_period_sec: 60
cpu_utilization:
target_utilization: 0.80
package.json
"scripts": {
"start": "serve -s ./build",
"prestart": "npm i -g serve",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
}
cloudbuild.yaml
steps:
- name: gcr.io/cloud-builders/npm
entrypoint: 'npm'
args: [ install ]
- name: gcr.io/cloud-builders/npm
entrypoint: 'npm'
args: [ run, build, --prod ]
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'npm'
args: [ app, deploy, '[public/app.yaml]', --version=$SHORT_SHA ]
If you're using the GAE standard, keep in mind that it currently supports Node.js 10 and 12 runtimes.
Also, as the official GCP documentation mentions:
By default, the runtime starts your application by running node
server.js. If you specify a start script in your package.json file,
the runtime runs the specified start script instead.
"scripts": {
"start": "node app.js"
}
Please, have a look at this GitHub repository, which contains a simple app for GAE standard, which you could use as a reference.
EDIT
You need to make sure that your start script is starting a web server that responds to HTTP requests on the port specified by the PORT environment variable, typically 8080 (link).
Here you can see an example for the GAE flexible environment, where the app.js contains
// Start the server
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
console.log(`App listening on port ${PORT}`);
console.log('Press Ctrl+C to quit.');
});
While the package.json has
"scripts": {
"start": "node app.js"
}
This error can be reproduced when deploying that sample on GAE with Cloud Build after removing the start script or if the app is not starting the server.
I want to deploy my app build in vue which use CLI 3.0.
My package.json:
"scripts": {
"serve": "vue-cli-service serve",
"postinstall": "npm run build",
"build": "vue-cli-service build",
"lint": "vue-cli-service lint",
"test:unit": "vue-cli-service test:unit",
"test:e2e": "vue-cli-service test:e2e"
}
I added "#vue/cli": "^3.0.0-rc.3" to devDependencies, but is don't see any changes.
Azure deploy result:
> npm run vue-cli-service build
npm ERR! missing script: vue-cli-service
Do you have any ideas?
I havent used azure yet, but try only with
npm run build
instead of
npm run vue-cli-service build
I assume you have a build pipeline that struggles with the message you have given.
I think what you are missing is a simple
npm install
After the install you are able to run
npm run build
Without the npm install before, threre is no vue-cli-service npm can find to build the application. I build my own vue-cli 3.0 app this way an deploy to azure like this from an Azure DevOps build pipeline.
or an other possibility is that you are missing another dependency. Add "#vue/cli-service": "^3.0.1" to your devDependencies. And as Daniel Gonzalez pointed out in the comments, there is no need for a postinstall script.
I have succeed building Vue with Vue CLI 3 in Azure.
Sharing my build file here
Azure pipeline YAML script
resources:
- repo: self
trigger: ['staging']
pool:
vmImage: 'Ubuntu 16.04'
steps:
- task: NodeTool#0
displayName: 'Use Node 10.x'
inputs:
versionSpec: 10.x
- script: |
npm install
npm run build-staging
displayName: 'npm install and build'
env:
NODE_ENV: staging
- task: ArchiveFiles#2
displayName: Archive
inputs:
rootFolderOrFile: '$(build.sourcesDirectory)/dist'
includeRootFolder: false
archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.SourceVersion)_$(Build.BuildId).zip'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: build'
inputs:
ArtifactName: build
package.json
...
"scripts": {
"serve": "vue-cli-service serve --port 9001",
"build": "vue-cli-service build",
"build-staging": "NODE_ENV=production vue-cli-service build --mode staging",
"build-production": "NODE_ENV=production vue-cli-service build --mode production",
"lint": "vue-cli-service lint"
},
...
Only way I have found to host Vue-CLI build files in Azure is this:
Create a StorageV2 in Azure. Make this a static website (under settings). Make index.html the index document name.
Run: npm run build
Install Azure Storage explorer
Open the new storage you created in step 1 in Azure Storage Explorer, go to Blob Containers, $web
Copy the build files from the ./dist folder to the $web folder.
Open a web browser, enter the endpoint URL.
Im trying to use gitlab runner to test and build my node server but I've run into a small issue when trying to automate tests. In my package.json I have scripts
"scripts": {
"start": "node app.js",
"test-init": "node ./node_modules/jasmine/bin/jasmine.js init",
"test": "set NODE_ENV=Dev&& node ./node_modules/jasmine/bin/jasmine.js"
},
So NODE_ENV=Dev will load a different settings file. One that uses the mongodb url "mongodb://mongo/DBName" and when I run npm test on localhost the server crashes(as its supposed to) because it cant connect to mongo using the Dev setttings file. But when I run the project in GitLab on a runner it wont connect to the db as it uses the non-dev settings file which has a url. Is there any reason in the GitLab-ci why the NODE_ENV is not being set?
Below is my GitLab-ci.yml
image: node:latest
stages:
- build
- test
cache:
paths:
- node_modules/
services:
- mongo
install_dependencies:
stage: build
script:
- npm install
artifacts:
paths:
- node_modules/
test_with_lab:
stage: test
script:
- npm run test-init
- npm test
This is because the docker images run on gitlab are linux based and therefore the set command won't work.
There are two solutions.
Solution 1
Use cross-env npm module as documented here by
doing the following:
Install cross-env like so:
npm install --save-dev cross-env
Then edit your package.json to this:
"scripts": {
"start": "node app.js",
"test-init": "node ./node_modules/jasmine/bin/jasmine.js init",
"test": "cross-env NODE_ENV=Dev node ./node_modules/jasmine/bin/jasmine.js"
},
Solution 2
Just modify the script for linux instead, much quicker and simpler. Here is how it should look. Note npm run test won't work on windows anymore. To avoid this use the first solution above.
"scripts": {
"start": "node app.js",
"test-init": "node ./node_modules/jasmine/bin/jasmine.js init",
"test": "NODE_ENV=Dev node ./node_modules/jasmine/bin/jasmine.js"
},
Note: solution 1 is better in the long run while solution 2 is quick but dirty