I've recently started using AdonisJS for API development.
I'm using AWS Elastic Beanstalk together with AWS CodeCommit and AWS CodePipeline to deploy new code on each git push.
Since .env file is not present in git repository, I've added env variables through Elastic Beanstalk web console.
But deployment failed when I tried to run node ace migration:run command.
Activity execution failed, because:
Error: ENOENT: no such file or directory, open '/tmp/deployment/application/.env'
1 Env.load
/tmp/deployment/application/node_modules/#adonisjs/framework/src/Env/index.js:110
2 new Env
/tmp/deployment/application/node_modules/#adonisjs/framework/src/Env/index.js:42
3 Object.app.singleton [as closure]
/tmp/deployment/application/node_modules/#adonisjs/framework/providers/AppProvider.js:29
4 Ioc._resolveBinding
/tmp/deployment/application/node_modules/#adonisjs/fold/src/Ioc/index.js:231
5 Ioc.use
/tmp/deployment/application/node_modules/#adonisjs/fold/src/Ioc/index.js:731
6 AppProvider.boot
/tmp/deployment/application/node_modules/#adonisjs/framework/providers/AppProvider.js:337
7 _.filter.map
/tmp/deployment/application/node_modules/#adonisjs/fold/src/Registrar/index.js:147
8 arrayMap
/tmp/deployment/application/node_modules/lodash/lodash.js:639
(ElasticBeanstalk::ExternalInvocationError)
Then I've tried to add ENV_SILENT=true flag before each command as stated in AdonisJS documentation. But that did not help.
So then, I've tried to upload .env file on S3 bucket, and copy its contents during deployment.
But it seems it does not work, since I'm getting the same error (no .env file).
These are my 2 config files from .ebextensions folder
01_copy_env.config (I'm using x-xxxxxxxxxxxx here for security)
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Auth:
type: "s3"
buckets: ["elasticbeanstalk-us-east-x-xxxxxxxxxxxx"]
roleName:
"Fn::GetOptionSetting":
Namespace: "aws:autoscaling:launchconfiguration"
OptionName: "IamInstanceProfile"
DefaultValue: "aws-elasticbeanstalk-ec2-role"
files:
"/tmp/deployment/application/.env":
mode: "000755"
owner: root
group: root
authentication: "S3Auth"
source: https://elasticbeanstalk-us-east-x-xxxxxxxxxxxx.s3.us-east-2.amazonaws.com/variables.txt
02_init.config
container_commands:
01_node_binary:
command: "ln -sf `ls -td /opt/elasticbeanstalk/node-install/node-v10* | head -1`/bin/node /bin/node"
leader_only: true
02_migration:
command: "node ace migration:run"
03_init_seed:
command: "node ace seed"
The only time the whole thing works is when I add .env file to git and deploy it with the rest of the code. But that is not the way to go, so if anyone knows a solution to my problem I would really appreciate it. Thanks!
Add new variable ENV_SILENT = true on your global variables (Elastic Beanstalk)
Adonis documentation
Related
Can someone please provide information on how to deploy Strapi to AWS Elastic Beanstalk?
I have found many resources on how to deploy Strapi on many other different platforms such as Digital Ocean and Heroku, but I am very curious about deploying Strapi to Elastic Beanstalk. Is that possible and how can I do with that?
First you need an EBS application & environment (Web Server) running Node version 12 (as of now). You'll also need to change the package.json in your Strapi project and update the engines part, like this (major version must match EBS Node version):
"engines": {
"node": "12.X.Y", // minor (X) & patch (Y) versions are up to you
...
},
You must switch your project to use NPM instead of Yarn (EBS currently only supports NPM out-of-the-box), to do this I recommend a tool like synp.
Then create a Procfile which will describe how you want EBS to run your app:
web: npm run start
Then to deploy manually, you could first (in the project root) run npm install, then npm run build to build the Strapi Admin (React) application. After the Strapi Admin has been built, make sure to remove the node_modules folder, because EBS will automatically install dependencies for you. (*)
Last step is to zip the whole project (again, in project root, run: zip -r application.zip .), upload the zip to AWS EBS & let it do it's magic. Hopefully it should then install dependencies and start your application automatically.
Side note: When using some specific dependencies in your project (one example is sharp), the EBS may fail to install your dependencies, to fix this, add a .npmrc file to your project root with the following contents:
unsafe-perm=true
Side note #2: You need to set some environment variables in the EBS configuration panel in order for Strapi to work (like database credentials etc.).
(*) Although you could include node_modules in your app and zip it and upload to EBS (which could work), sometimes zipping node_modules may break some dependencies, so I recommend removing it and let EBS install dependencies for you.
If you want to deploy Strapi on Elastic Beanstalk with AWS CodePipeline the following steps worked for me:
Navigate to Elastic Beanstalk and Create a new application with the corresponding Node version for the application
Platform: Node.js
Platform Branch: Node.js 12 funning on 64bit Amazon Linux 2
Platform Version: 5.4.6
Select Sample Application to start (we will connect this to AWS CodePipeline in a later step)
Set up the code repository on GitHub (if one doesn’t already exist)
Navigate to AWS CodeBuild and select create build project
In the Source Section connect to your Github Repository
In the Environment Section select the following configurations
Environment Image: Manage image
Operating System: Ubuntu
Runtimes: Standard
Image: aws/codebuild/standard:5.0
Role name: AWS will create one for you
Buildspec
Select “Use a buildspec file” - We will have to add a buildspec.yml file to our project in step 4
Leave the other default settings and continue with Create build project
Update your Strapi Code
Add the Procfile, .npmrc, and update the package.json file accordingly as suggested by Richárd Szegh
Add the .ebignore file for Elastic Beanstalk
Add the following buildspec.yml and .ebignore file into your project
buildspec.yml
version: 0.2
phases:
install:
runtime-versions:
nodejs: 12
pre_build:
commands:
- npm install
build:
commands:
- npm run build
post_build:
commands:
- rm -rf node_modules
artifacts:
files:
- '**/*'
.ebignore
# dependencies
node_modules/
# repository/project stuff
.idea/
.git/
.gitlab-ci.yml
README.md
# misc
.DS_Store
# debug
npm-debug.log*
yarn-debug.log*
yarn-error.log*
# local env files
.env.local
.env.development.local
.env.test.local
.env.production.local
# non prod env files
.env.development
.env.test
Navigate to AWS CodePipeline
Click Create pipeline
Pipeline Settings
Pipeline name: Name accordingly
Service role: New Service Role
Role name: AWS will create a default name for you
Source Stage:
Connect to your repository in this case GitHub (Version 2)
Connect To Github
Repository Name: select repository accordingly
Branch Name: select branch accordingly
Build Stage:
Build Provider: AWS CodeBuild
Region: Select the region where the initial created the CodeBuild project Step 3
Project Name: Select the CodeBuild project you created
Environment Variables: Add any environment variables
Deploy Stage:
Deploy Provider: AWS Elastic Beanstalk
Region: Select the region where you initially created the EB
Application name: Select the Application Name you created in Step 1
Environment name: Select the Environment Name you created in Step 1
Create pipeline
Now you can push changes to the repository and CodePipeline will pick up the changes, run the build, and deploy to Elastic Beanstalk
This seem to work for me, AWS Elastic Beanstalk t3.small instance, I wanted to use Free tier t3.micro but it didn't work for me, it seems t3.micro 1GB memory was not enough, t3.small had 2GB memory.
1)
added deploy to scripts
package.json
"scripts": {
"deploy": "NODE_ENV=production npm run build && NODE_ENV=production npm run start"
},
create file .npmrc and add:
unsafe-perm=true
Create Procfile and add:
web: npm run deploy
I used AWS Pipeline to trigger EB deploy when I push update to Bitbucket (I can also disable Pipeline if not used to save $$$)
I used AWS RDS PostgreSQL Free tier, the latest version of PostgreSQL didn't have the Free tier version but previous version did have the Free tier option checkbox to select it
I used AWS S3 bucket to store images
In my front end application, I'm storing sensitive information in the environment and using them as following:
const client_secret = process.env.CLIENT_SECRET;
On local development, I use dotenv package to pass in the values in .env file
CLIENT_SECRET=XXXXX
The .env file is not committed.
I use CircleCI for my deployment process, and saved the CLIENT_SECRET value in CircleCI environment variables, but how can I pass into the application?
This is my CircleCI config.yml:
- deploy:
name: Deploy
command: |
ENVIRONMENT=${ENVIRONMENT:=test}
VERSION=`date "+%Y-%m-%dt%H%M"`
if [ "${ENVIRONMENT}" = "production" ]; then
APP_FILE=app-prod.yaml
else
APP_FILE=app.yaml
fi
gcloud app deploy ${APP_FILE} --quiet --version ${VERSION}
I can do this in app.yaml:
env_variables:
NODE_ENV: 'production'
CLIENT_SECRET: XXXXX
But I don't want to include the sensitive information into the .yaml file and commit them. Does anyone know any way I can pass environment values into the application?
I'm using Google Cloud Platform, and gcloud app deploy command doesn't seem to have a flag to include the environment variables.
Using bash script to create a .env file with environment variables manually
app.yaml.sh
#!/bin/bash
echo """
env: flex
runtime: nodejs
resources:
memory_gb: 4.0
disk_size_gb: 10
manual_scaling:
instances: 1
env_variables:
NODE_ENV: 'test'
CLIENT_SECRET: \"$CLIENT_SECRET\"
"""
config.yml
steps:
- checkout
- run:
name: chmod permissions
command: chmod -R 755 ./
- run:
name: Copy across app.yaml config
command: ./app.yaml.sh > ./app.yaml
- deploy:
name: Deploy
command: |
VERSION=`date "+%Y-%m-%dt%H%M"`
gcloud app deploy app.yaml --quiet --version ${VERSION}
Reading about it, it's indeed, as you mentioned, that the only "official" way to set environment variables, it's by setting them in the app.yaml - this article provides more information on it. Considering that, I went to search further and I have found this good question from the Community - accessible here - where some workarounds are provided.
For example, the one that you mentioned, about creating a second file with the values and call it in the app.yaml is a good one. You can them use the .gitignore for the file not exist in the repository - in case you are using one. Another option would be to use Cloud Datastore to store the information and use it in your application. This way, Datastore would keep this information secured and accessible for your application, without becoming public within your App Engine configuration.
I just thought a good idea of adding this information here, with the article and question included, in case you want more information! :)
Let me know if the information helped you!
Is there anyway to inject environment variables from Cloud Build into the App Engine Standard environment?
I do not want to push my environment variables to GitHub inside the app.yaml or .env. Thus, when Cloud Build pulls and deploys it is missing the .env file and the server is unable to complete some requests.
I am trying to avoid using Datastore as the async nature of Datastore will make the code a lot more messy. I tried to use encrypted secrets found here, but that doesn't seem to work as I added the secrets to app deploy and they do not make their way into the deployment, so I assume this is not the use case for Cloud Build.
I also tried the tutorial here, to import the .env file into App Engine Standard from storage, but since Standard does not have local storage I assume it goes into the void.
So is there anyway to inject the .env into App Engine Standard environment without using Datastore, or committing app.yaml or .env to change control? Potentially using Cloud Build, KMS, or some type of storage?
Here is what I tried for cloudbuild.yaml:
steps:
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
secretEnv: ['SECRET1', 'SECRET2', 'SECRET3', 'SECRET4', 'SECRET5']
timeout: "1600s"
secrets:
- kmsKeyName: projects/<Project-Name>/locations/global/keyRings/<Key-Ring-Name>/cryptoKeys/<Key-Name>
secretEnv:
SECRET1: <encrypted-key-base64 here>
SECRET2: <encrypted-key-base64 here>
SECRET3: <encrypted-key-base64 here>
SECRET4: <encrypted-key-base64 here>
SECRET5: <encrypted-key-base64 here>
Here is a tutorial on how to securely store env vars in your cloud build (triggers) settings and import them into your app.
Basically there are three steps:
Add your env vars to the 'variables' section in one of your build trigger settings
Screenshot of where to add variables in build triggers
By convention variables set in the build trigger must begin with an underscore (_)
Configure cloudbuild.yaml (on the second step in the code example) to read in variables from your build trigger, set them as env vars, and write all env vars in a local .env file
Add couldbuild.yaml (below) to your project root directory
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: node:10.15.1
entrypoint: npm
args: ["run", "create-env"]
env:
- 'MY_SECRET_KEY=${_MY_SECRET_KEY}'
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
timeout: "1600s"
Add create-env script to package.json
"scripts": {
"create-env": "printenv > .env"
},
Read env vars from .env to your app (config.js)
Install dotenv package
npm i dotenv -S
Add a config.js to your app
// Import all env vars from .env file
require('dotenv').config()
export const MY_SECRET_KEY = process.env.MY_SECRET_KEY
console.log(MY_SECRET_KEY) // => Hello
Done! Now you may deploy your app by triggering the cloud build and your app will have access to the env vars.
I have another solution, if someone is still interested in this. This should work on all languages, because environment variables are added directly into app.yaml file
Add substitution variable in build trigger (as described in this answer).
Add environment variables to app.yaml in a way they can be easily substituted with build trigger variables. Like this:
env_variables:
SECRET_KEY: %SECRET_KEY%
Add a step in cloudbuild.yaml to substitute all %XXX% variables inside app.yaml with their values from build trigger.
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
args:
- '-c'
- |
sed -i 's/%SECRET_KEY%/'${_SECRET_KEY}'/g' app.yaml
gcloud app deploy app.yaml
The highfivebrian answer is great, but I'm adding my slightly different solution.
1). In the root project folder we need the cloudbuild.yaml file but I'll call it buildsetttings.yaml, because
first one name have a problem
In buildsetttings.yaml I added this code:
steps:
- name: node
entrypoint: npm
args: ['install']
- name: node
entrypoint: npm
env:
- 'DB_URL=${_DB_URL}'
- 'SENDGRID_API_KEY=${_SENDGRID_API_KEY}'
- 'CLIENT_ID=${_CLIENT_ID}'
args: ['run', 'create-app-yaml']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
buildsetttings.yaml will be create app.yaml file in the Cloud Build, using a npm create-app-yaml command.
Tip: app.yaml file we will then use to deploy our app to GCP App Engine.
2). In the root folder(near buildsetttings.yaml) we need to create create-app-yaml.js which will run in Cloud Build after it is called from buildsetttings.yaml.
In buildsetttings.yaml I added this code:
require('dotenv').config();
const fs = require('fs');
const appYamlContent = `runtime: nodejs14
env_variables:
DB_URL: ${process.env.DB_URL}
SENDGRID_API_KEY: ${process.env.SENDGRID_API_KEY}
CLIENT_ID: ${process.env.CLIENT_ID}`;
fs.writeFileSync('./app.yaml', appYamlContent);
This code using a npm package dotenv(add it to package.json) and get variables from Cloud Build Trigger Variables and create with they app.yaml file.
3). app.yaml file was created in the Cloud build and our last step(name: 'gcr.io/cloud-builders/gcloud') in the buildsetttings.yaml, using app.yaml file, deploy the project to the Google Cloud App Engine.
Success!
In short, it works like this: buildsetttings.yaml run "create-app-yaml.js" in the Cloud Build, after which dynamically creates an app.yaml file by adding variables from Cloud Build Trigger Variables, then makes a deployment in the App Engine.
Notes:
Delete the file app.yamlin from you project, because it will be create dynamically in the Cloud Build. Also delete cloudbuild.yaml file, because instead we use buildsetttings.yaml.
package.json:
Cloud Build Trigger Variables:
As of 2020/11/13. It seem like .env will work only at that step and in the next step an invisible .env will no longer there.
If you get stuck do try consume that printed .env it in 1 step like this ...
in cloudbuild.yaml
# [START cloudbuild_yarn_node]
steps:
# Install
- name: node
entrypoint: yarn
args: ["install"]
# Build
- name: node
entrypoint: yarn
env:
- "FOO=${_FOO}"
args: ["env-build"]
and in package.json add this
{
"scripts": {
"env-build": "printenv > .env && yarn build",
}
}
in index.js
require('dotenv').config();
console.log(process.env.FOO);
Took me an hour to figure this out.
First, I created secret using gcp secret manager and uploaded my env file there.
Second, I called the secret in cloudbuild.yaml on run time and created a file with name of '.env' using echo.
Example
steps:
- id: "Injecting ENV"
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
args:
- '-c'
- |
echo $$ENV > .env
secretEnv: ['ENV']
availableSecrets:
- versionName: projects/<Project-Name>/secrets/environment-variables/versions/1
env: 'ENV'
timeout: 900s
Based on your preferences that you have highlighted (Cloud Build, KMS). The Google Secrets link that you had mentioned involves storing sensitive data at build or runtime using Cloud KMS: KeyRing and CryptoKey. However, Google offers other Secret Management Solutions using Cloud KMS as well.
Here are a couple of other options you can use while storing Secrets:
Option 1 : You can store Secrets in code that are encrypted with a key from Cloud KMS.
(This is typically used by encrypting your secret at the application layer.)
Benefit: Provides a layer of security from insider threats because it restricts access to the code with a corresponding key.
[You can find some additional information about these options on the Google Documentation here.]
Option 2: You can Store Secrets inside a Google Storage Bucket where your data is at rest encryption. (Similar to option 1 this has the ability to limit access to secrets to a small group of Developers.)
Benefit: Storing your secrets in a separate location ensures that if a breach of your code repository has occurred, your secrets may still be protected.)
[Note: Google recommends that you use two projects for proper separation of duties. One project will use Cloud KMS to manage the keys and the other project will use Cloud Storage to store the secrets.]
If the options listed above still do not meet your needs, I have found a StackOverflow question that shares a similar objective as your project. (i.e: Storing environment variables in GAE without Datastore)
The solution provided on this link illustrates the use of storing keys in a client_secrets.json file that gets excluded when uploading to git by listing it in .gitignore. You can find some Google examples (Python) of usage here.
I am deploying a new Node.js application to Google App Engine. I use Docker for local development so I have the following files in my root folder:
Dockerfile
.dockerignore
docker-compose.yml
docker-compose.debug.yml
When I run gcloud app deploy, I get the following error:
ERROR: (gcloud.app.deploy) There is a Dockerfile in the current directory, and the runtime field in /Users/Nag/Code/project/web-service/app.yaml is currently set to [runtime: nodejs]. To use your Dockerfile to build a custom runtime, set the runtime field to [runtime: custom]. To continue using the [nodejs] runtime, please remove the Dockerfile from this directory.
As per the Google App Engine app.yaml settings, I can include a skip_files key to ignore certain files. I included skip_files key in my app.yaml like this but it still throws the same error:
# [START app_yaml]
runtime: nodejs
env: flex
skip_files:
- /docker/i
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
env_variables:
NODE_ENV: development
# [END app_yaml]
I would like to continue using the nodejs runtime when the application is running on App Engine and use Docker only on my local machine. I can't Git ignore the Docker files because other developers on the team need access to it in the repository. Can someone please help me understand why App Engine still recognizes my Docker files even though I am skipping them? Thanks.
I am trying to deploy a node.js app to appEngine but he doesn't seem to like my app.yaml.
D:\projects\Personal\Project>gcloud app deploy
ERROR: (gcloud.app.deploy) An error occurred while parsing file: [D:\projects\Personal\Project\app.yaml]
No URLMap entries found in application configuration
in "D:\projects\Personal\Project\app.yaml", line 10, column 12
Here's the app.yaml:
runtime: nodejs
env: standard
skip_files:
- ^node_modules$
env_variables:
USER: 'dbUser'
PASSWORD: 'DBPass'
DB: 'URI to DB'
Am I missing something here?
Could it be that it's because of my folder structure?
I have following structure:
- client
- server
---- server.js
- app.yaml
So the app.yaml is not in the server folder as otherwise it doesn't include the client folder...
Now that nodejs8 is supported in the standard environment (still in beta at this time), you must update your local gcloud environment to be able to deploy. Updating your gcloud components will make this warning go away.
Run
gcloud components update
Update your app.yaml file
See Google's github for the most basic example required: https://github.com/GoogleCloudPlatform/nodejs-docs-samples/blob/master/appengine/building-an-app/build/app.yaml
Update your package.json
Specify the appropriate compatible node version, example:
"engines": {
"node": ">=8.0.0"
},
Full Tutorial
On building a nodejs8 app in the App Engine standard environment: https://cloud.google.com/appengine/docs/standard/nodejs/building-app/
The standard environment deploys significantly faster than the flex environment.
There is no standart environment with nodejs in AppEngine.
You must use env: flex.
see this.