I have the following Serverless configuration file:
service: aws-node-scheduled-cron-project
frameworkVersion: '2 || 3'
plugins:
- serverless-plugin-typescript
provider:
name: aws
runtime: nodejs14.x
lambdaHashingVersion: 20201221
region: us-west-2
# Imports all the enviromental variables in `.env.yml` file
environment: ${file(./env.yml)}
functions:
...
env.yml
DATABASE_HOST: 127.0.0.1
DATABASE_USER: root
DATABASE_PASSWORD: your_mysql_root_password
DATABASE_TABLE: laravel
DATABASE_PORT: 3306
DATABASE_USE_SSL: false
NOTIFICATION_SERVICE_URL: http://localhost:4000
Now I would like to change DATABASE_TABLE on the fly when invoking the function locally. I have tried:
export DATABASE_TABLE=1-30-task-template-schedule && npx serverless invoke local --function notifyTodoScheduleFullDay
but the variable DATABASE_TABLE gets overwritten by the one in env.yml. Is it possible to do it via command line?
On your yml file you can declare the table name as ${opt:tablename,'DEFAULT'}.
This line means that you are going to give the name as a parameter from the terminal command like this serverless deploy ... -tablename NAME_OF_THE_TABLE if you do not give it as a parameter it takes the default name you can give it. This can generate a name on the fly.
Related
I have recently started working on a project in which we are using serverless framework. We are using docker to make dev environment easier to setup.
As part of this docker setup we have created a script that creates S3 buckets & tables among other things. We were earlier defining environment variables in the docker-compose file and were accessing them in our nodejs app. For purposes of deployment to other environments our devops team defined a few environment variables in the serverless.yaml file resulting in environment configs being present at two places. We are now planning to move all the environment configs defined in our docker-compose file to serverless.yaml. This works well for our lambdas functions as they are able to read these configs, but it doesn't work for the standalone setup script that we have written.
I tried using this plugin(serverless-scriptable-plugin) in an attempt to be able to read these env variables but still unable to do so.
Here is my serverless.yaml file
service:
name: my-service
frameworkVersion: '2'
configValidationMode: error
provider:
name: aws
runtime: nodejs14.x
region: 'us-east-1'
profile: ${env:PROFILE, 'dev'}
stackName: stack-${self:provider.profile}
apiName: ${self:custom.environment_prefix}-${self:service.name}-my-api
environment: ${self:custom.environment_variables.${self:provider.profile}}
plugins:
- serverless-webpack
- serverless-scriptable-plugin
- serverless-offline-sqs
- serverless-offline
functions:
myMethod:
handler: handler.myHandler
name: ${self:custom.environment_prefix}-${self:service.name}-myHandler
events:
- sqs:
arn:
Fn::GetAtt:
- MyQueue
- Arn
resources:
Resources:
MyQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: ${self:custom.queuename}
Tags:
- Key: product
Value: common
- Key: service
Value: common
- Key: function
Value: ${self:service.name}
- Key: region
Value: ${env:REGION}
package:
individually: true
custom:
webpack:
webpackConfig: ./webpack.config.js
includeModules: true
serverless-offline:
host: 0.0.0.0
port: 3000
serverless-offline-sqs:
apiVersion: '2012-11-05'
endpoint: http://sqs:9324
region: ${self:provider.region}
accessKeyId: root
secretAccessKey: root
skipCacheInvalidation: false
localstack:
stages:
- local
lambda:
mountCode: true
debug: true
environment_prefixes:
staging: staging
production: production
dev: dev
environment_prefix: ${self:custom.environment_prefixes.${self:provider.profile}}
queuename: 'myQueue'
environment_variables:
dev:
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
BUCKET_NAME: my-bucket
S3_URL: http://localstack:4566
SLS_DEBUG: '*'
scriptable:
commands:
setup: node /app/dockerEntrypoint.js
In my DockerFile I try executing script using sls setup CMD. I initially thought using sls command might expose these environment variables defined in serverless.yaml file but it doesn't seem to happen.
Is there any other way this can be achieved? I am trying to access these variables using process.env which works for lambdas but not for my standalone script. Thanks!
There's not a good way to get access to these environment variables if you're running the lambda code as a script.
The Serverless Framework injects these variables into the Lambda function runtime configuration via CloudFormation.
It does not insert/update the raw serverless.yml file, nor does it somehow intercept calls to process.env via the node process.
You'll could use the scriptable plugin to run after package, and then export each variable into your local docker environment. But that seems pretty heavy for the variables in your env.
Instead, you might consider something like dotenv, which will load variables from a .env file into your environment.
There is a serverless-dotenv plugin you could use, and then your script could also call dotenv before running.
In my front end application, I'm storing sensitive information in the environment and using them as following:
const client_secret = process.env.CLIENT_SECRET;
On local development, I use dotenv package to pass in the values in .env file
CLIENT_SECRET=XXXXX
The .env file is not committed.
I use CircleCI for my deployment process, and saved the CLIENT_SECRET value in CircleCI environment variables, but how can I pass into the application?
This is my CircleCI config.yml:
- deploy:
name: Deploy
command: |
ENVIRONMENT=${ENVIRONMENT:=test}
VERSION=`date "+%Y-%m-%dt%H%M"`
if [ "${ENVIRONMENT}" = "production" ]; then
APP_FILE=app-prod.yaml
else
APP_FILE=app.yaml
fi
gcloud app deploy ${APP_FILE} --quiet --version ${VERSION}
I can do this in app.yaml:
env_variables:
NODE_ENV: 'production'
CLIENT_SECRET: XXXXX
But I don't want to include the sensitive information into the .yaml file and commit them. Does anyone know any way I can pass environment values into the application?
I'm using Google Cloud Platform, and gcloud app deploy command doesn't seem to have a flag to include the environment variables.
Using bash script to create a .env file with environment variables manually
app.yaml.sh
#!/bin/bash
echo """
env: flex
runtime: nodejs
resources:
memory_gb: 4.0
disk_size_gb: 10
manual_scaling:
instances: 1
env_variables:
NODE_ENV: 'test'
CLIENT_SECRET: \"$CLIENT_SECRET\"
"""
config.yml
steps:
- checkout
- run:
name: chmod permissions
command: chmod -R 755 ./
- run:
name: Copy across app.yaml config
command: ./app.yaml.sh > ./app.yaml
- deploy:
name: Deploy
command: |
VERSION=`date "+%Y-%m-%dt%H%M"`
gcloud app deploy app.yaml --quiet --version ${VERSION}
Reading about it, it's indeed, as you mentioned, that the only "official" way to set environment variables, it's by setting them in the app.yaml - this article provides more information on it. Considering that, I went to search further and I have found this good question from the Community - accessible here - where some workarounds are provided.
For example, the one that you mentioned, about creating a second file with the values and call it in the app.yaml is a good one. You can them use the .gitignore for the file not exist in the repository - in case you are using one. Another option would be to use Cloud Datastore to store the information and use it in your application. This way, Datastore would keep this information secured and accessible for your application, without becoming public within your App Engine configuration.
I just thought a good idea of adding this information here, with the article and question included, in case you want more information! :)
Let me know if the information helped you!
No cfnRole warned and unnecessary files was created after deploy
Serverless: Safeguards Processing...
Serverless: Safeguards Results:
Summary --------------------------------------------------
passed - no-unsafe-wildcard-iam-permissions
passed - framework-version
warned - require-cfn-role
passed - allowed-runtimes
passed - no-secret-env-vars
passed - allowed-regions
passed - allowed-stages
Details --------------------------------------------------
1) Warned - no cfnRole set
details: http://slss.io/sg-require-cfn-role
Require the cfnRole option, which specifies a particular role for CloudFormation to assume while deploying.
I had been go to the site that write in details.
details: http://slss.io/sg-require-cfn-role
Anyway, I don't know how to fix it.
s_hello.py & s_hello2.py always generated after deploy.
This is my serverless.yaml file
service: myapp
app: sample-app
org: xxx
provider:
name: aws
runtime: python3.7
stage: dev
region: us-east-1
package:
individually: true
functions:
hello:
handler: src/handler/handler.hello
hello2:
handler: src/handler2/handler2.hello2
It's always happen although follow this site .
My Lambda-function will create "s_xxx.py (Where xxx is handler.py file.
I solved this issue creating a cfn-role in AWS IAM following these steps:
Roles -> Create Role -> AWS Service -> Select Cloud Formation from the list
Next: Permisions
You need to choose all the policies you need to deploy your lambda function (S3FullAccess, SQSFullAccess, LambdaFullAccess...)
There's one that it's mandatory AWSConfigRole who allows to serverless framework to get this role.
After setting the role, you need to copy its arn and create behind provider level cfnRole like this:
provider:
name: aws
runtime: python3.7
stage: ${opt:stage, 'dev'}
profile: ${self:custom.deploy-profile.${opt:stage, 'dev'}}
region: us-west-2
environment:
TEMP: "/tmp"
cfnRole: arn:aws:iam::xxxxxxxxxx:role/cfn-Role
That's work for me, I hope to help!
I am creating a CI/CD pipeline in Cloud Build of a very basic Node.js app with deployment to GCP appengine standard.
None-secret environment variables are stored in app.yaml file. But of course I don't want to put my secrets there. In fact I don't want to put them in any file any where (encrypted or not) since this file will end up on the AppEngine instance and can be "viewed" by a "bad admin". There are many samples out there that suggests to encrypt/decrypt complete files (and some times even code) but I don't want to go down that path.
I am looking for a way to set secret environment variables "in memory" as part of the CI/CD pipeline. Anyone?
I added none secrets in the app.yaml file (env_variables) - works fine
Added encrypted secrets into my cloudbuild.yaml file (secrets) - no error
Added secretEnv: into a build steps but value don't end up as process.env.[KEY] in app engine
cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/npm'
args: ['install']
dir: "appengine/hello-world/standard"
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "test-app.yaml"]
dir: "appengine/hello-world/standard"
secretEnv: ['API_KEY', 'API_URL']
secrets:
- kmsKeyName: projects/XXXXXXXX/locations/global/keyRings/customintegrations-secrets/cryptoKeys/integration-secrets
secretEnv:
API_KEY: XXQAoHgKKoHBKOURrUU2RqU+ki8XyqmTjz+ns+MEWp5Kx3hQBpgSQgATFQ5yRdW4m1TLNqNRIdHIqVJi8tn8jFrtlHIEouOzNDe/ASlOT0ZQBfl9Rf7xlvOHAa667poBq2hEoMNvOclxUQ==
API_URL: YYQAoHgKKklo08ZsQF+/8M2bmi9nhWEtb6klyY4rNthUhSIhQ8oSQQATFQ5ywKOxaM/TLwGDmvMtCpl/1stXOOK0kgy42yipYbw/J/QZL68bMat1u4H3Hvp/GMbUVIKEb9jwUtN2xvbL
I was hoping that the secretEnv: ['API_KEY', 'API_URL'] would make the decrypted values accessable in code (process.env.API_KEY) in app engine.
Here is a full tutorial on how to securely store env vars in your cloud build (triggers) settings and import them into your app.
Basically there are three steps:
Add your env vars to the 'variables' section in one of your build trigger settings
Screenshot of where to add variables in build triggers
By convention variables set in the build trigger must begin with an underscore (_)
Configure cloudbuild.yaml (on the second step in the code example) to read in variables from your build trigger, set them as env vars, and write all env vars in a local .env file
Add couldbuild.yaml (below) to your project root directory
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: node:10.15.1
entrypoint: npm
args: ["run", "create-env"]
env:
- 'MY_SECRET_KEY=${_MY_SECRET_KEY}'
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
timeout: "1600s"
Add create-env script to package.json
"scripts": {
"create-env": "printenv > .env"
},
Read env vars from .env to your app (config.js)
Install dotenv package
npm i dotenv -S
Add a config.js to your app
// Import all env vars from .env file
require('dotenv').config()
export const MY_SECRET_KEY = process.env.MY_SECRET_KEY
console.log(MY_SECRET_KEY) // => Hello
Done! Now you may deploy your app by triggering the cloud build and your app will have access to the env vars.
Using secrets from Secrets Manager
Your sample would become:
steps:
- name: 'gcr.io/cloud-builders/npm'
args: ['install']
dir: "appengine/hello-world/standard"
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy", "test-app.yaml"]
dir: "appengine/hello-world/standard"
secretEnv: ['API_KEY', 'API_URL']
availableSecrets:
secretManager:
- versionName: projects/$PROJECT_ID/secrets/<secret name>/versions/latest
env: API_KEY
- versionName: projects/$PROJECT_ID/secrets/<secret name 2>/versions/latest
env: API_URL
Add a cloud trigger step in your cloudbuild.yaml to add place holders in your app.yaml file
steps:
- name: "gcr.io/cloud-builders/gcloud"
secretEnv: ['API_KEY','API_URL']
entrypoint: 'bash' args:
- -c
- |
echo $'\n API_KEY: '$$API_KEY >> app.yaml
echo $'\n API_URL: '$$API_URL >> app.yaml
gcloud app deploy
availableSecrets: secretManager:
- versionName: projects/012345678901/secrets/API_KEY
env: 'API_KEY'
- versionName: projects/012345678901/secrets/API_URL
env: 'API_URL'
look following reference app.yaml
runtime: nodejs
service: serviceone
env_variables:
PROJECT_ID: demo
PORT: 8080
Reference by: https://stackoverflow.com/users/13763858/cadet
Is there anyway to inject environment variables from Cloud Build into the App Engine Standard environment?
I do not want to push my environment variables to GitHub inside the app.yaml or .env. Thus, when Cloud Build pulls and deploys it is missing the .env file and the server is unable to complete some requests.
I am trying to avoid using Datastore as the async nature of Datastore will make the code a lot more messy. I tried to use encrypted secrets found here, but that doesn't seem to work as I added the secrets to app deploy and they do not make their way into the deployment, so I assume this is not the use case for Cloud Build.
I also tried the tutorial here, to import the .env file into App Engine Standard from storage, but since Standard does not have local storage I assume it goes into the void.
So is there anyway to inject the .env into App Engine Standard environment without using Datastore, or committing app.yaml or .env to change control? Potentially using Cloud Build, KMS, or some type of storage?
Here is what I tried for cloudbuild.yaml:
steps:
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
secretEnv: ['SECRET1', 'SECRET2', 'SECRET3', 'SECRET4', 'SECRET5']
timeout: "1600s"
secrets:
- kmsKeyName: projects/<Project-Name>/locations/global/keyRings/<Key-Ring-Name>/cryptoKeys/<Key-Name>
secretEnv:
SECRET1: <encrypted-key-base64 here>
SECRET2: <encrypted-key-base64 here>
SECRET3: <encrypted-key-base64 here>
SECRET4: <encrypted-key-base64 here>
SECRET5: <encrypted-key-base64 here>
Here is a tutorial on how to securely store env vars in your cloud build (triggers) settings and import them into your app.
Basically there are three steps:
Add your env vars to the 'variables' section in one of your build trigger settings
Screenshot of where to add variables in build triggers
By convention variables set in the build trigger must begin with an underscore (_)
Configure cloudbuild.yaml (on the second step in the code example) to read in variables from your build trigger, set them as env vars, and write all env vars in a local .env file
Add couldbuild.yaml (below) to your project root directory
steps:
- name: node:10.15.1
entrypoint: npm
args: ["install"]
- name: node:10.15.1
entrypoint: npm
args: ["run", "create-env"]
env:
- 'MY_SECRET_KEY=${_MY_SECRET_KEY}'
- name: "gcr.io/cloud-builders/gcloud"
args: ["app", "deploy"]
timeout: "1600s"
Add create-env script to package.json
"scripts": {
"create-env": "printenv > .env"
},
Read env vars from .env to your app (config.js)
Install dotenv package
npm i dotenv -S
Add a config.js to your app
// Import all env vars from .env file
require('dotenv').config()
export const MY_SECRET_KEY = process.env.MY_SECRET_KEY
console.log(MY_SECRET_KEY) // => Hello
Done! Now you may deploy your app by triggering the cloud build and your app will have access to the env vars.
I have another solution, if someone is still interested in this. This should work on all languages, because environment variables are added directly into app.yaml file
Add substitution variable in build trigger (as described in this answer).
Add environment variables to app.yaml in a way they can be easily substituted with build trigger variables. Like this:
env_variables:
SECRET_KEY: %SECRET_KEY%
Add a step in cloudbuild.yaml to substitute all %XXX% variables inside app.yaml with their values from build trigger.
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
args:
- '-c'
- |
sed -i 's/%SECRET_KEY%/'${_SECRET_KEY}'/g' app.yaml
gcloud app deploy app.yaml
The highfivebrian answer is great, but I'm adding my slightly different solution.
1). In the root project folder we need the cloudbuild.yaml file but I'll call it buildsetttings.yaml, because
first one name have a problem
In buildsetttings.yaml I added this code:
steps:
- name: node
entrypoint: npm
args: ['install']
- name: node
entrypoint: npm
env:
- 'DB_URL=${_DB_URL}'
- 'SENDGRID_API_KEY=${_SENDGRID_API_KEY}'
- 'CLIENT_ID=${_CLIENT_ID}'
args: ['run', 'create-app-yaml']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
buildsetttings.yaml will be create app.yaml file in the Cloud Build, using a npm create-app-yaml command.
Tip: app.yaml file we will then use to deploy our app to GCP App Engine.
2). In the root folder(near buildsetttings.yaml) we need to create create-app-yaml.js which will run in Cloud Build after it is called from buildsetttings.yaml.
In buildsetttings.yaml I added this code:
require('dotenv').config();
const fs = require('fs');
const appYamlContent = `runtime: nodejs14
env_variables:
DB_URL: ${process.env.DB_URL}
SENDGRID_API_KEY: ${process.env.SENDGRID_API_KEY}
CLIENT_ID: ${process.env.CLIENT_ID}`;
fs.writeFileSync('./app.yaml', appYamlContent);
This code using a npm package dotenv(add it to package.json) and get variables from Cloud Build Trigger Variables and create with they app.yaml file.
3). app.yaml file was created in the Cloud build and our last step(name: 'gcr.io/cloud-builders/gcloud') in the buildsetttings.yaml, using app.yaml file, deploy the project to the Google Cloud App Engine.
Success!
In short, it works like this: buildsetttings.yaml run "create-app-yaml.js" in the Cloud Build, after which dynamically creates an app.yaml file by adding variables from Cloud Build Trigger Variables, then makes a deployment in the App Engine.
Notes:
Delete the file app.yamlin from you project, because it will be create dynamically in the Cloud Build. Also delete cloudbuild.yaml file, because instead we use buildsetttings.yaml.
package.json:
Cloud Build Trigger Variables:
As of 2020/11/13. It seem like .env will work only at that step and in the next step an invisible .env will no longer there.
If you get stuck do try consume that printed .env it in 1 step like this ...
in cloudbuild.yaml
# [START cloudbuild_yarn_node]
steps:
# Install
- name: node
entrypoint: yarn
args: ["install"]
# Build
- name: node
entrypoint: yarn
env:
- "FOO=${_FOO}"
args: ["env-build"]
and in package.json add this
{
"scripts": {
"env-build": "printenv > .env && yarn build",
}
}
in index.js
require('dotenv').config();
console.log(process.env.FOO);
Took me an hour to figure this out.
First, I created secret using gcp secret manager and uploaded my env file there.
Second, I called the secret in cloudbuild.yaml on run time and created a file with name of '.env' using echo.
Example
steps:
- id: "Injecting ENV"
name: 'gcr.io/cloud-builders/gcloud'
entrypoint: bash
args:
- '-c'
- |
echo $$ENV > .env
secretEnv: ['ENV']
availableSecrets:
- versionName: projects/<Project-Name>/secrets/environment-variables/versions/1
env: 'ENV'
timeout: 900s
Based on your preferences that you have highlighted (Cloud Build, KMS). The Google Secrets link that you had mentioned involves storing sensitive data at build or runtime using Cloud KMS: KeyRing and CryptoKey. However, Google offers other Secret Management Solutions using Cloud KMS as well.
Here are a couple of other options you can use while storing Secrets:
Option 1 : You can store Secrets in code that are encrypted with a key from Cloud KMS.
(This is typically used by encrypting your secret at the application layer.)
Benefit: Provides a layer of security from insider threats because it restricts access to the code with a corresponding key.
[You can find some additional information about these options on the Google Documentation here.]
Option 2: You can Store Secrets inside a Google Storage Bucket where your data is at rest encryption. (Similar to option 1 this has the ability to limit access to secrets to a small group of Developers.)
Benefit: Storing your secrets in a separate location ensures that if a breach of your code repository has occurred, your secrets may still be protected.)
[Note: Google recommends that you use two projects for proper separation of duties. One project will use Cloud KMS to manage the keys and the other project will use Cloud Storage to store the secrets.]
If the options listed above still do not meet your needs, I have found a StackOverflow question that shares a similar objective as your project. (i.e: Storing environment variables in GAE without Datastore)
The solution provided on this link illustrates the use of storing keys in a client_secrets.json file that gets excluded when uploading to git by listing it in .gitignore. You can find some Google examples (Python) of usage here.