I am using the serverless framework for running lambda functions on AWS.
In my serverless.yml there are environment variables that are fetched from SSM.
When I write integration tests for the code, I need the code to have the environment variables and I can't find a good way to do this.
I don't want to duplicate all the variables definitions just for the tests, they are already defined in the serverless.yml. Also, some are secrets and I can't commit them to source conrol, so I would have to also repeat them in the ci environment.
Tried using the serverless-jest-plugin but it is not working and not well maintained.
Ideas I had for solutions:
Make the tests exec sls invoke - this will work but would mean that the code cannot be debugged, I won't know the test coverage, and it will be slow.
Parse the serverless.yml myself and export the env variables - possible but rewriting the logic of pulling the SSM variables just for tests seems wrong.
Any ideas?
The solution we ended up using is a serverless plugin called serverless-export-env.
After adding this plugin you can run serverless export-env to export all the resolved environment variables to an .env file. This resolves ssm parameters correctly and made integration testing much simpler for us.
BTW, to get the environment variables set from the .env file use the the dotenv npm package.
Credit to grishezz for finding the solution
You can run node with --require option to inject .env file to a serverless command.
Create .env at the project root with package.json, and list variables in .env.
Install serverless and dotenv in the project by yarn add -D serverless dotenv.
Run a command like node -r dotenv/config ./node_modules/.bin/sls invoke.
Then, you can get environment variables in the handler process.env.XXX.
Are you looking to do mocked unit tests, or something more like integration tests?
In the first case, you don't need real values for the environment variables. Mock your database, or whatever requires environment variables set. This is actually the preferable way because the tests will run super quickly with proper mocks.
If you are actually looking to go with end-to-end/integration kind of approach, then you would do something like sls invoke, but from jest using javascript. So, like regular network calls to your deployed api.
Also, I would recommend not to store keys in serverless.yml. Try the secret: ${env:MY_SECRET} syntax instead (https://serverless.com/framework/docs/providers/aws/guide/variables#referencing-environment-variables), and use environment variables instead. If you have a ci/cd build server, you can store your secrets there.
After searching I did my custom solution
import * as data from './secrets.[stage].json'
if( process.env.NODE_ENV === 'test'){
process.env = Object.assign( data, process.env );
}
//'data' is the object that has the Serverless environment variables
The SLS environment variables in my case at the file secrets.[stage].json
Serverless.yml has
custom:
secrets: ${file(secrets.[stage].json)}
Related
We currently have multiple environments and we would like to init datadogRum only for production.
In the _app.tsx I tried something like this :
import { datadogRum } from '#datadog/browser-rum';
if (process.env.NODE_ENV=== 'production') {
datadogRum.init({...});
datadogRum.startSessionReplayRecording();
}
But this is not working because env variables are not available here ...
Do you have any work around for this ?
I succeed using #bwest comment.
I used nextJs getInitialProps in order to transmit my env variable.
And I put my datadog init code in a useEffect(() => {...} , []) hook to only execute the code once.
With nextjs environment variables can be available to front-end code.
They will be interpolated inside your code using some webpack plugin (something like EnvironmentPlugin or DefinePlugin).
From this Nextjs documentation:
Next.js comes with built-in support for environment variables, which allows you to do the following:
Use .env.local to load environment variables
Expose environment variables to the browser by prefixing with NEXT_PUBLIC_
Suppose I have a .env file like these:
#dev variable
PORT=3000
#production
PORT=3030
And I get these variables using process.env, how can I manage to sometimes use the dev variable and other times the Production variable
You can create multiple .env files like .dev.env, .prod.env, and load them based on NODE_ENV. using this
Storing configuration in environment variables is the way to go, and exactly what is recommended by the config in the 12-Factor App, so you're already starting with the right foot.
The values of these variables should not be stored with the code, except maybe the ones for your local development environment, which you can even assume as the default values:
port = process.env.PORT || '3000';
For all other environments, the values should be stored in a safe place like Vault or AWS Secrets Manager, and then are only handled by your deployment pipeline. Jenkins, for example, has a credentials plugin to handle that.
I am trying to publish Gatsbyjs by Azure Static web app.
I have a plugin (gatsby-source-contentful).
I need to pass variables like:
{
resolve: `gatsby-source-contentful`,
options: {
spaceId: process.env.CONTENTFUL_SPACE_ID,
accessToken: process.env.CONTENTFUL_ACCESS_TOKEN,
},
},
Error:
Running 'npm run build'...
> gatsby-starter-default#0.1.0 build /github/workspace
> gatsby build
success open and validate gatsby-configs - 0.021s
error Invalid plugin options for "gatsby-source-contentful":
- "accessToken" is required
- "spaceId" is required
not finished load plugins - 0.905s
Where can I pass this?
Thanks.
For Azure Static Web Apps there is two ways to set environment variables one for front-end and one for back-end scenarios.
Since you are using Gatsby, I guess its safe to assume you are building your front-end. For that you will need to add the environment variables on your build configuration (azure-static-web-apps-.yml).
Like so:
env: # Add environment variables here
CONTENTFUL_SPACE_ID: <your-id>
Here is the link for that in documenation.
Not to be confused with this one which is used for defining backend environment variables.
They are called environment variables. They are intended to store sensitive data such as tokens, identifiers, etc, and they shouldn't be pushed in your repository, so you should ignore them (in your .gitignore file).
By default, Gatsby creates 2 environments without noticing you, one for each compilation method:
gatsby develop: uses .env.development
gatsby build: uses .env.production
Note: you can change this behavior if needed to add your own environments using NODE_ENV custom commands.
So, to pass your data to your gatsby-config.js you just need to create two files (.env.development and .env.production) at the root of your project. Then, add the following snippet at the top of your gatsby-config.js:
require("dotenv").config({
path: `.env.${process.env.NODE_ENV}`,
})
Note: dotenv is already a dependency of Gatsby so you don't need to install it again
This will tell Gatsby where to take the environment variables.
You just remain to populate both environment files. Look for the credentials in Contentful and add them in the files using the sane naming than you've set in your gatsby-config.js:
CONTENTFUL_SPACE_ID=123456789
CONTENTFUL_ACCESS_TOKEN=123456789
Keep also in mind that when dealing with Azure, Netlify, AWS, or similar CI/CD tools, you'll need to provide to the server the same environment files to avoid a code-breaking when pushing the changes.
So I have the simplest example on a node machine running with a react-redux app with webpack (Though I don't think any of this matters for the issue expect it being on nodejs).
Specific calls get a value pack:
console.log(process.env.NODE_ENV); // output: 'development'
General calls get nothing back:
console.log(process.env); // output: {}
What am I missing here?
Addition info the might be relevant:
I am using dotenv for the test environment.
I am using dotenv-webpack for the development environment.
I am not using neither of those for the production environment deployed to Heroku
The problem persists on all environments.
The issue with process.env variable being empty in browser is because browser doesn't have real access to the process of the node.js. It's run inside the browser though.
Usage of process.env.ANYTHING is usually achieved by plugins like https://webpack.js.org/plugins/define-plugin/ which just simply replace any occurrence of process.env.ANYTINHG with env variable during BUILD time. It really does just simple str.replace(/process.env.ANYTING/value/) this needs to be done during build time as once you output dist bundle.js you don't have access to the ENV variables.
Replacing during build time
Therefore you need to be sure that when you are producing production build e.g with yarn build you are using webpack.DefinePlugin and replacing those process.env calls with current ENV values. They can't be injected in runtime.
Injecting in runtime
When you need to access env variables in runtime it's basically impossible in JavaScript in browser. There are some sort of hacks for example for NGINX which can serialize current env variables to the window.ENV variable and in your app you will not use process.env but window.ENV. So you need to either have ENV variables available while you are building the application or build mechanism which will dynamically output current ENV as json to window and access with react. If you are using docker it can be done with ENTRYPOINT otherwise you need some bash script which will always output current ENV variables as JSON to the index.html of your app
Actually I have a nodejs express app with its config file for params like host, port, JWT token, DB params and more.
The question is if it could have sense to keep those params directly on environment variables (whitout any config file) and acces them without the need of do the "require" for config in all components and modules.
All examples I see uses a config file, probably something about security or memory?
config file is usually for setting the default values for your environment variables,
which is needed when you are writing the test cases and need to use default values or mock values,
and also you will have all the env variables at one place which is better management.
so if you have an environment variable x,
in config file you can keep it as
config.x = process.env.x || 'defaultVale or mockValue'
A config file lets your very quickly set the entire environment of a machine - eg S3 buckets, API urls, access keys, etc. If you separate these into separate process.env.VARIABLE then you would need to set each of these...for which you would likely make a script...and now you have an environment file again!
To access environment variables you can use process.env.VARIABLE in your nodejs code (is always a string), as long as the variable is set before the process is started.
Another possibility is using an .env files in nodejs. I think you have to npm install dotenv in your application. Ideally different instances (dev, prod....) have its own .env file, and you dont have to call require("dotenv") every time if you want to access the environment variable. Call it in the very beginning i.e) in app.js and you can access the environment variable in any of the sub-files.