I have a small Sinatra app I'm running on Heroku that uses a single admin password, plus a couple API authentication keys.
Where's the best place to store these things? Do I put them in environment variables, and use
heroku config:add ADMIN_PASSWORD=foobar
? Or do I use a config file that contains them, and I simply don't commit the config file?
I stick API keys and that sort of thing in a config yaml, like so
development:
twitter_api_key: stringstringstring
chunky: bacon
production:
twitter_api_key: gnirtsgnirtsgnirts
foo: bar
then use Sinatra's builtin set to handle the data.
configure do
yaml = YAML.load_file(settings.config + "/config.yaml")[settings.environment.to_s]
yaml.each_pair do |key, value|
set(key.to_sym, value)
end
end
And I can then access them from the settings object. I'm not sure why you wouldn't commit the config file, though . . . there's no major security risk here, since only those paths that you've explicitly defined can be accessed via the web. I guess the admin password could be stored in the same manner if you don't want to put it in a database, but I would at least encrypt it with a salt.
Just be careful not to step on Sinatra's Configuration settings when defining your own.
EDIT:
I think I just realized why you would prefer not to commit the config file. If you're working on an open source project, you certainly wouldn't want to commit the config file to your open source repo, but you would need to commit the file to Heroku in order for it to work. If this is the case, I'd either:
Use two separate local repos: one for the open source project, and one for the heroku project. Just set the open source project as an upstream repository in the Heroku project, then you can fetch changes.
Put both the API keys and encrypted/salted password in a database; MongoHQ offers a free tier to Heroku users as an addon for simple nosql storage using MongoDB.
Related
i'm using node and github actions for a simple twitter api script.
twitter credentials are stored in github secrets
other than hard coding the twitter credentials and then changing the code to use secrets before pushing to github how do i test locally?
I need some way to retrieve the credentials locally if running locally and from secrets if running on github.
what is best practice way to do this please?
Essentially, what David is trying to say is that you'll need to store your credentials in two separate places:
Inside GitHub Actions Secrets
Your local filesystem inside a gitignored .env file
GitHub Actions Secrets
GitHub Actions Secrets uses encryption to store the secret value, and this information is only decrypted while the workflow is running. Note that the secrets are not stored in the codebase. Instead, this is managed entirely via the GitHub web interface, and perhaps the GitHub API. I do want to emphasise again that none of this is stored inside your git repository as that would essentially be considered a security leak.
Now, this works fine when running in GitHub Actions, but the question you have is how do you run locally. This is where the .env file comes into play.
Running Locally with .env and dotenv
Create the following file, and add the same value you used as the secret in GitHub Actions secrets inside the .env file as a key/value pair. Let's assume the secret name is MY_SECRET_CRED and the value is abcdefg:
.env:
MY_SECRET_CRED=abcdefg
Next, add this file to .gitignore. You do not want to commit this, because it would violate your security, and you also don't need it to be committed, since GitHub has stored the credential as a secret. You only need this file locally.
.gitignore:
.env
The purpose of adding the file to gitignore is because you do not want to commit this file. This file is local to your workstation only.
Next, install the dotenv npm module:
$ npm i dotenv
In your main file, be sure to require the dotenv module. This module reads the contents of .env and loads the key value pairs up as environment variables:
require('dotenv').config()
Tying it all together
In your application code, you're probably already using the secret in GitHub Actions as some kind of environment variable. If running in GitHub Actions, you can load the secret as an environment variable with the same name you used in the .env file:
- name: Test application
env:
MY_SECRET_CRED: ${{ secrets.MY_SECRET_CRED }}
run: npm run test
In the above example, the Node.js process can access the secret as the environment variable MY_SECRET_CRED
doSomethingWithSecret(process.env.MY_SECRET_CRED);
But do note that, if you try to print it, GitHub will mask it, since it knows it's a secret that shouldn't be leaked.
When running locally, the environment variable is loaded into the Node.js process via the dotenv module.
Downside to .env files
The downside to .env files is that the secret is stored in plain text in your workstation, and it's difficult to share with other developers. Ideally, we don't want to share secrets through chat systems or email for the same reasons we don't want to commit and push them.
One solution is to use a secret manager or key management store to encrypt secrets locally. Your local application then decrypts them when running the application. This also means you can commit the encrypted data directly to the GitHub repository. One example of this is Google KMS.
I have a Sails app and I wanted to add one of the config files (connections.js) to version control. This file has the host and port numbers of the database, but doesn't have any app secrets or passwords. Is it safe to add this to git or would it still be considered bad practice?
All connection parameters should be in config files.
You can use something like config package to define all needed parameters, their default values and create config files for all configurations you use (such as a development, a production, a tests configurations).
Obviously these config files should not be stored in version control. You can store a template for a default config file.
When some connection params are defined in code then after some time you will have to change them if your version control host changed its IP-address or something else.
Definitely, it's not safe to commit/push your credentials to any source repository. It's not even safe to store them in production as plain. A solution is to read values of connection strings and similar items from environment. In a Node.js it similar to this code:
const pool = new Pool.Pool({ connectionString: process.env.DATABASE_URL });
It's a common practice in cloud solutions. Sometime there are even more securities applied. In the Heroku, connection string is stored in an environment variable while it is changed each few hours.
I'm relatively new to Node.js and Express and I was following a guide on adding authentication using JWT to my site found here: Medium. I'm confused by one point of this guide, about using secrets, that says:
This secret will be read by JWT library while creating and validating tokens. In production, we need to store this secret in environment variable instead of a file.
In the guide, they placed the secret inside a config file. I wish the guide described why exactly the JWT secret should be an environment variable instead of in a config file, but it's pretty vague. So why should secrets be environment variables, and what are other best practices for using secrets? I've decided to use Passport & Express Sessions for authentication instead of JWT, but I'd think it still applies to session secrets.
My next question is how would I exactly set a secret in an environment variable? I'm not using the guide anymore, but how would somebody that is using it convert that config file into an environment variable? And lastly, what do you usually use environment variables for in a typical Node.js app?
Why should a secret be an environment variable, instead of being stored in a config file?
While working on projects, you will end up uploading your code on github, which is accessible to everyone. If you store your secrets in a config file, anybody with a github account will be able to read it, and hence it is a security risk. Storing the secrets as environment variables ensure their safe-keeping. Your config file should read these values from the process.env object. Something like this:
const jwtSecretKey = process.env.JWT_SECRET_KEY;
const googleApiKey = process.env.GOOGLE_API_KEY;
const serverPort = process.env.PORT || 8000; // 8000 is the default value in case if the env variable has not been set
module.exports = {
jwtSecretKey: jwtSectetKey,
googleApiKey: googleApiKey,
serverPort: serverPort
}
And any other piece of code using these secrets should require the config file.
const config = require('./config');
...
jwt.verify(token, config.jwtSecretKey);
Along with storing secrets as environment variables, you should also store any environment specific values as environment variables. For example, say your NodeJS server (thats is connected to a hosted Database) is run on three environments - Development, QA and PROD. Each of these environments will have different information regarding the DB HOST and PORT, the environment should connect to. Environment Variables are a good way to store the DB Host and Port on each individual environment, and to use the same code across all environments to connect to different databases by reading the environment variable.
How to store the values as environment variables - One way of doing it is manually, though I wouldn't recommend it. One way is use of shell scripts. It actually depends upon your infrastructure.
You should also add your *.env to the .gitignore file So pushing it to the public will ignore the file and you have only one file with all the environment variables.
Maybe the original question was more about why you should not hard-code secrets into your codebase? Hopefully that is fairly obvious to most people. So ideally your secret and non secret "app config" should live separate from your codebase making your app "configurable". The standard way to do this:
have a development config file (e.g. config.js or .env) that is added to .gitignore so it is never checked in.
have a development sample config file (e.g. config.sample.js or sample.env) for new developers to use as a template for their own development configs (this is checked in and does not contain real secrets).
in production deployment this config file is generated or provisioned with production config and secrets by the devops team.
the path to the config file (location of file is not a secret) might be specified by an environment variable for the app process.
If you are concerned about security you won't put secrets into ENV.
(Yes, I know this is a controversial topic for some people. Sorry.)
You can't really prevent unprivileged users from getting access to ENV information. So if you put secrets in env, the bar is lowered for an attacker because they don't necessarily need a privileged account to see your app's secrets running under a different user.
if your app spawns any subprocesses, your secrets will be exposed to them because they inherit your environment.
if you have debug process logging at a system level, process ENV can be exposed to unprivileged users because the system doesn't consider ENV to be secret.
If you are provisioning a production server, you can just as easily put your secrets in a .env file as a systemd EnvironmentFile using the familiar environment variable format.
However the trick is you want to treat this as secret app-loaded config rather than loading it into process environment (via systemd, bash, npm scripts, etc) or process.env within your app.
dotenv by default (when you call .config()) populates process.env. so if you want to use dotenv securely, you can parse your .env config using dotenv.parse so it doesn't put the secrets from your .env into process.env.
This would allow you to provision a .env file with production secrets and use a development .env file that you add to .gitignore so you don't commit your development file. However application developers need to remember to use .parse instead of .config or else it is a security risk.
Alternatively if you don't want to use dotenv since its primary design is focused on loading up ENV... just use a json or js config file that exports a config object (e.g. config.js). Same as .env you can have a dev version that is not committed and a production one that is auto-provisioned. This can be required within the app where needed and avoids the risk (you have with dotenv) of accidentally populating secrets into the environment. Of course bad app code can always still leak secrets, but at least they won't be leaked by default due to how the secrets are exposed to the app.
The downside of all this is that ENV gave us a very standard way to pass config to any type of app/process/script (ie not just nodejs) in production environments. ENV being insecure means we have to come up with a (ideally standard) mechanism for passing secure config to a variety of apps and scripts in production.
In bash (as an example) you can source a .env file and as long as those vars aren't exported they are just "local" variables to the current bash process (ie not inherited by sub-processes). Not using export is akin to not populating process.env. Parsing json config in simple bash scripts...not so fun without helper apps.
You would like application developers to not have to think about production secrets/config or security and give them a standard config retrieval mechanism that works for local development, testing and production.
You would like this standard mechanism to be straight forward and relatively low-risk of them inadvertently opening up security issues on the app side (exposing production secrets to sub-processes). Of course nothing protects you 100% as if the app has access to the secret there are many ways it could leak it (writing to file, making http calls, etc)...but at least it wouldn't be due to the config loading/passing mechanism.
We are currently looking at options for addressing this for a variety of apps in production...so very happy if someone has well thought out suggestions or experience solving this in a clean, general purpose way.
I would be inclined to stick with the .env because we already support EnvironmentFile provisioning and every app can deal with that file format (e.g. can source a .env with bash)...the problem is how to prevent apps/developers from accidentally using dotenv.config instead of dotenv.parse to keep this out of the app environment.
So we are probably just sticking with config.sample.{env,json,js} so the devlops team has a template to populate with production secrets. Config is loaded by the node app via require so it doesn't populate process.env. We also are adding command-line flag (or ENV) to our apps to allow the location of this config file to be specified so devops has the option to place the production config file outside of the app deploy directory (which can make continuous deployment a bit easier). The downside is that a config.js won't work easily for other languages, but devops can fairly easily script converting an ENV style variable format into other targets.
As I said, keeping secrets out of your ENV is for people that are concerned about security. Many blogs/guides/tutorials online show secret passing through ENV. The popularity of dotenv which (by default) loads secrets into the process environment also might lead you to believe this is the right way. Security is always a tradeoff between risk, effort/cost and inconvenience. Personally I think if you auto-provision production it isn't a big deal to fix this...the issue is more about coming up with a semi-standard mechanism that can work across a variety of apps/code-languages.
References:
https://systemd-devel.freedesktop.narkive.com/ibRtoqPS/environment-variable-security
https://github.com/motdotla/dotenv#parse
https://github.com/motdotla/dotenv/pull/378
I don't wan't to put sensitive Credentials (Like API keys, passwords etc) into my branch.
For this, GitLab (and other CI/CD-Services) are able to set Environment-Variables. They will be injected into the script during deployment process.
I know about two ways for GitLab to set them:
Via UI: Project ⇒ Settings ⇒ CI/CD ⇒ Variables
Via .gitlab-ci.yml
As in my opinion the first way is the secure one, because no files with credentials are saved in the git repo but it's also the more complicated way because I have to set each single variable by hand via GitLab-GUI.
With the second way, the issue is that .gitlab-ci.yml is saved into the gitlab repo so that the credentials are not secure.
Question:
Is there a way to define the ENV-Vars in a file and provide it to GitLab, without putting them into the branch?
Or is there another way to create those ENV-Vars easily and securely in GitLab?
Is there a way to define the ENV-Vars in a File?
Yes, in UI settings you mentioned you can specify variables type to be variable (key:value) or file (in Key will be passed path to secret file with content from value input).
So file variable seems like what you are looking for.
Readme and docs provide good description for variables. Personally I find very useful other options: variable masking and protected state.
Thanks makozaki,
that was a good hint, but there are some special requirements:
First go to your Project ⇒ Settings ⇒ CI/CD ⇒ Variables and add them like this:
The Key will be turned in a File-Name and the values you entered will be as Data inside this File.
In the Job-Logs of the CI-Process it provides the full Path to your new generated File... it looks like this:
ENV_PRODUCTION: '/builds/yourProjectGroup/gatsby_netlifycms_starter.tmp/ENV_PRODUCTION',
Notice
you can't choose the easy way and name it .env.production to use it
with dotenvBecause the Key-Field don't allow special-characters like .,
Now as you got the File with all your config-values, you can easily implement it in your Application (e.g. with dotenv).
For my Gatsby implementation it looks like this.
require("dotenv").config({
path: process.env.ENV_PRODUCTION ? process.env.ENV_PRODUCTION : `.env.${process.env.NODE_ENV}`,
})
const config = require('gatsby-plugin-config').default;
What's happening here? The dotenv.config() is checking, if your self-created ENV_PRODUCTION exists... if yes, it will be used. Else it will use the general .env.<yourNodeEnviroment> one.
So with the above given Values in ENV_PRODUCTION, you are able to access FIRST_VALUE within your Application.
This way it's easy to have an .env.development for running your application ot local machine, and using ENV_PRODUCTION env from Gitlab on production.
We're planning a secure Node.JS server, which uses several third-party web services. Each requires credentials that will need to be configured by the operations team.
Clearly they could simply put them in plain text in a configuration file.
Microsoft .NET seems to offer a better option with DPAPI (Data Protection API) - see Credential storage best practices. Is there a way to make this available through IISNode? Or is there any other option to secure such credentials within Node-JS configuration?
There's an extensive discussion of several options here, including the two suggested by xShirase:
http://pmuellr.blogspot.co.uk/2014/09/keeping-secrets-secret.html
User-defined services solves the problem, but only for Cloud Foundry.
This blog http://encosia.com/using-nconf-and-azure-to-avoid-leaking-secrets-on-github/ points out that you can often set environment variables separately on servers, and suggests using nconf to read them and config files separately.
I still wonder if there are specials for IIS?
There is 2 ways to do it securely :
First one is to use command line parameters when you launch your app.
These parameters are then found in process.argv
So, node myapp.js username password would give you :
process.argv[0]=node
process.argv[1]=/.../myapp.js (absolute path)
process.argv[2]=username
process.argv[3]=password
Second is to set the credentials as ENV variables. It is generally considered as the best practice as only you have access to these variables.
You would have to set the variables using the export command, than you'd access it in process.env
I currently had to do the exact same thing for my External API credentials. this is what i did
install node-config module
create a folder and file called config/config.js
here require(config) module
In local box it reads the configuation from local.json file
i have dummy values in local.json for api key and shared secret
on my QA environment i export two variables NODE_ENV="QA" and NODE_CONFIG_DIR="path to my configuation folder on qa server"
node-config module reads configuation from "path to your config folder / QA.json"
now i have real api key and credential in QA.json
here you can use an encryption to encrypt these values and put it back in QA.json
in your app get these config values and decrypt use it in your rest call
hope this helps.
so your config can live in the same container as node code.
refer to this for encryption and decryption
http://lollyrock.com/articles/nodejs-encryption/