I got a Visual Studio solutions with multiple projects (function apps and a web API) and a datalayer that is shared between all the projects. I've set up the solution so all projects share the same config (appsettings.json) based on this article: https://andrewlock.net/sharing-appsettings-json-configuration-files-between-projects-in-asp-net-core/
All projects are based on .net core.
I've set up a build and a release pipeline for the dev environment. But I need a Test and production environment. How do I transform the shared config before releasing it to the test and production environment?
You don't. That's not how configuration works in ASP.NET Core. Configuration is overridden, not transformed. There's an order of ops to how the different configuration sources are applied, which is basically the order in which they're registered. The default is JSON < Environment-specific JSON < User Secrets < Environment Variables < Command-line Arguments.
If you need configuration to vary by environment, you're going to rely on the environment-specific JSON files (for general config) or environment variables and/or something like Azure Key Vault (for secrets). Since all of these come later in the configuration registration, any value you set there will override the values in your appsettings.json.
For things like environment-specific JSON, which is loaded is dependent on the value of ASPNETCORE_ENVIRONMENT, which can be set as an environment variable or passed as a command-line argument --environment. In either case, the value set corresponds to the {environment} portion of appsettings.{environment}.json. In other words, if you set the environment as Production, then appsettings.Production.json will be loaded into the config, if it is present. Environment variables are tied to the environment itself so are not dependent on any particular environment value.
Related
I'm developing a little server made in node, hapijs, nodemon, etc.
It's a basic api rest which will grow with ongoing dev.
I need to have different variables for dev. and production. I actually have only one .env file. I've read it is not recommended to have 2 separate files for this.
How should I modify my app.js to have two situations?
run nodemon locally in my pc while in dev and local variables
when deploying to heroku, use production variables
Thanks a lot in advance,
As you've probably already done. Write your code to use environment variables. (whether you run locally or on production, that's the same code.).
const ACCESS_KEY = process.env.ACCESS_KEY;
Your .env file then contains ONLY your local settings, for debug on your local computer. You can add .env in your .gitignore file to make sure it doesn't get pushed to your git repository.
Production settings by contrast shouldn't be in any file at all. They should only be configured directly in the settings of your cloud provider.
if you're using Azure, they should be in an Azure Key Vault
if you're using AWS, they should be in the AWS Parameter Store
if you're using Heroku, then they should be configured in Heroku's settings.
Heroku settings
It's possible to do this from the "Settings" tab in your heroku app dashboard. There is a section "Config vars".
When heroku launches your application, it will define the configured config variables as environment variables. And you will be able to access them with process.env just as you would with the environment variables which were defined in your .env file during development.
CLI
The dashboard makes it easy to get an overview and to manage the keys. Perhaps even more conveniently, you can also do this with the heroku cli tool straight from the commandline.
To get a list of your current environment variables, run.
heroku config
To add a new key from the CLI.
heroku config:set ACCESS_KEY=adfsqfddqsdf
All of this is also described in the official documentation of Heroku.
Generally, you would generate your env file at build time. For example, using AWS SSM / or some kind of Vault that is secure, you store your secrets like db passwords. The env file is a template that gets compiled with the right env vars for the target deployment.
Also, you can have dummy variables in your env template that you commit to git. Then add a .gitignore file with an entry to your env template to ensure you don't commit any secrets to the env file. Then locally you compile your file for local, during your staging build for staging, during prod build for prod, etc.
As the app gets larger, this allows you to provision credentials per person / per environment. You add the associated secrets / permissions to the vault. Allow the people/environments access to those secrets, and then you can control access in a pretty fine grained fashion.
I suggest using an npm package for handling different environments variables and keys. (or implement it by yourself)
Alongside with .env file
1- use .env file to store credentials and secrets
2- Reference these .env variables via different package that provides separate file for each environment
suggested package : https://www.npmjs.com/package/config
I used this approach in one of my projects and made my life easier.
A widely adopted best practice is to inject at runtime the application settings (secrets and environment config).
It is safer (secrets are NOT stored in the source code, bundles or packages/images) and portable (as you deploy to more environments you only need to define suitable values - no code changes, recompilation or repackaging).
Single .env file
Define a single .env file: your application needs the same properties (with different values obviously) anywhere.
On your local development environment config the .env file for development: you don't commit either package this file.
Production Deployment
Define the runtime configuration: on Heroku use Config Vars to create an environment variable for each property defined in the .env file, for example
# .env
API_TOKEN = dev1
Create a Config Var API_TOKEN with the production value: this is injected at application-startup and never stored/exposed.
This approach is language-agnostic (in Java you might have a .properties instead but the principle is the same) and works with different hosting providers: you would deploy the same app/package while configuring the environment settings accordingly.
[Disclamer] : this post focus exclusively on CI / build aspects and has nothing to do with CD / deployment
Say I have a project hosted on Gitlab involving some source code compilation. For the present purpose, let's say I wan't Maven to create two cars (STAGE PROD) of some java project every time a developer merges on master. Those cars:
will be stored on the project registry
need to contain the environment variables defined for this project.
Can I legitimately assume that those environment variables will remain safe (i.e. private) if the project is compiled with shared gitlab runners based on the assumption that Docker runners are ephemeral ? Is there a (better) way to enforce privacy despite using shared runners?
One possible alternative would be using Protected environment variables.
issue 1452 show them in "CI/CD" panel of Settings.
By prefixing them with STAGE_ or PROD_, your maven job could use the right variable depending on the current car.
By their protected nature, they would not be exposed.
Azure DevOps XDT Transform tasks allow you to build release profiles that transform the base config file with settings that are specific to each environment, such as a connection string that points to different db servers for different environments. The app.dev.config file has transformations for the dev environment, app.qa.config for qa, etc, which are applied during the deployment to the base app.config file.
I need to take this one step further and deploy custom config files for each individual server in a load balanced environment. For example, the DEV environment has two servers dev1.mysite.com and dev2.mysite.com that are load balanced by dev.mysite.com. Each of the two servers needs specific settings in the config file deployed to that server.
I don't (yet) see a way in Azure DevOps to do this. Part of the solution might be to set up variables with the setting that needs to be applied to each environment/server but I haven't figured out how to apply the correct variable to each config.
You can use task Magic Chunks to apply the variable to each config.
You can search for Magic Chunks task in your pipeline and install it to your organization. Then add Config transform task before the deployment task to update the config file with specific setting. For below example settings of magic chunk task:
As above screenshot shows, You can reference your pipeline variables in the tasks.
There are other extension tasks like RegEx Find & Replace you can use to replace the variables in the config files.
In Node.js, I want to set config property value based on platform (dev, staging, qa, production) it will get deployed.
So for example for dev and staging environment, I want to set value '234'
And for prod, i want to set value '456'.
for deployment, i am using VSTS.
Shall I make the use of deployment variables?
For setting config property based, please take a look at this question: Node.js setting up environment specific configs to be used with everyauth
In VSTS Release, you could use environment scoped value per environment.
Share values across all of the tasks within one specific environment
by using environment variables. Use an environment-level variable for
values that vary from environment to environment (and are the same for
all the tasks in an environment). You define and manage these
variables in the Variables tab of an environment in a release
pipeline.
Source Link: Custom variables
If you want to change values, suggest you to use Replace Tokens task Visual Studio Team Services Build and Release extension that replace tokens in files with variable values.
Before I was shown Octopus Deploy I thought that environment dependent appSettings should be a part of Build Configuration in project properties.
Now in times of software as a service it is a deployment process that buckles everything up.
I want my environment configuration in release process to just open web.config and substitute appsettings and connection strings based on variable names i defined for the release definition.
How can I do it? The closest I could get was Magic chunks. The problem with it is that I have to give it a json with the mapping and I have to define it for each environment separately, so it makes no use of environment variables, really, or at least you have to define it in "enviroment variables" section and then, again, in each process of environment.
You can use the "Tokenizer" task in "Release Management Utility tasks" extension or "Replace Tokens" task.
These tasks can replace the strings in a file with the custom variables in definition.