As per the Prefect Configuration guide I've set a local config file (i.e. in my current working directory) via export PREFECT__USER_CONFIG_PATH="./config.toml", however, this requires the environmental variable to be set each shell session.
I tried setting the config in a Python script with prefect.config.user_config_path = "./config.toml" but this doesn't seem to load the config file in the same way as export because:
(from docs) "The configuration file is parsed when Prefect is first imported and is available as a live object in prefect.config."
For my dummy config.toml data:
[api]
host = "localhost"
port = "5432"
url = "https://${api.host}:${api.port}"
prefect.config via environmental variable:
...'task_runner': {'default_class': 'prefect.engine.task_runner.TaskRunner'}}, 'api': {'host': 'localhost', 'port': 5432, 'url': 'https://localhost:5432'}, 'user_config_path': './config.toml'}>
prefect.config via script:
...'task_runner': {'default_class': 'prefect.engine.task_runner.TaskRunner'}}, 'user_config_path': './config.toml'}>
So config.toml variables aren't loaded in second case
Any ideas?
(Application: I want to set a 'Prefect' secret in my local config.toml file hidden from source control)
Is there a reason you want the config.toml file in your present working directory? The default location for user config files is ~/.prefect/config.toml, so if you place your configuration there it will work with no additional setup.
If you do need the configuration file in your present working directory, you might try adding the export statement to your ~/.bashrc (or whatever shell equivalent you might need) so that it's always in your environment.
Related
I'm using bitbucket pipelines as a build server.
I need to pass environmental variables from a host machine into a .env file which will then set the var values to be used in the build.
For example, lets say an environmental variable in a docker container running the build is AWS_ACCESS_KEY_ID.
In my .env file I'd like something like the following:
ACCESS_KEY=${AWS_ACCESS_KEY_ID}
I would then run the build and the ACCESS_KEY var would have a value equal to the env var in the docker container.
My current idea for a solution right now involves replacing values with sed, but that feels pretty hacky. Example:
.env file contains the following line:
ACCESS_KEY=<_access_key_replace_me_>
sed "s/<_access_key_replace_me_>/${AWS_ACCESS_KEY_ID}/g" .env
Any better solution than this?
I'm deploying a node.js app on Heroku dyno and using config module that requires me to define a system variable NODE_CONFIG_DIR with the location of the config folder.
The config folder is located on my project's root.
I tried to define the system variable NODE_CONFIG_DIR using the following values, all failed:
./config
~/config
app/config
~/app/config
./app/config
$HOME/config
$HOME/app/config
I keep getting this error:
WARNING: No configurations found in configuration directory:app/config
(replace app/config with any of the values above)
I manage to set a system variable, but its value is not pointing the right place.
What is the correct way to refer to the root of my tree when using a system variable in Heroku?
Based on documentation - if config folder is in the root of your application you should not need to specify $NODE_CONFIG_DIR env variable.
From node-config documentation:
Node-config reads configuration files in the './config' directory for the running process, typically the application root. This can be overridden by setting the $NODE_CONFIG_DIR environment variable to the directory containing your configuration files. It can also be set from node, before loading Node-config:
process.env["NODE_CONFIG_DIR"] = __dirname + "/configDir/";
const config = require("config");
$NODE_CONFIG_DIR can be a full path from your root directory, or a relative path from the process if the value begins with ./ or ../.
You could use above code to set it from your node code.
You were close: /app is the correct path. You can verify it by running heroku run bash.
It was my bad...
Both answers are correct but didn't solve my issue.
The problem was that I used lowercase for my configuration file name while the NODE_ENV value was uppercase.
I have a docker-compose file that contains a secret for my database.
port: 4466
managementApiSecret: ${DB_SECRET}
So I would like to use Docker-Compose's Variable Substitution (https://docs.docker.com/compose/compose-file/#variable-substitution) to be able to provide my secrets through an environment variable instead of my yaml file.
Is this at all possible using the "Application Settings" variables on my Azure Linux Docker Instance?
It doesn't seem to work for me, I've tried both ${VAR} and $VAR syntax and neither worked. I also tried with secrets that only contain alphanumerics and numbers.
Thanks in advance.
Frank
Environment variables that you need to start the container (for example: you want to include build number in container name) can be added to .env file in same directory as the docker-compose.yml file.
sample .env file:
DB_SECRET=foo
run: docker-compose config and verify that the variable has been substituted with the value you have in .env file
Also, I recommend using managementApiSecret:"${DB_SECRET}" (note the quotes around the variable) in your docker-compose.yml file
There might be azure specific way to share secrets but I didn't try that yet.
If you want to pass in environment variables that the container needs, then https://docs.docker.com/compose/compose-file/#env_file is what you want. Those environment vars will become part of environment inside docker container.
I have .env file at my project root directory.
How should I handle .env file for dev, qa, stage and prod?
Should include them in git repo? if not where I put them? different folder on external drive for example?
What is the correct extensions? .env.qa or .qa.env?
If I want to build my bundle using webpack to the dist folder (server side), should I include the env file or manually copy it to the dist folder?
You should not check-in your env files into any source control. Any of those secrets will be forever available to anyone having access to the repo until the history is rewritten to remove them.
If you use AWS services, for example, I would suggest using the Secrets Manager.
Any environment variables introduced to Webpack should not be secrets but be configuration values. Anyone who can view source can read those values. If you need to have environment-specific configurations, the Webpack DefinePlugin will replace vars like MY_API_HOST with their values with the following config:
const plugins = [
new webpack.DefinePlugin({
MY_API_HOST: JSON.stringify('https://my-domain.com/api/'),
MY_API_VERSION: JSON.stringify('v2')
})
]
Config module is a easy way to address the different env specific values. Read about config module at - https://www.npmjs.com/package/config. You will have a config folder in the repository with env specific files and I like this approach as the files are in the repository but very well separated.This provides a really easy way to set default values, override the environment specific values etc. It is also very convenient to use the different environment specific files by setting the appropriate node environment variable(export NODE_ENV=development or acceptance or production).
setting environmental variables for a Hubot is pretty easy on the production server. However when I want to test the bot locally, I need the env vars inside a file. I already have the file .env for env vars that heroku is using for running locally.
But I can't seem to find a way to load env vars inside the Hubot scripts from a file.
Merry Christmas :-)
okay it's possible with hubot-env.
https://www.npmjs.com/package/hubot-env
The following command will load the file from a relative path:
hubot env load --filename=[filename].
It previously didn't work for me because I had HUBOT_ENV_BASE_PATH set on my mac so the command searched in the wrong folder for the file.