I'm creating a VM instance through cloud function in GCE.I want to add some environment variables to the Instance during creation.
I'm referring this code for instance creation:
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/compute/api/create_instance.py
I don't want to add this in startup script, because already I'm running a set of tasks in startup script, and i want one of those tasks to use these environment variables. Is there any other way,like passing values in config, while creating instance ?
When you create a new Cloud-Function, you can expand the bottom menu named:
Environment variables, networking, timeouts and more
and set your environment variables from there.
Edit: I'd like to specify that environment variables are used by Cloud-Function itself on the main.py. However, you should may be interested into Instance Metadata & Metadata Server.
You can set environment variables by using gcloud command or through GCP Console [1]:
By using gcloud command:
You can use the --set-env-vars flag to define a variable using the gcloud command-line.
e.g.:
gcloud functions deploy FUNCTION_NAME --set-env-vars env1=whatever,env2=whatever FLAGS...
*Note: The --set-env-vars and --env-vars-file flags for environment variables are destructive. That is, they replace all current variables with those provided at deployment. To make additive changes, use the --update-env-vars flag described in the next section.
e.g.:
gcloud functions deploy FUNCTION_NAME --update-env-vars env1=whatever
Through GCP Console:
Open the Functions Overview page in the GCP Console:
GO TO THE CLOUD FUNCTIONS OVERVIEW PAGE.
Click Create function.
Fill in the required fields for your function.
Expand the advanced settings by clicking More.
In the Environment variables section, set variables by clicking Add variable.
References:
[1] https://cloud.google.com/functions/docs/env-var#setting_environment_variables
Related
i have CI/CD environment in azure devops.i specially want to get the user name who pushed his code to trigger pipeline.i want to use this info as a variable inside the pipeline.do you guys know how can i do that ?
Actually we do have a predefined Build variable .
We can directly use $(Build.RequestedFor) to get the the original user who kick off pipeline
If you want to pass this variable to any other customized variable/parametes.
You could use Logging Commands to set value during your build pipeline. ##vso[task.setvariable variable=testvar The first task can set a variable, and following tasks in the same phase are able to use the variable. The variable is exposed to the following tasks as an environment variable.
Define and modify your variables in a script
To define or modify a variable from a script, use the task.setvariable
logging command. Note that the updated variable value is scoped to the
job being executed, and does not flow across jobs or stages. Variable
names are transformed to uppercase, and the characters "." and " " are
replaced by "_".
For example, Agent.WorkFolder becomes AGENT_WORKFOLDER. On Windows,
you access this as %AGENT_WORKFOLDER% or $env:AGENT_WORKFOLDER. On
Linux and macOS, you use $AGENT_WORKFOLDER.
More details please take a look at this tutorial Define and modify your variables in a script You could also look at this blog: Use Azure DevOps Variables Inline powershell in build and release pipelines
Azure Devops passes that information to pipelines. Check Build.RequestedFor variable: Build variables (DevOps Services), How are the identity variables set?
In Git or TFVC by the Continuous integration (CI) triggers, the
Build.RequestedFor and Build.RequestedForId values are based on the
person who pushed or checked in the changes.
How to use variables you can find here: Define variables
In my projects Docker file I have some environment variables, like this:
ENV ACCEPT_EULA=Y
ENV SA_PASSWORD=Password
ENV MSSQL_PID=Developer
ENV MSSQL_TCP_PORT=1433
And I would like to pass the password here as an environment variable set in my pipeline.
In Azure DevOps I have two pipelines. One for building the solution and one for building and pushing docker images to DockerHub. There are options to set variables in both these pipelines:
I have set the password in both pipelines and edited my password in the Dockerfile to look like this:
ENV SA_PASSWORD=$(SA_PASSWORD)
But that does not seem to be working. What is the correct way of passing environment variables from Azure DevOps into a Docker image?
Also, is this a safe way of passing secrets? Is there any way someone could read secrets from a Docker image?
Thanks!
You can set an ARG var_name and reference ENV to the ARG variables. Then you can replace those variables when docker build the image $ docker build --build-arg var_name=$(VARIABLE_NAME)
For example the add ARG in dockerfile, and have the ENV variable refer to it:
ARG SECRET
ENV ACCEPT_EULA=Y
ENV SA_PASSWORD=$SECRET
ENV MSSQL_PID=Developer
ENV MSSQL_TCP_PORT=1433
You can use dock build task and dock push task separately, as buildandpush command cannot accept arguments. And set a variable SECRET in your pipeline.
The set the Build Arguments SECRET= $(SECRET) to replace the ARG SECRET
You can also refer to a similar thread.
I am using the Replace Tokens extension for exactly tasks like this: https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens
However, putting secrets into your Dockerfile might not be the best idea. Usually you would provide secrets or generally runtime configuration as environment variables when you actually execute the container.
I suggest to set the environment variables at runtime. If you are deploying to an Azure App Service, app settings are injected into the process as environment variables automatically.
You can then use the same image for multiple environments. With the Deploy Azure App Service task in a release pipeline, you can change the app settings for each environment.
https://learn.microsoft.com/en-us/azure/app-service/configure-custom-container?pivots=container-linux#configure-environment-variables
Also, is this a safe way of passing secrets? Is there any way someone
could read secrets from a Docker image?
This questions really depends on the appproach and the importance of your image here. Usually there are 2 ways that somebody would look at to achieve this. Build arguments and Environment variables.
Build Arguments are usually declarted in the dockerfile, and are supplied using --build-arg parameter to the docker builder (docker build). Nothe that the docker build will complain if you declare an argument and not pass it during build if you also didnt supply a default value during declaration. These are available only when the image is being built. The subsequent containers from this image will not have access to the ARG variables, as long as you dont set them again using ENV.
ENV are environment varriables which can be declared and set from the dockerfile or at the os-level. The ENV variables are then available for use in subsequent instructions. These are available during the image build and all the subsequent containers from this specific image will have access to these variables. So if you ssh/exec into the container and take a look at which environment variables are set you will find them.
But that does not seem to be working. What is the correct way of
passing environment variables from Azure DevOps into a Docker image?
buildAndPush command will ignore ARG by default in the task-inputs.. So either use a bash step to build the image or separate build and push tasks as described earlier.
In release, choose deploy azure app service task. Provide required properties at App settings section under Application and Configuration Settings option.
I am deploying a model to AKS via AML using the python-sdk, and I am facing a problem accessing the environment variables defined for the Environment object myenvused for the deployment.
# add environment variable
myenv.environment_variables = {'SOME_ENV_VARIABLE': 'ABC'}
# register to workspace
myenv.register(ws)
This environment object is stated in the inference configuration for the deployment:
myenv = Environment.get(workspace=ws,name="myenv")
inference_config = InferenceConfig(entry_script='score.py',
source_directory=os.path.abspath(__file__ + "//.."),
environment=myenv,
enable_gpu=True,
description="...")
When the model executes, the init() method in the entry_script score.py should be able to access these environment variables by calling os.environ['SOME_ENV_VARIABLE']. However, that is not working. The conda and pip packages defined in myenv are present in the image.
Shouldn't it be possible to access these env variables from the entry_script?
Environment variables set on environment object are deprecated. Those are runtime variables set during the execution and implementation moved to the RunConfiguration.
https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.runconfig.runconfiguration?view=azure-ml-py#azureml-core-runconfig-runconfiguration-environment-variables
Variables baked into the image could be set in the dockerfile of the environment or in your base image.
Sorry if some of the docs are misleading, we are working on fixing those.
You can create an environment by instantiating Environment object and then setting its attributes: set of Python packages, environment variables and others.
Specify environment variables:
myenv.environment_variables = {"MESSAGE":"Hello from Azure Machine Learning"}
You can add environment variables to your environment. These then become available using os.environ.get in your training script.
Please follow the below for using environments for inferencing.
I just tested environment_variables from an Environment ojb. It works. I also included a couple of environment variables for an inference environment.
python sdk v1.38.0.
The solution I used more or less the same as what was described above. After model was deployed in ACI, the environment variables are visible from properties (Containers -> Properties) e.g., enter image description here
I've defined Variable Group which downloads secrets from Key Vault.
Looks like that unlike other variables, secrets aren't set automatically as environment variables.
I've tried using a bash script to take those 'task variables' and set them as environment variables but they were gone by the next task:
export ENV1=$(someSecretTaskVariable)
I'm using npm task which can't be provided with environment variables via the UI and the yaml is read only.
How should this be done?
If you want to create an environment variable that is passed to subsequent Azure DevOps tasks, maybe try this :
echo '##vso[task.setvariable variable=ENV1]$(someSecretTaskVariable)'
instead of export ENV1=$(someSecretTaskVariable)
Set variables in scripts
I dont think you can do this via UI, but via yaml you would do this:
- task: xxx
env:
ENV1=$(someSecretTaskVariable)
apparently you can do this:
Unlike a normal variable, they are not automatically decrypted into
environment variables for scripts. You can explicitly map them in,
though.
To pass a secret to a script, use the Environment section of the
scripting task's input variables.
seems like with UI you can only do this with scripting tasks
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=classic%2Cbatch#secret-variables
Essentially, I need a way to inject environment variables for my app since I don't want to check a .env file into my repo. I was hoping to run something like API_URL=api.example.com npm run build, but it looks like you can't prepend things before the task command.
Is there a way to do this or is there a better way to create environment variables for a node app hosted in Azure?
You can add some build variables to the build definition and then reference those in your build steps somewhere. For example, for your API_URL add a build variable with the same name and value. If you need the variable to be secret for any reason (passwords, etc.) just click the lock icon next to the value field.
Then add a new cmd task to your build and move it to the top to set your environment variables before you start your build. The way you reference the build variables is like this...
set API_URL=$(Build.API_URL)
In the UI it will look like this:
I added two cmd tasks to a test build just to show that it is working. The first one is used to set the environment variable and then I used the second to dump all the environment variables so I could see them. You can see in the build logs that it worked.
If you want to reference the build variables using something other than the command line you can find examples of the different ways to reference build variables here. There are also examples of how to use secret variables on that page.
EDIT:
Since you have some concerns about it not being available in your node app I tested it out in the console real quick to show that it will definitely work.
Using yaml config you can switch to script task and then do something like this:
- script: 'npm run'
workingDirectory: 'src'
displayName: 'npm run'
env:
{ API_URL: 'api.example.com' }
tehbeardedone's answer is great for the build machine, and the answer is similar for when running your app in an Azure app service, which I'm thinking is what you really need. This is for if you need your access to these .env files just previous to or after your app has started (post build).
To the Azure portal we go.
Navigate to your app service.
Click "Configuration" under the "Settings" heading.
For each of the variables you'll need during a production run of your app you'll need to:
Click "New application setting"
Paste the name of the variable (exactly as it appears in your .env) into the "Name" field.
Similarly, paste the value of the variable into the next field.
Check the "deployment slot setting" checkbox. As nebulous as this field is, it's necessary. When I didn't do this, the app didn't have access to this variable after running the start command.
Restart your app. If you deployment when well, then your app should have access to the variables add in the manner specified above.