I am trying to run a machine learning experiment in azureml.
I can't figure out how to get the workspace context from the control script. Examples like this one in the microsoft docs use Workspace.from_config(). When I use this in the control script I get the following error:
"message": "We could not find config.json in: [path] or in its parent directories. Please provide the full path to the config file or ensure that config.json exists in the parent directories."
I've also tried including my subscription id and the resource specs like so:
subscription_id = 'id'
resource_group = 'name'
workspace_name = 'name'
workspace = Workspace(subscription_id, resource_group, workspace_name)
In this case I have to monitor the log and authenticate on each run as I would locally.
How do you get the local workspace from a control script for azureml?
Using Workspace.from_config() method:
The workspace configuration file is a JSON file that tells the SDK how to communicate with your Azure Machine Learning workspace. The file is named config.json, and it has the following format:
{"subscription_id": "<subscription-id>",
"resource_group": "<resource-group>",
"workspace_name": "<workspace-name>"}
IMPORTANT: This JSON file must be in the directory structure that contains your
Python scripts or Jupyter Notebooks. It can be in the same directory,
a subdirectory named .azureml, or in a parent directory.
Alternatively, use the get method to load an existing workspace without using configuration files: (in your case, your code is missing the .get())
ws = Workspace.get(name="myworkspace",subscription_id='<azure-subscription-id>',resource_group='myresourcegroup')
What is the development system that you are using? A DSVM in the AML workspace or your local dev system?
If it is your local then use this to write config file to your project root directory under the path /.azureml/config.json
from azureml.core import Workspace
subscription_id = 'xxxx-xxxx-xxxx-xxxx-xxxx'
resource_group = 'your_resource_group'
workspace_name = 'your_workspace_name'
try:
ws = Workspace(subscription_id = subscription_id, resource_group =
resource_group, workspace_name = workspace_name)
ws.write_config()
print('Library configuration succeeded')
except:
print('Workspace not found')
or else if it DSVM, then you are all set, Workspace.from_config() should work.
Note: You will have to see .config directory under your user name in AML studio.
This had no answers for 10 months, and now they are coming in :). I figuerd this out quite a while ago but haven't gotten around to posting the answer. Here it is.
From the training script, you can get the workspace from the run context as follows:
from azureml.core import Run
Run.get_context()
ws = run.experiment.workspace
Related
Trying to run and debug a pipeline locally. Pipeline is imeplemented with azure.ml.component.dsl.pipeline. When I try to set default_compute_target='local', the compute target cannot be found:
local not found in workspace, assume this is an AmlCompute
...
File "/home/amirabdi/miniconda3/envs/stm/lib/python3.8/site-packages/azure/ml/component/run_settings.py", line 596, in _get_compute_type
raise InvalidTargetSpecifiedError(message="Cannot find compute '{}' in workspace.".format(compute_name))
azure.ml.component._util._exceptions.InvalidTargetSpecifiedError: InvalidTargetSpecifiedError:
Message: Cannot find compute 'local' in workspace.
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Cannot find compute 'local' in workspace."
}
}
The local run, for example, can be achieved with azureml.core.ScriptRunConfig.
src = ScriptRunConfig(script="train.py", compute_target="local", environment=myenv)
run = exp.submit(src)
We have different types of compute targets and one of those is local computer.
Create an experiment
from azureml.core import Experiment
experiment_name = 'my_experiment'
experiment = Experiment(workspace=ws, name=experiment_name)
Select the compute target where we need to run
compute_target='local'
If the compute_target is not mentioned or ScriptRunConfig is not mentioned, then AzureML will run the script locally
from azureml.core import Environment
myenv = Environment("user-managed-env")
myenv.python.user_managed_dependencies = True
Create the script job, based on the procedure mentioned in link
Submit the experiment
run = experiment.submit(config=src)
run.wait_for_completion(show_output=True)
To check for the troubleshooting the procedure, check with link
I have a repository with several separate configs which share some modules, and reference those modules using relative paths that look like ../../modules/rabbitmq. The directories are setup like this:
tf/
configs/
thing-1/
thing-2/
modules/
rabbitmq/
cluster/
...
The configs are setup with a remote backend to use TF Cloud for runs and state:
terraform {
backend "remote" {
hostname = "app.terraform.io"
organization = "my-org"
workspaces {
prefix = "config-1-"
}
}
}
Running terraform init works fine. When I try to run terrform plan locally, it gives me an error saying:
Initializing modules...
- rabbitmq in
Error: Unreadable module directory
Unable to evaluate directory symlink: lstat ../../modules: no such file or
directory
...as if the modules directory isn't being uploaded to TF Cloud or something. What gives?
It turns out the problem was (surprise, surprise!) it was not uploading the modules directory to TF Cloud. This is because neither the config nor the TF Cloud workspace settings contained any indication that this config folder was part of a larger filesystem. The default is to upload just the directory from which you are running terraform (and all of its contents).
To fix this, I had to visit the "Settings > General" page for the given workspace in Terraform Cloud, and change the Terraform Working Directory setting to specify the path of the config, relative to the relevant root directory - in this case: tf/configs/config-1
After that, running terraform plan displays a message indicating which parent directory it will upload in order to convey the entire context relevant to the workspace. 🎉
Update #mlsy answer with a screenshot. Using Terraform Cloud with free account. Resolving module source to using local file system.
terraform version
Terraform v1.1.7
on linux_amd64
Here is the thing I worked for me. I used required_version = ">= 0.11"
and then put all those tf files which have provider and module in a subfolder. Kept the version.tf which has required providers at root level. Somehow I have used the same folder path where terraform.exe is present. Then Built the project instead of executing at main.tf level or doing execution without building. It downloaded all providers and modules for me. I am yet to run on GCP.
enter image description here - Folder path on Windows
enter image description here - InteliJ structure
enter image description hereenter image description here
Use this source = "mhmdio/rabbitmq/aws
I faced this problem when I started. Go to hashicorp/terraform site and search module/provider block. They have full path. The code snippets are written this way. Once you get path run Terraform get -update Terraform init - upgrade
Which will download modules and provider locally.
Note: on cloud the modules are in repo but still you need to give path if by default repo path not mapped
I have similar issue, which I think someone might encounter.
I have issue where in my project the application is hosted inside folder1/folder2. However when I run terraform plan inside the folder2 there was an issue because it tried to load every folder from the root repository.
% terraform plan
Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.
Preparing the remote plan...
The remote workspace is configured to work with configuration at
infrastructure/prod relative to the target repository.
Terraform will upload the contents of the following directory,
excluding files or directories as defined by a .terraformignore file
at /Users/yokulguy/Development/arepository/.terraformignore (if it is present),
in order to capture the filesystem context the remote workspace expects:
/Users/yokulguy/Development/arepository
â•·
│ Error: Failed to upload configuration files: Failed to get symbolic link destination for "/Users/yokulguy/Development/arepository/docker/mysql/mysql.sock": lstat /private/var/run/mysqld: no such file or directory
│
│ The configured "remote" backend encountered an unexpected error. Sometimes this is caused by network connection problems, in which case you could retry the command. If the issue persists please open a support
│ ticket to get help resolving the problem.
╵
The solution is sometime I just need to remove the "bad" folder whic is the docker/mysql and then rerun the terraform plan and it works.
Running pipeline failed with the following error.
User program failed with ValueError: ZIP does not support timestamps before 1980
I created Azure ML Pipeline that call several child run. See the attached codes.
# start parent Run
run = Run.get_context()
workspace = run.experiment.workspace
from azureml.core import Workspace, Environment
runconfig = ScriptRunConfig(source_directory=".", script="simple-for-bug-check.py")
runconfig.run_config.target = "cpu-cluster"
# Submit the run
for i in range(10):
print("child run ...")
run.submit_child(runconfig)
It seems timestamp of python script (simple-for-bug-check.py) is invalid.
My Python SDK version is 1.0.83.
Any workaround on this ?
Regards,
Keita
One workaround to the issue is setting the source_directory_data_store to a datastore pointing to a file share. Every workspace comes with a datastore pointing to a file share by default, so you can change the parent run submission code to:
# workspacefilestore is the datastore that is created with every workspace that points to a file share
run_config.source_directory_data_store = 'workspacefilestore'
if you are using RunConfiguration or if you are using an estimator, you can do the following:
datastore = Datastore(workspace, 'workspacefilestore')
est = Estimator(..., source_directory_data_store=datastore, ...)
The cause of the issue is the current working directory in a run is a blobfuse mounted directory, and in the current (1.2.4) as well as prior versions of blobfuse, the last modified date of every directory is set to the Unix epoch (1970/01/01). By changing the source_directory_data_store to a file share, this will change the current working directory to a cifs mounted file share, which will have the correct last modified time for directories and thus will not have this issue.
Need some assistance regarding using files with in modules from terraform enterprise. Git top folder structure is like this:
modules
README.md
main.tf
With in the modules the folder structure is like this:
modules
main.tf
file1.json
the main.tf within modules referring these file1.json through
like below
resource "aws_iam_policy" "deny_bill_policy" {
name = "OpsPolicy"
path = "/"
policy = "${file("${path.module}/file1.json")}"
}
The same program runs without any issues from my localpc to deploy on aws but when i run the same through terraform enterprise which pulls repo from git throwing the following error.
module.policy_roles.aws_iam_policy.deny_bill_policy: file: open file1.json: no such file or directory in: ${file("${path.module}/file1.json")}
fyi - there is no previous/old .terraform dir existed. Seems TFE handling module/paths is different. some one please assist me here.
The Hashicorp Consul repository contains a Terraform module for launching a Consul cluster in AWS. The module references several files that are found in the parent directory of the module under the shared/scripts directory here https://github.com/hashicorp/consul/tree/master/terraform
However, when I reference the module in one of my .tf files and run terraform get to download the module, the required files under shared/scripts/ are not included with the downloaded module files, leading to errors like the one described here
My module section in Terraform looks like this:
module "consul" {
source = "github.com/hashicorp/consul/terraform/aws"
key_name = "example_key"
key_path = "/path/to/example_key"
region = "us-east-1"
servers = "3"
platform = "centos7"
}
Is there anyway to have terraform get pull in files that live outside the module directory?
Thanks
From looking at what those files do, I'd just copy the one's you need (depending on whether you're deploying on debian or rhel), which will be 2/3 files and feeding them into provisioner "file":
https://www.terraform.io/docs/provisioners/file.html