I have lots of run files created by running PyTorch estimator/ ScriptRunStep experiments that are saved in azureml blob storage container. Previously, I'd been viewing these runs in the Experiments tab of the ml.azure.com portal and associating tags to these runs to categorise and load the desired models.
However, a coworker recently deleted my workspace. I created a new one which is connected to the previously-existing blob container, the run files therefore still exist and can be accessed on this new workspace, but they no longer show up in the Experiment viewer on ml.azure.com. Neither can I see the tags I'd associated to the runs.
Is there any way to load these old run files into the Experiment viewer or is it only possible to view runs created inside the current workspace?
Sample scriptrunconfig code:
data_ref = DataReference(datastore=ds,
data_reference_name="<name>",
path_on_datastore = "<path>")
args = ['--data_dir', str(data_ref),
'--num_epochs', 30,
'--lr', 0.01,
'--classifier', 'int_ext' ]
src = ScriptRunConfig(source_directory='.',
arguments=args,
compute_target = compute_target,
environment = env,
script='train.py')
src.run_config.data_references = {data_ref.data_reference_name:
data_ref.to_config()}
Sorry for your loss! First, I'd make absolutely sure that you can't recover the deleted workspace. Definitely worthwhile to open an priority support ticket with Azure.
Another thing you might try is:
create a new workspace (which will create a new storage account for you for the new workspace's logs)
copy your old workspace's data into the new workspace's storage account.
Related
I’m following this article to create ML pipelines with the new SDK.
So I started by loading the first class
from azure.ai.ml import MLClient
and then I used it to authenticated on my workspace
ml_client = MLClient(
credential=credential,
subscription_id=subscription_id,
resource_group_name=resource_group_name,
workspace_name=" mmAmlsWksp01",
)
However, I can’t understand how I can retrieve the objects it refers to. For example, it contains a “workspaces” member, but if I run
ml_client.workspaces["mmAmlsWksp01"]
, I get the error “'WorkspaceOperations' object is not subscriptable”.
So I tried to run
for w in ml_client.workspaces.list():
print(w)
and it returns the workspace details (name, displayName, id…) for a SINGLE workspace, but not the workspace object.
In fact, the ml_client.workspaces object is a
<azure.ai.ml._operations.workspace_operations.WorkspaceOperations at 0x7f3526e45d60>
, but I don’t want a WorkspaceOperation, I want the Workspace itself. How can I retrieve it?
You need to use the get method on workspaces e.g.
ml_client.workspaces.get.("mmAmlsWksp01")
I am new to Azure batch. I have to perform a task on nodes in the pool.
The approach I am using is that I have the code that I want to run on the node. I am making a zip of the jar of the .class file and uploading to my Azure storage account and then getting the application Id and putting it in ApplicationPackageReference and adding to my task in job.
String applicationId= "TaskPerformApplicationPack";
ApplicationPackageReference reference = new ApplicationPackageReference().withApplicationId(applicationId);
List<ApplicationPackageReference> list = new ArrayList<ApplicationPackageReference>();
list.add(reference);
TaskAddParameter taskToAdd = new TaskAddParameter().withId("mytask2").withApplicationPackageReferences(list);
taskToAdd.withCommandLine(String.format("java -jar task.jar"));
batchClient.taskOperations().createTask(jobId, taskToAdd);
Now when I run this, my task fails giving an error that
access for one of the specified Azure Blob(s) is denied
How can I run a particular code on a node using azure batch job tasks?
I think a good place to start is: (I have covered most of the helpful links along with the guided docs below, they will elaborate the use of environment level variable etc, also I have included few sample links as well.) hope material and sample below will help you. :)
Also I would recommend to recreate your pool if it is old which will ensure you have the node running at the latest version.
Azure batch learning path:
Samples & demo link or look here
Detailed walk through depending on what you are using i.e. CloudServiceConfiguration or VirtualMachineConfiguration link.
Further to add from the article: also look in here: Application Packages with VM configuration
In particular this link will take you through the guide process of how to use it in your code: Also be it resource file or package you need to make sure that they are uploaded and available for use at the batch level.
along with sample like: (below is a pool level pkg example)
// Create the unbound CloudPool
CloudPool myCloudPool =
batchClient.PoolOperations.CreatePool(
poolId: "myPool",
targetDedicatedComputeNodes: 1,
virtualMachineSize: "small",
cloudServiceConfiguration: new CloudServiceConfiguration(osFamily: "4"));
// Specify the application and version to install on the compute nodes
myCloudPool.ApplicationPackageReferences = new List<ApplicationPackageReference>
{
new ApplicationPackageReference {
ApplicationId = "litware",
Version = "1.1" }
};
// Commit the pool so that it's created in the Batch service. As the nodes join
// the pool, the specified application package is installed on each.
await myCloudPool.CommitAsync();
For the Task level form the link above a sample is: (make sure you have followed the steps correctly mentioned here.
CloudTask task =
new CloudTask(
"litwaretask001",
"cmd /c %AZ_BATCH_APP_PACKAGE_LITWARE%\\litware.exe -args -here");
task.ApplicationPackageReferences = new List<ApplicationPackageReference>
{
new ApplicationPackageReference
{
ApplicationId = "litware",
Version = "1.1"
}
};
further to add: be it CloudServiceCOhnfiguration or VirtualMachineConfiguration, An application package is **a .zip file** that contains the application binaries and supporting files that are required for your tasks to run the application. Each application package represents a specific version of the application. from reference: 4
I gave it a shot and tried and was successful, so I am not able to replicate the error above and seems like you might be missing something.
I’ve uploaded some files to Blob storage, and now I’m using the OnStart method to retrieve those files and run them. Right now I’m working locally.
Using the following code:
using (var fileStream = System.IO.File.OpenWrite(#"C:\testfolder"))
{
blob.DownloadToStream(fileStream);
}
Results in a “Access to the path 'C:\testfolder' is denied.” error.
What do you think is causing this? And - will this be an issue once the project is actually pushed up to Azure? I can change permissions locally, but I'm hoping that once it's actually in a live worker role, it won't be an issue.
Any help would be awesome :)
Scratch that - it looks like the C:\testfolder should specify the file name, not the location. I've changed it to C:\testfolder\test.txt and it works just fine :).
How do I create new instances of some role via C# using Azure emulator? Is there some guide about that? There are some manuals about creating instances in the cloud, not in emulator.
So far I know that:
I need to change config-file. Is it config in sln-file or in some temp-delpoyment folder?
I need to use csrun tool. How to pick params?
UPD
Got it.
To change count or instances on emulator, you have to:
update 'ServiceConfiguration.cscfg' file in bin-folder
run 'csrun' tool with params: string.Format("/update:{0};\"{1}\"", deploymentId, "<path to ServiceConfiguration.cscfg>")
where deploymentId:
// get id from RoleEnvironment with regex
var patternt = Regex.Escape("(") + #"\d+" + Regex.Escape(")");
var input = RoleEnvironment.DeploymentId;
var m = Regex.Match(input, patternt);
var deploymentId = m.ToString().Replace("(", string.Empty).Replace(")", string.Empty);
If you have troubles running csrun via code, read this:
http://social.msdn.microsoft.com/Forums/en/windowsazuredevelopment/thread/62ca1372-2388-4181-9dbd-8fbba470ea77
In local emulator, you need to modify the CSCFG file under the deployment .csx folder, instead of your source code folder, since the local emulator will fire your application up from that folder.
Once you modified the saved your CSCFG file, for example the count of the instances you can retrieve the new value from your code immediately. But if you want the local emulator detect this changes and perform the related actions, such as increase the VMs or invoke the Configuration_Changed method, you need to execute
csrun /update:;
You can retrieve the deployment id from the compute emulator UI.
You can find the instance count in the ServiceConfiguration.cscfg in your Azure project
I am trying to run a C executable on Azure. I have many workerRoles and they continuously check a Job Queue. If there is a job in the queue, a worker role runs an instance of the C executable as a process according to the command line arguments stored in a job class. The C executable creates some log files normally. I do not know how to access those created files. What is the logic behind it? Where are the created files stored? Can anyone explain me? I am new to azure and C#.
One other problem is that all of the working instances of the C executable need to read a data file. How can I distribute that required file?
First, realize that in Windows Azure, your worker role is simply running inside a Windows 2008 Server environment (either SP2 or R2). When you deploy your app, you would deploy your C executable as well (or grab it from blob storage, but that's a bit more advanced). To find out where your app lives on disk, call Environment.GetEnvironmentVariable("RoleRoot") - that returns a path. You'd typically have your app sitting in a folder called AppRoot under the role root. You'd find your C executable there.
Next, you'll want your app to write its files to an output directory you specify on the command line. You can set up storage in your local VM with your role's properties. Look at the Local Storage tab, and configure a named local storage area:
Now you can get the path to that storage area, in code, and pass it as a command line argument:
var outputStorage = RoleEnvironment.GetLocalResource("MyLocalStorage");
var outputFile = Path.Combine(outputStorage.RootPath, "myoutput.txt");
var cmdline = String.Format("--output {0}", outputFile);
Here's an example of launching your myapp.exe process, with command line arguments:
var appRoot = Path.Combine(Environment.GetEnvironmentVariable("RoleRoot")
+ #"\", #"approot");
var myProcess = new Process()
{
StartInfo = new ProcessStartInfo(Path.Combine(appRoot, #"myapp.exe"), cmdline)
{
CreateNoWindow = false,
UseShellExecute = false,
WorkingDirectory = appRoot
}
};
myProcess.WaitForExit();
Normally you'd set CreateNoWindow to true, but it's easier to debug if you can see the command shell window.
Last thing: Once your app is done creating the file, you'll want to either:
Process it and delete it (it's not in a durable place so eventually it'll disappear)
Change your storage to use a Cloud Drive (durable storage)
Copy your file to a blob (durable storage)
In production, you'll want to add exception-handling, and you can re-route stdout and stderr to be captured. But this sample code should be enough to get you started.
OOPS - one more 'one more thing': When adding your 'myapp.exe' to your project, be SURE to go to its Properties, and set 'Copy to Output Directory' to 'Copy Always' - otherwise your myapp.exe file won't end up in Windows Azure and you'll wonder why things don't work.
EDIT: Pushing results to a blob - a quick example
First get set up a storage account and add to your role's Settings. Say you called it 'AzureStorage' - now set it up in code, get a reference to a blob container, get a reference to a blob within that container, and then perform a file upload to the blob:
CloudStorageAccount storageAccount = CloudStorageAccount.FromConfigurationSetting("AzureStorage");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer outputfiles = blobClient.GetContainerReference("outputfiles");
outputfiles.CreateIfNotExist();
var blobname = "myoutput.txt";
var blob = outputfiles.GetBlobReference(blobname);
blob.UploadFile(outputFile);
In Azure land you shouldn't write to the file system. You should write to SQL Azure, Table Storage or most likely in this case Blob storage (basically, I think you should think of blob storage as the old file system)
This is because:
You could have multiple instances running and you will end up having different files on different instances (which are just virtual machines)
Your instance could potentially be moved at any moment and you would lose the info on the file system as it's not part of your deployment package.
Using one of the three storage options will provide a central repository for all of your instances to access and it will be persisted over a redeployment.