I am trying to deploy a model to a managed online endpoint in Azure Machine Learning.
(Along the lines of https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-managed-online-endpoints).
This works fine with a publically accessible AML workspace, but not with our workspace inside our private VNET.
With an AML workspace/storage account in a private VNET the deployment fails:
I found that this is directly related to the network setting of the storage account. This is the setting that fails. Note that allowing Azure Services does not mitigate the problem:
Is this is blind spot of managed endpoints, which is simply not yet supported or is this a bug?
The problem can be reproduced with the sample code at
https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/managed/sample
i.e.
az ml online-deployment create --name blue -f endpoints/online/managed/sample/blue-deployment.yml
The 'troubleshooting' guide in the error message refers to the importance of accessibility of the storage account (and Azure Container Registry), but does not consider the usecase, where AML is inside a private VNET:
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-online-endpoints?tabs=cli#authorization-error
If workspace and storage are private, you need to make egress_public_network_access flag disabled. This flag is required to establish private endpoint connections from managed online deployment to your private resources. Do not forget to approve PE connections.
Doc for Managed Online Endpoint network isolation
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-online-endpoint?tabs=model
Related
My goal is to run an exe file stored in a private Azure Blob container.
The exe is simple : it creates a text file, write the current datetime in it, and then push it to the private Azure Blob container.
This has to be sent from Azure Data Factory. To do this, here is my environment :
Azure Data Factory running with the simple pipeline :
https://i.stack.imgur.com/txQ9r.png
Private storage account with the following configuration :
https://i.stack.imgur.com/SJrGX.png
A linked service connected to the storage account :
https://i.stack.imgur.com/8xW5l.png
A private managed virtual network approved :
https://i.stack.imgur.com/G2DH3.png
A linked service connected to an Azure Batch :
https://i.stack.imgur.com/Yaq6C.png
A batch account linked to the right storage account
A pool running on this batch account
Two things that I need to add in context :
When I set the storage account to public, it works and I find the text file in my blob storage. So the process works well, but there is a security issue somewhere I can't find.
All the resources (ADF, Blob storage, Batch account) used have a role has contributor/owner of the blob with a managed identity.
Here is the error I get when I set the storage account to private :
{
"errorCategory":0,
"code":"BlobAccessDenied",
"message":"Access for one of the specified Azure Blob(s) is denied",
"details":[
{
"Name":"BlobSource",
"Value":"https://XXXXXXXXXXXXXXXXX/testv2.exe?sv=2018-03-28&sr=b&sig=XXXXXXXXXXXXXXXXXX&sp=r"
},
{
"Name":"FilePath",
"Value":"D:\\batch\\tasks\\workitems\\XXXXXXXXXXX\\job-1\\XXXXXXXXXXXXXXXXXXXXXXXX\\testv2.exe"
}
]
}
Thank you for your help!
Solution found Azure community support :
Check Subnet information under Network Configuration from the Azure portal > Batch Account > Pool > Properties. Take note and write the information down.
Navigate to the storage account, and select Networking. In the Firewalls and virtual networks setting, select Enable from selected virtual networks and IP addresses for Public network access. Add the Batch pool's subnet in the firewall allowlist.
If the subnet doesn't enable the service endpoint, when you select it, a notification will be displayed as follows:
The following networks don't have service endpoints enabled for 'Microsoft.Storage'. Enabling access will take up to 15 minutes to complete. After starting this operation, it is safe to leave and return later if you don't wish to wait.
Therefore, before you add the subnet, check it in the Batch virtual network to see if the service endpoint for the storage account is enabled.
After you complete the configurations above, the Batch nodes in the pool can access the storage account successfully.
I'm trying to connect blob storage from my RG storage account to the Data tab in my synapse workspace, but I get the following error: "The public network interface on this Workspace is not accessible. To connect to this Workspace, use the Private Endpoint from inside your virtual network or enable public network access for this workspace."
Public network access to my workspace must be disabled for company reasons. I made private endpoint connections on my synapse resource to Dev, Sql, and Sql-On-Demand, but I'm not sure where to go from there.
Thanks!
Go to Azure Synapse -> Manage -> Managed private endpoints -> +New and add private endpoints.
Accessing blob storage : If you already created linked service the follow below image. If not, please created and follow this Ms Doc for creating linked service.
Fastest way to access Azure blob storage
For for information follow this reference by Dennes Torres.
We are currently migrating to a new Azure Subscription and are having issues executing Azure Functions that worked as expected in our old Azure Subscription. The man difference between our old Subscription and our new Subscription is that we have set up a Virtual Network with Subnets and have deployed our Resources behind the Subnets.
We have also had to migrate from an Azure App Service in the old Subscription to a Azure App Environment in the new Subscription.
Our Azure environment consist of:
App Service Environment
App Service Plan I1
The Azure App Environment and Storage Containers are on the same Virtual Network but different Sub Nets. The Function is using a Managed Identity which has Owner Role on Storage Account.
The code listed below worked just fine in our old environment which did not contain the Virtual Network, but fails in our new environment.
Any guidance would be greatly appreciated.
The Azure function which connects to Azure Storage works when run locally from Visual Studio 2019, but fails when run from Azure portal.
Code Snippet below:
This section works just fine:
string storageConnectionString = XXXXConn.ConnectionETLFileContainer();//Get Storage connection string
var myDirectory = "XXXX/Uploads"; ///XXXX-etl-file-ingest/ABSS/Uploads/ CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();// Create a CloudBlobClient object for credentialed access to Azure Blob. CloudBlobContainer blobContainer = blobClient.GetContainerReference("XXXX-etl-blobfile-ingest");// Get a reference to the Blob Container we created previously. CloudBlobDirectory blobDirectory = blobContainer.GetDirectoryReference(myDirectory);// Get a reference to the Blob Directory.
var blobs = blobDirectory.ListBlobs(useFlatBlobListing: true); //set useFlatBlobListing as true
This statement fails: Failure occurs when trying to iterate through the Blob files and get specific file info.
foreach (var myblob in blobs)
In the azure portal open storage account blade under that go to configuration blade , you will be able to see the list of networks for which your storage account has allowed access to.Once you have the allowed network list kindly check if the function app is on one of those networks if not then you need to get the network on which your function app is hosted added to the list.
Update 2:
The simplest explanation/cause that I found is when an App Service or Function App has the setting WEBSITE_VNET_ROUTE_ALL set to 1, all traffic to public endpoints is blocked. So if your Storage Account has no private endpoint configured, requests to it will fail.
Docs: "To block traffic to public addresses, you must have the application setting WEBSITE_VNET_ROUTE_ALL set to 1."
https://learn.microsoft.com/en-us/azure/app-service/web-sites-integrate-with-vnet#network-security-groups
Update 1:
My answer below was only a workaround for my problem. Turns out I did not link the Private DNS Zone (this is created for you when you create a new Private Endpoint) to my VNET.
To do this, go to your Private DNS Zone in the Azure Portal and click on Virtual network links in the left menu bar. There add a new link to the VNET your Function is integrated in.
This may not have been relevant for the OP, but hopefully it will help others.
Original answer:
In my case this was solved by enabling the Microsoft.Storage Service Endpoint on the App Service's subnet (dedicated subnet).
I'm setting up a Storage Account so I can Dynamically create and use a persistent volume with Azure Files in Azure Kubernetes Service (AKS). Doing this to:
Have a PV and PVC for the database
A place to store the application files
AKS does create a storage account in the MC_<resource-group>_<aks-name>_<region> resource group that is automatically created. However, that storage account is destroyed if the node size/VM is changed (not node count), so it shouldn't be used since you'll lose your files and database if you need a node size/VM with more resources.
This documentation, nor any other I've really come across, says what the best practice is for the Connectivity method:
Public endpoint (all networks)
Public endpoint (selected networks)
Private endpoint
The first option sounds like a bad idea.
The second option allows me to select a virtual network, and there are two choices:
MC_<resource-group>_<aks-name>_<region>... again, doesn't seem like a good idea because if the node size/VM is changed, the connection will be broke.
aks-vnet-<number>... not sure what this is, but looks like it is part of the previous resource group so will also be destroyed in the previously mentioned scenario.
The third option contains a number of options some of which are included the second option.
So how should I securely set this up for AKS to share files with the application and persist database files?
EDIT
Looking at the both the "Firewalls and virtual networks" and "Private endpoint connections" for the storage account that comes with the AKS node, it looks like it is just setup for "All networks"... so maybe having that were my actual PV and PVC will be stored isn't such an issue...? Could use some clarity on the topic.
not sure where the problem lies. all the assets generated by AKS are tied to AKS lifecycle. if you delete AKS it will delete the MC_* resource group (and that it 100% right). Not sure what do you mean about storage account being destroyed, it wouldn't get destroyed unless you remove the pvc and set the delete action to reclaim.
Reading: https://learn.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv
As for the networking part, selected networks with selecting the AKS nodes network should be the way to go. you can figure that network out by looking at the AKS nodes or the AKS agent pool definition(s). I dont think this is configurable only using kubernetes primitives, so that would be a manual\scripted action after storage account is created.
I'm trying to create an Azure Container Instance in the portal with a private image. The validation seems to pass but I can't click OK. The button is disabled.
The private registry is an Azure Container Registry in the same subscription.
Looks like this was resolved on your own after waiting between attempts. For future reference if you need to deploy a private image from the portal you need to include the hostname of the image like "myprivateregistry/nginx:latest".
We've also improved some error handling so deployment failures should also be easier to troubleshoot.