I have data stored in Azure Table Storage and want to secure it such that only my API (a function app) can read and write data.
What is best practice and how can I do this? I thought setting --default-action on the network rules to Deny for the Storage, plus adding a --bypass Logging Metrics AzureServices would shut down access but enable my Azure services, but this did not work.
I then looked at creating a Managed Service Identity (MSI) for the function app and adding RBAC to the Storage Account, but this did not work either. It doesn't look like MSIs are supported for Table Storage Access Azure Table Storage with Azure MSI
Am I missing or misunderstanding something? How do I secure the data in the tables in the Storage account, and is this even possible?
As the link you provided, azure table storage does not support Azure MSI, and it only support Shared Key (storage account key) and Shared access signature (SAS).
You must use Shared Key authorization to authorize a request made against the Table service if your service is using the REST API to make the request.
To encode the signature string for a request against the Table service made using the REST API, use the following format:
StringToSign = VERB + "\n" +
Content-MD5 + "\n" +
Content-Type + "\n" +
Date + "\n" +
CanonicalizedResource;
You can use Shared Key Lite authorization to authorize a request made against any version of the Table service.
StringToSign = Date + "\n"
CanonicalizedResource
For more details, you could refer to this article.
For securing Azure Table Storage data you do below network configurations -
Use selected network instead of public network. This configuration is available under "Firewalls and virtual networks" of storage account.
Second step which you can do is to either move the data to Azure Key Vault or use an encryption key stored in Azure Key Vault to encrypt required fields of Azure Table Storage. This way you won't face Azure Key Vault's throttling limits - https://learn.microsoft.com/en-us/azure/key-vault/general/service-limits#secrets-managed-storage-account-keys-and-vault-transactions
Related
For sake of safety I wish to use geo-replication / secondary Blob Storage container for a data source for AzureML Datastore. So I do the following:
New Datastore
Enter name + Azure Blob Storage + Enter manually
For URL I paste "Secondary Blob Service Endpoint" value from "Storage account endpoints" and I add container name at the end, e.g. https://somedata-secondary.blob.core.windows.net/container-name
Select subscription ID
I select the resource group in which somedata is hosted,
I add account key taken from "Access keys" section, I tried also with SAS token
After finalizing, the new datastore seem to appear in the list but it is impossible to Browse (preview), throwing the error "Invalid host".
What is the correct way of doing this?
Is it possible at all to access this geo-replication / secondary Blob Storage as datastore?
Please check with below points:
Initially please check if Share Access Token (SAS) token is outdated or expired
Please note that Both primary and geo-secondary are required to have the same service tier and strongly recommended that the geo-secondary is configured with the same backup storage redundancy and compute size as the primary.
Note: You can only access your storage account by its primary name. In the event of failover, that name will be mapped to the alternate datacenter.
There are two disadvantages of GRS redundancy:
Replication between regions is asynchronous and so data is propagated with a small delay
The second region cannot be accessed or read until the storage account fails over
Active geo-replication - Azure SQL Database | Microsoft Docs
As the replicated endpoint will be https://account-secondary.blob.core.windows.net. Note that this DNS entry won’t even be registered unless read access geo redundant replication is enabled.
The access keys for your storage account are the same for both the primary and secondary endpoints. You can use the same primary (or secondary) access key for the secondary too.
I have created a PowerShell script to update a key vault with new blob storage key every time when the key rotates but the problem I have is how do I update the apps with the new blob storage key.
I have used Set-AzKeyVaultAccessPolicy to give the apps access to the key vault secret which contains the latest storage key. I have a logic app which uses blob storage but when the key is rotated the blob storage within the logic app show an error. This is the error I encounter:
{
"status": 403,
"message": "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\r\nclientRequestId: 7c8edbdf-e7c9-4658-8370-54102589213e",
"source": "azureblob-uks.azconn-uks-01.p.azurewebsites.net"
}
Is there a way for the logic app to get the latest key from the key vault.
Logic app does support a connector to connect your logic app to Azure KeyVault to retrieve the keys: https://learn.microsoft.com/en-us/connectors/keyvault/
My suggestion is to write a specific script which would discard the rotated keys, then retrieve the new ones as soon as they are available in Azure Key vault.
When I develop for Azure I usually start copying in some keyvault client code so only keyvault urls will be in my settings file, no secrets can ever end up my git repositories.
After starting to make Azure functions I realized that it was not possible to do this for the trigger connection string for e.g. service bus or blob storage.
The recommended approach seems to connect the app to keyvault directly in Azure when deployed, and just manage secrets locally in Secret Manager, like suggested in
this article
I am not developing alone, so while I am not adverse to using a tool like Secret Manager, I need to still have my offline secrets connected to the Azure keyvault! If others change anything.
Question: How do I manage secrets offline in a way that is synchronized with Azure keyvault?
it was not possible to do this for the trigger connection string for e.g. service bus or blob storage.
In short, it's possible.
Here are steps you could follow and refer to the detailed article.
1.Add a System Assigned Managed Identity to the Azure Function.
2.Go to the Access Control section of your Key Vault and click on Add a role assignment blade.
3.Go to your Key Vault and click on Access Policies and then click on Add service principal with secret GET permission.
4.When you use ServiceBusTrigger, you set ServiceBusConnectionString in Function ->Configuration ->Application settings.
public static void Run([ServiceBusTrigger(_topicName, _subscriptionName, Connection = "ServiceBusConnectionString")] string mySbMsg, ILogger log)
{ ....
}
5.Now you change the value of ServiceBusConnectionString to the Azure Key Vault reference with #Microsoft.KeyVault(SecretUri=Secret URI with version). Then you could run your function successfully with Key Vault.
I need to load external data (in blob storage) to my Azure data warehouse using Polybase. I had it working fine when I was using Classic Azure Storage.
Recently, I have to update our Storage to ARM and I could not figure out how to set up the firewall rule on the ARM Storage to my Azure data warehouse. If I set the firewall to "All networks" everything works seamlessly. However, I cannot let the blob wide open.
I tried using nslookup to find the outbound ip for our Azure Data warehouse and put the value into the Firewall of the Storage; I got "This request is not authorized to perform this operation." error
Is there a way I can find the ip address for an Azure Data warehouse? Or I should use different approach to make it work?
Any Suggestions are appreciated.
Kevin
Under the section 1.1 Create a Credential, it states:
Don't skip this step if you are using this tutorial as a template for loading your own data. To access data through a credential, use the following script to create a database-scoped credential, and then use it when defining the location of the data source.
-- A: Create a master key.
-- Only necessary if one does not already exist.
-- Required to encrypt the credential secret in the next step.
CREATE MASTER KEY;
-- B: Create a database scoped credential
-- IDENTITY: Provide any string, it is not used for authentication to Azure storage.
-- SECRET: Provide your Azure storage account key.
CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
WITH
IDENTITY = 'user',
SECRET = '<azure_storage_account_key>'
;
-- C: Create an external data source
-- TYPE: HADOOP - PolyBase uses Hadoop APIs to access data in Azure blob storage.
-- LOCATION: Provide Azure storage account name and blob container name.
-- CREDENTIAL: Provide the credential created in the previous step.
CREATE EXTERNAL DATA SOURCE AzureStorage
WITH (
TYPE = HADOOP,
LOCATION = 'wasbs://<blob_container_name>#<azure_storage_account_name>.blob.core.windows.net',
CREDENTIAL = AzureStorageCredential
);
Edit: (additional way to access Blobs from ADW through the use of SAS):
You also can create a Storage linked service by using a shared access signature. It provides the data factory with restricted/time-bound access to all/specific resources (blob/container) in the storage.
A shared access signature provides delegated access to resources in your storage account. You can use a shared access signature to grant a client limited permissions to objects in your storage account for a specified time. You don't have to share your account access keys. The shared access signature is a URI that encompasses in its query parameters all the information necessary for authenticated access to a storage resource. To access storage resources with the shared access signature, the client only needs to pass in the shared access signature to the appropriate constructor or method. For more information about shared access signatures, see Shared access signatures: Understand the shared access signature model.
Full document can be found here
I am using an API (node.js) to generate a read only shared access signature for an iOS app using Azure Mobile Services. The API generates the SAS using the following code...
var azure = require('azure-storage');
var blobService = azure.createBlobService(accountName, accountKey);
var sas = blobService.generateSharedAccessSignature("containerName", null, sharedAccessPolicy);
This works great when I want a SAS for access to one container. But I really need access to all containers in the storage account. I could obviously do this with a separate API call for each container but this would require hundreds of extra calls.
I have looked everywhere for a solution but I can't get anything to work, I would very much appreciate knowing if there is a way to generate a SAS for all containers in a storage account?
You can construct an account-level SAS, where you get to specify:
services to include (blob, table, queue, file)
resource access (e.g. container create & delete)
permissions (e.g. read, write, list)
protocol (e.g. https only, vs http+https)
Just like a service-specific SAS, you get to specify expiry date (and optionally start date).
Given your use case, you can tailor your account SAS to be just for blobs; there's no need to include unneeded services (in your case, tables/queues/files).
More specifics are documented here.