I want to pass credentials, to Azure blob component, without having to create and register a bean.
I have a route which consumes a file and routes it to an Azure Blob
I want this route to be generic.
The challenge I am facing is.
I have to support multiple storage accounts
We are supposed to fetch the credentials from a key vault. The credentials can change any time and I don't have control. The blob component requires a bean for credentials. So the only option I am left is a bean. So while I was able to achieve this, wondering, if camel should be able to receive the credentials, not the object or client , but the account name and key as an option. Camel doesn't support key vault which is kind of a bummer.I might be able to help with implementing the keyVault solution
This is the bean I wrote to achieve what I wanted.
from(file://somedir).to(bean:azureService?sendFiletoAzure(name,key,container);
AzureService{
public void sendFileToAzure(String storageAccountName,
String storageAccountKey, String containerName, Exchange exchange) {
StorageCredentials credentials = new StorageCredentialsAccountAndKey(
storageAccountName, storageAccountKey);
SimpleRegistry registry = new SimpleRegistry();
GenericFile<?> file = (GenericFile<?>) exchange.getIn().getBody();
String blobName = (String) exchange.getIn().getHeader("blobName");
CamelContext context = new DefaultCamelContext(registry);
registry.put("blobCredentials", credentials);
String endpointUri = String.format(azureEndpointUri, storageAccountName,
containerName, blobName);
context.createProducerTemplate().sendBody(endpointUri, file);
}
}
Related
I am using libraries Microsoft.Azure.Storage.Blob 11.2.3.0 and Microsoft.Azure.Storage.Common 11.2.3.0 to connect to an Azure BlobStorage from a .NET Core 3.1 application.
When I started working on this, I had been given connection strings that gave me full access to the BlobStorage (or rather, the entire cloud storage account). Based upon those, I chose to write my connection code "defensively", making use of Exists() and CreateIfNotExists() from the CloudBlobContainer class to ensure the application would not fail when a container was not yet existing.
Now, I'm connecting a BlobStorage container using a SAS. While I can freely retrieve and upload blobs within the container like this, unfortunately, it seems that I am not allowed to do anything on the container level. Not only CreateIfNotExists, but even the mere querying of existence by Exists() throws a StorageException saying
This request is not authorized to perform this operation.
The documentation does not mention the exception.
Is there any way to check preemptively whether I am allowed to check the container's existence?
I have tried looking into the container permissions retrieved from GetPermissions, but that will throw an exception, as well.
The only other alternative I can see is to check for container existence within a try-catch-block and assume existence if an exception is thrown ...
There's a no definitive way to identify if an operation can be performed using a SAS token other than performing that operation and catching any exception that may be thrown by the operation. The exception that is of your interest is Unauthorized (403).
However you can try to predict if an operation can be performed by looking at the SAS token. If it is a Service SAS Token and not an Account SAS Token, that means all the account related operations are not not allowed. The way to distinguish between an Account SAS token and a Service SAS token is that the former will contain attributes like SignedServices (ss) and SignedResourceTypes (srt).
Next thing you would want to do is look for SignedPermissions (sp) attribute in your SAS token. This attribute will tell you what all operations are possible with the SAS token. For example, if your SAS token is a Service SAS token and if it includes Delete (d) permission, that would mean you can use this SAS token to delete a blob.
Please see these tables for the permissions/allowed operations combinations:
Service SAS Token: https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas#permissions-for-a-directory-container-or-blob
Account SAS Token: https://learn.microsoft.com/en-us/rest/api/storageservices/create-service-sas#permissions-for-a-directory-container-or-blob
Please note that the operation might still fail for any number of reasons like SAS token has expired, account key has changed since the generation of SAS token, IP restrictions etc.
I tried in in my system to check whether the container exist or not able check it and if container not exists created container and able to upload file.
You need to give proper permission for your SAS Token
const string sasToken = “SAS Token”
const string accountName = "teststorage65";
const string blobContainerName = "example";
const string blobName = "test.txt";
const string myFileLocation = #"Local Path ";
var storageAccount = new CloudStorageAccount(storageCredentials, accountName, null, true);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference(blobContainerName);
var result=blobContainer.Exists();
if (result == true)
{
Console.WriteLine("Container exists");
}
else
{
// blobContainer.CreateIfNotExists();
Console.WriteLine("Conatiner not exists");
Console.WriteLine("Creating Container "+ blobContainerName);
blobContainer.CreateIfNotExists();
}
// blobContainer.CreateIfNotExists();
//Console.WriteLine("Creating Container ");
CloudBlockBlob cloudBlob = blobContainer.GetBlockBlobReference(blobName);
cloudBlob.UploadFromFile(myFileLocation);
OUTPUT
I've created and published (to Azure) a working and fairly simple Azure Function App (HTTP Trigger) which inserts data retrieved from an API call to an Azure SQL database after authenticating via User identity (if testing locally) or Managed Identity (when running on VM).
Everything is functional, however, I now need to encrypt one of the SQL table columns which I have done using SSMS. The next step as I understand it is authenticating a provider to access the CMK via Azure Key Vault (I'm following this Microsoft guide).
I'm wondering in the following code how to InitializeAzureKeyVaultProvider without using applicationId and clientKey from Azure App registration resource, but with a user or managed identity role. Or, if there's any other way to get/use applicationId, and clientKey without creating/using an Azure App registration resource.
Is there a newer / easier way to access Azure Key Vault for Always Encrypted Columns sql queries?
static void InitializeAzureKeyVaultProvider() {
_clientCredential = new ClientCredential(applicationId, clientKey);
SqlColumnEncryptionAzureKeyVaultProvider azureKeyVaultProvider = new SqlColumnEncryptionAzureKeyVaultProvider(GetToken);
Dictionary<string, SqlColumnEncryptionKeyStoreProvider> providers = new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>();
providers.Add(SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, azureKeyVaultProvider);
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(providers);
}
}
Here is the other way I've been attempting, however, upon installing and using Microsoft.Data.SqlClient.AlwaysEncrypted.AzureKeyVaultProvider, I get the error following this code sample:
private static bool EncryptionProviderInitialized = false;
private static void InitializeAzureKeyVaultProvider()
{
if (!EncryptionProviderInitialized)
{
SqlColumnEncryptionAzureKeyVaultProvider akvProvider = null;
#if DEBUG
if (Debugger.IsAttached)
{
Console.WriteLine("Debugger attached - configuring KeyVaultProvider via VisualStudioCredential");
akvProvider = new SqlColumnEncryptionAzureKeyVaultProvider(new VisualStudioCredential());
}
if (akvProvider == null)
#endif
akvProvider = new SqlColumnEncryptionAzureKeyVaultProvider(new ManagedIdentityCredential());
SqlConnection.RegisterColumnEncryptionKeyStoreProviders(customProviders: new Dictionary<string, SqlColumnEncryptionKeyStoreProvider>(capacity: 1, comparer: StringComparer.OrdinalIgnoreCase)
{
{ SqlColumnEncryptionAzureKeyVaultProvider.ProviderName, akvProvider}
});
EncryptionProviderInitialized = true;
}
}
ERROR:
[2021-10-08T01:36:24.209Z] Executed 'EncryptedSQLMigration' (Failed, Id=1323dcbb-e671-4ed4-8a7c-6259447326c5, Duration=537ms)
[2021-10-08T01:36:24.209Z] System.Private.CoreLib: Exception while executing function: EncryptedSQLMigration. FreshFirstApproach: Method not found: 'Microsoft.Extensions.Primitives.StringValues Microsoft.AspNetCore.Http.IQueryCollection.get_Item(System.String)'.
Initially, I got that same error, however, with Microsoft.Extensions.Logging.Abstractions - upon removing the ILogger from my main function just for the sake of moving on as I wasn't able to solve that issue either, I now get this Microsoft.Extensions exception.
Any help with my goal of using Always Encrypted Columns with Azure Function App and Azure Key Vault is very much appreciated!
Thank you very much.
It is not possible to use applicationId and clientKey without creating or using an Azure App registration resource. There is an alternative way where you can pass clientId and clientSecret as shown below, but here also you will need application registration.
static void InitializeAzureKeyVaultProvider()
{
string clientId = ConfigurationManager.AppSettings["AuthClientId"];
string clientSecret = ConfigurationManager.AppSettings["AuthClientSecret"];
_clientCredential = new ClientCredential(clientId, clientSecret);
....
....
}
As for user or managed identity, if you check this document you won't find the Azure Key Vault in the list of services that support managed resources.
So managed service identity should be created for Azure Function instead of Azure Key Vault. Check this Retrieve Azure Key Vault Secrets using Azure Functions and Managed Service Identity document for more information.
From my question here I understand that I can set up an application registration in Active Directory, and that I can use the application ID and a key that I set up within the application registration in order to authenticate.
Where is an example on how to do that?
What has the combination of the application ID (which I understand to also be called the client ID) and the key I add to the keys collection got to do with the Service Principal?
[Update]
From this link about service principals
If I understand it correctly we are no longer talking about "application key", we are talking about "application credentials". I am guessing this is the same thing?
The following paragraph has me hopelessly confused about the difference between "application credentials", "sign in credentials", and "service principal's credentials":
"To sign in with a service principal, use the -ServicePrincipal argument with the Connect-AzureRmAccount cmdlet. You will also need the service princpal's application ID, sign-in credentials, and the tenant ID associate with the service principal. In order to get the service principal's credentials as the appropriate object, use the Get-Credential cmdlet. This cmdlet will display a dialog box to enter the service principal user ID and password into."
[Update]
From the answer to my question here I have been able to run HelloKeyVault using the following app settings:
VaultUrl, AuthClientId and AuthCertThumbprint
There is no mention of a service principal or "key" or a "token"
I am just trying to understand the instructions at https://learn.microsoft.com/en-gb/azure/key-vault/key-vault-get-started at this stage.
You typically use the service principal to deploy / manage your resources within a CI / CD environment like VSTS or within PowerShell scripts. Check Sign in with a service principal
From the sample application here after fixing the bug in the Powershell script reported here
I was able to run the HelloKeyVault source
class Program
{
static KeyVaultClient keyVaultClient;
static InputValidator inputValidator;
static void Main(string[] args)
{
KeyBundle keyBundle = null; // The key specification and attributes
SecretBundle secret = null;
CertificateBundle certificateBundle = null;
string keyName = string.Empty;
string secretName = string.Empty;
string certificateName = string.Empty;
string certificateCreateName = string.Empty;
inputValidator = new InputValidator(args);
ServiceClientTracing.AddTracingInterceptor(new ConsoleTracingInterceptor());
ServiceClientTracing.IsEnabled = inputValidator.GetTracingEnabled();
var clientId = ConfigurationManager.AppSettings["AuthClientId"];
var cerificateThumbprint = ConfigurationManager.AppSettings["AuthCertThumbprint"];
var certificate = FindCertificateByThumbprint(cerificateThumbprint);
var assertionCert = new ClientAssertionCertificate(clientId, certificate);
keyVaultClient = new KeyVaultClient((authority, resource, scope) => GetAccessToken(authority, resource, scope, assertionCert),
new InjectHostHeaderHttpMessageHandler());
// etc
This shows that we can get the token using the AuthClientId and the AuthCertThumbprint
In this case guess the Application Id is given by the AuthClientId and the "key to authenticate" is given by the AuthCertThumbprint
The Service principal is not mentioned as being necessary.
I am having a blob container which ACL is set up to allow full public read access so anyone can read and list the blobs in that container.
I store the files so the WPF client app my clients use could read them but I don't want to allow them to modify/delete/create files.
Does anyone knows what connection string should be used in this scenario?
I hoped to specify the connection string without the account key and/or shared access key due to the fact blobs are public but that didn't work - StorageAccount.Parse throws FormatException
As mentioned by the previous answers, the best practice is usually to control the access to your blob container using shared access signatures (SAS) or a stored access policy. These can be used to create an access token (string) you can pass to your client without revealing your account key.
However, it is also possible to specify the level of public read access to the blobs and metadata saved in the container. Public access is the level of read permission automatically given an anonymous user that is in possession the public access url for the container or blob. You cannot use public access to give anonymous users write permissions to the container. If you need to give write permission to users that are not in possession of the account key of your Azure storage account, then you will need to provide those users with a token in the form of a url the references a shared access signature or a shared access policy.
If the public access to the blob container is not currently off (private,) anonymous user will be able to read all blobs in the container using a public access url such as the following.
http://grassy.blob.core.windows.net/container1/image2.jpg
When you create the container, you can set the value of the publicAccess property to the appropriate constant of the BlobContainerPublicAccessType enum. The value of the publicAccess property can be one of the following three constants which specify the level of public read access.
• BLOB – The public can read the content and metadata of blobs within this container, but cannot read container metadata or list the blobs within the container.
• CONTAINER – The public can read blob content and metadata and container metadata, and can list the blobs within the container.
• OFF – Specifies no public access. Only the account owner can read resources in this container.
So in this case the public access level might be set to CONTAINER. For example:
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException
{
Account creds = new Account();
final String storageConnectionString = creds.getstorageconnectionstring();
CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer container = blobClient.getContainerReference("container1");
container.createIfNotExist();
BlobContainerPermissions containerPermissions = new BlobContainerPermissions();
containerPermissions.setPublicAccess(BlobContainerPublicAccessType.CONTAINER);
container.uploadPermissions(containerPermissions);
BlobContainerPublicAccessType access1 = containerPermissions.getPublicAccess();
System.out.println("Public access to " + container.getName() + " is set to:
" + access1);
}
If the public access level on container1 has been set to CONTAINER, an anonymous user should be able to list the blobs in container1 knowing only the storage account AccountName ("grassy") and the container name, but without needing to know the AccountKey. For example, an anonymous application might use java code similar to the following:
public static void main(String[] args) throws InvalidKeyException, URISyntaxException, StorageException, FileNotFoundException, IOException
{
URI baseuri = new URI("http://grassy.blob.core.windows.net");
CloudBlobClient blobclient = new CloudBlobClient(baseuri);
CloudBlobContainer container = blobclient.getContainerReference("container1");
for (ListBlobItem blobItem : container.listBlobs()){System.out.println(blobItem.getUri());}
}
However, as discussed, it is a better practice to avoid giving anonymous users access. Instead control access to the container using a SAS or policy and pass on the token to only known users.
StorageAccount is not meant to connect to public blobs as far as I know. You simply can get at the public blobs via public URL by using something like WebClient or any other tool that can download data over public http/https endpoint.
You could use shared access signature for that purpose. What you could do is create the SAS on a blob container which only allows list and read permissions on the blob container and then distribute that SAS URI to your clients. Your code could then create an instance of BlobContainer object using that SAS URI.
Here's the sample code for listing blobs in a blob container using SAS URI:
static void ListBlobsWithStorageClientLibrary(string blobContainerSasUri)
{
CloudBlobContainer blobContainer = new CloudBlobContainer(new Uri(blobContainerSasUri));
var blobs = blobContainer.ListBlobs(null, true);
foreach (var blob in blobs)
{
Console.WriteLine(blob.Uri);
}
}
Other alternative is to create an instance of StorageCredentials object using SAS token: http://msdn.microsoft.com/en-us/library/windowsazure/jj682529.aspx. Then you could create an instance of CloudStorageAccount object using that StorageCredentials object.
I wrote a detailed post on using Shared Access Signatures with blob storage which you can read here: http://gauravmantri.com/2013/02/13/revisiting-windows-azure-shared-access-signature/
I am trying to access a blob stored in a private container in Windows Azure. The container has a Shared Access Signature but when I try
to access the blob I get a StorgeClientException "Server failed to authenticate the request. Make sure the Authorization header is formed
correctly including the signature".
The code that created the container and uploaded the blob looks like this:
// create the container, set a Shared Access Signature, and share it
// first this to do is to create the connnection to the storage account
// this should be in app.config but as this isa test it will just be implemented
// here:
// add a reference to Microsoft.WindowsAzure.StorageClient
// and Microsoft.WindowsAzure.StorageClient set up the objects
//storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["ConnectionString"]);
blobClient = storageAccount.CreateCloudBlobClient();
// get a reference tot he container for the shared access signature
container = blobClient.GetContainerReference("blobcontainer");
container.CreateIfNotExist();
// now create the permissions policy to use and a public access setting
var permissions = container.GetPermissions();
permissions.SharedAccessPolicies.Remove("accesspolicy");
permissions.SharedAccessPolicies.Add("accesspolicy", new SharedAccessPolicy
{
// this policy is live immediately
// if the policy should be delatyed then use:
//SharedAccessStartTime = DateTime.Now.Add(T); where T is some timespan
SharedAccessExpiryTime =
DateTime.UtcNow.AddYears(2),
Permissions =
SharedAccessPermissions.Read | SharedAccessPermissions.Write
});
// turn off public access
permissions.PublicAccess = BlobContainerPublicAccessType.Off;
// set the permission on the ocntianer
container.SetPermissions(permissions);
var sas = container.GetSharedAccessSignature(new SharedAccessPolicy(), "accesspolicy");
StorageCredentialsSharedAccessSignature credentials = new StorageCredentialsSharedAccessSignature(sas);
CloudBlobClient client = new CloudBlobClient(storageAccount.BlobEndpoint,
new StorageCredentialsSharedAccessSignature(sas));
CloudBlob sasblob = client.GetBlobReference("blobcontainer/someblob.txt");
sasblob.UploadText("I want to read this text via a rest call");
// write the SAS to file so I can use it later in other apps
using (var writer = new StreamWriter(#"C:\policy.txt"))
{
writer.WriteLine(container.GetSharedAccessSignature(new SharedAccessPolicy(), "securedblobpolicy"));
}
The code I have been trying to use to read the blob looks like this:
// the storace credentials shared access signature is copied directly from the text file "c:\policy.txt"
CloudBlobClient client = new CloudBlobClient("https://my.azurestorage.windows.net/", new StorageCredentialsSharedAccessSignature("?sr=c&si=accesspolicy&sig=0PMoXpht2TF1Jr0uYPfUQnLaPMiXrqegmjYzeg69%2FCI%3D"));
CloudBlob blob = client.GetBlobReference("blobcontainer/someblob.txt");
Console.WriteLine(blob.DownloadText());
Console.ReadLine();
I can make the above work by adding the account credentials but that is exactly what I'm trying to avoid. I do not want something
as sensitive as my account credentials just sitting out there and I have no idea on how to get the signature into the client app without having the account credentials.
Any help is greatly appreciated.
Why this?
writer.WriteLine(container.GetSharedAccessSignature(new SharedAccessPolicy(), "securedblobpolicy"));
and not writing the sas string you already created?
It's late and I could easily be missing something but it seems that you might not be saving the same access signature that you're using to write the file in the first place.
Also perhaps not relevant here but I believe there is a limit on the number of container-wide policies you can have. Are you uploading multiple files to the same container with this code and creating a new container sas each time?
In general I think it would be better to request a sas for an individual blob at the time you need it with a short expiry time.
Is "my.azurestorage.windows.net" just a typo? I would expect something there like "https://account.blob.core.windows.net".
Otherwise the code looks pretty similar to the code in http://blog.smarx.com/posts/shared-access-signatures-are-easy-these-days, which works.