It's not a good sign when the method I'm asking about, GetUserDelegationKey, yields zero search results on SO. Good luck, me.
I have a C# console app, .Net framework 4.8, using Azure.Storage.Blobs and Azure.Identity that will run on customer servers and access Azure blob storage to hold some stuff. I'm doing all of this with the library, not rolling my own REST. Built with VS2019, testing on Win10.
The plan is to use a single Azure storage account that I own, and create one Container per customer project with per-customer credentials that permit them only their own container. Projects never ever talk to each other.
I could set up credentials in the Azure portal by hand, but I am stubbornly trying to do this in software, where a simple project-management app connects as the project app's service principal (which I defined in Azure AD), creates the container, then creates the shared access signatures with a limited lifetime.
The storage account name / container name / access signature would then be configured on the customer server.
I'm having a terrible time.
Note: this is using the newer BlobClient mechanisms, not the older CloudBlob stuff. Dunno if that matters.
This is all documented here at Microsoft, and following even the simple example gets me the same failure.
using System;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Identity;
namespace Azure.Test
{
class Program
{
static void Main(string[] args)
{
var serviceClient = new BlobServiceClient(
new Uri("https://stevestorageacct.blob.core.windows.net"),
new DefaultAzureCredential(true)); // true=pop up login dlg
/*BOOM*/ UserDelegationKey key = serviceClient.GetUserDelegationKey(
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddDays(30));
// use the key to create the signatures
}
}
}
Even though this program couldn't be simpler, it fails every time with an XML error calling GetUserDelegationKey
Unhandled Exception: Azure.RequestFailedException: The value for one of the XML nodes is not in the correct format.
RequestId:c9b7d324-401e-0127-4a4c-1fe6ce000000
Time:2020-05-01T00:06:21.3544489Z
Status: 400 (The value for one of the XML nodes is not in the correct format.)
ErrorCode: InvalidXmlNodeValue
The XML being sent is supposed to be super simple, I think just the start/end dates for validity, but I have no idea how to get to it to inspect, and http is forbidden for this kind of call, so no Wireshark.
It also fails the same way when I use my application's service principal:
static void Main(string[] args)
{
var tokenCredential = new ClientSecretCredential(
"xxxx-xxxx-xxxx-xxxxx", // tenant ID
"yyyy-yyyy-yyyy-yyyyy, // application ID
"**************"); // client secret
var serviceClient = new BlobServiceClient(
new Uri("https://stevestorageacct.blob.core.windows.net"),
tokenCredential);
UserDelegationKey key = serviceClient.GetUserDelegationKey(
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddDays(30));
// ALSO: boom
I'm really at a loss.
I suppose I could try rolling my own REST and playing with it that way, but it doesn't feel like this should be necessary: this kind of error feels like a bug even if I'm doing something wrong. XML nodes?
Also open to entirely different ways of approaching this problem if they are superior, but would like to at least find out why this is failing.
I've had some issues with this also. The first things to try is removing the start time (pass null) or setting it ~15 minutes in the past. This is to avoid clock skew between the requesting pc and azure servers.
The second thing to verify is that the user that you are using has the "Storage Blob Data Contributor" role on the storage account. I had to grant it at the storage account level in the end otherwise it just refused to work for me. However in your use case it might be that you need to grant it at the container level to allow you to have one container per client.
Hope this helps.
Related
I am building an infrastructure wrapper around Azure Blob Storage for a shared component in a .Net Core 6 suite of applications, both web and desktop-based applications. I’m using version 12.14.1 of Azure.Storage.Blobs.
public BlobContainerItem[] GetContainers(string sasToken)
{
blobUri = "https://spiffyaccountname.blob.core.windows.net?" + sasToken;
var client = new BlobServiceClient(blobUri);
var containers = client.GetBlobContainers(); // Succeeds!
var array = containers.array() // Fails with Azure.RequestFailedException
return array;
}
Exception Details:
Azure.RequestFailedException: This request is not authorized to perform this operation using this resource type.
RequestId:...
Time:...
Status: 403 (This request is not authorized to perform this operation using this resource type.)
ErrorCode: AuthorizationResourceTypeMismatch
I don't think it's a code issue but rather a configuration issue, as creating and deleting containers work fine, though I haven't been able to find the setting to fix it.
More than likely the reason you are getting this error is because of incorrect permissions in your SAS token.
Please check for 2 things:
Use Account SAS - Listing containers require that you use Account SAS instead of Service SAS.
Use proper permissions - Listing containers require that your account SAS has SignedServices as Blob (ss=b), SignedResourceTypes as Service (srt=s) and SignedPermission should have at least List permission (sp=l).
I'm migrating a servicebus client application from Microsoft.Azure.ServiceBus to use the current library Azure.Messaging.ServiceBus.
The application is a Worker Process running on a virtual machine in windows azure.
The VM has a system assigned managed identity which grants it access to service bus and we have been using it successfully with the old library for over a year.
On the old library we created a client using this connection string
Endpoint=sb://MyNamespace.servicebus.windows.net/;Authentication=Managed Identity
When I put that connection string into the constructor of Azure.Messaging.ServiceBus.ServiceBusClient I get the following error
The connection string used for an Service Bus client must specify the Service Bus namespace host and either a Shared Access Key (both the name and value) OR a Shared Access Signature to be valid. (Parameter 'connectionString')
I've been trawling through documents for some time now with no progress.
Is there anyway to make this work?
Ideally I would continue to use the connection string - developer machines do not have system assigned ID's so we develop with key based connection strings and let devops swap in the correct prod connection string.
UPDATE
Following on from Jesse's answer managed identity has to go trough a separate constructor which requires a namespace instead of an endpoint and an instance of ManagedIdentityCredential.
As I mentioned not all environments where we deploy have managed aged identities, some require a SharedAccessKey based connection string.
Instead introducing new "identity type" configuration parameters into our build process I've used a factory method to parse the connection string and call the correct constructor overload. Where its a managed identity It extracts the namespace from the endpoint setting.
I Hope its useful for others
private static ServiceBusClient CreateServiceBusClient(string connectionString)
{
var cs = new DbConnectionStringBuilder();
cs.ConnectionString = connectionString;
if (cs.ContainsKey("Authentication") &&
"Managed Identity".Equals(cs["Authentication"].ToString(), StringComparison.OrdinalIgnoreCase))
{
string endpoint = cs["Endpoint"].ToString() ?? String.Empty;
if (endpoint.StartsWith(#"sb://", StringComparison.OrdinalIgnoreCase)) endpoint = endpoint.Substring(5);
if (endpoint.EndsWith(#"/")) endpoint = endpoint.Substring(0, endpoint.Length - 1);
return new ServiceBusClient(endpoint, new ManagedIdentityCredential());
}
return new ServiceBusClient(connectionString);
}
it needs the Azure.Identity package and the namespace System.Data.Common for the connection string builder.
The clients in the Azure.Messaging.ServiceBus package support connection strings only in the format that the Azure portal returns them.
The ;Authentication=Managed Identity token that you've included in your connection string is not a known token and is ignored, so the client does not have the information needed to perform authorization. A managed identity cannot be specified via connection string.
To use a managed identity, you'll use one of the constructor overloads that accepts a fully qualified namespace and a TokenCredential. An example can be found in the package Overview docs. Any of the Azure.Identity credentials can be used; you may want to take take a look at the managed identity section of the Azure.Identity overview.
I have an netcoreapp2.2 containerized application that uses azure key vault to store keys and also uses:
app.UseAuthentication();
And
services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
I am building/running a docker image in a hosted linux environment under App Services. I am using the azure container registry and dev ops pipe line to maintain my app. Azure controls the deployment process and the "docker run" command.
My app works great, however in the container logs I see:
2019-12-13T17:18:12.207394900Z [40m[1m[33mwarn[39m[22m[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
2019-12-13T17:18:12.207436700Z No XML encryptor configured. Key {...} may be persisted to storage in unencrypted form.
...
2019-12-13T17:18:14.540484659Z Application started. Press Ctrl+C to shut down.
I realize there are many other posts on this that allude to using other storage mechanisms, however I am using key vault to store my sensitive data. JWT is all handled by key vault. I have a few application settings that control static variables for DEV/QA/PROD but they are not sensitive data at all.
I am also not sure what key is being stored in memory as all my sensitive keys are completely outside of the application and are called by:
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
new KeyVaultClient.AuthenticationCallback(
azureServiceTokenProvider.KeyVaultTokenCallback));
config.AddAzureKeyVault(
$"https://{builtConfig["MY_KEY_VAULT_ID"]}.vault.azure.net/",
keyVaultClient,
new DefaultKeyVaultSecretManager());
I am having a difficult time understanding why this warning is being thrown and also if I should take additional steps to mitigate the issue. I have not personally seen side effects, and app restarts do not seem to have any effect as I am using bearer tokens and other concerns such as token expiration, password resets and the like are not applicable.
So I am left with asking are there any additional steps I can take to avoid this warning? Do I need to ensure that there is a better data at rest mechanism for any configuration settings that may be in my linux environment? Can I safely ignore this warning?
It took me a while to find a way that suited the needs that I have for my application but I wanted to lend some clarity to a number of other stack answers that just did not make sense to me and how I finally understood the problem.
TLDR; Since I was already using key vault, I was confusing how .net core works. I didn't realize that config.AddAzureKeyVault() has nothing to do with how .net core decides to store data at rest on your app service.
When you see this warning:
No XML encryptor configured. Key {GUID} may be persisted to storage in unencrypted form.
it really doesn't matter what GUID was being set: that string of data was not being stored encrypted at rest.
For my risk analysis any information that is not being encrypted at rest is a bad idea as it could mean at anytime in the future some sort of sensitive data could leak and then be exposed to an attacker. In the end, I chose to classify my data at rest as sensitive and err on the side of caution with a potential attack surface.
I have been struggling to try and explain this in a clear and concise way and it is difficult to sum up in a few words. This is what I learned.
Access control (IAM) is your friend in this situation as you can declare a system assigned identity for your application and use role based accessed control. In my case I used my application identity to control access to both key vault and azure storage with RBAC. This makes it much easier to get access without SAS tokens or access keys.
Azure storage will be the final destination for the file you are creating, but it will be the vault that controls the encryption key. I created an RSA key in key vault, and that key is what encrypts the XML file that is throwing the original error.
One of the mistakes I was making in my head was that I wanted two write the encrypted XML to key vault. However, that is not really the use case Microsoft describes. There are two Mechanisms: PersistKeysTo and ProtectKeysWith. As soon as I got that through my thick head, it all made sense.
I used the following to remove the warning and create encrypted data at rest:
services.AddDataProtection()
// Create a CloudBlockBlob with AzureServiceTokenProvider
.PersistKeysToAzureBlobStorage(...)
// Create a KeyVaultClient with AzureServiceTokenProvider
// And point to the RSA key by id
.ProtectKeysWithAzureKeyVault(...);
I had already used RBAC for my application with key vault (with wrap/unwrap permissions), but I also added Storage Blob Data Contributor to the storage account.
How you create your blob is up to you, but one gotcha is creating the access token synchronously:
// GetStorageAccessToken()
var token = new AzureServiceTokenProvider();
return token.GetAccessTokenAsync("https://storage.azure.com/")
.GetAwaiter()
.GetResult();
Then I called it from a method:
var uri = new Uri($"https://{storageAccount}.blob.core.windows.net/{containerName}/{blobName}");
//Credentials.
var tokenCredential = new TokenCredential(GetStorageAccessToken());
var storageCredentials = new StorageCredentials(tokenCredential);
return new CloudBlockBlob(uri, storageCredentials);
After this hurdle was overcame, putting the encryption in was straight forward. The Keyvault ID is the location of the encryption key you are using.
https://mykeyvaultname.vault.azure.net/keys/my-key-name/{VersionGuid}
And creating the client is
var token = new AzureServiceTokenProvider();
var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(token.KeyVaultTokenCallback));
services.AddDataProtection()
.ProtectKeysWithAzureKeyVault(client, keyVaultId);
I also have to give credit to this blog: https://joonasw.net/view/using-azure-key-vault-and-azure-storage-for-asp-net-core-data-protection-keys as this pointed me in the right direction.
https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/default-settings?view=aspnetcore-2.2 this also pointed out why keys are not encrypted
https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles - RBAC for apps
https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-3.1 this was confusing at first but has a good warning about how to grant access and limit access in production.
Might be you have to configure your data protection policy to use CryptographicAlogrithms as follow:
.UseCryptographicAlgorithms(new AuthenticatedEncryptorConfiguration()
{
EncryptionAlgorithm = EncryptionAlgorithm.AES_256_CBC,
ValidationAlgorithm = ValidationAlgorithm.HMACSHA256
});
Also, following are few warning which you get around Data protection policy
ASP.Net core DataProtection stores keys in the HOME directory (/root/.aspnet/DataProtection-Keys) so when container restart keys are lost and this might crash the service.
This can be resolve by persisting key at
Persist key at the persistent location (volume) and mount that volume
to docker container
Persist key at the external key store like Azure or Redis
More details about ASP.NET DataProtection:
https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-3.1
https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/introduction?view=aspnetcore-3.1
To mount an external volume (C:/temp-kyes) to docker container volume (/root/.aspnet/DataProtection-Keys) using following command
docker run -d -v /c/temp-keys:/root/.aspnet/DataProtection-Keys container-name
Also, You need to update your Starup.cs - ConfigureServices to configure DataProtection policy
services.AddDataProtection().PersistKeysToFileSystem(new DirectoryInfo(#"C:\temp-keys\"))
.UseCryptographicAlgorithms(new AuthenticatedEncryptorConfiguration()
{
EncryptionAlgorithm = EncryptionAlgorithm.AES_256_CBC,
ValidationAlgorithm = ValidationAlgorithm.HMACSHA256
});
I have developed a .net console application which have these main characteristics :-
Integrate with SharePoint online REST API, to retrieve some list items, and modify the items fields.
Will run daily #1 am for example.
I will host this console application inside Azure Function app.
The Azure account does not have any permission on the sharepoint tenant, as the Azure account and the sharepoint online are on different domains.
so i am not sure under which account the console application will be running?
Will it runs under the current Azure account? if this is the case, then this will not work as the azure account is on different domain and does not have any permission on the sharepoint (and it shouldn't have)?
OR
I can define a service account for the Azure function app to run under it, where in this case i can define the service account to be an authorized account inside sharepoint online?
OR
i need to define the username/password inside the console application itself? i do not like to approach, as i will be exposing the password inside the console application. also changing the password for the username, means that we will need to update the console application accordingly..
so can anyone advice on this please?
Thanks
EDIT
code for managing the console application authentication :-
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint.Client;
namespace O365SPProject
{
class Program
{
private class Configuration
{
public static string ServiceSiteUrl = "https://<tenant>.sharepoint.com";
public static string ServiceUserName = "<user>#<tenant>.onmicrosoft.com";
public static string ServicePassword = "xxxxxxxxxx";
}
static ClientContext GetonlineContext()
{
var securePassword = new SecureString();
foreach (char c in Configuration.ServicePassword)
{
securePassword.AppendChar(c);
}
var onlineCredentials = new SharePointOnlineCredentials(Configuration.ServiceUserName, securePassword);
var context = new ClientContext(Configuration.ServiceSiteUrl);
context.Credentials = onlineCredentials;
return context;
}
static void Main(string[] args)
{
var ClientContext=GetonlineContext();
Web web = clientContext.Web;
// do somethings
}
}
}
There are multiple parts to your question, so I'll answer it accordingly.
1. Which option out of the 3 you mentioned (or if there is a different better option :)), should you use to manage your configuration data/service account identity
OPTION 4 (similar to your option 2 with subtle difference):
You should take your service account identity and configuration data out of your console application completely and pass them in through "Application Settings" for your Azure Function App.
This option is similar to the option 2 you had in your question, as you keep the information outside of console app code
I can define a service account for the Azure function app to run under
it, where in this case i can define the service account to be an
authorized account inside sharepoint online?
but difference is that I am not saying that you will be able to define a service account for your Azure function app to run under (because you can't control the account that Azure function will run under, Microsoft infrastructure takes care of it), instead you will pass it to your console app as a secure configuration data and your console app will use it. More on security/encryption later while comparing the options.
I actually took your console application code from question, created a console app and used it in a timer triggered Azure function to get it working. So these steps are from a working sample. I used the "Microsoft.SharePointOnline.CSOM" nuget package in my console app, and had to upload some of the dependency dlls along with exe in order for it to run. Feel free to ask for more details on doing this part if you run into issues.
Adding Application Settings - Navigate your Azure Function App and Click on "Application Settings"
Add Settings for all items that you want to take out of your console application and control from outside. I did it for all 3 items I saw, but this is up to you.
Then change your code to use these settings. I have shown the exact code changes at the end.
OPTION 5
Registering a new application in Azure AD to represent your Azure function.
You should register a new application in your Azure AD and use this identity to access SharePoint online.
You will need to grant permissions to SharePoint online for this application (NOTE: permission assignment will not be as granular or detailed as in case of your service account approach, I'll explain more while comparing the options)
You will need to associate a certificate with your AzureAD application to help in authentication.
While authenticating to SharePoint online, you will not be directly able to use the SharePointOnlineCredentials class as in your code today, but instead send the bearer token in 'Authorization' header for the http request.
Here is blog post that walks through detailed steps involved in this option 5.
NOTE: This blog still leaves out the certificate details like password in function code at the end, which will not be ideal and you will need to move it out to App Settings or Azure Key Vault ideally.
2. Which account will the .NET console application run under and a Quick Comparison of all options
It's an arbitrary IIS App Pool account, as pointed out by #Mitch Stewart, other SO posts and is evident in the output I get for my function, it's exact value in my run came out to be "IIS APPPOOL\mawsFnPlaceholder0_v1 ". See the image at the bottom. You already have some good info shared on this, so I'll not repeat. Only thing I'll add is that this account will be controlled by the infrastructure hosting your function app and will be designed more towards taking care of isolation/other concerns in a shared infrastructure where many function apps can run, so trying to control/change it may not be the way to go right now.
Option 1 (from your question) - Giving permissions to an IIS app pool account for your SharePoint Online site, especially when you don't control the account may not be a good idea.
Option 2 (from your question) - It would have been better than the other 2 options you mentioned, but you can't really control this account.
Option 3 (from your question)- Embedding this information deep into your console application will be a maintenance issue as well as not the most secure option unless you start reading form a vault etc. Maintenance issues will remain no matter what you do because it's embedded in compiled code, which it shouldn't be.
Option 4 - This is better than previous 3 options, because it separates the concern of code from configuration and identity information, no recompilation needed for updates. Also note that whatever you store in App Settings configurations is encrypted by default (with good governance of key rotation) and is the recommended way. These values are decrypted only just before execution of your app and loaded into process memory. Look detailed discussion in this link, I have also given a small relevant excerpt below -
Provide documentation about encrypt/decrypt settings
Even with this option you could store them in a key vault and then your setting would be the URL of the key vault secret that has the actual information.
Option 5 - This option makes use of Azure AD based identity to authenticate with SharePoint Online which is good part.
It does come with some additional effort and some limitations though, so you will need to consider if these limitations are acceptable or not in your scenario:
Permissions for SharePoint online will not be as granular/detailed as a user being given permissions from inside SharePoint Users/Groups interfaces (no site/list/folder/item level specific permissions etc). In this approach, you will give the permissions as part of setting up Azure AD application and you will only get generic options like these (shown in screenshot below)
Microsoft has some well documented limitations in this scenario, which you can read here: What are the limitations when using app-only
So overall, I would suggest you choose option 4 or option 5, or a combination of both for your implementation depending on which limitations are acceptable in your scenario.
3. Code Changes to use App Settings
Just the important Change
public static string ServiceSiteUrl = Environment.GetEnvironmentVariable("ServiceSiteUrl");
public static string ServiceUserName = Environment.GetEnvironmentVariable("ServiceUserName");
public static string ServicePassword = Environment.GetEnvironmentVariable("ServicePassword");
Full Code in a working Sample (I replaced do something with reading the title and Url for SharePoint Web object):
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint.Client;
using System.Security;
using System.Security.Principal;
namespace O365SPProject
{
class Program
{
private class Configuration
{
public static string ServiceSiteUrl = Environment.GetEnvironmentVariable("ServiceSiteUrl");
public static string ServiceUserName = Environment.GetEnvironmentVariable("ServiceUserName");
public static string ServicePassword = Environment.GetEnvironmentVariable("ServicePassword");
}
static ClientContext GetonlineContext()
{
var securePassword = new SecureString();
foreach (char c in Configuration.ServicePassword)
{
securePassword.AppendChar(c);
}
var onlineCredentials = new SharePointOnlineCredentials(Configuration.ServiceUserName, securePassword);
var context = new ClientContext(Configuration.ServiceSiteUrl);
context.Credentials = onlineCredentials;
return context;
}
static void Main(string[] args)
{
var ClientContext = GetonlineContext();
ClientContext.Load(ClientContext.Web);
ClientContext.ExecuteQuery();
Console.WriteLine("This app found web title as: {0} and URL as: {1}",
ClientContext.Web.Title, ClientContext.Web.Url);
Console.WriteLine("Console app is running with identity {0}", WindowsIdentity.GetCurrent().Name);
}
}
}
OUTPUT on executing Azure Function
The SharePoint REST API supports OAuth. Here's a promising article. Although, this might be a bit much for you intentions. Alternatively, you can try using basic auth (username + password). To guard against plain text passwords, you can store them in Azure Key Vault.
Edit
The current user of an Azure function is the identity of the IIS app pool.
Using the new Windows Azure SDK for .NET, I want to get a list of all virtual machines.
Piecing together the early documentation, here's what I came up with:
// This is a custom cert I made locally. I uploaded it to Azure's Management Certificates and added it to my local computer's cert store, see screenshot below.
X509Certificate2 myCustomCert = await this.GetAzureCert();
var credentials = new CertificateCloudCredentials(mySubscriptionId, myCustomCert);
using (var cloudServiceClient = CloudContext.Clients.CreateCloudServiceManagementClient(credentials))
{
credentials.InitializeServiceClient(cloudServiceClient); // Is this required? The next call fails either way.
// This line fails with exception: "The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription."
var services = await cloudServiceClient.CloudServices.ListAsync(CancellationToken.None);
}
My first thought was the cert was bad, but I am able to successfully call the Azure REST API using my custom certificate. As you can see below, it is properly added to the Azure Management Certificates and associated with my subscription:
What am I doing wrong?
Here's another option - rather than upload a cert, try pulling your management cert out of your publishsettings file and using the X509Certificate's constructor that takes a byte[]. Then, pass that parameter the result of a call to Convert.FromBase64String, passing it the string representation of your management certificate from your publishsettings file.
Also, take a look at the Compute management client rather than the Cloud Service Management client. There are more features specific to the compute stack in that client at this time. The code below is a demonstration of such an approach. Note, my _subscriptionId and _managementCert fields are both strings, and I just hard-code them to the values from my publishsettings file as I described above.
public async static void ListVms()
{
using (var client = new ComputeManagementClient(
new CertificateCloudCredentials(_subscriptionId,
new X509Certificate2(Convert.FromBase64String(_managementCert)))
))
{
var result = await client.HostedServices.ListAsync();
result.ToList().ForEach(x => Console.WriteLine(x.ServiceName));
}
}
There's a parameterless ListAsync method that's an extension method. Try importing the Microsoft.WindowsAzure.Management namespace (or the Microsoft.WindowsAzure.Management.Compute namespace). Once you see the parameterless ListAsync method you should be good. I'll also mock up some code to resemble what you're trying to accomplish and offer up a more comprehensive answer by the end of the day.