Using the new Windows Azure SDK for .NET, I want to get a list of all virtual machines.
Piecing together the early documentation, here's what I came up with:
// This is a custom cert I made locally. I uploaded it to Azure's Management Certificates and added it to my local computer's cert store, see screenshot below.
X509Certificate2 myCustomCert = await this.GetAzureCert();
var credentials = new CertificateCloudCredentials(mySubscriptionId, myCustomCert);
using (var cloudServiceClient = CloudContext.Clients.CreateCloudServiceManagementClient(credentials))
{
credentials.InitializeServiceClient(cloudServiceClient); // Is this required? The next call fails either way.
// This line fails with exception: "The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription."
var services = await cloudServiceClient.CloudServices.ListAsync(CancellationToken.None);
}
My first thought was the cert was bad, but I am able to successfully call the Azure REST API using my custom certificate. As you can see below, it is properly added to the Azure Management Certificates and associated with my subscription:
What am I doing wrong?
Here's another option - rather than upload a cert, try pulling your management cert out of your publishsettings file and using the X509Certificate's constructor that takes a byte[]. Then, pass that parameter the result of a call to Convert.FromBase64String, passing it the string representation of your management certificate from your publishsettings file.
Also, take a look at the Compute management client rather than the Cloud Service Management client. There are more features specific to the compute stack in that client at this time. The code below is a demonstration of such an approach. Note, my _subscriptionId and _managementCert fields are both strings, and I just hard-code them to the values from my publishsettings file as I described above.
public async static void ListVms()
{
using (var client = new ComputeManagementClient(
new CertificateCloudCredentials(_subscriptionId,
new X509Certificate2(Convert.FromBase64String(_managementCert)))
))
{
var result = await client.HostedServices.ListAsync();
result.ToList().ForEach(x => Console.WriteLine(x.ServiceName));
}
}
There's a parameterless ListAsync method that's an extension method. Try importing the Microsoft.WindowsAzure.Management namespace (or the Microsoft.WindowsAzure.Management.Compute namespace). Once you see the parameterless ListAsync method you should be good. I'll also mock up some code to resemble what you're trying to accomplish and offer up a more comprehensive answer by the end of the day.
Related
It's not a good sign when the method I'm asking about, GetUserDelegationKey, yields zero search results on SO. Good luck, me.
I have a C# console app, .Net framework 4.8, using Azure.Storage.Blobs and Azure.Identity that will run on customer servers and access Azure blob storage to hold some stuff. I'm doing all of this with the library, not rolling my own REST. Built with VS2019, testing on Win10.
The plan is to use a single Azure storage account that I own, and create one Container per customer project with per-customer credentials that permit them only their own container. Projects never ever talk to each other.
I could set up credentials in the Azure portal by hand, but I am stubbornly trying to do this in software, where a simple project-management app connects as the project app's service principal (which I defined in Azure AD), creates the container, then creates the shared access signatures with a limited lifetime.
The storage account name / container name / access signature would then be configured on the customer server.
I'm having a terrible time.
Note: this is using the newer BlobClient mechanisms, not the older CloudBlob stuff. Dunno if that matters.
This is all documented here at Microsoft, and following even the simple example gets me the same failure.
using System;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Identity;
namespace Azure.Test
{
class Program
{
static void Main(string[] args)
{
var serviceClient = new BlobServiceClient(
new Uri("https://stevestorageacct.blob.core.windows.net"),
new DefaultAzureCredential(true)); // true=pop up login dlg
/*BOOM*/ UserDelegationKey key = serviceClient.GetUserDelegationKey(
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddDays(30));
// use the key to create the signatures
}
}
}
Even though this program couldn't be simpler, it fails every time with an XML error calling GetUserDelegationKey
Unhandled Exception: Azure.RequestFailedException: The value for one of the XML nodes is not in the correct format.
RequestId:c9b7d324-401e-0127-4a4c-1fe6ce000000
Time:2020-05-01T00:06:21.3544489Z
Status: 400 (The value for one of the XML nodes is not in the correct format.)
ErrorCode: InvalidXmlNodeValue
The XML being sent is supposed to be super simple, I think just the start/end dates for validity, but I have no idea how to get to it to inspect, and http is forbidden for this kind of call, so no Wireshark.
It also fails the same way when I use my application's service principal:
static void Main(string[] args)
{
var tokenCredential = new ClientSecretCredential(
"xxxx-xxxx-xxxx-xxxxx", // tenant ID
"yyyy-yyyy-yyyy-yyyyy, // application ID
"**************"); // client secret
var serviceClient = new BlobServiceClient(
new Uri("https://stevestorageacct.blob.core.windows.net"),
tokenCredential);
UserDelegationKey key = serviceClient.GetUserDelegationKey(
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddDays(30));
// ALSO: boom
I'm really at a loss.
I suppose I could try rolling my own REST and playing with it that way, but it doesn't feel like this should be necessary: this kind of error feels like a bug even if I'm doing something wrong. XML nodes?
Also open to entirely different ways of approaching this problem if they are superior, but would like to at least find out why this is failing.
I've had some issues with this also. The first things to try is removing the start time (pass null) or setting it ~15 minutes in the past. This is to avoid clock skew between the requesting pc and azure servers.
The second thing to verify is that the user that you are using has the "Storage Blob Data Contributor" role on the storage account. I had to grant it at the storage account level in the end otherwise it just refused to work for me. However in your use case it might be that you need to grant it at the container level to allow you to have one container per client.
Hope this helps.
I am trying to access Azure Key Vault from my Azure App Service. I followed the steps outlined on this documentation: https://learn.microsoft.com/en-us/azure/key-vault/managed-identity (turned on system assigned identity for the app service, updated the access policy of the key vault to include the app with Get,List secret permissions).
However, when I run the web application, it is not able to get the secret from the key vault and my web service hits the following error:
502 - Web server received an invalid response while acting as a gateway or proxy server.
There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.
This is what my code looks like:
AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
KeyVaultClient keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = keyVaultClient.GetSecretAsync(KeyVaultUrl);
authenticationKey = secret.Result.Value;
The service gets stuck on the secret.Result.Value line. Is there something else I need to do?
This is much easier with the new package, like Azure.Security.KeyVault.Secrets. Together with Azure.Identity, you can just pass a DefaultAzureCredential like in our samples.
var client = new SecretClient(
new Uri("https://myvault.vault.azure.net"),
new DefaultAzureCredential());
KeyVaultSecret secret = await client.GetSecretAsync("secret-name");
string authenticationKey = secret.Value;
The DefaultAzureCredential is optimized to work for managed identity, service principals from the environment, and interactive logins to support the same code running both in production and on developer machines. The new libraries are also faster with fewer allocations, and have much better diagnostics - defaulted to on when using Azure Application Monitor.
They target netstandard2.0 so should be compatible with the older packages these replace. Would you be able to upgrade? We're only making critical fixes for the older packages, and recommending people upgrade to Azure.* packages intead of the older Microosft.Azure.* packages.
As for the problem itself, it's hard to say without knowing when you're calling this in your application. During startup? What version of .NET? What are you using for your ASP.NET application framework?
While it's probably not the cause of the problem, it's hard to ignore that you're calling an async method synchronously, which can also cause problems. If you're in an async method, you should write your code like so:
var secret = await keyVaultClient.GetSecretAsync(KeyVaultUrl);
authenticationKey = secret.Value;
If not, call:
var secret = keyVaultClient.GetSecretAsnc(KeyVaultUrl).GetAwaiter().GetResult();
This is not recommended, though. In the new packages I mentioned above, we have both sync and async versions that are either sync or async all the way through the call stack and safer to use. Generally, though, you should use async calls - especially for network traffic like accessing Key Vault because, depending on what thread you call it, it can hang your UI.
When you test in local:
Add your vs signed account into azure keyvault. Go to keyvault> Access policy> add your account with get secret permmission.
When you publish to azure:
1.Enable webapp MSI.
2.Go to keyvault> Access policy> add your webapp's service principal with get secret permission.
The code:
AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
KeyVaultClient keyVaultClient = new KeyVaultClient(new Microsoft.Azure.KeyVault.KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
var secret = keyVaultClient.GetSecretAsync("https://yourkevaultname.vault.azure.net/secrets/secretname/437d301daxxxxxx");
var authenticationKey = secret.Result.Value;
ViewBag.Message = authenticationKey.ToString();
I have developed a .net console application which have these main characteristics :-
Integrate with SharePoint online REST API, to retrieve some list items, and modify the items fields.
Will run daily #1 am for example.
I will host this console application inside Azure Function app.
The Azure account does not have any permission on the sharepoint tenant, as the Azure account and the sharepoint online are on different domains.
so i am not sure under which account the console application will be running?
Will it runs under the current Azure account? if this is the case, then this will not work as the azure account is on different domain and does not have any permission on the sharepoint (and it shouldn't have)?
OR
I can define a service account for the Azure function app to run under it, where in this case i can define the service account to be an authorized account inside sharepoint online?
OR
i need to define the username/password inside the console application itself? i do not like to approach, as i will be exposing the password inside the console application. also changing the password for the username, means that we will need to update the console application accordingly..
so can anyone advice on this please?
Thanks
EDIT
code for managing the console application authentication :-
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint.Client;
namespace O365SPProject
{
class Program
{
private class Configuration
{
public static string ServiceSiteUrl = "https://<tenant>.sharepoint.com";
public static string ServiceUserName = "<user>#<tenant>.onmicrosoft.com";
public static string ServicePassword = "xxxxxxxxxx";
}
static ClientContext GetonlineContext()
{
var securePassword = new SecureString();
foreach (char c in Configuration.ServicePassword)
{
securePassword.AppendChar(c);
}
var onlineCredentials = new SharePointOnlineCredentials(Configuration.ServiceUserName, securePassword);
var context = new ClientContext(Configuration.ServiceSiteUrl);
context.Credentials = onlineCredentials;
return context;
}
static void Main(string[] args)
{
var ClientContext=GetonlineContext();
Web web = clientContext.Web;
// do somethings
}
}
}
There are multiple parts to your question, so I'll answer it accordingly.
1. Which option out of the 3 you mentioned (or if there is a different better option :)), should you use to manage your configuration data/service account identity
OPTION 4 (similar to your option 2 with subtle difference):
You should take your service account identity and configuration data out of your console application completely and pass them in through "Application Settings" for your Azure Function App.
This option is similar to the option 2 you had in your question, as you keep the information outside of console app code
I can define a service account for the Azure function app to run under
it, where in this case i can define the service account to be an
authorized account inside sharepoint online?
but difference is that I am not saying that you will be able to define a service account for your Azure function app to run under (because you can't control the account that Azure function will run under, Microsoft infrastructure takes care of it), instead you will pass it to your console app as a secure configuration data and your console app will use it. More on security/encryption later while comparing the options.
I actually took your console application code from question, created a console app and used it in a timer triggered Azure function to get it working. So these steps are from a working sample. I used the "Microsoft.SharePointOnline.CSOM" nuget package in my console app, and had to upload some of the dependency dlls along with exe in order for it to run. Feel free to ask for more details on doing this part if you run into issues.
Adding Application Settings - Navigate your Azure Function App and Click on "Application Settings"
Add Settings for all items that you want to take out of your console application and control from outside. I did it for all 3 items I saw, but this is up to you.
Then change your code to use these settings. I have shown the exact code changes at the end.
OPTION 5
Registering a new application in Azure AD to represent your Azure function.
You should register a new application in your Azure AD and use this identity to access SharePoint online.
You will need to grant permissions to SharePoint online for this application (NOTE: permission assignment will not be as granular or detailed as in case of your service account approach, I'll explain more while comparing the options)
You will need to associate a certificate with your AzureAD application to help in authentication.
While authenticating to SharePoint online, you will not be directly able to use the SharePointOnlineCredentials class as in your code today, but instead send the bearer token in 'Authorization' header for the http request.
Here is blog post that walks through detailed steps involved in this option 5.
NOTE: This blog still leaves out the certificate details like password in function code at the end, which will not be ideal and you will need to move it out to App Settings or Azure Key Vault ideally.
2. Which account will the .NET console application run under and a Quick Comparison of all options
It's an arbitrary IIS App Pool account, as pointed out by #Mitch Stewart, other SO posts and is evident in the output I get for my function, it's exact value in my run came out to be "IIS APPPOOL\mawsFnPlaceholder0_v1 ". See the image at the bottom. You already have some good info shared on this, so I'll not repeat. Only thing I'll add is that this account will be controlled by the infrastructure hosting your function app and will be designed more towards taking care of isolation/other concerns in a shared infrastructure where many function apps can run, so trying to control/change it may not be the way to go right now.
Option 1 (from your question) - Giving permissions to an IIS app pool account for your SharePoint Online site, especially when you don't control the account may not be a good idea.
Option 2 (from your question) - It would have been better than the other 2 options you mentioned, but you can't really control this account.
Option 3 (from your question)- Embedding this information deep into your console application will be a maintenance issue as well as not the most secure option unless you start reading form a vault etc. Maintenance issues will remain no matter what you do because it's embedded in compiled code, which it shouldn't be.
Option 4 - This is better than previous 3 options, because it separates the concern of code from configuration and identity information, no recompilation needed for updates. Also note that whatever you store in App Settings configurations is encrypted by default (with good governance of key rotation) and is the recommended way. These values are decrypted only just before execution of your app and loaded into process memory. Look detailed discussion in this link, I have also given a small relevant excerpt below -
Provide documentation about encrypt/decrypt settings
Even with this option you could store them in a key vault and then your setting would be the URL of the key vault secret that has the actual information.
Option 5 - This option makes use of Azure AD based identity to authenticate with SharePoint Online which is good part.
It does come with some additional effort and some limitations though, so you will need to consider if these limitations are acceptable or not in your scenario:
Permissions for SharePoint online will not be as granular/detailed as a user being given permissions from inside SharePoint Users/Groups interfaces (no site/list/folder/item level specific permissions etc). In this approach, you will give the permissions as part of setting up Azure AD application and you will only get generic options like these (shown in screenshot below)
Microsoft has some well documented limitations in this scenario, which you can read here: What are the limitations when using app-only
So overall, I would suggest you choose option 4 or option 5, or a combination of both for your implementation depending on which limitations are acceptable in your scenario.
3. Code Changes to use App Settings
Just the important Change
public static string ServiceSiteUrl = Environment.GetEnvironmentVariable("ServiceSiteUrl");
public static string ServiceUserName = Environment.GetEnvironmentVariable("ServiceUserName");
public static string ServicePassword = Environment.GetEnvironmentVariable("ServicePassword");
Full Code in a working Sample (I replaced do something with reading the title and Url for SharePoint Web object):
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint.Client;
using System.Security;
using System.Security.Principal;
namespace O365SPProject
{
class Program
{
private class Configuration
{
public static string ServiceSiteUrl = Environment.GetEnvironmentVariable("ServiceSiteUrl");
public static string ServiceUserName = Environment.GetEnvironmentVariable("ServiceUserName");
public static string ServicePassword = Environment.GetEnvironmentVariable("ServicePassword");
}
static ClientContext GetonlineContext()
{
var securePassword = new SecureString();
foreach (char c in Configuration.ServicePassword)
{
securePassword.AppendChar(c);
}
var onlineCredentials = new SharePointOnlineCredentials(Configuration.ServiceUserName, securePassword);
var context = new ClientContext(Configuration.ServiceSiteUrl);
context.Credentials = onlineCredentials;
return context;
}
static void Main(string[] args)
{
var ClientContext = GetonlineContext();
ClientContext.Load(ClientContext.Web);
ClientContext.ExecuteQuery();
Console.WriteLine("This app found web title as: {0} and URL as: {1}",
ClientContext.Web.Title, ClientContext.Web.Url);
Console.WriteLine("Console app is running with identity {0}", WindowsIdentity.GetCurrent().Name);
}
}
}
OUTPUT on executing Azure Function
The SharePoint REST API supports OAuth. Here's a promising article. Although, this might be a bit much for you intentions. Alternatively, you can try using basic auth (username + password). To guard against plain text passwords, you can store them in Azure Key Vault.
Edit
The current user of an Azure function is the identity of the IIS app pool.
I have an HTTP-triggered function running on Google Cloud Functions, which uses require('googleapis').sheets('v4') to write data into a docs spreadsheet.
For local development I added an account via the Service Accounts section of their developer console. I downloaded the token file (dev-key.json below) and used it to authenticate my requests to the Sheets API as follows:
var API_ACCT = require("./dev-key.json");
let apiClient = new google.auth.JWT(
API_ACCT.client_email, null, API_ACCT.private_key,
['https://www.googleapis.com/auth/spreadsheets']
);
exports.myFunc = function (req, res) {
var newRows = extract_rows_from_my_client_app_request(req);
sheets.spreadsheets.values.append({
auth: apiClient,
// ...
resource: { values:newRows }
}, function (e) {
if (e) res.status(500).json({err:"Sheets API is unhappy"});
else res.status(201).json({ok:true})
});
};
After I shared my spreadsheet with my service account's "email address" e.g. local-devserver#foobar-bazbuzz-123456.iam.gserviceaccount.com — it worked!
However, as I go to deploy this to the Google Cloud Functions service, I'm wondering if there's a better way to handle credentials? Can my code authenticate itself automatically without needing to bundle a JWT key file with the deployment?
I noticed that there is a FUNCTION_IDENTITY=foobar-bazbuzz-123456#appspot.gserviceaccount.com environment variable set when my function runs, but I do not know how to use this in the auth value to my googleapis call. The code for google.auth.getApplicationDefault does not use that.
Is it considered okay practice to upload a private JWT token along with my GCF code? Or should I somehow be using the metadata server for that? Or is there a built-in way that Cloud Functions already can authenticate themselves to other Google APIs?
It's common to bundle credentials with a function deployment. Just don't check them into your source control. Cloud Functions for Firebase samples do this where needed. For example, creating a signed URL from Cloud Storage requires admin credentials, and this sample illustrates saving that credential to a file to be deployed with the functions.
I'm wondering if there's a better way to handle credentials? Can my
code authenticate itself automatically without needing to bundle a JWT
key file with the deployment?
Yes. You can use 'Application Default Credentials', instead of how you've done it, but you don't use the function getApplicationDefault() as it has been deprecated since this Q was posted.
The link above shows how to make a simple call using the google.auth.getClient API, providing the desired scope, and have it decide the credential type needed automatically. On cloud functions this will be a 'Compute' object, as defined in the google-auth-library.
These docs say it well here...
After you set up a service account, ADC can implicitly find your
credentials without any need to change your code, as described in the
section above.
Where ADC is Application Default Credentials.
Note that, for Cloud Functions, you use the App Engine service account:
YOUR_PROJECT_ID#appspot.gserviceaccount.com, as documented here. That is the one you found via the FUNCTION_IDENTITY env var - this rather tripped me up.
The final step is to make sure that the service account has the required access as you did with your spreadsheet.
I have my X509Certificate stored in a database (in byte[]) so that my application can retrieve the certificate and use it to sign my JWTs.
My x509Certificate is passed off a .pfx file that I generated on my machine, however now it sits in a database as a string of bytes.
My application works perfectly fine locally when I run it. The application can correctly create an instance of that X509Certificate2 and use it for my requirements, however the problem arises when I try to use it in my azurewebsites web application.
Basically I can not access the certificates' PrivateKey instance variable, I get an exception
System.Security.Cryptography.CryptographicException: Keyset does not exist
And I am re-instantiating the certificate with this
var cert = new X509Certificate2(myCertInBytes, myCertPass,
X509KeyStorageFlags.PersistKeySet |
X509KeyStorageFlags.MachineKeySet |
X509KeyStorageFlags.Exportable);
I am using ASPNET 5 rc1-update1. I have also tried running this on a different machine and it works fine, only have this issue when I publish to Azure. And to also add something else, This application was working when I was running the same project that was running using DNX version beta7
Any help appreciated.
The problem is the Azure Web Apps restricts access to the machines private key store, since it's a shared hosting environment, and you don't fully own the machine. As a workaround, you can load a cert. This blog post describes the best practice on how to do so:
https://azure.microsoft.com/en-us/blog/using-certificates-in-azure-websites-applications/
Please note that this only works for Basic Tier and above (not Free or Shared tier).
This can also be done from a .cer file as follows, however it should be noted that this is not best-practices since you're storing a secure credential with your code, in an insecure format.
public X509Certificate2 CertificateFromStrings(String certificateString64, String privateKeyXml)
{
try
{
var rsaCryptoServiceProvider = new RSACryptoServiceProvider();
rsaCryptoServiceProvider.FromXmlString(privateKeyXml);
var certificateBytes = Convert.FromBase64String(certificateString64);
var x509Certificate2 = new X509Certificate2(certificateBytes);
x509Certificate2.PrivateKey = rsaCryptoServiceProvider;
return x509Certificate2;
}
catch
{
return null;
}
}