Azure Functions bug. I get the error in the portal
Error:
We are not able to retrieve the runtime master key. Please try again later.
Session Id: d13fceebd4ea4cb1b7fb3d3829dd1406
Timestamp: 2017-08-24T20:04:23.555Z
I've tried all of the suggestions here:
https://blogs.msdn.microsoft.com/jpsanders/2017/05/09/function-app-error-we-are-not-able-to-retrieve-the-runtime-master-key/
I'm using the runtime version 1.0.10917 but I've tried ~1 and get the same result.
This seems to occur when I delete the function from the portal and then recreate it. It consistently happens after that for every function we have. The first time the function is created, it seems to work.
This is the exception you're hitting
System.Security.Cryptography.CryptographicException : The payload was invalid.
at Microsoft.AspNetCore.DataProtection.Cng.CbcAuthenticatedEncryptor.DecryptImpl(Byte* pbCiphertext,UInt32 cbCiphertext,Byte* pbAdditionalAuthenticatedData,UInt32 cbAdditionalAuthenticatedData)
at Microsoft.AspNetCore.DataProtection.Cng.Internal.CngAuthenticatedEncryptorBase.Decrypt(ArraySegment`1 ciphertext,ArraySegment`1 additionalAuthenticatedData)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData,Boolean allowOperationsOnRevokedKeys,UnprotectStatus& status)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData,Boolean ignoreRevocationErrors,Boolean& requiresMigration,Boolean& wasRevoked)
at Microsoft.Azure.WebJobs.Script.WebHost.DataProtectionKeyValueConverter.Unprotect(Key key) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\DataProtectionKeyValueConverter.cs : 43
at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.ReadHostSecrets(HostSecrets hostSecrets) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.cs : 383
at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.GetHostSecretsAsync() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.cs : 83
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetOrCreateExtensionKey(String extensionName) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.cs : 71
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetExtensionWebHookRoute(String extensionName) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.cs : 64
at Microsoft.Azure.WebJobs.Extensions.EventGrid.EventGridExtensionConfig.Initialize(ExtensionConfigContext context)
at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders(ExtensionConfigContext context)
at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices(JobHostConfiguration config)
at Microsoft.Azure.WebJobs.JobHost.PopulateStaticServices()
at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider(JobHost host) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.cs : 362
at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadCustomExtensions() at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 670
at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize() at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 510
at Microsoft.Azure.WebJobs.Script.ScriptHost.Create(IScriptHostEnvironment environment,IScriptEventManager eventManager,ScriptHostConfiguration scriptConfig,ScriptSettingsManager settingsManager) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 937
It's very hard for users to discover these errors for their app because the runtime doesn't log them anywhere the UX can query.
This issue is tracking the exception: https://github.com/Azure/azure-webjobs-sdk-script/issues/1832
We are still not entirely sure why this is happening. Did you republish your keys from another application by any chance? (edit: or delete and recreate the app with the same name) these keys are encrypted by a function app specific key, and won't work outside that context.
fix: copied from Fabio Cavalcante
can you delete (or rename) the host.json file in
d:\home\data\Functions\secrets\ and retry? This will force the runtime
to re-create those secrets for that environment. Keep in mind that
this would also change your master and default keys.
Switching the Azure Function storage account to a new storage account can trigger a similar issue.
Error: We are not able to get the key swaggerdocumentationkey.
Please check the runtime logs for any errors or try again later.
This is likely related to the storage account being uninitialized with the new site. I can't confirm the behavior, but it seems that when switching storage accounts the new account doesn't initialize a new site and will cause similar issues.
i deleted the azure functions and renamed it. After that it started working fine. Looks like even when you delete the azure functions, traces are there in azure logs and screwing up something.
Also make sure you keep separate azure blob storage for every functions, if you use it in more than one azure function, then it is giving some weird errors. I think they are using it for logging and authentication purposes.
Related
I have function app where I have one HttpTrigger and 3 BlobTrigger functions. After I deployed it, http trigger is working fine but for others functions which are blob triggers, it gives following errors
"Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function " for one function
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener' for function
" for another two
I verified with other environments and config values are same/similar so not sure why we are getting this issue in one environment only. I am using consumption mode.
Update: When file is placed in a blob function is not getting triggered.
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function
I was observed the same message when working on the Azure Functions Queue Trigger:
This message doesn't mean the error in function. Due to timeout of Function activity, this message will appear in the App Insights > Traces.
I have stopped sending the messages in the Queue for some time and has been observed the traces like Web Job Host Stopped and if you run the function again or any continuous activity is present in the Function, then this message will not appear in the traces.
If you are using elastic Premium and has VNET integrated, the non-http trigers needs Runtime scale monitoring enabled.
You can find Function App-->Configuration--> Function runtime settings and turn on Runtime scale monitoring.
If function app and storage account which holds the metadata of the function Private linked, you will need to add the app settings WEBSITE_CONTENTOVERVNET = 1.
Also, make sure you have private linked for blob, file, table and queue on storage account.
I created ticket with MS to fix this issue. After analysis I did some code changes as
Function was async but returning void so changed to return Task.
For the trigger I was using connection string from app settings. But then I changed it to azureWebJobStorage(even though bobth were same) in function trigger attribute param
It started working. So posting here in case it is helpful for others
I have also asked this question in the MSDN Azure forums, but have not received any guidance as to why my function goes idle.
I have an Azure function running on a Consumption plan that goes idle (i.e. does not respond to new messages on the ServiceBus trigger queue) despite following the instructions outlined in this GitHub issue:
The configuration for the function is the following json:
{
"ConnectionStrings": {
"MyConnectionString": "Server=tcp:project.database.windows.net,1433;Database=myDB;User ID=user#project;Password=password;Encrypt=True;Connection Timeout=30;"
},
"Values": {
"serviceBusConnection": "Endpoint=sb://project.servicebus.windows.net/;SharedAccessKeyName=SharedAccessKeyName;SharedAccessKey=KEY_HERE",
}
}
And the function signature is:
public static void ProcessQueue([ServiceBusTrigger("queueName", AccessRights.Listen, Connection = "serviceBusConnection")] ...)
Based on the discussion in the GitHub issue, I believed that having either a serviceBusConnection entry OR an AzureWebJobServiceBus entry should be enough to ensure that the central listener triggers the function when a new message is added to the ServiceBusQueue, but that is proving to not be the case.
Can anyone clarify the difference between how those two settings are used, or notice anything else with the settings I provided that might be causing the function to not properly be triggered after a period of inactivity?
I suggest there are several possible causes for this behavior. I have several Azure subs and only one of them had issues with Storage/Service Bus-based triggers only popping up when app is not idle. So far I have observed that actions listed below will prevent triggers from working correctly:
Creating any Storage-based trigger, deleting (for any reason) the triggering object and re-creating it.
Corrupting azure function input parameters by deleting/altering associated objects without recompiling a function
Restarting functions app when one of the functions fails to compile/bind to trigger OR input parameter and hangs may cause same problems.
It has also been observed that using legacy Connection Strings setting for trigger binding will not work.
Clean deploy of an affected function app will most likely solve the problem if it was caused by any of the actions described above.
EDIT:
It looks like this is also caused by setting Authorization/Authentication on the functions app, but I have not yet figured out if it happens in general or when Auth has specific configuration. Tested on affected Azure sub by disabling auth at all - function going idle after 30-40 mins, queue trigger still initiates an execution, though with a delay as expected. I have found an old bug related to this, but it says issue resolved.
I am having problems getting the Microsoft.Azure.Documents library to initialize the client in an azure worker role. I'm using Nuget Package 0.9.1-preview.
I have mimicked what was done in the example for azure document
When running locally through the emulator I can connect fine with the documentdb and it runs as expected. When running in the worker role, I am getting a series of NullReferenceException and then ArgumentNullException.
The bottom System.NullReferenceException that is highlighted above has this call stack
so the nullReferenceExceptions start in this call at the new DocumentClient.
var endpoint = "myendpoint";
var authKey = "myauthkey";
var enpointUri = new Uri(endpoint);
DocumentClient client = new DocumentClient(endpointUri, authKey);
Nothing changes between running it locally vs on the worker role other then the environment (obviously).
Has anyone gotten DocumentDb to work on a worker role or does anyone have an idea why it would be throwing null reference exceptions? The parameters getting passed into the DocumentClient() are filled.
UPDATE:
I tried to rewrite it being more generic which helped at least let the worker role run and let me attached a debugger. It is throwing the error on the new DocumentClient. Seems like some security passing is null. Both the required parameters on initialization are not null. Is there a security setting I need to change for my worker role to be able to connect to my documentdb? (still works locally fine)
UPDATE 2:
I can get the instance to run in release mode, but not debug mode. So it must be something to do with some security setting or storage setting that is misconfigured I guess?
It seems I'm getting System.Security.SecurityExceptions - only when using The DocumentDb - queues do not give me that error. All Call Stacks for that error seem to be with System.Diagnostics.EventLog. The very first Exception I see in the Intellitrace Summary is System.Threading.WaitHandleCannotBeOpenedException.
More Info
Intellitrace summary exception data:
top is the earliest and bottom is the latest (so System.Security.SecurityException happens first then the NullReference)
The solution for me to get rid of the security exception and null reference exception was to disable intellitrace. Once I did that, I was able to deploy and attach debugger and see everything working.
Not sure what is between the null in intellitrace and the DocumentClient, but hopefully it's just in relation to the nuget and it will be fixed in the next iteration.
unable to repro.
I created a new Worker Role. Single instance. Added authkey & endoint config to cscfg.
Created private static DocumentClient at WorkerRole class level
Init DocumentClient in OnStart
Dispose DocumentClient in OnStop
In RunAsync inside loop,
execute a query Works as expected.
Test in emulator works.
Deployed as Release to Production slot. works.
Deployed as Debug to Staging with Remote Debug. works.
Attached VS to CloudService, breakpoint hit inside loop.
Working solution : http://ryancrawcour.blob.core.windows.net/samples/AzureCloudService1.zip
(I'm quite new to Windows Azure development, so I hope I'm using the right terms.)
We have an Azure Worker Role that is supposed to fetch data stored in Blob Storage.
Somehow we occasionally get the following error message:
Microsoft.WindowsAzure.StorageClient.StorageServerException: The server encountered an unknown failure: The remote name could not be resolved: 'XXX.blob.core.windows.net' ---> System.Net.WebException: The remote name could not be resolved: 'XXX.blob.core.windows.net'
This seems strange, since requests only a second before and/or after works as expected.
If I understand things correctly, the CloudBlob class has internal retry functionality. It seems that this is not considered as a "retryable" error. Is this perhaps handled by the Transient Error Handling Block (Topaz), or do we have to handle this specific error in some other way?
I'm running into a problem with getting SSL to work in the Development Fabric. I'm running a clean install of Windows 8 Pro with Visual Studio 2012 Ultimate and the October 2012 Azure SDK for .NET. IIS8 is not installed, only IIS Express, which claims to support HTTPS so I'm hoping that's not the issue.
Running VS 12 as administrator, I've created a blank VS solution, added a new (.NET 4.5) cloud service with a new ASP.NET MVC 4 Internet web application project, and hit F5. Everything works fine. Then, when I add an SSL certificate to the web role and replace the HTTP endpoint (port 80) with an HTTPS endpoint (port 443, with the certificate), hitting F5 produces the following error message:
Windows Azure Tools for Microsoft Visual Studio
There was an error attaching the debugger to the role instance 'deployment18(32).WindowsAzureCloudService.Mvc4WebRole_IN_0' with Process Id: 4892'. Unable to attach. Access is denied.
Note, the last part ("Access is denied") comes in a few variations, a particularly pleasant one being "Catastrophic failure". :)
The only message in the VS Output window ('General' output) is:
Windows Azure Tools: Warning: Remapping private port 443 to 444 in role 'Mvc4WebRole' to avoid conflict during emulation.
The Compute Emulator UI is not much help; just before the instance disappears, this is the only console output that I get consistently (sometimes other messages appear, but sporadically every few runs; I'm not sure how to capture these):
[fabric] Role Instance: deployment18(33).WindowsAzureCloudService.Mvc4WebRole.0
[fabric] Role state Unknown
[fabric] Role state Suspended
[fabric] Role state Busy
[fabric] Role state Unhealthy
[fabric] Role state Stopped
The certificate was obtained from a CA and properly imported into the Local Machine/Personal/Certificates store as a .pfx with private key, extended properties, and marked as exportable, for what it's worth.
When I attempt to publish the service to Azure, I get one build (validation) warning about the database connection string (which I assume is irrelevant):
The connection string 'DefaultConnection' is using a local database '(LocalDb)\v11.0' in project 'Mvc4WebRole'. This connection string will not work when you run this application in Windows Azure. To access a different database, you should update the connection string in the web.config file.
Probably more important, the deployment actually fails with the following history in the Windows Azure Activity Log window:
9:00:25 AM - Warning: There are package validation warnings.
9:00:25 AM - Preparing deployment for WindowsAzureCloudService - 1/3/2013 8:59:55 AM with Subscription ID '<...>' using Service Management URL 'https://management.core.windows.net/'...
9:00:25 AM - Connecting...
9:00:26 AM - Object reference not set to an instance of an object.
9:00:26 AM - Deployment failed with a fatal error
Can someone help me troubleshoot this issue? I've rebooted a few times. ;)
Thanks in advance!
EDIT (Jan. 3, 4:44 PM): I have a few ideas that might help me make progress, but some are pretty drastic so any advice would be appreciated:
Is there a way to capture all the output from the Compute Emulator (Dev Fabric) to a log file so I can review it? (System.Diagnostic.Trace calls from my service won't help, since I don't even get as far as the RoleEntryPoint when using HTTPS!) I figured this out; see next edit.
That null pointer exception during the Azure deployment has me worried. Is it worthwhile to try reinstalling the Azure SDK, and if so, how should I go about doing a clean install of it?
Has anyone seen a problem of this sort disappear when switching to using full IIS for the emulator? (That seems unlikely since IIS vs. IIS Express should have no relevance to the Azure deployment.)
EDIT (Jan. 4, 10:15 AM): Bad news: I tried the suggestion to grant Read access to the certificates, but it didn't help in my case. Good news: I managed to capture one of those sporadic messages in the Compute Emulator UI before it shut down; it was a bit of info from some diagnostics. Not helpful in and of itself, but it revealed where the Development Fabric was storing its temporary files:
[Diagnostics] Information: C:\Users\Lars\AppData\Local\dftmp\Resources\0005155d-4592-40f4-812e-18793b26576c\directory\DiagnosticStore\Monitor
The GUID portion gets recreated for every deployment, and it is deleted when the deployment goes away (as it always does in my case). But in the parent directory ('dftmp'), there are a few helpful directories that I then monitored during a new deployment: DevFCLogs, DFAgentLogs, and IISConfiguratorLogs. I guess that answers the first question I had yesterday! :)
DFAgentLogs\DFAgent.log: (41KB) No useful information. A bunch of "Failure to read pipe" messages and failures to get the role/deployment instance ID, which I assume are just noise.
DevFCLogs\DevFabric--2013.01.04--<...>.log: (510 KB) No useful information. I skimmed the file and also searched for 'error', 'failure', 'not found', 'certificate', and 'Mvc4WebRole_IN_0'; none of those showed any hints of what was going on.
IISConfiguratorLogs\IISConfigurator.log: (6 KB) Now we're making progress!! :) Can someone tell me what this means? (In the meantime, I'm off ILSpy-hunting... fun fun...)
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:08.915] Using IIS Express appdomain
(...)
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:08.936] Adding binding 127.255.0.0:444: to site deployment18(40).WindowsAzureCloudService.Mvc4WebRole_IN_0_Web
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:10.484] Caught exception
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:10.487] Exception:System.Runtime.InteropServices.COMException (0x800401F3): Invalid class string (Exception from HRESULT: 0x800401F3 (CO_E_CLASSSTRING))
Server stack trace:
at Microsoft.Web.Administration.Interop.IAppHostProperty.get_Value()
at Microsoft.Web.Administration.ConfigurationElement.GetPropertyValue(IAppHostProperty property)
at Microsoft.Web.Administration.Binding.get_CertificateHash()
at Microsoft.Web.Administration.BindingCollection.Add(Binding binding)
at Microsoft.WindowsAzure.ServiceRuntime.IISConfigurator.WasManager.DeploySite(String roleId, WASite roleSite, String appPoolName, String sitePath, String iisLogsRootFolder, String failedRequestLogsRootFolder, List1 bindings, List1 protocols, FileManager fileManager, WAAppPool defaultAppPoolSettings, String roleGuid, String& appPoolSid, List`1 appPoolsAdded, String configPath)
EDIT (Jan. 4, 11 AM): ILSpy wasn't much help; the exception is being thrown at an interop point (we knew that already) while trying to get the hash of a certificate in order to set up the binding (we knew that too). Does anyone know what COM object would need to be registered in order to get a certificate hash for a binding in Microsoft.Web.Administration? Or how I could intercept the interop call to find out? Bonus points if you can tell me why this is happening in the first place. :)
I've had similar problem on two computers. On both cases installing IIS solved the problem.
It seems to be enough to just install the IIS (via add/remove Windows components). You don't need to start using it. The installation changes something and after that my IIS Express started working again with HTTPS from Visual Studio.
There is a discussion on similar issue on MSDN Social:
http://social.msdn.microsoft.com/Forums/nl-NL/windowsazuredevelopment/thread/ad362016-16f6-459a-8022-9307aa5f910e
And the issue has been also raised on Microsoft connect:
https://connect.microsoft.com/VisualStudio/feedback/details/758533
In my case the error in the log files was:
IISConfigurator Information: 0 : [00007644:00000007, 2013.01.17
00:39:18.523] Exception:System.Runtime.InteropServices.COMException
(0x800401F3): Invalid class string (Exception from HRESULT: 0x800401F3
(CO_E_CLASSSTRING))
I found the log files from C:\Users\\AppData\Local\dftmp\IISConfiguratorLogs directory.
When running locally with a private key cert for SSL, you'll need to give the user the emulator app is running under access to the private key. Open mmc.exe and add the Certificates >> Local Computer Snap-In to view your certificate. Right Click on the certificate, then All Tasks >> Manage Private Keys - then add IUSR and Network Service with at least read access.
For deployment to azure, you'll need to upload the certificate to the Cloud Service and make sure the certificate is valid for the domain.
Follow step 11 from http://www.microsoft.com/en-us/download/details.aspx?id=35448. From this SO post