Azure WebJob Service Bus Trigger Fails to Complete - azure

I have an Azure WebJob set to run continuously which uses the Service Bus triggers to read queues and topics. As the queues are processed, messages get sent into a topic and I have one topic to manage updates to cache.
Temporarily, I disabled the cache updates by simply returning from the function so that the message is cleared out of the topic.
public static void ProcessSearchUpdate([ServiceBusTrigger("events", "SearchClientUpdates", AccessRights.Listen)] BrokeredMessage message, System.IO.TextWriter log)
{
return;
}
Most of the time, things go as planned and the messages simply are read and discarded. However, sometimes, I get the following exception which I'm assuming is something from the webjobs sdk trying to write logs.
As you can see, my method is not doing anything and should therefore not timeout.
When it fails, it prevents any trigger from executing which is quite annoying. It seems whatever is failing in the SDK has a 5 minute timeout and it prevents anything from running for 5 minutes, plus the restart time.
Microsoft.WindowsAzure.Storage.StorageException: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 774
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1162
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1285
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 114
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<SaveAndCloseAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
Request Information
RequestID:
RequestDate:
StatusMessage:

The exception related to the WebJob log system. Azure WebJob SDK will periodically flush the logs to blob storage. To solve the timeout issue, we could increase the timeout value of CloudBlobClient that WebJob SDK uses for this. Steps below are for your reference.
Step 1. Update the version of Microsoft.Azure.WebJobs to 2.0.0 using NuGet.
Step 2. Implement our own StorageClientFactory which could reset the timeout value of blob client.
public class MyStorageClientFactory : Microsoft.Azure.WebJobs.Host.StorageClientFactory
{
public override CloudBlobClient CreateCloudBlobClient(StorageClientFactoryContext context)
{
CloudBlobClient client = context.Account.CreateCloudBlobClient();
client.DefaultRequestOptions.ServerTimeout = TimeSpan.FromMinutes(60);
return client;
}
}
Step 3. Let JobHost use the custom StorageClientFactory.
JobHostConfiguration configuration = new JobHostConfiguration();
configuration.StorageClientFactory = new MyStorageClientFactory();
var host = new JobHost(configuration);

Related

Azure table storage Client takes a lot of time retrieving records

We have a windows service running on our server machines which listens to an Event Bus, handles the events by inserting or retrieving records from Azure Table Storage.
The problem is, during peak hours almost every day it takes a lot of time to process a request made to Azure table storage, due to which there is a huge spike in the unprocessed events in the event bus.
This is the timeout error:
{"EndOfTail":false,"UserFeeds":null,"Message":"The operation has timed out.","StackTrace":" at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Core\\Executor\\Executor.cs:line 649\r\n at Microsoft.WindowsAzure.Storage.Table.TableQuery`1.<>c__DisplayClass7.<ExecuteInternal>b__6(IContinuationToken continuationToken) in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\ClassLibraryCommon\\Table\\TableQuery.cs:line 430\r\n at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.<LazyEnumerable>d__0`1.MoveNext() in c:\\Program Files (x86)\\Jenkins\\workspace\\release_dotnet_master\\Lib\\Common\\Core\\Util\\CommonUtility.cs:line 322\r\n at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()\r\n at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)\r\n at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)\r\n at Rebtel.Service.UserFeed.DataLayer.UserFeedRepository.<Execute>d__31.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\DataLayer\\UserFeedRepository.cs:line 531\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Rebtel.Service.UserFeed.DataLayer.UserFeedRepository.<GetUserFeedsByTypes>d__17.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\DataLayer\\UserFeedRepository.cs:line 254\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Rebtel.Service.UserFeed.BusinessLayer.Actions.GetUserFeedAction.<Execute>d__7.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\BusinessLayer\\Actions\\GetUserFeedAction.cs:line 68\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Rebtel.Service.UserFeed.UserFeedServiceImplementation.<GetActions>d__17.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\BackendServices\\Backend\\Rebtel.Service.UserFeed\\UserFeedServiceImplementation.cs:line 182\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Rebtel.Platform.Service.ServiceRequestInterceptor.<TryInvokeTaskWithServiceReplyResult>d__10`1.MoveNext() in C:\\TeamCity\\buildAgent\\work\\18bc44b16693ee8a\\Platform\\Platform\\Rebtel.Platform.Backend\\Service\\ServiceRequestInterceptor.cs:line 0","Result":501}
And it took like 603,060 milliseconds to get this response.
The problem is not with the scalability of the Azure storage account, because if we restart the windows service again with over a million events waiting it processes them instantly.
One of the queries looks like this:
query.WherePartitionKey(QueryComparisons.Equal, userId).AndRowKey(QueryComparisons.NotEqual, LastActionRowIdTableEntity.UniqueRowKey);
The strange thing is when I look at the monitoring graphs of the Azure table storage the average server latency is not more than 200ms.
Do you guys have any idea why this is happening?
Edit: I have verified that we have taken into account all of the recommendations on Microsoft table storage documentation for performance.
On the ServicePoint
Nagle Algo is Off.
Max connections are set to 1000.

Service Fabric Stateless Identity Server 3 service fails under load

We are in the process of migrating a stateless services based on Identity Server 3 (https://github.com/IdentityServer/IdentityServer3) to service fabric.
The project runs fine on a local development box and on a production cluster under normal development load but when put into production at about 20-30 requests per second it quickly stops responding to requests and the configured health checks in ARR go Unhealthy.
The service is fronted by a IIS ARR (Application Request Routing) cluster that does SSL off-loading.
The identity server logs output the following two errors, one seemingly related to inbound requests and another related to outbound requests to azure storage used for identity server persistence.
Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Collabco.Myday.Identity.IdSvr.BaseStore`1.<ExecuteQueryAsync>d__21.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\BaseStore.cs:line 258
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Collabco.Myday.Identity.IdSvr.ScopeStore.<GetScopesAsync>d__3.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\ScopeStore.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer3.Core.Endpoints.DiscoveryEndpointController.<GetConfiguration>d__11.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Endpoints\Connect\DiscoveryEndpointController.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.System.Web.Http910180.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
Request Information
RequestID:
RequestDate:
StatusMessage:
and another exception we see is:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.HttpListenerRequest'.
at System.Net.HttpListenerRequest.CheckDisposed()
at System.Net.HttpListenerRequest.get_LocalEndPoint()
at System.Net.HttpListenerRequest.get_IsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.GetServerIsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.get_ServerIsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.PropertiesTryGetValue(String key, Object& value)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.TryGetValue(String key, Object& value)
at Microsoft.Owin.OwinContext.Get[T](String key) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.Owin.OwinHttpRequestContext.get_IsLocal() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.Owin.OwinHttpRequestContext.get_IncludeErrorDetail() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Net.Http.System.Web.Http910180.HttpRequestMessageExtensions.CreateErrorResponse(HttpRequestMessage request, HttpStatusCode statusCode, Func`2 errorCreator) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.Handle(ExceptionHandlerContext context) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.LastChanceExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.ExceptionHandlerExtensions.<HandleAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Owin.PassiveAuthenticationMessageHandler.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
After approximately 5-10 minutes passed after removing load without recycling anything the service comes back to life. Service fabric detects no failures also during the unstable period.
Anyone any ideas?
The solution to this was to:
Ensure that timeout exceptions were being caught and handled and logged, in some cases they were not.
Tweaked the retry policy to make retries closer together (100ms delay) and limited to no more than 10 seconds. Also implemented logging of retries. In reality i'm getting no retries so far.
Increased the connection limit on the table service endpoint to 1000.
The most significant factor appears to be number 3 although some improvement was noted by implementing 1 which i did first which is kind of obvious really.
To do this I use a static constructor on my repository class (base store class in identity server terms) has the following code which depends on a StorageAccount instance. All but the last line already existed.
var tableServicePoint = System.Net.ServicePointManager.FindServicePoint(storageAccount.TableEndpoint);
tableServicePoint.UseNagleAlgorithm = false;
tableServicePoint.Expect100Continue = false;
tableServicePoint.ConnectionLimit = 1000;
The following article helped with this: https://azure.microsoft.com/en-gb/documentation/articles/storage-performance-checklist/
In summary my conclusion is the default connection limit (2 or 10 different documentation conflicts) was causing request to table storage to queue up and eventually timeout and ultimately cause the service to fall over.
To confirm setting the connection limit wasn't necessary when using azure web apps so there is something different about the way service fabric / owin self-host projects work in terms of their use of connections that affects azure storage.

System.InvalidOperationException: No authentication handler is configured to handle the scheme: Automatic

Scenario:
Create a New Asp.Net Core (version 1.0.0) project
Choose the Web API template
Add the [Authorize] attribute on the default ValuesController
Run the application
If I run the application with IIS and make a GET request to http://localhost:60513/api/values I get the expected 401 Unauthorized error
If however I run the application with Kestrel (e.g: dotnet run) and make a GET request to http://localhost:5000/api/values I get a 500 Internal Server Error with the following exception at kestrel:
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
Request starting HTTP/1.1 GET http://localhost:5000/api/values
info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
Authorization failed for user: .
warn: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
info: Microsoft.AspNetCore.Mvc.ChallengeResult[1]
Executing ChallengeResult with authentication schemes ().
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HKUMMBBBQ6AU": An unhandled exception was thrown by the application.
System.InvalidOperationException: No authentication handler is configured to handle the scheme: Automatic
at Microsoft.AspNetCore.Http.Authentication.Internal.DefaultAuthenticationManager.<ChallengeAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.ChallengeResult.<ExecuteResultAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeResultAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
Request finished in 282.8427ms 200
My question is Why my application has different result depending on the server hosting it? Why Kestrel and IIS handle the Authorization differently?
Please note that there are similar questions in StackOverflow like this or this other but they all are for more complex scenarios where filters or middleware are involved.
I do not have any middleware in the AspNet pipeline other than MVC, and all the code but the [Authorize] attribute is automatically generated by the AspNet Web Api template.
As per this thread in AspNet Security, the Authorize attribute on any action or controller requires at least one auth middleware in the pipeline in order to issue challenges.
When using IIS the IIS middleware is used but when using Kestrel there is no auth middleware dealing with this therefore we need to add our own.

Publish to Azure App Service crashes after upgrade to asp.net core rc2

When I try to publish a Asp.net core RC2 application to Azure app service, Visual Studio will crash when I press the last Publish -button in the wizard. On the previous page I can press Preview and it will list all changes. Sometimes it seems to crash even before I press Publish.
When I debug the crash with another instance of Visual Studio I get this information:
An unhandled exception of type 'System.InvalidOperationException'
occurred in PresentationFramework.dll
Additional information: DialogResult can be set only after Window is
created and shown as dialog.
or
An unhandled exception of type 'System.NullReferenceException'
occurred in mscorlib.dll
I tried running VS in safe mode but when I go to the Publish wizard is says
The 'CommonAzureToolsPackage' package did not load correctly
and in ActivityLog.xml I see
823
2016/06/02 07:27:13.770
Error
VisualStudio
SetSite failed for package [CommonAzureToolsPackage][Object reference not set to an instance of
an object.]:{ at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.Authentication.KeychainUtil.<<SetAccountManager>b__0>d__1.MoveNext()
---
End of stack trace from previous location where exception was thrown
---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.Authentication.KeychainUtil.<GetVSAccountProviderAsync>d__4.MoveNext()
---
End of stack trace from previous location where exception was thrown
---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.KeychainEndpointProvider.<GetInstanceAsync>d__0.MoveNext()
---
End of stack trace from previous location where exception was thrown
---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.EndpointProviderFactory.<CreateEndpointProviderList>d__3.MoveNext()
---
End of stack trace from previous location where exception was thrown
---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.EndpointProviderFactory.<GetEndpointProviderAsync>d__0.MoveNext()
---
End of stack trace from previous location where exception was thrown
---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.EndpointProviderFactory.GetEndpointProvider()
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.CommonAzureToolsPackage.<.ctor>b__1()
at System.Lazy1.CreateValue()
at
System.Lazy1.LazyInitValue()
at
System.Lazy1.get_Value()
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.CommonAzureToolsPackage.<.ctor>b__2()
at System.Lazy1.CreateValue()
at
System.Lazy1.LazyInitValue()
at
System.Lazy1.get_Value()
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.CommonAzureToolsPackage.<RegisterServices>b__61(IServiceContainer
c, Type t)
at
Microsoft.VisualStudio.Shell.Package.GetService(Type
serviceType)
at
Microsoft.VisualStudio.Shell.Package.System.IServiceProvider.GetService(Type
serviceType)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.IServiceProviderExtensions.GetService[InterfaceType,ServiceType](IServiceProvider
serviceProvider)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.ServerExplorer.AzureNodeStateManager.Initialize(IServiceProvider
serviceProvider)
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.CommonAzureToolsPackage.InitializeAzureNodeStateManager()
at
Microsoft.VisualStudio.WindowsAzure.CommonAzureTools.CommonAzureToolsPackage.Initialize()
at
Microsoft.VisualStudio.Shell.Package.Microsoft.VisualStudio.Shell.Interop.IVsPackage.SetSite(IServiceProvider
sp)}
{1E863AE9-93F6-42FD-A301-D3391F876208}
80004003 - E_POINTER
Object reference not set to an instance of an object.
Just noticed that if I authenticate to Azure through Server Explorer -> Azure beforehand I will be able to publish.
This was a bug in the Visual Studio Azure tooling and was later fixed.

Errors when running any stateful service on local cluster

Running Service Fabric 4.4.87.9494 on Windows 8.1
I can successfully deploy and run a stateless service or a stateful or stateless actor on my local cluster, but when I try to deploy a stateful service, I get the following error in the Service Fabric Explorer:
Event Warning Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open. Error = System.IO.FileNotFoundException (-2147024894)
The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
at System.Fabric.Data.Log.Interop.NativeLog.IKPhysicalLogManager.EndOpenLogContainer(IFabricAsyncOperationContext Context, IKPhysicalLogContainer& Result)
at System.Fabric.Data.Log.Interop.PhysicalLogManager.<OpenLogContainerAsync>b__5(IFabricAsyncOperationContext Context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.Data.Log.LogManager.<OnOpenPhysicalLogAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<OpenPhysicalLogAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<CreateLogFileAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.LogManager.<OpenAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.ServiceFabric.Replicator.LoggingReplicator.<OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.<OpenAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.TransactionalReplicator.<OpenAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.StatefulServiceReplica.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceBase.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
I see the same error whether I create a default stateful service with Visual Studio, or use any of the samples from the getting started repo, or add a stateful service to an existing app. The same thing happens regardless of whether I use Visual Studio to deploy it or run the Powershell commands. However, if I deploy the same application to Azure, all is well.
I've tried this on two other Windows 8.1 computers at work with the same results, however I tried it on my home machine (Windows 10) and did not see the problem. This suggests it might be something in the Group policy that's affecting things. I've looked at the resultant group policy and can't find anything that I would have thought could have caused such a problem. I've tried disabling the virus checker during deployment too.
I tried to use Process Monitor to figure out what file it's looking for but couldn't find anything obvious.
Any pointers on how to progress this would be very much appreciated. I want to get the team all up & running and I can't if we can't create stateful services!
This is likely a bug that was in version 4.4. Please upgrade to version 4.5 as this would have the fix needed.

Resources