We encounter a very bizarre issue with Azure AD library in a asp.net web api app. When we try to get token using AcquireTokenAsync with a Certificate, there is no response and then time out.
Bizarre part:
If we copy the same code in a console app, console app works, get token instantly.
if we use fiddler to start a proxy which overtake our company's default proxy, webapi app works.
The testing code we are using is exactly from the documentation.
Sample code we use :
public async Task < string > GetTokenAsync() {
var context = new AuthenticationContext(authority);
var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadOnly);
var results = store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, false);
var cert = new ClientAssertionCertificate(clientid, results[0]);
var token = await context.AcquireTokenAsync(resource, cert).AccessToken;
return token;
}
Basically the code should work properly since it is from documentation directly. is it something relative to the networking?
The Error we get in short:
at System.Web.Http.Results.ExceptionResult..ctor(Exception exception, IDependencyProvider dependencies)
at System.Web.Http.ApiController.InternalServerError(Exception exception)
at WebApplication1.Controllers.TokenController.d__0.MoveNext() in C:\Users\QiaoH\source\repos\WebApplication1\WebApplication1\Controllers\TokenController.cs:line 55 --- End of stack trace from previous location where exception was thrown
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.TaskHelpersExtensions.d__1`1.MoveNext()
End of stack trace from previous location where exception was thrown
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.d__1.MoveNext() --- End of stack trace from previous location where exception was thrown
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.d__5.MoveNext()
End of stack trace from previous location where exception was thrown
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.d__15.MoveNext()
Problem solved:
Due to the proxy, we have to enabled default proxy with default credential.
We put below in web.config and it works.
<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
</defaultProxy>
</system.net>
Related
I have an Azure WebJob set to run continuously which uses the Service Bus triggers to read queues and topics. As the queues are processed, messages get sent into a topic and I have one topic to manage updates to cache.
Temporarily, I disabled the cache updates by simply returning from the function so that the message is cleared out of the topic.
public static void ProcessSearchUpdate([ServiceBusTrigger("events", "SearchClientUpdates", AccessRights.Listen)] BrokeredMessage message, System.IO.TextWriter log)
{
return;
}
Most of the time, things go as planned and the messages simply are read and discarded. However, sometimes, I get the following exception which I'm assuming is something from the webjobs sdk trying to write logs.
As you can see, my method is not doing anything and should therefore not timeout.
When it fails, it prevents any trigger from executing which is quite annoying. It seems whatever is failing in the SDK has a 5 minute timeout and it prevents anything from running for 5 minutes, plus the restart time.
Microsoft.WindowsAzure.Storage.StorageException: Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromStream(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 774
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadFromByteArray(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1162
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndUploadText(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 1285
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 114
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<UpdateOutputBlob>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Loggers.UpdateOutputLogCommand.<SaveAndCloseAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
Request Information
RequestID:
RequestDate:
StatusMessage:
The exception related to the WebJob log system. Azure WebJob SDK will periodically flush the logs to blob storage. To solve the timeout issue, we could increase the timeout value of CloudBlobClient that WebJob SDK uses for this. Steps below are for your reference.
Step 1. Update the version of Microsoft.Azure.WebJobs to 2.0.0 using NuGet.
Step 2. Implement our own StorageClientFactory which could reset the timeout value of blob client.
public class MyStorageClientFactory : Microsoft.Azure.WebJobs.Host.StorageClientFactory
{
public override CloudBlobClient CreateCloudBlobClient(StorageClientFactoryContext context)
{
CloudBlobClient client = context.Account.CreateCloudBlobClient();
client.DefaultRequestOptions.ServerTimeout = TimeSpan.FromMinutes(60);
return client;
}
}
Step 3. Let JobHost use the custom StorageClientFactory.
JobHostConfiguration configuration = new JobHostConfiguration();
configuration.StorageClientFactory = new MyStorageClientFactory();
var host = new JobHost(configuration);
We are in the process of migrating a stateless services based on Identity Server 3 (https://github.com/IdentityServer/IdentityServer3) to service fabric.
The project runs fine on a local development box and on a production cluster under normal development load but when put into production at about 20-30 requests per second it quickly stops responding to requests and the configured health checks in ARR go Unhealthy.
The service is fronted by a IIS ARR (Application Request Routing) cluster that does SSL off-loading.
The identity server logs output the following two errors, one seemingly related to inbound requests and another related to outbound requests to azure storage used for identity server persistence.
Microsoft.WindowsAzure.Storage.StorageException: The client could not finish the operation within specified timeout. ---> System.TimeoutException: The client could not finish the operation within specified timeout.
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass1`1.<CreateCallback>b__0(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Collabco.Myday.Identity.IdSvr.BaseStore`1.<ExecuteQueryAsync>d__21.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\BaseStore.cs:line 258
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Collabco.Myday.Identity.IdSvr.ScopeStore.<GetScopesAsync>d__3.MoveNext() in C:\Dev\myday-identity\IdentityServer\IdSvr\ScopeStore.cs:line 43
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer3.Core.Endpoints.DiscoveryEndpointController.<GetConfiguration>d__11.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Endpoints\Connect\DiscoveryEndpointController.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Threading.Tasks.System.Web.Http910180.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
Request Information
RequestID:
RequestDate:
StatusMessage:
and another exception we see is:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.HttpListenerRequest'.
at System.Net.HttpListenerRequest.CheckDisposed()
at System.Net.HttpListenerRequest.get_LocalEndPoint()
at System.Net.HttpListenerRequest.get_IsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.GetServerIsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.get_ServerIsLocal()
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.PropertiesTryGetValue(String key, Object& value)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.CallEnvironment.TryGetValue(String key, Object& value)
at Microsoft.Owin.OwinContext.Get[T](String key) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.Owin.OwinHttpRequestContext.get_IsLocal() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.Owin.OwinHttpRequestContext.get_IncludeErrorDetail() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Net.Http.System.Web.Http910180.HttpRequestMessageExtensions.CreateErrorResponse(HttpRequestMessage request, HttpStatusCode statusCode, Func`2 errorCreator) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.Handle(ExceptionHandlerContext context) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.DefaultExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.LastChanceExceptionHandler.HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
at System.Web.Http.ExceptionHandling.ExceptionHandlerExtensions.<HandleAsyncCore>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Owin.PassiveAuthenticationMessageHandler.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext() in c:\local\identity\server3\IdentityServer3\source\Core\Configuration\DiscoveryOptions.cs:line 0
After approximately 5-10 minutes passed after removing load without recycling anything the service comes back to life. Service fabric detects no failures also during the unstable period.
Anyone any ideas?
The solution to this was to:
Ensure that timeout exceptions were being caught and handled and logged, in some cases they were not.
Tweaked the retry policy to make retries closer together (100ms delay) and limited to no more than 10 seconds. Also implemented logging of retries. In reality i'm getting no retries so far.
Increased the connection limit on the table service endpoint to 1000.
The most significant factor appears to be number 3 although some improvement was noted by implementing 1 which i did first which is kind of obvious really.
To do this I use a static constructor on my repository class (base store class in identity server terms) has the following code which depends on a StorageAccount instance. All but the last line already existed.
var tableServicePoint = System.Net.ServicePointManager.FindServicePoint(storageAccount.TableEndpoint);
tableServicePoint.UseNagleAlgorithm = false;
tableServicePoint.Expect100Continue = false;
tableServicePoint.ConnectionLimit = 1000;
The following article helped with this: https://azure.microsoft.com/en-gb/documentation/articles/storage-performance-checklist/
In summary my conclusion is the default connection limit (2 or 10 different documentation conflicts) was causing request to table storage to queue up and eventually timeout and ultimately cause the service to fall over.
To confirm setting the connection limit wasn't necessary when using azure web apps so there is something different about the way service fabric / owin self-host projects work in terms of their use of connections that affects azure storage.
Scenario:
Create a New Asp.Net Core (version 1.0.0) project
Choose the Web API template
Add the [Authorize] attribute on the default ValuesController
Run the application
If I run the application with IIS and make a GET request to http://localhost:60513/api/values I get the expected 401 Unauthorized error
If however I run the application with Kestrel (e.g: dotnet run) and make a GET request to http://localhost:5000/api/values I get a 500 Internal Server Error with the following exception at kestrel:
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
Request starting HTTP/1.1 GET http://localhost:5000/api/values
info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]
Authorization failed for user: .
warn: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
info: Microsoft.AspNetCore.Mvc.ChallengeResult[1]
Executing ChallengeResult with authentication schemes ().
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HKUMMBBBQ6AU": An unhandled exception was thrown by the application.
System.InvalidOperationException: No authentication handler is configured to handle the scheme: Automatic
at Microsoft.AspNetCore.Http.Authentication.Internal.DefaultAuthenticationManager.<ChallengeAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.ChallengeResult.<ExecuteResultAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeResultAsync>d__32.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
Request finished in 282.8427ms 200
My question is Why my application has different result depending on the server hosting it? Why Kestrel and IIS handle the Authorization differently?
Please note that there are similar questions in StackOverflow like this or this other but they all are for more complex scenarios where filters or middleware are involved.
I do not have any middleware in the AspNet pipeline other than MVC, and all the code but the [Authorize] attribute is automatically generated by the AspNet Web Api template.
As per this thread in AspNet Security, the Authorize attribute on any action or controller requires at least one auth middleware in the pipeline in order to issue challenges.
When using IIS the IIS middleware is used but when using Kestrel there is no auth middleware dealing with this therefore we need to add our own.
Running Service Fabric 4.4.87.9494 on Windows 8.1
I can successfully deploy and run a stateless service or a stateful or stateless actor on my local cluster, but when I try to deploy a stateful service, I get the following error in the Service Fabric Explorer:
Event Warning Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open. Error = System.IO.FileNotFoundException (-2147024894)
The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
at System.Fabric.Data.Log.Interop.NativeLog.IKPhysicalLogManager.EndOpenLogContainer(IFabricAsyncOperationContext Context, IKPhysicalLogContainer& Result)
at System.Fabric.Data.Log.Interop.PhysicalLogManager.<OpenLogContainerAsync>b__5(IFabricAsyncOperationContext Context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.Data.Log.LogManager.<OnOpenPhysicalLogAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<OpenPhysicalLogAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<CreateLogFileAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.LogManager.<OpenAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.ServiceFabric.Replicator.LoggingReplicator.<OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.<OpenAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.TransactionalReplicator.<OpenAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.StatefulServiceReplica.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceBase.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
I see the same error whether I create a default stateful service with Visual Studio, or use any of the samples from the getting started repo, or add a stateful service to an existing app. The same thing happens regardless of whether I use Visual Studio to deploy it or run the Powershell commands. However, if I deploy the same application to Azure, all is well.
I've tried this on two other Windows 8.1 computers at work with the same results, however I tried it on my home machine (Windows 10) and did not see the problem. This suggests it might be something in the Group policy that's affecting things. I've looked at the resultant group policy and can't find anything that I would have thought could have caused such a problem. I've tried disabling the virus checker during deployment too.
I tried to use Process Monitor to figure out what file it's looking for but couldn't find anything obvious.
Any pointers on how to progress this would be very much appreciated. I want to get the team all up & running and I can't if we can't create stateful services!
This is likely a bug that was in version 4.4. Please upgrade to version 4.5 as this would have the fix needed.
I'm using Azure mobile services with .net backend, with MvvmCross using Xamarin Pcl based solution.
The service returns me the results when I run it interactively from the default browser interface provided by Azure Mobile Services. Also I'm using offline data sync feature. I can push the data to the server and it works but now when I try to pull the data from the server.
{Microsoft.WindowsAzure.MobileServices.MobileServiceInvalidOperationException: The request could
not be completed. () at
Microsoft.WindowsAzure.MobileServices.MobileServiceHttpClient.d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.MobileServices.MobileServiceHttpClient.d__1d.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at
Microsoft.WindowsAzure.MobileServices.MobileServiceHttpClient.<RequestAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at
Microsoft.WindowsAzure.MobileServices.MobileServiceTable.d__f.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at
Microsoft.WindowsAzure.MobileServices.MobileServiceTable.<ReadAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at
Microsoft.WindowsAzure.MobileServices.Sync.PullAction.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.MobileServices.Sync.TableAction.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.MobileServices.Sync.MobileServiceSyncContext.
d__3b.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.MobileServices.Sync.MobileServiceSyncContext.d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at
AgLiveMobile.Core.Services.EmployeeService.d__4.MoveNext()}
I even did debug the service from client to server and the result are comings but I'm unable to get any idea why this happens. I'm testing on my Windows Phone 8.1 device with Wifi connection on.
Please help.
After further analyzing the http response in the exception dialog of VS. The actual error was 404 Not found. Then further debugging the issue it turned out to be that the application url of Azure Mobile Services should be https and not http
It would be good if the product team can find this out and give proper error message, which can save lot of time and devs doesn't get frustrated.
I had the same issue, but my problem was, that the UpdateAsync tries to get an item which doesn't exists in my database.
I have solved this issue with the SyncHandler and the following catch block:
catch (Exception ex) when (ex is MobileServiceInvalidOperationException)
{
var error = (MobileServiceInvalidOperationException)ex;
if (operation.Kind != MobileServiceTableOperationKind.Update)
{
continue;
}
if (error.Response.StatusCode != HttpStatusCode.NotFound)
{
continue;
}
var azureMobileBackend = new AzureMobileBackend();
var table = await azureMobileBackend.GetSyncTable<...>();
await table.PurgeAsync(null, table.Where(x => x == operation.Item.ToObject<...>()), new CancellationToken());
}
With my code, I check, whether Mobile Apps tries to update an object which doesn't exist in the database and if, delete it in the local database. I'm not sure, whether this is the best way, but it is okay for my case.
I think the problem cause, if the application is shutting down, while the UpdateAsync method is working.