Errors when running any stateful service on local cluster - azure

Question

Running Service Fabric 4.4.87.9494 on Windows 8.1
I can successfully deploy and run a stateless service or a stateful or stateless actor on my local cluster, but when I try to deploy a stateful service, I get the following error in the Service Fabric Explorer:
Event Warning Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open. Error = System.IO.FileNotFoundException (-2147024894)
The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
at System.Fabric.Data.Log.Interop.NativeLog.IKPhysicalLogManager.EndOpenLogContainer(IFabricAsyncOperationContext Context, IKPhysicalLogContainer& Result)
at System.Fabric.Data.Log.Interop.PhysicalLogManager.<OpenLogContainerAsync>b__5(IFabricAsyncOperationContext Context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.Data.Log.LogManager.<OnOpenPhysicalLogAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<OpenPhysicalLogAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.<CreateLogFileAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.LogManager.<OpenAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.ServiceFabric.Replicator.LoggingReplicator.<OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.<OpenAsync>d__2b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.TransactionalReplicator.<OpenAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.StatefulServiceReplica.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceBase.<System.Fabric.IStatefulServiceReplica.OpenAsync>d__0.MoveNext()
I see the same error whether I create a default stateful service with Visual Studio, or use any of the samples from the getting started repo, or add a stateful service to an existing app. The same thing happens regardless of whether I use Visual Studio to deploy it or run the Powershell commands. However, if I deploy the same application to Azure, all is well.
I've tried this on two other Windows 8.1 computers at work with the same results, however I tried it on my home machine (Windows 10) and did not see the problem. This suggests it might be something in the Group policy that's affecting things. I've looked at the resultant group policy and can't find anything that I would have thought could have caused such a problem. I've tried disabling the virus checker during deployment too.
I tried to use Process Monitor to figure out what file it's looking for but couldn't find anything obvious.
Any pointers on how to progress this would be very much appreciated. I want to get the team all up & running and I can't if we can't create stateful services!

Answer 1

This is likely a bug that was in version 4.4. Please upgrade to version 4.5 as this would have the fix needed.