I am working with Azure function app. I used a tool called the Durable Functions Monitor (see: https://github.com/microsoft/DurableFunctionsMonitor) to attempt to cancel an in-progress orchestrator function since I no longer wished for it to complete. To do this I found the instance in the Durable Functions Monitor UI (linked above) and first "terminated" the orchestration and then "purged" it. I did this a couple times.
Since then in the log stream on my function app I am seeing the following error message:
2022-12-22T23:38:44Z [Error] An error occurred while processing messages on XXXXXXXXXXXXX-workitems: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist.
---> Microsoft.WindowsAzure.Storage.StorageException: The specified blob does not exist.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken)
at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 156
at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId) in /_/src/DurableTask.AzureStorage/TimeoutHandler.cs:line 133
at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 136
Is this error related to my actions? If so:
For the present time, how can I stop this error from logging. What process is looking for this blob and how to do tell it to stop doing so
How can I safely terminate an orchestrator function in the future to prevent this state of affairs.
Bonus question:
I noticed that sometimes terminating an orchestrator does not seem to also terminate a long-ish (~ 10 min) running activity function. Is there a way to terminate an orchestration and any activities it has called using any available tool?
Related
I have an Azure build pipeline which uses a private build machine in our corporate network. As part of the pipeline a windows dll file and an exe file is created. Both files should be uploaded using a PublishBuildArtifacts#1 task:
- task:
inputs:
PathtoPublish: 'batchfiles/SSMD'
ArtifactName: 'ssmd-$(SSMD_VERSION).0.$(Build.BuildId)-$(GIT_VERSION)'
publishLocation: 'Container'
When uploading the windows dll file the following error occurs:
Fail to upload 'C:\agent-azure-v2.174.1\_work\1\s\batchfiles\SSMD\SSMD-21.0.0.106.dll' due to 'Forbidden'.
The upload of the exe file works fine.
Thanks for your help!
Detailed log information:
Starting: PublishBuildArtifacts
==============================================================================
Task : Publish build artifacts
Description : Publish build artifacts to Azure Pipelines or a Windows file share
Version : 1.158.3
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts
==============================================================================
Async Command Start: Upload Artifact
Uploading 1 files
Total file: 1 ---- Processed file: 0 (0%)
Total file: 1 ---- Processed file: 0 (0%)
Fail to upload 'C:\agent-azure-v2.174.1\_work\1\s\batchfiles\SSMD\SSMD-21.0.0.106.dll' due to 'Forbidden'.
Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Forbidden
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState)
at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Boolean uploadFirstChunk, Object userState, Boolean compressStream)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.UploadAsync(IAsyncCommandContext context, Int32 uploaderId, CancellationToken token)
Detail upload trace for file that fail to upload: ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll
Begin chunking upload file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll', chunk size '4194304 Bytes', total chunks '2'.
Attempt '1' for uploading chunk '1' of file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll'.
Generate new HttpRequest for uploading file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll', chunk '1' of '2'.
Start uploading file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll' to server, chunk '1'.
Chunk '1' attempt '1' of file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll' fail to send request to server. Error: Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Forbidden
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).
Backoff 8.74 seconds before attempt '2' chunk '1' of file 'ssmd-21.0.0.106-gae26b4b/SSMD-21.0.0.106.dll'.
[...]
Fail to upload windows dll due to 'Forbidden' (using PublishBuildArtifacts to Azure DevOps)
According to your description, uploading the exe file is successful but the dll file fails.
That means our task or pipeline is correct. Then combined with the error log:
Fail to upload 'C:\agent-azure-v2.174.1\_work\1\s\batchfiles\SSMD\SSMD-21.0.0.106.dll' due to 'Forbidden'.
So we can speculate there should be a proxy block .dll file of your private build machine or corporate network. You should check it.
To verify it, you could try to use hosted agent to verify this issue.
I am downloading a large amount of files >1000 from Azure Datalake V2 and I am continuously getting exception:
The SSL connection could not be established, see inner exception.
<--- Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
<--- An existing connection was forcibly closed by the remote host.
Stacktrace:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
The code:
var downloadTasks = job.Files.AsParallel().Select(x => Download(x));
await Task.WhenAll(downloadTasks);
private async Task Download(DownloadableFile file)
{
try
{
var options = new BlobRequestOptions
{
ParallelOperationThreadCount = 8,
DisableContentMD5Validation = true,
StoreBlobContentMD5 = false
};
var xzBlob = await _cloudBlobFileService.GetBlockBlobReference(file.FilePath);
await xzBlob.DownloadToFileAsync(file.LocalFilePath, FileMode.Create, null, options, null);
}
catch (Exception e)
{
_log.LogCritical(e, "Error downloading " + file.FilePath);
}
}
I have also added this:
ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;
ServicePointManager.Expect100Continue = false;
Using .Net core 3.1 and WindowsAzure.Storage 9.3.3
to program.cs main method in webjob
We used to have a blobstorage configured without datalake, but after switching to datalake this has appeared. It doesn't affect the application to much as skipped downloads are retried later. It would however be nice to know whats causing it.
You could start by trying the new storage SDK that hit GA in November, though I can't guarantee that would solve the issue. It is a complete rewrite
While its not possible to pinpoint exactly just from the error message, there are a couple things to look at:
Network errors. It is the by far most likely cause, though it is interesting that it works consistently with your old blob storage account. Increasing the timeout may lower the frequency of network errors, and retry logic will help overcome them.
Using unbounded parallelism is not recommended. ParallelOperationThreadCount is for uploads not downloads, so it isn't throttling the requests in this case. The default limit for server-side connections in .NET is 10, and it is recommended to increase this when using .NET Core, it is something to consider. If you are accessing the same blob or partition too many times concurrently you can start to bump into the concurrent connections limits in Storage.
I'm using a Webjob with the Windows Azure Storage SDK. When a new item shows up in a Queue, a method in my class is invoked. According to the SDK docs, if I take a TextWriter as a parameter to my method, the SDK will provide me with a TextWriter that I can write to which will show up in the Webjob's logging infrastructure. This makes it pretty easy to diagnose issues and troubleshoot things.
public async static void ProcessQueueMessage([QueueTrigger("queueName")]MyModelType model, TextWriter logger)
{
await logger.WriteLineAsync(string.Format("Processing Item {0}", model.SasUrl));
// Some work here
await logger.WriteLineAsync(string.Format("Done Processing Item {0}", model.SasUrl));
}
However, very frequently, within the body of my method, the TextWriter is being disposed of. I'm getting the following exception on the 2nd logger.WriteLineAsync:
System.ObjectDisposedException was unhandled by user code
HResult=-2146232798
Message=Cannot write to a closed TextWriter.
Source=mscorlib
ObjectName=""
StackTrace:
at System.IO.__Error.WriterClosed()
at System.IO.StringWriter.Write(Char[] buffer, Int32 index, Int32 count)
at System.IO.TextWriter.WriteLine(String value)
at System.IO.TextWriter.SyncTextWriter.WriteLine(String value)
at System.IO.TextWriter.SyncTextWriter.WriteLineAsync(String value)
at NameOfProject.Program.<ProcessQueueMessage>d__8.MoveNext() in c:\Dev\Path\To\Program.cs:line 173
InnerException:
I can't find others having this problem, so I can't imagine there is a bug in the SDK or webjobs infrastructure.
Is there a way to tell if the logger is disposed of ahead of the call?
Is there a way to create a new logger within my method that will participate in the WebJobs logging subsystems and UI?
That's because your method returns void. Try returning Task instead
I am intermittently facing the ErrorCode:SubStatus with Windows Azure and the App Fabric cache in my ASP.NET application. This brings my whole web application to a standstill until I reset it making Azure no longer viable.
I am only storing very small strings in Session state and only have a very small number of users. I can’t imagine that I could be over any of the usage quotas (at http://msdn.microsoft.com/en-us/library/gg602420.aspx#C_BKMK_FAQ8)
I would like to find out which quota if any I am exceeding and why. How can I find out if and why I am being throttled, or if there is any other issue that might be causing this.
Is there any way to find the Cache Size (I know this is in the Management Portal but it always reports ir over 95% lower than my 128MB limit), Transactions Per Hour, Bandwidth MB Per Hour and Concurrent Connections?
Stack trace:
Application_Error: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)Stack Trace: at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
at Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region, IMonitoringListener listener)
at Microsoft.ApplicationServer.Caching.DataCache.<>c_DisplayClass25.b_24()
at Microsoft.ApplicationServer.Caching.MonitoringListenerFactory.EmptyListener.Microsoft.ApplicationServer.Caching.IMonitoringListener.Listen[TResult](Func1 innerDelegate)
at Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value, TimeSpan timeout)
at Microsoft.Web.DistributedCache.DataCacheWrapper.Put(String key, Object value, TimeSpan timeout)
at Microsoft.Web.DistributedCache.DataCacheForwarderBase.<>c__DisplayClass10.<Put>b__f()
at Microsoft.Web.DistributedCache.DataCacheForwarderBase.<>c__DisplayClass2e1.b__2d()
at Microsoft.Web.DistributedCache.DataCacheRetryWrapper.PerformCacheOperation(Action action)
at Microsoft.Web.DistributedCache.DataCacheForwarderBase.PerformCacheOperation[TResult](Func`1 func)
at Microsoft.Web.DistributedCache.DataCacheForwarderBase.Put(String key, Object value, TimeSpan timeout)
at Microsoft.Web.DistributedCache.BlobBasedSessionStoreProvider.SetAndReleaseItemExclusive(HttpContextBase context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
at Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
at System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs)
at System.Web.SessionState.SessionStateModule.OnEndRequest(Object source, EventArgs eventArgs)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) on page
Take a look at the Windows Azure Service Dashboard. Go to the bottom, and under Status History, select AppFabric Caching. Look for periods of service degradation or interruption on the days you saw this error (including today), for your given data center.
Hope that helps...
There is an object limit as 8MB which causes that error message when you are trying to post something more than that
I am trying to do a server side doc/docx to pdf conversion using Word Automation Services. I am using the basic sample from MSDN, Link: http://msdn.microsoft.com/en-us/library/ff742315.aspx and encounter an obscure error every time ConversionJob's Start() method is called. Here's the exception;
Source: mscorlib
Message: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.
InnerException: System.ServiceModel.FaultException: An error occurred when verifying security for the message.
StackTrace: Server stack trace: at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst) at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo) at Microsoft.SharePoint.SPSecurityContext.<>c_DisplayClass7.b_6() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurityContext.GetProcessSecurityTokenForServiceContext() at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForServiceContext(Uri contextUri) at Microsoft.SharePoint.SPChannelFactoryOperations.InternalCreateChannelActingAsLoggedOnUser[TChannel](ChannelFactory1 factory, EndpointAddress address, Uri via) at Microsoft.SharePoint.SPChannelFactoryOperations.CreateChannelActingAsLoggedOnUser[TChannel](ChannelFactory1 factory, EndpointAddress address) at Microsoft.Office.Word.Server.Service.ConfigChannelFactory1.CreateChannel() at Microsoft.Office.Word.Server.Service.WordServiceApplicationProxy.AddJob(Ucid ucid, Nullable1 subscriptionId, SPUserToken userToken, ConversionJobSettings settings, String name) at Microsoft.Office.Word.Server.Conversions.ConversionJob.Start() at VisualWebPartProject1.VisualWebPart1.VisualWebPart1UserControl.Page_Load(Object sender, EventArgs e)
I tried the same code inside one of our sharepoint sites, still got the same error. Trying this inside a Wcf Service has same result. I made a visual web part for this and same error. Changing user accounts or RunWithElevatedPrivileges didn't help either.
Since the sample requires Microsoft.Sharepoint and Microsoft.Office.Word.Server.Conversions namespaces, I have to run this code on some server having sharepoint installed. So IMHO, the time skew issue is not related as the code is ran over the same server where Word Automation Services are present. I expect some configuration problems on the sharepoint server side, but I don’t know what to check.
Let me know if any other details are required, thanks.
Without having more specifics around the code you're running it's hard to say. Take a look at the overall Word Automation Services content in the SDK.
Also, check out the topics under "Configuring Word Automation Services for Development" in the SDK in particular.