We have an Umbraco website (version 7.5.11) hosted on Azure Web Apps.
We are experiencing the following exception intermittently (3 times within the past 3 weeks). Once the exception occurs it brings the website down until we republish the home node in Umbraco. At all other times the website is working as expected, including retrieving image files from the server.
Exception type: IOException
Exception message: An unexpected network error occurred. at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func1 getCacheItem, Nullable1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, CacheDependency dependency) at Umbraco.Core.Cache.HttpRuntimeCacheProvider.GetCacheItem(String cacheKey, Func1 getCacheItem, Nullable1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles) at Umbraco.Core.Cache.DeepCloneRuntimeCacheProvider.GetCacheItem(String cacheKey, Func1 getCacheItem, Nullable1 timeout, Boolean isSliding, CacheItemPriority priority, CacheItemRemovedCallback removedCallback, String[] dependentFiles) at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetCacheValues(Int32 id, Func2 func) at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia(Int32 id) at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetById(UmbracoContext umbracoContext, Boolean preview, Int32 nodeId) at Umbraco.Web.PublishedCache.ContextualPublishedCache1.GetById(Boolean preview, Int32 contentId) at Umbraco.Web.PublishedContentQuery.DocumentById(Int32 id, ContextualPublishedCache cache, Object ifNotFound) at Umbraco.Web.PublishedContentQuery.Media(Int32 id) at Umbraco.Web.UmbracoHelper.Media(String id)
The media file exists, and republishing the home node brought the site back online.
At the time of the exception, no code changes were deployed and no pages were updated / published within Umbraco.
Has anyone experienced something similar, or any ideas what the root cause is?
According to the source code of PublishedMediaCache.cs, the exception is often caused by following issue.
Examine index is corrupted.
Here is a thread on umbraco forum which related to your issue.
Examine corruption issues
And here is the solution for this issue from #Shannon Deminick.
If you are using Azure web apps and are NOT auto-scaling, you should use these settings:
useTempStorage="Sync"
use this feature to store local index files: http://issues.umbraco.org/issue/U4-7614
Remove the {machinename} token from your index path
RebuildOnAppStart="true" - since this should only happen one time
If you are using Azure web apps and are load balancing w/ auto-scaling your front-end workers then:
useTempStorage="Sync"
use this feature to store local index files: http://issues.umbraco.org/issue/U4-7614
You must have the {machinename} token from your index path
RebuildOnAppStart="true" - so that when new sites come online, their indexes are built
... yes in some cases this might not be ideal, please see: https://our.umbraco.org/forum/extending-umbraco-and-using-the-api/74731-examine-corruption-issues#comment-244293
Related
Today I've adjusted the App Service Log settings for my Azure App Serivce: a .NET Core 2.1 Web API. Soon after I changed the Application Logging (file system) setting from Information to Warning, my application stopped responding. I immediately kicked of a .NET Trace using the Diagnose and solve problems feature.
This Trace report showed the following:
system.private.corelib!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync
microsoft.aspnetcore.hosting!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1+<>c[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].<.cctor>b__9_0
system.private.corelib!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].MoveNext
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
Same time, the Azure Blob log showed these lines:
2019-07-06 14:04:06.476 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6Q", Request id "0HLO233NHLK6Q:000003BF": An unhandled exception was thrown by the application.
System.OverflowException: Arithmetic operation resulted in an overflow.
at System.IO.StreamWriter.Write(String value)
at Microsoft.Extensions.RazorViews.BaseView.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2019-07-06 14:04:19.245 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6T", Request id "0HLO233NHLK6T:000000EF": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: chars
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
These stacktraces don't show any of my own code, they seem framework related.
Is there a message or a hint in these error stack traces that I can use to prevent these situations?
Many thanks in advance
I know it's too late but maybe my answer will help someone else.
I received the same exceptions as you. But it was not Azure App Service but IIS. The problem was in incorrect configuration somewhere in Startup.cs. You webhost tried to show error page but at the same time App Service (or IIS in my case) tried to shut down or restart it.
This thread helped me a lot https://github.com/aspnet/KestrelHttpServer/issues/1737
I am having issues with the preview of our sharepoint office documents, it is not working, i have checked the OWA Server and the owa services are running . I also checked the binding using Get-SPWOPIBinding and it returned the bindings to the owa server. I have also set the zone to external-https using
Set-SPWOPIZone –zone “external-https”
because i saw that the zone was binded to external-https in my sharepoint servers. but still the issue is still there.not sure why and I don't understand the ULS logs either. these are the ULS logs for the correlation id of one of the documents i was trying to preview.
Unexpected Exception in
SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage
'DistributedLogonTokenCache' - Exception 'System.ArgumentException:
Max connections value should be in the range 1 to 100. Parameter
name: value at
Microsoft.ApplicationServer.Caching.DataCacheFactoryConfiguration.set_MaxConnectionsToServer(Int32
value) at
Microsoft.SharePoint.DistributedCaching.SPDistributedCachePointerWrapper.InitializeDataCacheFactory()'.
Token Cache: Failed to initialize SPDistributedSecurityTokenCache
Exception: 'System.ArgumentException: Max connections value should be
in the range 1 to 100. Parameter name: value at
Microsoft.ApplicationServer.Caching.DataCacheFactoryConfiguration.set_MaxConnectionsToServer(Int32
value) at
Microsoft.SharePoint.DistributedCaching.SPDistributedCachePointerWrapper.InitializeDataCacheFactory()
at
Microsoft.SharePoint.DistributedCaching.SPDistributedCache..ctor(String
name, TimeSpan timeToLive, SPDistributedCacheContainerType
containerType, Boolean encryptData) at
Microsoft.SharePoint.IdentityModel.SPDistributedSecurityTokenCache..ctor(String
name, TimeSpan timeToLive, SPDistributedCacheContainerType
containerType, Boolean encrptyData, TimeSpan
minimumTokenExpirationWindow) at
Microsoft.SharePoint.IdentityModel.SPDistributedSecurityTokenCacheInitializer.Init(Object
state)'.
i'm stuck and i need assistance
Also check in Event Viewer and see if you can get an error code(s). Looks like AppFabric (Distributed Cache) is having an issue, but it may be a symptom of something else, sometimes only one of several issues. There is a fine tuning script available here, but make sure you know what you're doing. Try setting the Max Connections to 1, test it, and then bump it up to 10 and see if you have any issues. Might want to do this during a maintenance window as a restart may be required.
After a deploy to our Azure Web App, we are getting 500 timeouts on any request to the service:
500 - The request timed out.
The web server failed to respond within the specified time.
This has come out of the blue and we cannot determine what's causing it. It seems to take around 230s consistently to time out.
I've enabled all the diagnostic logs in the portal:
But I honestly don't quite know what to look for in the logs. I've scoured through all the files in the following folders but nothing jumps out.
How can I troubleshoot this problem?
The trick to get debug messages is to set stdoutLogFile="D:\home\LogFiles\stdout.log" in your config, instead of the ..\logs path that you had. After changing that, you get an error file under D:\home\LogFiles. Here is the error you get:
Application startup exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentNullException: SMTP server password cannot be null or empty.
Parameter name: smtpPassword
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLogger..ctor(String recipient, String sender, String smtpUsername, String smtpPassword, String smtpHost, Int32 smtpPort, String environmentName, LogLevel minimumLevel)
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLoggerProvider.CreateLogger(String name)
at Microsoft.Extensions.Logging.Logger.AddProvider(ILoggerProvider provider)
at Microsoft.Extensions.Logging.LoggerFactory.AddProvider(ILoggerProvider provider)
at TransitApi.Api.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
So some kind of issue setting up the mail server. That causes the process to crash, and that it behaves poorly.
But I highly suggest that you upgrade to Core RC2, as RC1 is quite obsolete.
We have some solution that we built against a MOSS farm one of which includes a timer job. This job has been working just fine for months. Recently the administrator enlisted another server into the farm, and our timer job automatically started running on this new machine. As soon as this switch happened our timer job started yielding the error below (found this in the SP logs).
At first I thought it was a rights issue, but the timer service on the machine where it worked before and the new one are running under the same domain account. It seems to be failing while looping the site list in a site collection, on just one of the sites/webs (code snippet below). I know that this domain account has access to this because it works on the other box under same account. Does anyone have any ideas on why this cryptic error is occurring? Or if any special procedure needs to be done on this new machine to ensure it has proper ACL's for all databases in the MOSS farm?
Code:
public static void Main(string[] args)
{
SPSecurity.RunWithElevatedPrivileges(delegate() { setInputParameters(); });
}
private static void setInputParameters()
{
SPFarm farm = SPFarm.Local;
SPWebService service = farm.Services.GetValue<SPWebService>("");
foreach (SPWebApplication webApp in service.WebApplications)
{
foreach (SPSite siteCollection in webApp.Sites)
{
using(siteCollection)
{
siteCollection.CatchAccessDeniedException = false;
try
{
/* Here is the line that it fails on */
foreach (SPWeb web in siteCollection.AllWebs)
Exception:
The Execute method of job definition LMSDataImport (ID 4b37b285-ef8a-407c-8652-391639449790) threw an exception.
More information is included below.
The type initializer for 'Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1' threw an exception.
Exception stack trace:
at Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1.get_BackingList()
at Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1.GetEnumerator()
at Microsoft.SharePoint.Administration.SPAlternateUrlCollectionManager.LookupAlternateUrl(Uri canonicalRequestUri)
at Microsoft.SharePoint.Administration.SPAlternateUrl.LookupCore(Uri uri, SPFarm farm)
at Microsoft.SharePoint.Administration.SPWebApplication.Lookup(SPFarm farm, Uri requestUri, Boolean fallbackToHttpContext, SPAlternateUrl& alternateUrl, SiteMapInfo& hostHeaderSiteInfo, Boolean& lookupRequiredContext)
at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, SPUserToken userToken)
at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite)
at Microsoft.SharePoint.Administration.SPSiteCollection.get_Item(String strSiteName)
at Microsoft.SharePoint.Administration.SPSiteCollection.get_Item(Int32 index)
at Microsoft.SharePoint.Administration.SPSiteCollection.ItemAtIndex(Int32 iIndex)
at Microsoft.SharePoint.SPBaseCollection.SPEnumerator.System.Collections.IEnumerator.get_Current()
at LMSDataImporter.setInputParameters()
at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state)
at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2()
at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
at Axian.AxianCalendar.LMSDataImporter.Main(String[] args)
at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result)
Check the DLLs for SharePoint, do all exists and all are the same version? Try putting a catch for the TypeInitializationException, and see what is wrong inside that exception.
It's not a solution, but as a workaround in the interim, I think my suggestion to one of your other questions here:
How do you instruct a SharePoint Farm to run a Timer Job on a specific server?
will keep you running while you investigate further.
Check the NLB (Network Load Balancing) configuration. Most of the time SharePoint and applications integrated to it fails when NLB changes its state. There is a patch available to solve this problem. Just a suggestion. Not sure if this is the reason. But I have faced a similar issue and the root cause was an NLB bug
A Type init exception just means an exception occurred in the .ctor of the class (as you probably know). The real exception should be in the InnerException property - can you get your hands on this? Likely it's stemming from the database alright, from Microsoft.SharePoint.Administration.SPPersistedChildCollection InitializeFromDatabse method.
Can you look into the sharepoint logs (on that errant server) for information about the database error, it will be there. Reading logs are a pain, but not if you install the ULS Log Viewer feature from http://www.codeplex.com/features
Since the stacktrace has SPAlternateUrl tinkering furhter up the stack, perhaps your zones are misconfigured (and do not include a mapping for this new server's machine name) - granted, it shouldn't fail this bad, but what can you do.
You can filter the ULS logs by source.
-Oisin
We’re getting the following error message when we click on “Search Settings” for a Shared Services Provider: “Authentication failed because the remote party has closed the transport stream.”
This is a new server environment with two web front ends, one database server, and one index server, all running Windows 2003 x64.
Does anyone have any thoughts related to if this could be related to 64-bit, or what could be causing the error.
Here are the full details from ULS:
09/17/2008 16:30:34.13 w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86x4 High Configuring the Search Application web service Url to 'https://mushni-sptwb04q:56738/Shared%20Services%20Portal/Search/SearchAdmin.asmx'.
09/17/2008 16:30:34.14 w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High Exception caught in Search Admin web-service proxy (client). System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream. at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Threading.ExecutionContext.runTryCode(Object userData) at System.Runtime.Co...
09/17/2008 16:30:34.14* w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High ...mpilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size) at System.Net.ConnectStream.WriteHeaders(Boolean async) --- End of inner exception stack trace --- at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request) at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request) at System.Web.Services.Protocols.SoapHt...
09/17/2008 16:30:34.14* w3wp.exe (0x0E84) 0x030C Search Server Common MS Search Administration 86ze High ...tpClientProtocol.Invoke(String methodName, Object[] parameters) at Microsoft.Office.Server.Search.Administration.SearchWebServiceProxy.RunWithSoapExceptionHandling[T](String methodName, Object[] parameters)
I guess you find this exception in the index server, right?
Are you able to browse to 'https://mushni-sptwb04q:56738/Shared%20Services%20Portal/Search/SearchAdmin.asmx' from the index server?
It seems like SSL is not properly provisioned on the front-end servers. This might solve your issue:
Remove the SSL certificate of the front-end servers
Remove the index server from the farm
Move the search and index roles to one of the front-ends
Join the index server back to the farm
Add the index/search roles to the index server
Apply the SSL certificate (you can generate them using SelfSSL) to both front-ends
Be careful with SelfSSL, its better to use Use SSLDiag. SelfSSL has a bug where if you use it to assign certificates to multiple sites on the same box, only the last site will work. You can run SslDiag from the command line like so:
ssldiag /selfssl /V:999 /N:CN=<hostname> /S:<siteId>
Use metabase explorer to find the side it.
Could be an SSL issue.
Do have a look into profiles settings, do you get any error when accessing to the User Profiles settings for that same SSP?
I'm having the same problem. The "Office Server Web Services" (henceforth OSWS) site is available through HTTP on my app server, but not via HTTPS. It doesn't matter where I try to hit the HTTPS URL from, it just flat-out fails (read: no HTTP error code).
However, I have come up with some more information. When the app server was joined to the farm, it gave OSWS a different site identifier than exists in the rest of the farm.
I tried changing the site identifier, but that didn't work. I've also tried installing the IIS diagnostics toolkit. That pointed me towards the certificate that MOSS installed when the machine was joined to the farm. The line of interest is this one:
#WARNING: AcquireCredentialsHandle failed with error -2146893043(0x8009030d)
Unfortunately, it looks like Microsoft has embedded some information in the certificate that would prevent me from using SelfSSL or similar tools. Here's the subject (suitably scrubbed):
CN={hostname},L=951338967,OU=SharePoint,O=Microsoft
The "L" parameter matches the original (and incorrect) site identifier that the site was given and not the one that matches the rest of the farm.
My next step is to see if I can generate something that looks appropriate and install it with winhttpcertcfg.exe
We are also running x64 windows and moss 2007 with .net 3.5 sp1,same issues. I suspect this is the culprit.
To resolve this issue download the IIS6 resource kit and run the following command
Selfssl /s:(IIS ID of the Office Server Web Services site) /v:9999
Cheers,
-Ivan