We've been having a problem with Lucene indexing for a while already.
Basically quite often when we try to publish a content the indexing just throws an error like this:
2020-01-13 22:22:38,068 [P36840/D2/TLucene Merge Thread #0] ERROR Umbraco.Core.UmbracoApplicationBase - Unhandled exception in AppDomain (terminating)
Lucene.Net.Index.MergePolicy+MergeException: Exception of type 'Lucene.Net.Index.MergePolicy+MergeException' was thrown. ---> System.IO.IOException: read past EOF
at Lucene.Net.Index.IndexWriter.HandleMergeException(Exception t, OneMerge merge)
at Lucene.Net.Index.IndexWriter.Merge(OneMerge merge)
at Lucene.Net.Index.ConcurrentMergeScheduler.DoMerge(OneMerge merge)
at Lucene.Net.Index.ConcurrentMergeScheduler.MergeThread.Run()
--- End of inner exception stack trace ---
at Lucene.Net.Index.ConcurrentMergeScheduler.HandleMergeException(Exception exc)
at Lucene.Net.Index.ConcurrentMergeScheduler.MergeThread.Run()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
We get Secure Connection Failed on the public site as the result, the content is being saved but not included in cache so it does not display on the website.
Then we are unable to access Examine Management dashboard to restart indexes.
The error sometime slooks like this too:
2019-12-30 14:49:14,895 [P18524/D2/T28] ERROR Umbraco.Core.UmbracoApplicationBase - Unhandled exception in AppDomain (terminating)
System.IO.IOException: Cannot overwrite: D:\Inetpub\vhosts\mysite.com\httpdocs\App_Data\TEMP\ExamineIndexes\Internal\Index\_pe.cfs
at Lucene.Net.Store.FSDirectory.InitOutput(String name)
at Lucene.Net.Store.SimpleFSDirectory.CreateOutput(String name)
at Lucene.Net.Index.CompoundFileWriter.Close()
at Lucene.Net.Index.DocumentsWriter.CreateCompoundFile(String segment)
at Lucene.Net.Index.IndexWriter.DoFlushInternal(Boolean flushDocStores, Boolean flushDeletes)
at Lucene.Net.Index.IndexWriter.DoFlush(Boolean flushDocStores, Boolean flushDeletes)
at Lucene.Net.Index.IndexWriter.PrepareCommit(IDictionary`2 commitUserData, Boolean internal_Renamed)
at Lucene.Net.Index.IndexWriter.Commit(IDictionary`2 commitUserData)
at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.TimerRelease() in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1711
at Examine.LuceneEngine.Providers.LuceneIndexer.IndexCommiter.<ScheduleCommit>b__7_0(Object _) in C:\projects\examine-qvx04\src\Examine\LuceneEngine\Providers\LuceneIndexer.cs:line 1663
at System.Threading.TimerQueueTimer.CallCallbackInContext(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
In such case publishing a content becomes practically ineffective.
Any idea what's the cause? And how to fix it? I can see the index becomes corrupted but why?
I have checked several similar issues like these:
Umbraco Indexing Error Filling Up Logs
https://issues.apache.org/jira/browse/LUCENENET-527 but none of them is our case - I can't see any 0kB segment files.
EDIT
The error crashes the whole website causing Secure Connection Failed error and it happens when trying to perform a save on a content (no matter via CMS or the front site).
Related
I recently discovered that there are error logs created every 10 mins with the following content. I can't figure out why and where they come from.
AppInisghts is deactivated for the webapp...
New Log File(2022/04/17/17/918a59-5808.applicationLog.csv) has been created
date,level,applicationName,instanceId,eventTickCount,eventId,pid,tid,message,activityId
2022-04-17T17:08:35,Information,boxplanner,918a59,637858121159061901,0,5808,56,Automatically redeploying web job.,
2022-04-17T17:08:36,Information,boxplanner,918a59,637858121163988746,0,5808,56,Checking NuGet for latest agent.,
2022-04-17T17:08:36,Verbose,boxplanner,918a59,637858121167270466,0,5808,-1,"System.InvalidOperationException: An error occurred while loading packages from 'https://packages.nuget.org/api/v2': The underlying connection was closed: An unexpected error occurred on a send. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.TlsStream.CallProcessAuthentication(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at NuGet.RequestHelper.GetResponse()
at NuGet.HttpClient.GetResponse()
at NuGet.RedirectedHttpClient.GetResponseUri(HttpClient client)
at NuGet.RedirectedHttpClient.EnsureClient()
at System.Lazy`1.CreateValue()
at System.Lazy`1.LazyInitValue()
at System.Lazy`1.get_Value()
at NuGet.MemoryCache.GetOrAdd[T](Object cacheKey, Func`1 factory, TimeSpan expiration, Boolean absoluteExpiration)
at NuGet.RedirectedHttpClient.get_CachedClient()
at NuGet.RedirectedHttpClient.get_Uri()
at NuGet.DataServicePackageRepository.get_Context()
at NuGet.DataServicePackageRepository.FindPackagesById(String packageId)
--- End of inner exception stack trace ---
at NuGet.DataServicePackageRepository.FindPackagesById(String packageId)
at NuGet.PackageRepositoryExtensions.FindPackagesById(IPackageRepository repository, String packageId)
at Microsoft.ApplicationInsights.ExtensionManager.Models.AgentModel.GetLatestAgentFromNuGet()
at Microsoft.ApplicationInsights.ExtensionManager.Models.AgentModel.get_LatestNugetVersion()
at Microsoft.ApplicationInsights.ExtensionManager.Models.AgentModel.DeployWebJob(String trigger, String nugetVersion)",
AppInsights is deactivated for the web app.
The possibility of getting the telemetry information after deactivating the Application Insights from the Web App.
Maybe in your Application, you may hardcode the configuration of Application Insights in the Web APP.
Make sure if you use custom telemetry in your application, it must contain the Application Insights configuration.
To Avoid getting the telemetry information from a web app you can try to remove the hardcoded Application Insights configuration
Refer here to remove application insights in multiple ways.
I have a simple asp.net core web app (v2.1), that I deployed to a B1 (and I tried B2) Azure App Service on Linux. When I call dbContext.SaveChanges(), after adding one very simple entity, the request takes about 30 seconds before throwing the following error:
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
Here is the code. _dbContext is injected with scoped lifetime.
public async Task<IActionResult> SignIn([Bind("Email,Password,RedirectUrl")] SignInModel model) {
if (ModelState.IsValid) {
var user = _dbContext.Users.Include(u => u.Claims).FirstOrDefault(u => u.UserName == model.Email);
...
user.LastLogin = DateTimeOffset.Now;
await _dbContext.SaveChangesAsync();
...
return Redirect(String.IsNullOrWhiteSpace(model.RedirectUrl) ? "/" : model.RedirectUrl);
}
else {
return View(model);
}
}
During the 30 seconds, I see, via SSH, that a journal file exists beside by SQLite DB file. It's eventually deleted.
UPDATE: Here are the logs. You can see, that after a single update call, a lock exception is thrown exactly 30 seconds later. 30 seconds is the SQL command timeout. I'm watching the file system using a remote SSH shell, and the journal file is there for ~30 seconds. It's like the combination of the network share used by the app service, and the SQLite file locking logic, is broken or very slow.
2018-12-20T15:06:27.660624755Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
2018-12-20T15:06:27.660656156Z Request starting HTTP/1.1 POST http://insecuresite.azurewebsites.net/Account/SignIn application/x-www-form-urlencoded 56
2018-12-20T15:06:27.660797960Z [15:06:27 DBG] InsecureSite.Middleware.MyCookieAuthHandler:Constructor called.
2018-12-20T15:06:27.660875561Z [15:06:27 DBG] InsecureSite.Middleware.MyCookieAuthHandler:Constructor called.
2018-12-20T15:06:27.660885462Z [15:06:27 DBG] InsecureSite.Middleware.MyCookieAuthHandler:HandleAuthenticateAsync called.
2018-12-20T15:06:27.660890662Z [15:06:27 DBG] InsecureSite.Middleware.MyCookieAuthHandler:HandleAuthenticateAsync called.
2018-12-20T15:06:27.661837484Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
2018-12-20T15:06:27.661856585Z Route matched with {action = "SignIn", controller = "Account"}. Executing action InsecureSite.Controllers.AccountController.SignIn (InsecureSite)
2018-12-20T15:06:27.662465200Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
2018-12-20T15:06:27.662478400Z Executing action method InsecureSite.Controllers.AccountController.SignIn (InsecureSite) with arguments (InsecureSite.Models.SignInModel) - Validation state: Valid
2018-12-20T15:06:27.667736726Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Infrastructure[10403]
2018-12-20T15:06:27.667751427Z Entity Framework Core 2.1.4-rtm-31024 initialized 'AppDbContext' using provider 'Microsoft.EntityFrameworkCore.Sqlite' with options: None
2018-12-20T15:06:27.716864407Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Command[20101]
2018-12-20T15:06:27.716886507Z Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
2018-12-20T15:06:27.716892507Z PRAGMA foreign_keys=ON;
2018-12-20T15:06:27.776374136Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Command[20101]
2018-12-20T15:06:27.776410837Z Executed DbCommand (59ms) [Parameters=[#__model_Email_0='?' (Size = 18)], CommandType='Text', CommandTimeout='30']
2018-12-20T15:06:27.776514640Z SELECT "u"."UserId", "u"."FirstName", "u"."LastLogin", "u"."LastName", "u"."PasswordHash", "u"."UserName"
2018-12-20T15:06:27.776526140Z FROM "Users" AS "u"
2018-12-20T15:06:27.776531140Z WHERE "u"."UserName" = #__model_Email_0
2018-12-20T15:06:27.776536040Z ORDER BY "u"."UserId"
2018-12-20T15:06:27.776540740Z LIMIT 1
2018-12-20T15:06:27.778553489Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Command[20101]
2018-12-20T15:06:27.778567689Z Executed DbCommand (1ms) [Parameters=[#__model_Email_0='?' (Size = 18)], CommandType='Text', CommandTimeout='30']
2018-12-20T15:06:27.778840096Z SELECT "u.Claims"."UserClaimId", "u.Claims"."Claim", "u.Claims"."UserId", "u.Claims"."Value"
2018-12-20T15:06:27.778852696Z FROM "UserClaims" AS "u.Claims"
2018-12-20T15:06:27.778857796Z INNER JOIN (
2018-12-20T15:06:27.778862596Z SELECT "u0"."UserId"
2018-12-20T15:06:27.778869696Z FROM "Users" AS "u0"
2018-12-20T15:06:27.778874396Z WHERE "u0"."UserName" = #__model_Email_0
2018-12-20T15:06:27.778879897Z ORDER BY "u0"."UserId"
2018-12-20T15:06:27.780228429Z LIMIT 1
2018-12-20T15:06:27.780242129Z ) AS "t" ON "u.Claims"."UserId" = "t"."UserId"
2018-12-20T15:06:27.780247829Z ORDER BY "t"."UserId"
2018-12-20T15:06:27.789636555Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Command[20101]
2018-12-20T15:06:27.789651955Z Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
2018-12-20T15:06:27.789657656Z PRAGMA foreign_keys=ON;
2018-12-20T15:06:27.794111763Z [40m[32minfo[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Command[20101]
2018-12-20T15:06:27.794126763Z Executed DbCommand (4ms) [Parameters=[#p1='?', #p0='?'], CommandType='Text', CommandTimeout='30']
2018-12-20T15:06:27.794132363Z UPDATE "Users" SET "LastLogin" = #p0
2018-12-20T15:06:27.794280267Z WHERE "UserId" = #p1;
2018-12-20T15:06:27.794298667Z SELECT changes();
2018-12-20T15:06:57.833069471Z [41m[30mfail[39m[22m[49m: Microsoft.EntityFrameworkCore.Database.Transaction[20205]
2018-12-20T15:06:57.833107571Z An error occurred using a transaction.
2018-12-20T15:06:57.833113572Z Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
2018-12-20T15:06:57.833118772Z at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
2018-12-20T15:06:57.833123772Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
2018-12-20T15:06:57.833128672Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
2018-12-20T15:06:57.833133672Z at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText)
2018-12-20T15:06:57.833138672Z at Microsoft.Data.Sqlite.SqliteTransaction.Commit()
2018-12-20T15:06:57.833143372Z at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Commit()
2018-12-20T15:06:57.853805669Z [41m[30mfail[39m[22m[49m: Microsoft.EntityFrameworkCore.Update[10000]
2018-12-20T15:06:57.853833569Z An exception occurred in the database while saving changes for context type 'InsecureSite.Data.AppDbContext'.
2018-12-20T15:06:57.853928072Z Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
2018-12-20T15:06:57.853938272Z at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
2018-12-20T15:06:57.853943572Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
2018-12-20T15:06:57.854041474Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
2018-12-20T15:06:57.854051475Z at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText)
2018-12-20T15:06:57.854056675Z at Microsoft.Data.Sqlite.SqliteTransaction.Commit()
2018-12-20T15:06:57.854137377Z at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Commit()
2018-12-20T15:06:57.854146577Z at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(DbContext _, ValueTuple`2 parameters, CancellationToken cancellationToken)
2018-12-20T15:06:57.854208178Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IReadOnlyList`1 entriesToSave, CancellationToken cancellationToken)
2018-12-20T15:06:57.854283180Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.854292080Z at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.854299081Z Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
2018-12-20T15:06:57.854366282Z at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
2018-12-20T15:06:57.854384483Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
2018-12-20T15:06:57.854389683Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
2018-12-20T15:06:57.854455384Z at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText)
2018-12-20T15:06:57.854463685Z at Microsoft.Data.Sqlite.SqliteTransaction.Commit()
2018-12-20T15:06:57.854468185Z at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Commit()
2018-12-20T15:06:57.854529686Z at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(DbContext _, ValueTuple`2 parameters, CancellationToken cancellationToken)
2018-12-20T15:06:57.854652489Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IReadOnlyList`1 entriesToSave, CancellationToken cancellationToken)
2018-12-20T15:06:57.854673890Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.854748391Z at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.858109772Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
2018-12-20T15:06:57.858123973Z Executed action InsecureSite.Controllers.AccountController.SignIn (InsecureSite) in 30193.6715ms
2018-12-20T15:06:57.860885139Z [41m[30mfail[39m[22m[49m: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
2018-12-20T15:06:57.860899939Z An unhandled exception has occurred while executing the request.
2018-12-20T15:06:57.860905239Z Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
2018-12-20T15:06:57.861009242Z at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
2018-12-20T15:06:57.861018942Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
2018-12-20T15:06:57.861023842Z at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
2018-12-20T15:06:57.861120545Z at Microsoft.Data.Sqlite.SqliteConnectionExtensions.ExecuteNonQuery(SqliteConnection connection, String commandText)
2018-12-20T15:06:57.861130145Z at Microsoft.Data.Sqlite.SqliteTransaction.Commit()
2018-12-20T15:06:57.861134745Z at Microsoft.EntityFrameworkCore.Storage.RelationalTransaction.Commit()
2018-12-20T15:06:57.861237547Z at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(DbContext _, ValueTuple`2 parameters, CancellationToken cancellationToken)
2018-12-20T15:06:57.861311249Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IReadOnlyList`1 entriesToSave, CancellationToken cancellationToken)
2018-12-20T15:06:57.861320149Z at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.861392851Z at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
2018-12-20T15:06:57.861401851Z at InsecureSite.Controllers.AccountController.SignIn(SignInModel model) in /home/site/repository/InsecureSite/Controllers/AccountController.cs:line 57
2018-12-20T15:06:57.861463253Z at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
2018-12-20T15:06:57.861472053Z at System.Threading.Tasks.ValueTask`1.get_Result()
2018-12-20T15:06:57.861541855Z at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
2018-12-20T15:06:57.861550055Z at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
2018-12-20T15:06:57.861554655Z at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
2018-12-20T15:06:57.861629257Z at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
2018-12-20T15:06:57.861639057Z at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
2018-12-20T15:06:57.861718659Z at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
2018-12-20T15:06:57.861727059Z at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
2018-12-20T15:06:57.861791861Z at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
2018-12-20T15:06:57.861800861Z at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
2018-12-20T15:06:57.861805561Z at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
2018-12-20T15:06:57.861810161Z at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
2018-12-20T15:06:57.861880363Z at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
2018-12-20T15:06:57.861888363Z at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
2018-12-20T15:06:57.861948164Z at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
2018-12-20T15:06:57.862056667Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
2018-12-20T15:06:57.862066767Z Route matched with {action = "Error", controller = "Home"}. Executing action InsecureSite.Controllers.HomeController.Error (InsecureSite)
2018-12-20T15:06:57.867899207Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[1]
2018-12-20T15:06:57.867914108Z Executing action method InsecureSite.Controllers.HomeController.Error (InsecureSite) - Validation state: Valid
2018-12-20T15:06:57.868025910Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
2018-12-20T15:06:57.868045011Z Executed action method InsecureSite.Controllers.HomeController.Error (InsecureSite), returned result Microsoft.AspNetCore.Mvc.ViewResult in 0.0771ms.
2018-12-20T15:06:57.868147613Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor[1]
2018-12-20T15:06:57.868157914Z Executing ViewResult, running view Error.
2018-12-20T15:06:57.869182938Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor[4]
2018-12-20T15:06:57.869196139Z Executed ViewResult - view Error executed in 7.5623ms.
2018-12-20T15:06:57.869201439Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker[2]
2018-12-20T15:06:57.869206339Z Executed action InsecureSite.Controllers.HomeController.Error (InsecureSite) in 7.9125ms
2018-12-20T15:06:57.869222639Z [40m[32minfo[39m[22m[49m: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
2018-12-20T15:06:57.869228039Z Request finished in 30208.5835ms 500 text/html; charset=utf-8
2018-12-20T15:08:44 No new trace in the past 1 min(s).
There is not any other requests hitting this web app. I am the only one making requests.
Controllers can read data from the DB file without errors. I see that a couple queries right before the failed write take 3ms and 6ms.
I saw there was an issue with locking and multiple threads, but it is fixed, and I'm on a later version from the fix (2.1.4)
A solution for temporary databases
If the database file resides somewhere under /home (e.g. /home/Data/mydb.db or /home/site/wwwroot/mydb.db) the 'database is locked' error will occur.
I think this is due that all files under /home are persisted and reside in Azure Storage. Explained here
As solution use a different path. E.g. /tmp/mydb.db.
Or Path.Combine(Path.GetTempPath(), dbName);
Caveat: The db will be lost when the application restarts. So you can create the db in the temporary folder, then copy the db file to a persisted folder. Then access the db in the persisted folder with read only access. Add Mode=ReadOnly to the SQlite connection string (at least when using Microsoft.Data.Sqlite.SqliteConnection).
I fear there is no better solution when requiring Linux as OS. Microsoft should change this behaviour.
Can you share your update method? sometimes database is locked because C# code does not use database connection/transaction appropriately.
I suggest you have a look at your update method and see if database connection and transaction is used correctly. microsoft documentation might help
using (var context = new BloggingContext())
{
using (var transaction = context.Database.BeginTransaction())
{
try
{
context.Blogs.Add(new Blog { Url = "http://blogs.msdn.com/dotnet" });
context.SaveChanges();
context.Blogs.Add(new Blog { Url = "http://blogs.msdn.com/visualstudio" });
context.SaveChanges();
var blogs = context.Blogs
.OrderBy(b => b.Url)
.ToList();
// Commit transaction if all commands succeed, transaction will auto-rollback
// when disposed if either commands fails
transaction.Commit();
}
catch (Exception)
{
// TODO: Handle failure
}
}
}
To someone who still facing this issue:
Like #Josh mentioned, this is related to the Linux Azure App Service.
I also have a project, which is developed on my windows machine, and it works well, but when published to a Linux based Azure App Service, it will throw this error even for one simple INSERT from one request, I also confirmed from the App Service monitor that there is no con-current requests at that time.
I haven't figured out more details about the reason, but after changed the App Service to a Windows based, everything works well.
This is most often caused by open or pending connection requests that couldn't be processed because of some earlier process grabbing up the lock. You can try the following implementation, using which connection and commands can be disposed correctly in order.
private static void ExecuteNonQuery(string queryString)
{
using (var connection = new SQLiteConnection(
ConnectionString))
{
using (var command = new SQLiteCommand(queryString, connection))
{
command.Connection.Open();
command.ExecuteNonQuery();
}
}
}
Occasionally affecting some long time process and making users frustrated.
Googled and found nothing, one would expect 6616 or EventCode=20257 points to something but I could not find anything.
My guess is that under heavy load fails? I wonder if I move all plugins out of the sandbox would solve the issue.
Below is the event log:
LogName=Application SourceName=MSCRMSandboxService EventCode=20257
EventType=2 Type=Error ComputerName= TaskCategory=%1 OpCode=None
RecordNumber=2033209 Keywords=Classic Message=The Sandbox Host
service has encountered an unexpected error and has stopped. Source:
Microsoft.Crm.Sandbox.HostService.exe (6616) Error:
System.IO.IOException: The process cannot access the file
'SandboxWorker.{d60c83a8-807f-4121-8d16-86351fc98c02}.txt' because
it is being used by another process. at
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileInfo.Delete() at
Microsoft.Crm.Sandbox.SandboxWorkerManager.CleanupCrashFiles() at
Microsoft.Crm.Sandbox.SandboxWorkerManager.BackgroundThread(Object
job) at
System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx) at
System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ontextCallback callback, Object state, Boolean
preserveSyncCtx) at
System.Threading.ExecutionContext.Run(ExecutionContext
executionContext, ContextCallback callback, Object state) at
System.Threading.ThreadHelper.ThreadStart(Object obj)"
I've been writing a webjob with a blob watch (like http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx)
from time to time and without warning, I receive "invalid container name" exceptions from the line
host.RunAndBlock();
Despite not changing the container name in App.config, and while every other method of accessing the blob storage works.
Here's a stack trace for anyone who cares to go through it:
The thread '<No Name>' (0x17bc) has exited with code 0 (0x0).
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at BarcodeJob.Program.Main() in c:\Users\Nik\Documents\Visual Studio 2012\Projects\BarcodeJob\BarcodeJob\Program.cs:line 34
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
and the error message
System.FormatException: Invalid container name:
at Microsoft.WindowsAzure.Jobs.BlobClient.ValidateContainerName(String containerName)
at Microsoft.WindowsAzure.Jobs.CloudBlobPath..ctor(String blobInput)
at Microsoft.WindowsAzure.Jobs.LogRow.ToPath()
at Microsoft.WindowsAzure.Jobs.BlobLogListener.<GetRecentBlobWrites>d__0.MoveNext()
at Microsoft.WindowsAzure.Jobs.BlobListener.Poll(Action`1 callback, CancellationToken cancel)
at Microsoft.WindowsAzure.Jobs.Listener.PollBlobs(CancellationToken token)
at Microsoft.WindowsAzure.Jobs.Listener.Poll(CancellationToken token)
at Microsoft.WindowsAzure.Jobs.Worker.Poll(CancellationToken token)
at Microsoft.WindowsAzure.Jobs.JobHost.RunAndBlock(CancellationToken token, Action pauseAction)
at Microsoft.WindowsAzure.Jobs.JobHost.RunAndBlock(CancellationToken token)
at Microsoft.WindowsAzure.Jobs.JobHost.RunAndBlock()
at BarcodeJob.Program.Main() in c:\Users\Nik\Documents\Visual Studio 2012\Projects\BarcodeJob\BarcodeJob\Program.cs:line 30
This is more of a bug report than an actual question... So there's not a specific question to answer here, but I can confirm that callstack is certainly incriminating that it's a bug in the WebJobs SDK. We've filed a bug and expect to fix it in the Alpha 2 release. The releases are done via nuget, so Alpha 2 will show up at: http://www.nuget.org/packages/Microsoft.WindowsAzure.Jobs.Host
I am running the latest RavenDB (previously "stable", but now trying the latest "unstable" builds) on IIS 7.5. We have half a million docs and are hitting it fairly hard with a lot of writes as we need to update these docs fairly frequently.
We often see the below error in Raven Studio (and also "Object reference not set to an instance of an object." when inserting batches in a loop). The same also occurs when executing patch requests against large numbers of documents. We previously had large map/reduce indexes which caused a similar issue.
Is there anything we can do (perhaps via IIS to handle) this better and stop the database performance degrading or becoming unusable? Is the below a bug?
I also note our ESENT "Data" file has become large (10+Gb), perhaps (unwisely) due to a good number of fairly large docs stored in it previously. Is there any way to "shrink" this and make it smaller?
(note: I may edit this Q with a wrap-up a bit later).
Error reading RavenJArray from JsonReader.
System.Exception: Error reading RavenJArray from JsonReader.
at Raven.Json.Linq.RavenJArray.Load(JsonReader reader)
at Raven.Json.Linq.RavenJArray.Parse(String json)
at Raven.Client.Silverlight.Connection.Async.AsyncServerClient.b__22(Task`1 task)
at System.Threading.Tasks.Task`1.c__DisplayClass1a`1.b__19()
at System.Threading.Tasks.Task`1.InvokeFuture(Object futureAsObj)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
Inner StackTrace:
at Raven.Studio.Infrastructure.InvocationExtensions.Catch(Task parent, Action`1 action)
at Raven.Studio.Infrastructure.InvocationExtensions.Catch(Task parent)
at Raven.Studio.Infrastructure.Model.TimerTicked()
at Raven.Studio.Infrastructure.View.b__7(Model model)
at Raven.Studio.Infrastructure.View.InvokeOnModel(Object ctx, Action`1 action)
at Raven.Studio.Infrastructure.View.InvokeTimerTicked(Object ctx)
at Raven.Studio.Infrastructure.View.DispatcherTimerOnTick(Object sender, EventArgs eventArgs)
at MS.Internal.CoreInvokeHandler.InvokeEventHandler(Int32 typeIndex, Delegate handlerDelegate, Object sender, Object args)
at MS.Internal.JoltHelper.FireEvent(IntPtr unmanagedObj, IntPtr unmanagedObjArgs, Int32 argsTypeIndex, Int32 actualArgsTypeIndex, String eventName)
Could not parse json:
System.InvalidOperationException: Could not parse json:
---> System.Exception: Error reading RavenJObject from JsonReader.
at Raven.Json.Linq.RavenJObject.Load(JsonReader reader)
at Raven.Json.Linq.RavenJObject.Parse(String json)
--- End of inner exception stack trace ---
at Raven.Json.Linq.RavenJObject.Parse(String json)
at Raven.Client.Silverlight.Connection.Async.AsyncServerClient.b__58(Task`1 task)
at System.Threading.Tasks.Task`1.c__DisplayClass1a`1.b__19()
at System.Threading.Tasks.Task`1.InvokeFuture(Object futureAsObj)
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
Inner StackTrace:
at Raven.Studio.Infrastructure.InvocationExtensions.Catch(Task parent, Action`1 action)
at Raven.Studio.Infrastructure.InvocationExtensions.Catch(Task parent)
at Raven.Studio.Infrastructure.Model.TimerTicked()
at Raven.Studio.Infrastructure.View.b__7(Model model)
at Raven.Studio.Infrastructure.View.InvokeOnModel(Object ctx, Action`1 action)
at Raven.Studio.Infrastructure.View.InvokeTimerTicked(Object ctx)
at Raven.Studio.Infrastructure.View.DispatcherTimerOnTick(Object sender, EventArgs eventArgs)
at MS.Internal.CoreInvokeHandler.InvokeEventHandler(Int32 typeIndex, Delegate handlerDelegate, Object sender, Object args)
at MS.Internal.JoltHelper.FireEvent(IntPtr unmanagedObj, IntPtr unmanagedObjArgs, Int32 argsTypeIndex, Int32 actualArgsTypeIndex, String eventName)
Can you update your ravendb version? We have added additional error reporting that can help track that down.
The size of the db can be truncate, but only offline.
The way it works, we never give space back to the OS, but reserve it for our own usage.