Currently I am debugging a seemingly random error with our applications. I will attempt to try to provide as much detail as possible. Basically after a while, one of the application pools for an application on our server stops. I have to start it, or sometimes, restart Coldfusion or worse the machine to get things running again.
The Server logs indicate the pool was stopped because of the Rapid Fail Succession thing in IIS. (The App serves X number of errors in a short amount of time so IIS stops the application)
Going through the ColdFusion Error logs I get this one right around the time of the IIS error where it stops my Pool.
SEVERE: Error in getRealPathFromConn
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:124)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:391)... AND SO ON
If you really need the full trace I can provide it, but it is long. I am not sure what the getRealPathFromConn is about. I've tried Googling, but haven't come up with a conclusion yet. In the traces they do lead to different files in various applications, but the lines in question seem to have nothing in common with one another.
Any thoughts on what might be the cause or where I should look next would help a ton.
EDIT: I forgot to say, I "believe" this started Monday. The only real thing that changed prior to Monday was the following.
We have an app structure:
root/app1/application.cfc
root/app2/application.cfc
root/app3/application.cfc
root/app3/website1
root/app3/website2
root/app3/website3
I changed the structure so two of the websites have their own application.cfc because we are slowly rolling out a change in how the websites work. I am testing this on two of them for now. Since I made that change, the apppool that keeps crashing the the app pool for app1. Not any of the websites, but I feel like it could be related.
UPDATE Updated Coldfusion to most recent version. (10,0,7,283649) Systems ran for about a week and this morning crashed.
Coldfusion-error.log at that time:
Jan 25, 2013 8:09:11 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\\ColdFusion10\\cfusion\lib;C:\\ColdFusion10\\cfusion\jintegra\bin;C:\\ColdFusion10\\cfus ion\jintegra\bin\international;C:\\ColdFusion10\\cfusion\lib\oosdk\classes\win
Jan 25, 2013 8:09:12 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 25, 2013 8:09:12 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.23
Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services
Jan 25, 2013 8:09:21 AM org.apache.catalina.core.ApplicationContext log
INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 20.4-b02
Jan 25, 2013 8:09:29 AM org.apache.catalina.core.ApplicationContext log
INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646
Jan 25, 2013 8:09:35 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8012"]
Jan 25, 2013 8:09:35 AM com.adobe.coldfusion.launcher.Launcher run
INFO: Server startup in 25362 ms
Jan 25, 2013 8:10:04 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer
SEVERE: Error in getRealPathFromConn
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:319)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:445)
at org.apache.coyote.ajp.AjpProcessor.readString(AjpProcessor.java:425)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:407)
at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1154)
at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198)
at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)
at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4475)
at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:414)
at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:335)
at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:179)
at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)
at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)
at coldfusion.runtime.TemplateProxyFactory.getFullName(TemplateProxyFactory.java:1092)
at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:202)
at coldfusion.runtime.TemplateProxyFactory.resolveName(TemplateProxyFactory.java:159)
at coldfusion.runtime.TemplateProxyFactory.resolveFile(TemplateProxyFactory.java:120)
at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:138)
at coldfusion.cfc.CFCProxy.<init>(CFCProxy.java:84)
at coldfusion.runtime.AppEventInvoker.<init>(AppEventInvoker.java:64)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:226)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:219)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
The GetRealPathFromConn exception gets through about 20 times or so one right after the other.
In the event Viewer I have of course 5 Warnings right before IIS shut down the pool:
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '3732'. The process exit code was '0xff'.
A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3052'. The data field contains the error number.
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '7972'. The process exit code was '0xff'.
A process serving application pool 'domain.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4736'. The data field contains the error number.
A process serving application pool 'domain.com' terminated unexpectedly. The process id was '8016'. The process exit code was '0xff'.
Application pool 'domain.com' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
So, this issue is/was "related" to the fixed CF bug. However we are still having an issue, but this error is no longer in the logs.
The issue we are having we think deals with the IIS/Tomcat connector that Coldfusion uses. I am going to close this as it was fixed in the hotfix, but a related issue still persists.
I don't know how to migrate it to my ServerFault question: https://serverfault.com/questions/487643/tomcat-cf10-iis7-5-connector-tuning-help-503-errors
Related
We are using apache ignite for one of our caching solution,It is perfectly working fine.But while stopping the web container lots of error related to removal of ignite Thread Local is displaying the server log.Since we are not creating any ThreadLocal in our application,how can i remove those Thread local when stopping the server. Some of the errors are mentioned below.
SEVERE: The web application [/webapp-name] created a ThreadLocal with key of type [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1#5826d221]) and a value of type [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value [org.apache.ignite.internal.binary.BinaryThreadLocalContext#47a32716]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 08, 2021 10:39:34 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/webapp-name] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#63b81d02]) and a value of type [org.apache.ignite.internal.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder] (value [org.apache.ignite.internal.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder#58474af2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 08, 2021 10:39:34 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [/webapp-name] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal#67782fee]) and a value of type [org.apache.ignite.internal.binary.BinaryContextHolder] (value [org.apache.ignite.internal.binary.BinaryContextHolder#482bfe01]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
I want to integrate automatic digital signature capabilities in my application. I signed-up for DocuSign sandbox account and tried to build and run example code from https://github.com/docusign/docusign-signature-appliance-api-recipes/tree/master/dsa-rest/Hello-World-examples
While running java hello-world example I am getting error as
Feb 09, 2021 9:01:00 AM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://prime-dsa-devctr.docusign.net:8081: Connection reset by peer: socket write error
Feb 09, 2021 9:01:00 AM org.apache.http.impl.execchain.RetryExec execute
I tried running C# code also , but get similar error in calling REST Endpoint "https://prime-dsa-devctr.docusign.net:8081/sapiws/v1/digital_signature"
The underlying connection was closed: An unexpected error occurred on a receive
am I missing something here? I tried changing user credential used in code, but still error does not change.
Update: the issue should be resolved now.
The DSA team is working on this, this is not your issue. It's down at least when I'm typing this answer. I'll update it as soon as it's back up.
I've set up Azerothcore on a VPS following this tutorial. I've created an account but when I try to log in my client gets stuck on the authenticating phase.
I followed the tutorial completely except I had to download the data files from the recommended link in the AzerothCore Wiki, because my worldserver did not recognize the files provided in the tutorial.
I've checked the config files and database and everything seems ok. The address is what it should be (my VPS address) and ports seem to be ok, too. I've tried redownloading the client (WoWmane WotLK 3.3.5a client, with WoD models), checking my firewall (exceptions added for the WoW client) and checking the realmlist.wtf file and config file, to no avail. My folder is not read-only and I'm really lost now.
EDIT: I've now managed to get the 'Malformed package' error again. I started the auth server, then the world server, then tried to log in, then shut down both servers after the client got stuck again. I'll paste the relevant portion of the server log file:
2019-08-25 03:03:49 ERROR: WORLD: World initialized in 0 minutes 13 seconds
2019-08-25 03:03:49
2019-08-25 03:03:49 worldserver process priority class set to -15
2019-08-25 03:03:49 Max allowed socket connections 1024
2019-08-25 03:03:49 Starting up Auction House Listing thread...
2019-08-25 03:03:49 AzerothCore rev. 2f74802d03d5 2019-08-23 22:22:26 +0200 (master branch) (Unix, Release) (worldserver-daemon) ready...
2019-08-25 03:04:16 ERROR: WorldSocket::handle_input_header(): client (account: 0, char [GUID: 0, name: <none>]) sent malformed packet (size: 8, cmd: 1867972643)
2019-08-25 03:04:42 Auction House Listing thread exiting without problems.
2019-08-25 03:04:42 Halting process...
My application deployed in Weblogic instance is getting too slow sometimes. At that time, it's hitting the error related to Stuck Thread Time in Managed server log. Initially, when I noticed this, I did some research and increased the value of Max Stuck Thread Time to 800 seconds in place of 600 seconds. But, this didn't fix the issue. I got the following error again.
WatchRule: (SEVERITY = 'Error') AND ((MSGID = 'WL-000337') OR (MSGID = 'BEA-000337'))
WatchData: MESSAGE = [STUCK] ExecuteThread: '58' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "812" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl#42f38088[POST /****/faces/index.jsf]
", which is more than the configured time (StuckThreadMaxTime) of "800" seconds in "server-failure-trigger". Stack trace:
oracle.jbo.pcoll.PCollNode.objectAt(PCollNode.java:1753)
oracle.jbo.pcoll.PCollNode.objectAt(PCollNode.java:1753)
oracle.jbo.pcoll.PCollection.elementAt(PCollection.java:839)
oracle.jbo.server.QueryCollection.get(QueryCollection.java:2556)
oracle.jbo.server.ViewRowSetImpl.getRow(ViewRowSetImpl.java:5540)
oracle.jbo.server.ViewRowSetIteratorImpl.getRangeIndexOf(ViewRowSetIteratorImpl.java:1179)
oracle.jbo.server.ViewRowSetIteratorImpl.notifyRowUpdated(ViewRowSetIteratorImpl.java:3491)
We are using
ADF 12c for application development
Weblogic Version: 12.2.1 in windows server
Database : Oracle 11g
Jdk version : 1.8-65
Can anyone please advise me on the reason and possible solution for this issue?
Thanks in advance.
Increasing the stuck threads time will not resolve your issue. From the stack trace it looks like your application may be reading some file. There can be different reason for slowness
1) May be the file which you are reading is very large
2) May be slow network
3) May be IO latency
If possible you can introduce some debugging messages in your application. This will provide the details on the process which your application is running.
If you are running your application on linux/unix environment then you can monitor process activity inside the /var/proc directory
I am running Liferay 6.2 on WebLogic 12c server.
Out of nowhere it just stopped working.
This is the last thing I see before it throws a flurry of exceptions
<Jan 10, 2014 2:53:28 PM EST> <Notice> <LoggingService> <BEA-320400> <The log fi
le C:\Oracle_2\Middleware\user_projects\domains\liferay\servers\AdminServer\logs
\AdminServer.log will be rotated. Reopen the log file if tailing has stopped. Th
is can happen on some platforms, such as Windows.>
<Jan 10, 2014 2:53:28 PM EST> <Notice> <LoggingService> <BEA-320401> <The log fi
le has been rotated to C:\Oracle_2\Middleware\user_projects\domains\liferay\serv
ers\AdminServer\logs\AdminServer.log00369. Log messages will continue to be logg
ed in C:\Oracle_2\Middleware\user_projects\domains\liferay\servers\AdminServer\l
ogs\AdminServer.log.>
The errors are shown here http://www.pastebin.ca/2532946
Anyone have any ideas on this?
As you can see in the log files (see below excerpt of your log file), Liferay is not able to either get a handle to the HSQL database or the HSQL db might be corrupted when you updated it.
13:11:16,769 WARN [C3P0PooledConnectionPoolManager[identityToken->uArzPQ2m]-HelperThread-#4][BasicResourcePool:1851] com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask#933b16 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (3). Last acquisition attempt exception:
java.sql.SQLException: error in script file line: 15 unexpected token: AVG
So you need to answer below questions:
Did you use any Client tool to make changes to your HSQL db?
If yes, did you close the connection to HSQL database before starting Liferay?
If not, Liferay won't be able to acquire lock on your db and fail to start.
If not, did you make DB changes directly in the HSQL db file?
This is NOT Recommended. Rollback your changes and try to use HSQL client to make your db changes
HTH!
P.S. Is this issue duplicate of: https://stackoverflow.com/questions/21052236/weblogic-wont-start. If so, please delete that one.