Migrated an application to WebSphere v8 from v6 and started getting memory leaks. The primary suspect is org.apache.axis2. It looks like each time the application calls a web service, an object called ServiceClient is created by WAS8 and stored in something that's called ClientConfigurationContextStore and then never garbage collected. Has anybody had a similar issue?
Fixed the problem by forcing original axis 1.4 over supplied soap implementation. This was done by placing two files in WEB-INF/services of the application. First file is called javax.xml.soap.MessageFactory and contains 'org.apache.axis.soap.MessageFactoryImpl' and the second is called javax.xml.soap.SOAPConnectionFactory and contains 'org.apache.axis.soap.SOAPConnectionFactoryImpl'. So now in the code this: javax.xml.soap.SOAPConnectionFactory.newInstance() returns a org.apache.axis stuff while before it was returning com.ibm.ws.webservices stuff. No memory leaks anymore.
If you don't have the problem in WebSphere v6, it's possible it is a leak in v8 itself. But it's also possible that v8 is being more strict about something that v6 was letting you get away with.
Have you checked that you're reusing all the Axis2 client objects that you can, rather than recreating ones on every call that you don't need to recreate? I recall us having some leakage in Axis2 client code under WAS v6.1 and realizing that we were recreating objects that we could be reusing instead.
In one of our projects, we used Axis2 1.6.2 as service client. Application server was WebSphere 7 and in test environment it got out of memory from time to time. When i examined heap dump AxisConfiguration class had lots of AxisService class instances. I was instantiating ServiceClient for every request and i saw that sometimes garbage collection worked late to finalize this object. So we used ServiceClient singleton and that solved our problem.
Related
I have got problem while working with Java agent in Notes client. I need to upload the file from particular path in Notes client memo form. I am using Java agent. It uses Alfresco's API to upload file to alfresco server. It works fine in standalone Java application. It is working fine for the first time in Lotus Notes also. But when I want to update for the second time or if I run any Java agent which has JAR files imported for the second time, it throws the following error:
Notes Error : JVM : Attempt to retrieve java agent attachments failed.
I added necessary jar files in the server's jvm/lib/ext path. After restarting server, it works for first time, then it fails for second time. Any solutions would be appreciated.
The 'Attempt to retrieve java agent attachments failed' error on the second run is almost certainly caused by memory exhaustion. That means that on the first run, your agent is grabbing a lot of memory and failing to free it.
This can happen if your code is accessing a large number of objects in the lotus.domino.* classes and failing to call their recycle() method. (Each such object allocates some non-JVM memory, which is not freed by the JVM's garbage collector. It is essential that you call the recycle() method to free that memory. Usually, the finally clause is a good place to do this.)
I don't know anything about the Alfresco API, but it may also leak memory if it is not used properly.
It is known issue with memory leak if Java agent or Java script library contains JAR files. The bigger the sooner your server/amgr crashes. Not fixed since beginning of Java agents :-(.
Common workaround is to put all JARs into jvm/lib/ext folder, as #Simon O'Doherty mentioned.
Modern approach is to use XAgents or servlet.
I'm building an application using Google App Engine. The application consists of several servlets, one of which has a static member object that holds a lot of internal state. Multiple Android phones contact this servlet, causing the servlet to update the state of the static member object. However, when multiple phones happen to talk to the server at the same time, I get synchronization issues in which the static object is being modified by multiple threads at once. I've tried throwing in some synchronized blocks, but this does not seem to help.
I think the reason has something to do with how App Engine spawns threads for HTTP requests, but I'm not sure. What's the standard way to synchronize access to shared objects in App Engine? Thanks!
Your GAE app can be started on different servers, there can be, and will be few instances at one time. Any instance can be started or killed at any time, without any notice before. So, any in-memory state is useless.
You have to use or memcache service, or database instead.
I've set startMode="AlwaysRunning" attribute on my application pool and serviceAutoStartEnabled="true" attribute on my application in IIS configuration. I've even set up serviceAutoStartProvider and can see that "warm up" code is being executed. I also can see that w3wp process auto-starts after iisreset. Still, the first request to my ASP.NET MVC application is exactly as slow as without auto-start. Is there any point I'm missing or any way to easily debug this without a profiler?
Is this feature expected to affect first request performance at all? What is actually the bulk of work to do on the first request, given that the worker process is ready, .NET appdomain and even all .NET assemblies have been loaded?
I've been looking into this recently.
As far as I can tell, the autoStart feature will cause your IIS worker threads (by default, just the one for the pool) to JIT compile before the first request.
However, what is compiled appears to be just a bulk of the assemblies and dependencies, but not necessarily any methods.
When that first request happens, and your methods you've written get called for the first time, the JITer performs a final compile on those methods that have not yet been compiled.
The benefit of autoStart appears to be it lets .Net do 90% of the work up-front, but the last 10% is still paid for when the first request happens and those methods that were yet to be accessed get run for the first time.
Background...
I have to build a new (asp.net mvc) app that uses an existing class library that is complex and can't be rewritten at this stage. The main problem is that this class library has a huge initialisation hit - it takes up to 10 mins to load all its data into memory. This is fine for production environment where it performs fast after IIS has started up. However, for development this is a nightmare because every time you build the solution and start it up in a browser, it takes ages.
Possible Solution?
So, the idea was that Castle Windsor or IOC lifestyle can be used to hold this in memory so that only recycling the application pool will force an expensive reload. I remember having a problem before when Windsor was keeping code in memory so even after changing it and recompiling, IIS still had the old code running - in this scenario it was a problem, but in my new scenario this is exactly what I'd like.
Anyone know how this can be done? I have tried with a dummy project using Singleton lifestyle but after changing the mvc project it still reloads the class library.
If the data does serialize then you could store in a cache that will keep it's state when you recompile. For example, memcached runs as a separate process. You could change the bin or restart the dev server process and the cache will keep it's state. There's a provider for accessing memcacheD on codeplex.
Maybe you could serialize the contents of the loaded library and save it binary form on the disk. This could potentially speed up the load. It's a crazy idea, but then again, having a class library that takes 10 minutes to load is crazy, too.
Behavior:
Application is loaded and being used as expected.
Suddenly, a particular DLL can no longer be loaded. The error message is:
ActiveX component cannot create object.
In each case, the object had been created successfully many times before failure. All objects are marked for "retain in memory".
This error is cleared when the application pool is recycled. It may be hours or months before it is seen again.
Issue has happened within two hours of a refresh, as well as never happened in months of uptime.
Issue has happened with hundreds of simultaneous users (heavy usage) and also with 1-3 users.
While the issue is occurring, the process running that application pool cannot create the object that is failing. However it can create any other objects. Memory, CPU, and other resources all remain at normal usage. In addition, other processes (such as a stand-alone exe) can successfully create the object.
The first instance of the issue appeared in mid 2008. There have been less than fifty instances since then, despite a pool of hundreds of servers for it to occur on. All instances except one have failed on the same DLL.
DLL Failure Info:
most common - generic data structure implementing a b-tree, has no references other than to its interface. Code consists of arrays and one use of the vb6 Event functionality. The object has not been changed in any way since 2005.
one-time - interop to a .NET module. the failure is occurring when trying to create the interop object, not the .NET object. This object is updated a few times each year.
Application Environment:
IIS hosted application
VB6, classic ASP, some interop to minor .NET components
Windows Server 2003 / Windows Server 2008 (both have independently had the problem)
Attempts to Reproduce:
Using scripts (and real-life humans) to run the same end-user workflows that our logs reported the days before the issue occurred.
Using scripts to create/destroy suspected objects as fast as possible from multiple simultaneous sessions.
Wild speculation.
No intentional success, but it does manifest randomly on the servers on its own.
Troubleshooting:
Code reviews
Test harnesses to investigate upper limits of object creation / destruction
Verification of ability to create object outside of the process experiencing the issue
Monitoring of resources over time on servers under load
Review of IIS, error, and event logs to determine events leading up to issue
Questions:
Any ideas on how to reproduce the issue?
What could cause this behavior?
Ideas for bypassing the first two questions in favor of a fast solution?
The DLL isn't on a network drive is it? You can get "glitches" where the drive is not available momentarily that then means COM can't do what it needs and could then fail to notice the drive is available again.
I used Process Monitor to debug similar problem when accessing ADO/OLEDB stack. Turned out environment got corrupted at some point and ADO classes are registered with InprocServer32 being REG_EXPAND_SZ pointing to %CommonProgramFiles%\System\ado\msado15.dll or similar ot x64 OSes.
Also when you register an application with Restart Manager, on failure the process gets restarted by winlogon process whose environment is different than explorer's one and unfortunately is missing %CommonProgramFiles% -- ouch!
This seems like a random failure; some race condition.
Try VMWARE to record the state of the machine you run this dll on. When the error happens you can then replay the record and inspect the memory contents. That why you won't have to play try and catch the error. At least you will have a solid record of it.
While I can't provide a solution, try catching the error and retry loading the dll when this happens after a refresh to the environment.