Consistently, I am able to generate a Java Heap Space OutOfMemory exception with the following code on ColdFusion 9.01 (haven't tried earlier versions):
<cfset uuidGenerator = createObject("java", "java.util.UUID")>
<cfset transient = structNew()>
<cfloop from="1" to="100000" index="index">
<cfset transient[uuidGenerator.randomUUID().toString()] = true>
</cfloop>
The code above uses the Java UUID class because its faster than ColdFusion's. The structure itself does not exist after the request (i.e. it's not in some persistent scope such as application).
As a test, I generate a heap dump just after initializing the server. Then I run this code several times and see the tenured generation fill through jConsole. After, I run another heap dump. Using Eclipse Memory Analysis Tool's Leak report I can see one large object rooted on coldfusion.util.Key.
I'm asking here in hopes others have hit similar problem, and if so, what they've done to work around it.
Not an ideal solution, but until Adobe fix the memory leak internally you can get access to the private member ConcurrentHasMap on the coldfusion.util.Key object and manually clear it.
We have setup a scheduled task to execute this nightly and then do a GC immediately afterwards.
Compile this into a JAR file and put it somewhere in your ColdFusion class path.
import coldfusion.util.Key;
import java.lang.reflect.Field;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
class KeyEx
{
public KeyEx()
{
}
public void resetCache(Object k)
{
try
{
Field f = Key.class.getDeclaredField("keys");
f.setAccessible(true);
ConcurrentHashMap chm = (ConcurrentHashMap)f.get(k);
chm.clear();
}
catch (Exception ex)
{
System.out.println("ZOMG something went epically wrong!");
}
}
}
And then you can simply instantiate the new KeyEx object in coldfusion and call resetCache passing in the coldfusion.util.Key singleton.
<cfset keys = createObject("java", "coldfusion.util.Key") />
<cfset x = createObject("java", "KeyEx").init() />
<cfset x.resetCache(keys) />
These two example tests probably illustrate the problem a bit clearer:
-- MemoryLeak.cfm
<cfset transient = structNew() />
<cfset base = getTickCount() />
<cfloop from="1" to="10000" index="index">
<cfset transient[hash("#base##index#")] = true >
</cfloop>
<cfoutput>
Done
</cfoutput>
-- NoMemoryLeak.cfm
<cfset transient = structNew() />
<cfloop from="1" to="10000" index="index">
<cfset transient[hash("#index#")] = true >
</cfloop>
<cfoutput>
Done
</cfoutput>
Hit MemoryLeak.cfm on a CF 9.01+ box for 100 times and the memory leak is really obvious. Restart JVM and hit NoMemoryLeak.cfm as many times as you want, and the OldGen won't even notice it. I got up to 500,000 times before giving up.
I can't see OrangePips original bug# in the CF bug-base (looks like all old bugs went in the upgrade?), so I created a new one https://bugbase.adobe.com/index.cfm?event=bug&id=3119991 status currently confirmed & ToFix.
OK, so this is what I ended up doing to track down the root cause. Heap dumps were showing me that coldfusion.util.Key.keys was holding on to many objects in a ConcurrentHashMap of type coldfusion.util.Key after I ran a load test for awhile. So I figured out which .jar contained the .class - /lib/cfusion.jar - and decompiled it to see the source. What I saw is that it holds onto these references in a static private field whose keys are never removed.
Basically this object gets created every time a key never used before is inserted into a structure. So it's called from all over any ColdFusion application. I believe the intent is to facilitate speedups around lookups, case insensitivity, and comparison.
Accepting then that I cannot change code to fix the issue, as I'm sure it would violate a license agreement when used in a production environment, I did change code to help me understand why so many objects were being created in the first place. I basically added the following to the constructor:
try {
throw new RuntimeException();
} catch (Exception e) {
e.printStackTrace(System.out);
}
Then I compiled and put the changed class into cfusion.jar.
Fortunately, the stack trace generated includes .cfm file names and line numbers. I started CF from a command line (i.e. jrun.exe -start coldfusion) so I could see what was going to System.out easily and then kicked off my load test again. So the heuristic was finding what code was calling this a lot and potentially changing that.
This allowed me to quickly see a .cfm file that was being called with every request. So I can change that file to no longer create structure keys all the time and this has resulted in a much longer running VM. It doesn't fix the problem completely, but lowers memory consumption dramatically.
Related
crosspost: https://orchard.codeplex.com/discussions/474456
I'm on Orchard 1.7.2 and I get this error everywhere in my site (which I assume causes the load times to slow down):
Orchard.DisplayManagement.Implementation.DefaultDisplayManager - NullReferenceException thrown from Action`1 by System.Action`1[[Orchard.DisplayManagement.Implementation.ShapeDisplayingContext, Orchard.Framework, Version=1.7.2.0, Culture=neutral, PublicKeyToken=null]]
System.NullReferenceException: Object reference not set to an instance of an object.
So I traced the issue and found the function public string Slug within TermPart.cs (line 16).
public string Slug {
get { return this.As<AutoroutePart>().DisplayAlias; }
set { this.As<AutoroutePart>().DisplayAlias = value; }
}
I found out that this.As<AutoroutePart>() is null, and checking out the parts of the Content Item (via this.ContentItem.Parts), there is no AutoroutePart. Since it is null, a null reference exception is thrown when trying to retrieve the property DisplayAlias.
See image below:
Is this a known issue? Any workarounds? Is this what's causing my site to slow down?
Although having these exceptions definitely has an impact on performance I doubt it could cause a noticeable slowdown.
The exception is caused by Taxonomies's TermPart depending on AutoroutePart; i.e. if your content item doesn't use AutoroutePart (what should be pretty rare IMO) then you'll see this exception. Making TermPart depend on Autoroute is a questionable design decision (at least it could use IAliasAspect instead). Please open a bug report about the issue because this should be addressed somehow.
In the meantime you could attach AutoroutePart to the content type in question to get rid of the exceptions.
I have a application that sends out a few 100k emails each night, so to speed processing added some cfthreads.
This has caused some strange errors, and I've found that a variable created in one thread is being modified by another thread. From the documentation I've read, variables created in one thread should be visible only to that thread?
Made a simple test like so:
<cfthread
name="thread1"
action="run">
<cfsavecontent variable="local.template_body">
<cfinclude template="templates\6\2\bulletin_template.cfm">
</cfsavecontent>
<cfset tmpEmailBody = template_body>
</cfthread>
<cfthread
name="thread2"
action="run">
<cffile action="append"
file="C:\inetpub\error1.txt"
output="#tmpEmailBody#">
</cfthread>
The contents of "tmpEmailBody" successfully get written to the file.
The strange thing is if i remove the cfsavecontent section, and have:
<cfset tmpEmailBody = "test">, then the second thread raises and error that tmpEmailBody isnt defined, as I would expect.
Anyone know what's going on here?
I believe your running cfthread for the wrong purpose. It seems that the tmpEmailBody in thread 2 is dependent on thread1. You should not run dependent code in separate threads..
A better use of cfthread as pertaining to mailing may be
Thread1 {
select Emails from database where emails start from a-m
out your list of a-m
}
Thread 2 {
select Emails from database where emails start from n-z}
out your list n-z
}
Both of your threads run at the same time. The 2 queries can take place at the same time, but you cant save variables in 1 thread and hope that it matches up your call timing in another thread.
The variable tmpEmailBody is being created in the Variables scope, not in a scope that is limited to the thread. The threads will execute in random order and random times, so the error is caused because thread1 hasn't executed its last line before thread2 executes its first line.
All of the normal scopes are not thread safe when using cfthread (as opposed to being thread safe for ColdFusion request threads; aka page threads) If you want to make sure that a variable created/used in one thread is isolated from all other threads then you must use thread scoping. This is officially documented as Using thread data in the ColdFusion documentation.
I would have guessed that you were declaring tmpEmailBody outside of the two threads, but since changing the cfset to a static string gives the expected behaviour I would say there's an "issue" with cfsavecontent and it's writing to the Variables scope instead of thread-local scope, which should be filed as another scoping gotcha.
Since the ColdFusion documentation examples leave a bit to be desired I'll rewrite your code as if you wanted to pass the email body from one thread to the other. You've said this isn't your intended use, but it will show the various thread scopes. The following code copies values into different scopes when it doesn't need to, but done to hopefully make the different scopes more clear. And, as others have stated, the below task is a poor use of threads.
<cfthread
name="thread1"
action="run">
<cfset var template_body = "">
<cfsavecontent variable="template_body">
<cfinclude template="templates\6\2\bulletin_template.cfm">
</cfsavecontent>
<cfset thread.tmpEmailBody = template_body>
</cfthread>
<cfthread action="join" name="thread1" timeout="60">
<cfset Variables.tmpEmailBody = cfthread["thread1"]tmpEmailBody>
<cfthread
name="thread2"
action="run"
emailBody="#Variables.tmpEmailBody#">
<cffile action="append"
file="C:\inetpub\error1.txt"
output="#Attributes.emailBody#">
</cfthread>
My current application is using single instance of an object as a global variable for many of the main components, which I understand is considered inferior to using dependency injection.
I wish to make my applications open source in the future, but first I want to refactor the code to use the most recommended techniques for team collaboration so that other developers will be able to change my source code more easily.
Example of a shared resource: In the CFML language, you have the Server scope, which is shared memory that is available to any request for the entire server instance.
Here is my new design concept for managing changes to the Server scope:
Create a single instance of component named ServerMemoryManager which provides an interface for writing and reading to the server scope.
Any other code that needs to access the server scope will be injected with a reference to the single instance of the ServerMemoryManager via an init() function or a setServerMemoryManager() function.
Whenever a component reads/writes data to the ServerMemoryManager object, it will be able to internally lock the server scope so that no 2 threads can simultaneous write to the same piece of memory in the server scope.
Is this the best way to manage a shared resource (shared memory, filesystem, etc) that requires locking in order to be thread-safe?
Please describe any additional methods that can be used to manage a shared resource that requires locking during certain read/write operations which are considered best practices.
Edit: Based on the accepted answer, instead of locking scope="server", I will use named locks and manage the shared resources with more fine-grained locking. This may allow using multiple objects to manage the shared resources assuming they are all managing different keys in shared memory or files in the filesystem. For example, one application could have its own unique key or directory assigned to it so that it wouldn't conflict with another application trying to change a shared resource.
Edit2: I found I could use a single component named scope.cfc for each scope if I pass the scope into the init function when I create the object. I am now using fine-grained named locks. Let me know if it can be improved. The actual revised code now looks like this (I excluded the code for read, delete, clear). It also doesn't seem that it is required to have a single instance of the scope.cfc component anymore.
<cfcomponent>
<cfscript>
variables.scope=false;
variables.scopeName=false;
</cfscript>
<cffunction name="init" access="public" output="no" returntype="scope">
<cfargument name="scope" type="struct" required="yes">
<cfargument name="scopeName" type="string" required="yes">
<cfscript>
variables.scope=arguments.scope;
variables.scopeName=arguments.scopeName;
return this;
</cfscript>
</cffunction>
<cffunction name="write" access="public" output="no" returntype="boolean">
<cfargument name="key" type="string" required="yes">
<cfargument name="value" type="any" requires="yes">
<cfargument name="timeout" type="numeric" required="no" default="10">
<cftry>
<cflock type="exclusive" name="zcore-#variables.scopeName#-scope-#arguments.key#" timeout="#arguments.timeout#" throwontimeout="yes">
<cfscript>
variables.scope[arguments.key]=arguments.value;
</cfscript>
</cflock>
<cfcatch type="lock"><cfreturn false></cfcatch>
</cftry>
<cfreturn true>
</cffunction>
</cfcomponent>
** Edit3:** I tested the performance of reading from server scope through a component method like this and found it to be 20 times slower then reading the server scope directly when using a read only lock and 4 times slower without a lock. The overhead of an extra function call hundreds or thousands of times per request will be too slow. Tests done on Railo 3.3.x.
I prefer to build a large object in a non-public request and then set a single shared memory scope key then try to write an incomplete object to the scopes. Example:
<cfscript>
ts=structnew();
ts.largeObject=buildLargeObject();
server.cachedObject=ts;
</cfscript>
This lets you avoid locking across the entire application when you only write complete objects to shared memory since updating a single struct key is thread-safe. However, when you build the large object on startup, you need to be sure it is locked until that object is fully created.
I'm going to make the scope variable become directly readable by using the this scope instead of variables scope in the init function to avoid slowing down the application.
CFLOCK only prevents code from executing if every occurrence is locked the same way.
For example:
page1.cfm
<cflock type="exclusive" scope="server" timeout="10" >
<cfset application.xyz = 'abc'>
</cflock>
page2.cfm
<cfset application.xyz = '123'>
Page2.cfm is going to negate any locks you have on page1.cfm if page2 runs the same time page1 does. That said, it's good that you are locking inside your cfc so that each object doesn't have to be locked.
However, locking every occurrence isn't enough. The following won't do much good either.
page1.cfm
<cflock type="exclusive" scope="server" timeout="10" >
<cfset application.xyz = 'abc'>
</cflock>
page2.cfm
<cflock type="exclusive" scope="server" timeout="10" >
<cfset application.xyz = '123'>
</cflock>
This will halt the processing for every request of page1 and page2 but will not protect application.xyz on page1 from changes made to application.xyz on page2. To do this you need to give your locks a "name".
Locks name. Mutually exclusive with the scope attribute. Only one
request can execute the code within a cflocktag with a given name at a
time. Cannot be an empty string.
Permits synchronizing access to resources from different parts of an
application. Lock names are global to a ColdFusion server. They are
shared among applications and user sessions, but not clustered
servers.
Because you are creating multiple instances of your object, I believe serverMemoryManagerObject could interfere with serverMemoryManagerObject2 unless you name your lock.
Here is some more information about locking dos and don'ts
Locking code with cflock
CFLock And Negative Outcomes Think It Through
CFMythbusters - Countering Some Conventional Wisdom
If I want to parse a text field in SSJS there are 2 main tools. The built in JavaScript code and the newly converted #Functions. Are the #Functions slower then using pure javascript? Or is there no real difference?
viewScope.put("length", tmpStr.length)
vs.
viewScope.put("length:, #Length(tmpStr))
All SSJS is parsed into an AST (abstract syntax tree) at runtime. In other words, your code just remains a String until the exact moment that it is executed, at which point a parser examines that String to syntactically identify what the code contains: which characters denote variables, which are operators, functions, etc. Once that parsing is complete, the runtime engine is able to run Java code that is a rough approximation of what the JavaScript code was designed to do.
This is why SSJS is always slower than the directly equivalent Java: if you just write your code in Java to begin with, then it's compiled into bytecode the moment you build your project, but perhaps more importantly, at runtime it doesn't have to "guess" what code to run by parsing a String... it just runs the Java code you already defined.
On the other hand, there's nothing about this process that significantly distinguishes the SSJS implementation of various #Functions from "native" JavaScript; given that #Length(tmpStr) is just a wrapper for tmpStr.length, it doesn't surprise me that Sven is seeing a difference in execution time given enough iterations. But if your goal is optimization, you'll gain far more improvement by moving all code from SSJS blocks to bean methods than you will by eschewing the convenience of #Functions in favor of native JavaScript, because even native JavaScript has to be parsed into an AST. In that sense, there is no fundamental difference between the two.
UPDATE: there's a slight caveat to the AST parsing mentioned at the beginning of this answer. By default, the XPages runtime caches up to 400 unique SSJS expressions (you can override this limit via the ibm.jscript.cachesize property in the server's xsp.properties file). So if an expression is encountered that matches exactly (including whitespace) one that is already cached, Domino doesn't have to construct a new AST for that expression; it just references the tree already in cache. This is a MRU ("most recently used") cache, so the more frequently the same expression is encountered, the more likely it is to remain in the cache. Regardless of whether the AST is cached, it still has to be evaluated against the current context, and some of the JavaScript wrapper objects do have additional overhead compared to what you'd likely use instead if you were just coding directly in Java (for instance, {} becomes an ObjectObject, which is similar to a HashMap, but has additional features that support closures, which are just wasted if you're not using closures anyway). But the primary performance implication of this AST cache is that, unlike in most development contexts, duplication of code can actually be a good thing, if only in the sense that using the same exact expression over and over again allows all but the first instance of each to skip the language parsing and jump straight to invocation.
For me it seems that the #Formula is not as fast as using SSJS-code.
You can easily test by yourself with some code like this one (reload page multiple times to get serious results):
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core">
<xp:label id="label1">
<xp:this.value>
<![CDATA[#{javascript:
var start = java.lang.System.currentTimeMillis();
var testString = "0123456789";
var dummy;
for( var i=0; i<100000; i++ ){
dummy = #Length( testString )
}
var stop = java.lang.System.currentTimeMillis();
stop - start + " ms"}]]>
</xp:this.value>
</xp:label>
<xp:br></xp:br>
<xp:br></xp:br>
<xp:label id="label2">
<xp:this.value>
<![CDATA[#{javascript:
var start = java.lang.System.currentTimeMillis();
var testString = "0123456789";
var dummy;
for( var i=0; i<100000; i++ ){
dummy = testString.length;
}
var stop = java.lang.System.currentTimeMillis();
stop - start + " ms"}]]>
</xp:this.value>
</xp:label>
</xp:view>
I dont think #formula in SSJS is as fast as traditional #formula. One of the reasons would be that #formula is just another layer on top of SSJS functionality and therefore there is a bit more code to execute.
But that's just a wild guess.
As far as #DBLookup, I did some testing and found it was significantly slower then getting the view in SSJS, then doing a getDocumentByKey and getting fields from the found NotesDocument. At least 10 times slower when I looped four #DBLookups 100 times vs. getting the document 100 times and then getting the four fields.
Howard
tmpStr.length is browser native function (not in Javascript but code inside the browser and compiled)
#Length(tmpStr) is a javascript function, so it is interpreted by the browser
Edit: I've figured out the constructor for the singleton is getting called multiple times so it appears the classes are getting loaded more than once by separate class loaders. How can I make a global singleton in Tomcat? I've been googling, but no luck so far.
I have a singleton object that I construct like thus:
private static volatile KeyMapper mapper = null;
public static KeyMapper getMapper()
{
if(mapper == null)
{
synchronized(Utils.class)
{
if(mapper == null)
{
mapper = new LocalMemoryMapper();
}
}
}
return mapper;
}
The class KeyMapper is basically a synchronized wrapper to HashMap with only two functions, one to add a mapping and one to remove a mapping. When running in Tomcat 6.24 on my 32bit Windows machine everything works fine. However when running on a 64 bit Linux machine (CentOS 5.4 with OpenJDK 1.6.0-b09) I add one mapping and print out the size of the HashMap used by KeyMapper to verify the mapping got added (i.e. verify size = 1). Then I try to retrieve the mapping with another request and I keep getting null and when I checked the size of the HashMap it was 0. I'm confident the mapping isn't accidentally being removed since I've commented out all calls to remove (and I don't use clear or any other mutators, just get and put).
The requests are going through Tomcat 6.24 (configured to use 200 threads with a minimum of 4 threads) and I passed -Xnoclassgc to the jvm to ensure the class isn't inadvertently getting garbage collected (jvm is also running in -server mode). I also added a finalize method to KeyMapper to print to stderr if it ever gets garbage collected to verify that it wasn't being garbage collected.
I'm at my wits end and I can't figure out why one minute the entry in HashMap is there and the next it isn't :(
Another wild guess: is it possible the two requests are being served by different copies of your web app? Each would be in its own ClassLoader and thus have a different copy of the singleton.
Have you tried removing the outer check
if(mapper == null)
{
Thereby always hitting the Synchronized point, it's subtle stuff but possibly you're hitting the double-checked locking idiom problem. Described here and in many other articles.
Must admit I've never seen the problem actually bite someone before, but this sure sounds like it.
With this solution, the JVM guarantees that it's only one mapper and that's it's initialized before use.
public enum KeyMapperFactory {
;
private static KeyMapper mapper = new LocalMemoryMapper();
public static KeyMapper getMapper() {
return mapper;
}
}
This may not be the cause of your problem but you are using the faulty double-checked locking. See this,
http://en.wikipedia.org/wiki/Double-checked_locking#Usage_in_Java
I found a rather poor fix. I exported my code as a JAR and put it in $TOMCAT/lib and that worked. This is clearly a class loader issue.
Edit: Figured out the solution
Ok, I finally figured out the problem.
I had made my application the default application for the server by adding a to server.xml and setting the path to "". However, when I was accessing it through the URL http://localhost/somepage.jsp for somethings, but also the URL http://localhost/appname/anotherpage.jsp for other things.
Once I changed all the URLs to use http://localhost/ instead of http://localhost/appname the problem was fixed.