I have a scheduled task which should run every minute. To block the file from being called from outside, I want to compare host and calling IP adress like so:
<cfif NOT structKeyExists( "variables", caller )>
<cfset variables.caller = createObject("java", "java.net.InetAddress")>
</cfif>
<cfif variables.caller.getLocalHost().getHostAddress() EQ cgi.remote_addr>
// run task
</cfif>
Question:
I'm not really used to working with objects in Coldfusion, so would this be a way to make sure the object is created once and re-created every minute when the task is being called? Since this is running in my admin section, should I create the object on application scope right away?
Thanks for some input!
When you run a request even if the request is a scheduled task, all the variables are destroyed when the request has been completed. Only variables like session, application, and server persist between requests.
One approach to making sure that pages only run on the machine that they are on is to:
<cfif cgi.remote_addr EQ "127.0.0.1">
// run task
</cfif>
Related
I have a ColdFusion script which performs a task and then redirects to itself using the <cflocation> tag, passing in the next ID to be processed. The script stops working after EXACTLY 51 rows. I thought it was a bug in the code, so I updated the ID to be process to 52, and it runs from ID 52 through 102 and stops. Again, 51 rows exactly.
does anyone know if there is some limit in ColdFusion on the cflocation tag? Or could this be something IIS is doing?
When I use JavaScript window.location method, it works fine and does not stop until the end - as expected.
Updated - Explanation
The reason the script redirects to itself after each ID is processed is because ColdFusion does not release variables/memory back to the JVM until after the request is complete. So, for a request that performs several hundred complicated tasks, the memory usage continues to grow until the request officially completes. This causes an ERROR OUT OF MEMORY in the JVM. From what I understand, the best way to correct this is to split the items out into separate requests, that way ColdFusion will release the request variables/memory back to the JVM. Thus, why we are processing 1 item and then redirecting to self passing in the next ID to process. This is to keep ColdFusion seeing the requests as unique and keep memory consumption down.
It's an "ugly hack" but, maybe you'd like to try in case you can't find another option.
I was able to test and run this until completion as a scheduled task by executing curl from ColdFusion, so it requires curl available.
<cfparam name="url.id" default="0" />
<cfset url.id = url.id + 1 />
<!--- // your task code --->
<cffile
action = "append"
file = "C:\inetpub\wwwroot\foo\redir-log.txt"
output = '#url.id#'>
<!--- // your task code --->
<!--- // condition to end the loop --->
<cfif (url.id lt 300)>
<cfset args = [
'http://127.0.0.1:53227/redir.cfm?id=#url.id#'
] />
<cfexecute
name="curl"
arguments="#args#"
variable="output"
errorVariable="error"
timeout="0"
/>
<cfdump var="#output#" />
<hr />
<cfdump var="#error#" />
<hr />
</cfif>
I am not sure about a limit being there in ColdFusion/IIS, but I think browsers have a fail safe in order to restrict infinite loop. In my browser(Chrome), it seems the limit is 19.
The ways to overcome this is to make sure that the HTML loads, so that we do not confuse the session. Then redirect using either of the following.
window.location
<meta http-equiv="refresh" content="0;url=http://example.com">.
UPDATE: See MarkLogic 8 - Stream large result set to a file - JavaScript - Node.js Client API for someone's answer on how to do this in Javascript. This question is specifically asking about XQuery.
I have a web application that consumes rest services hosted in node.js.
Node simply proxies the request to XQuery which then queries MarkLogic.
These queries already have paging setup and work fine in the normal case to return a page of data to the UI.
I need to have an export feature such that when I put a URL parameter of export=all on a request, it doesn't lookup a page anymore.
At that point it should get the whole result set, even if it's a million records, and save it to a file.
The actual request needs to return immediately saying, "We will notify you when your download is ready."
One suggestion was to use xdmp:spawn to call the XQuery in the background which would save the results to a file. My actual HTTP request could then return immediately.
For the spawn piece, I think the idea is that I run my query with different options in order to get all results instead of one page. Then I would loop through the data and create a string variable to call xdmp:save with.
Some questions, is this a good idea? Is there a better way? If I loop through the result set and it does happen to be very large (gigabytes) it could cause memory issues.
Is there no way to directly stream the results to a file in XQuery?
Note: Another idea I had was to intercept the request at the proxy (node) layer and then do an xdmp:estimate to get the record count and then loop through querying each page and flushing it to disk. In this case I would need to find some way to return my request immediately yet process in the background in node which seems to have some ideas here: http://www.pubnub.com/blog/node-background-jobs-async-processing-for-async-language/
One possible strategy would be to use a self-spawning task that, on each iteration, gets the next page of the results for a query.
Instead of saving the results directly to a file, however, you might want to consider using xdmp:http-post() to send each page to a server:
http://docs.marklogic.com/xdmp:http-post?q=xdmp:http-post&v=8.0&api=true
In particular, the server could be a Node.js server that appends each page as it arrives to a file or any other datasink.
That way, Node.js could handle the long-running asynchronous IO with minimal load on the database server.
When a self-spawned task hits the end of the query, it can again use an HTTP request to notify Node.js to close the file and report that the export is finished.
Hping that helps,
This is an extension of this question.
In my models, every one requires a companyId to be set on creation and every one requires models to be filtered by the same session held companyid.
With sails.js, I have read and understand that session is not available in the model unless I inject it using the controller, however this would require me to code all my controller/actions with something very, very repetitive. Unfortunate.
I like sails.js and want to make the switch, but can anyone describe to me a better way? I'm hoping I have just missed something.
So, if I understand you correctly, you want to avoid lots of code like this in your controllers:
SomeModel.create({companyId: req.session.companyId, ...})
SomeModel.find({companyId: req.session.companyId, ...})
Fair enough. Maybe you're concerned that companyId will be renamed in the future, or need to be further processed. The simplest solution if you're using custom controller actions would be to make class methods for your models that accept the request as an argument:
SomeModel.doCreate(req, ...);
SomeModel.doFind(req, ...);
On the other hand, if you're on v0.10.x and you can use blueprints for some CRUD actions, you will benefit from the ability to override the blueprints with your own code, so that all of your creates and finds automatically use the companyId from the session.
If you're coming from a non-Node background, this might all induce some head-scratching. "Why can't you just make the session available everywhere?" you might ask. "LIKE THEY DO IN PHP!"
The reason is that PHP is stateless--every request that comes in gets essentially a fresh copy of the app, with nothing in memory being shared between requests. This means that any global variables will be valid for the life of a single request only. That wonderful $_SESSION hash is yours and yours alone, and once the request is processed, it disappears.
Contrast this with Node apps, which essentially run in a single process. Any global variables you set would be shared between every request that comes in, and since requests are handled asynchronously, there's no guarantee that one request will finish before another starts. So a scenario like this could easily occur:
Request A comes in.
Sails acquires the session for Request A and stores it in the global $_SESSION object.
Request A calls SomeModel.find(), which calls out to a database asynchronously
While the database does its magic, Request A surrenders its control of the Node thread
Request B comes in.
Sails acquires the session for Request B and stores it in the global $_SESSION object.
Request B surrenders its control of the thread to do some other asynchronous call.
Request A comes back with the result of its database call, and reads something from the $_SESSION object.
You can see the issue here--Request A now has the wrong session data. This is the reason why the session object lives inside the request object, and why it needs to be passed around to any code that wants to use it. Trying too hard to circumvent this will inevitably lead to trouble.
Best option I can think of is to take advantage of JS, and make some globally accessible functions.
But its gonna have a code smell :(
I prefer to make a policy that add the companyId inside the body.param like this:
// Needs to be Logged
module.exports = function(req, res, next) {
sails.log.verbose('[Policy.insertCompanyId() called] ' + __filename);
if (req.session) {
req.body.user = req.session.companyId;
//or something like AuthService.getCompanyId(req.session);
return next();
}
var err = 'Missing companyId';
//log ...
return res.redirect(307, '/');
};
Googlers if you've got a heap dump with a root of coldfusion.runtime.CFDummyComponent read on.
Update 2/22/2011
Marc Esher of MXUnit fame found the exact same bug in a different context. His solution involves a large loop over a query solved by going from query="name" to from="1" to="#name.recordcount#" index="row". Another approach that works is using <cfthread> inside the loop as such:
<cfloop ...>
<cfset threadName = "thread" & createUuid()>
<cfthread name="#threadName#">
<!--- do stuff --->
</cfthread>
<cfthread action="join" name="#threadName#">
</cfloop>
This is very effective when you run into situations where you need to do things inside the loop like queries and <cfmodule> inside <cffunction> so that the memory consumed is only for that iteration.
Old Question
Hoping someone else can confirm or tell me what I'm doing wrong. I am able to consistently reproduce an OOM running by calling the file oom.cfm (shown below). Using jconsole I am able to see the request consumes memory and never releases it until complete. The issue appears to be calling <cfmodule> inside of <cffunction>, where if I comment out the <cfmodule> call things are garbage collected while the request is running.
ColdFusion version: 9,0,1,274733
JVM Arguments
java.home=C:/Program Files/Java/jdk1.6.0_18
java.args=-server -Xms768m -Xmx768m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/ -Djava.security.policy={application.home}/servers/41ep8/cfusion.ear/cfusion.war/WEB-INF/cfusion/lib/coldfusion.policy -Djava.security.auth.policy={application.home}/servers/41ep8/cfusion.ear/cfusion.war/WEB-INF/cfusion/lib/neo_jaas.policy -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=56033
Test Case
oom.cfm (this calls template.cfm below - Adobe Bug #85736)
<cffunction name="fun" output="false" access="public" returntype="any" hint="">
<cfset var local = structNew()/>
<!--- comment out cfmodule and no OOM --->
<cfmodule template="template.cfm">
</cffunction>
<cfset size = 1000 * 200>
<cfloop from="1" to="#size#" index="idx">
<cfset fun()>
<cfif NOT idx mod 1000>
<cflog file="se-err" text="#idx# of #size#">
</cfif>
</cfloop>
template.cfm
<!--- I am empty! --->
Update #2 (cfthread case from Elliott Sprehn - Adobe ColdFusion Bug #83359)
<cfthread name="test">
<cfloop from="1" to="10000" index="i">
<cflog text="This is very bad.">
<cflock name="test" timeout="10">
</cflock>
</cfloop>
<!--- Sleep a very long time (10 minutes) --->
<cfset sleep(600000)>
</cfthread>
I've not run into this before, but here's what I think is going on:
Each time cfmodule is called, a new memory space is created for it (which, IIRC, is the main difference between it and cfinclude).
Because you are calling the cfmodule within the function, the cfmodule memory space technically belongs to that function's memory space.
The function's memory is protected from garbage collection until the function is done.
Result: heap fills, and you get an OOM error.
I don't think calling this a memory leak is correct, as it is behaving correctly, and when the function completes, the garbage collector can clear the hold on that memory. However, I can see how it might be inconvenient.
This problem manifests with lots of tags unfortunately. I've seen this with cflock inside cfthread. Write a very long running loop in a cfthread that uses cflock, you'll run out of memory eventually. It takes a long time, but it happens. I bet the retention problem exists in regular requests too, but you don't usually have a loop that runs hundreds of thousands of times with a cflock inside so no one notices.
I reported this bug a long time ago, but it never got fixed:
http://www.elliottsprehn.com/cfbugs/bugs/83359
The best solution for now is to not use cfmodule inside a loop like this. Custom tags really weren't intended for calling 20k times in a single request. You're going to want to use UDFs instead. cfmodule is extremely expensive anyway and using a UDF will be noticeably faster.
Here is a discussion of a possibly related Coldfusion version 9 cfc memory leak problem: http://forums.adobe.com/thread/1034324?start=0&tstart=0
See this bug report on it: https://bugbase.adobe.com/index.cfm?event=bug&id=3124148
I don't believe Adobe released a fix for verion 9.01 but supposedly this problem is fixed in version 10. There are workarounds for most people (depending on the scope of their problem) for this not unlike what's been described here.
hey guys I'm trying to create a timer which counts the time spent on a page using a thread heres what I have so far:
<cfset session.time=0>
<cfthread name="timer" action="run">
<cfscript>
counter = 0;
while (counter <9000) {
sleep(1000);
session.time++;
counter ++;
}
</cfscript>
</cfthread>
page 2:
<cfoutput>#session.time#</cfoutput>
page 2 gives me 0 every time anyone see a problem?
edit:
I changed line 1 of my code to <cfset session.time=100> and now page 2 says 100, its like the stuff inside the cfscript loop isn't reassigning session.time
This works for me.
Are you sure you are using the Developer or Enterprise edition of ColdFusion and the threads have actually kicked off? I think only those editions support multi-threading.
One way to do verity your threads working is to use cfstat - you should see one request running even though your page has returned.
Another is to write output from your spawned thread - use the code snipped below to write to System.out - ideally you'll need to have CF running as a console task to do this.
<cfset session.time=0 />
<cfthread name="timer" action="run">
<cfscript>
counter = 0;
while (counter <9000) {
sleep(1000);
session.time++;
sys = createObject("java", "java.lang.System");
sys.out.println("*** [DEBUG] - #timeformat(now(),'HH:mm:ss' )# - session.time=#session.time# ");
counter ++;
}
</cfscript>
</cfthread>
Your code works fine for me. You have got an Application.cfm page setup to enable session management i.e:
<cfapplication name="#hash(getCurrentTemplatePath())#"
sessiontimeout="#createTimeSpan(0,0,20,0)#" sessionmanagement="true"/>
Code executed inside a thread has its own scope, including session. I would set a variable within the tread and then access it from within the threads scope.
i.e.
Change session.time++; to thread.time++; and then use cfthread[timer].time to get the thread's time.
Although this may not hold if you enable session management like some of the other posts discuss.