In my HTA app there are some actions which are executing some heavily time-consuming tasks on a server. For example an action uses an old ActiveX component to read some fileproperties (like Subject and Comments) of files in a particular folder (0 - ~200 files in a folder).
Originally this was done by setting an interval, and reading the fileproperties file by file. The slowing down of the app was acceptable when connected to the server using fast local connections. But now, as remote working has significantly increased, and the remote connections are magnitudes slower than the intranet connections, the interval is not suitable for the task anymore.
To make the app faster during the filepropety search, I outsourced the code to a wsh job. The results are stored in a file, which of existence an interval (5 secs) is observing. However, some users are still experiencing remarkable slow-down of the app, even when polling the file existence with the said interval of 5 secs.
Now I wanted to know, if there is an event or some other internal mechanism, which I could use to detect when the wsh script has done its job? And if possible, even perhaps there's a way to send the results directly from the wsh job to HTA, without using the intermediate temporal file at all?
Here's some simplified code for the actual task performed in the wsh file and HTA app. HTA has the HTML5 DTD and it's running in Edge-mode using IE11. ui is an utility library, the referred propertynames are hopefully describing the usage accurate enough.
WSF:
<package>
<job id="getFileProps">
<script language="JScript">
(function () {
var topRoot = WScript.Arguments(0), // The starting folder <String>
fso = WScript.CreateObject('Scripting.FileSystemObject'), // Filesystem object <ActiveXObject>
fileProps = readProps(topRoot), // Fileprops, the result <Array>
file; // The result file to read in HTA <FileObject>
function readProps (root) {
var fileProperties = [];
// A bunch of code reading the fileproperties on a disk
return fileProperties;
}
file = fso.openTextFile(topRoot + '\\$fileprops$.txt', 2, true);
file.Write(fileProps.join(',');
file.Close();
WScript.Quit();
)());
</script>
</job>
</package>
JSCRIPT:
ui.winShell.Exec('WScript //Job:getFileProps ' + '"' + ui.appRoot + '"');
function fpCheck () {
var file, fileProps;
try {
file = ui.fileIO.OpenTextFile('$fileprops$.txt', 1, false);
} catch (err) {
file && file.Close();
setTimeout(fpCheck, 5000);
return;
}
// Write the fileprops to the view
}
If someone is interested in the fileproperty reader, its name is DSOFile.OleDocumentProperties. It originates somewhere to the deep internet archives, but I can't recall where I've loaded it from.
Eventually what you need is Inter-Process Communication.
Neither HTA or WSH do offer something built-in for IPC.
So, I'm going share a hacky solution that I rarely use.
The trick is to use a shared object in the two ends like a client-server scenario.
We're going to use and Internet Explorer object as an IPC server, the methods we will use are GetProperty and PutProperty.
Since Internet Explorer windows can be enumerated by using Shell objects's Windows() method, it's possible to access the object created in HTA from another application.
Examine and create the following two files, then start test.hta and click Start WSF button.
test.hta:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<title>IPC</title>
<script>
sharedObject = {
sendData: function(data){
// blocking here is not a good idea
// so we process the data within another callback
// leaving the method as soon as possible
var receivedData = data;
setTimeout(function(){alert("Data received: " + receivedData)}, 50);
}
};
shareName = "ipcShare";
serverId = createIPCServer(shareName);
function createIPCServer(shareName){
var ie = new ActiveXObject("InternetExplorer.Application");
ie.PutProperty(shareName, sharedObject);
window.onbeforeunload = function(){ ie.Quit(); };
return ie.HWND; // window handle
}
function starWSFJob() {
alert("please wait about 5 seconds...");
var wshShell = new ActiveXObject("Wscript.Shell");
wshShell.Run('wscript.exe test.wsf //Job:TestIPC /serverId:"' + serverId + '" /shareName:"' + shareName + '"');
}
</script>
</head>
<body>
<button onclick="starWSFJob()">Start WSF</button>
</body>
</html>
test.wsf:
<package>
<job id="TestIPC">
<script language="JScript">
function getSharedObject(serverId, shareName){
var shellApp = new ActiveXObject("Shell.Application");
var windows = new Enumerator(shellApp.Windows());
for (;!windows.atEnd();windows.moveNext()){
var window = windows.item();
if(window.HWND == serverId) {
return window.GetProperty(shareName);
}
}
}
function App() {
var serverId = Number(WScript.Arguments.Named("serverId"));
var shareName = WScript.Arguments.Named("shareName");
var sharedObject = getSharedObject(serverId, shareName);
if(!sharedObject){
WScript.Echo("shared object not found.");
return 1;
}
// simulate long running job
WScript.Sleep(5000);
// send data
sharedObject.sendData("Hello from WSF Job!");
return 0;
}
WScript.Quit(App());
</script>
</job>
</package>
Would your scenario support use of a dictionary object or array list to store just filename and lastupdated, and to only retrieve the full properties set (into a second list) for new or changed files (and deletes). Depends on how frequently the files are coming and going or being updated I would guess. This could be quicker than generating the entire file properties dataset if most of the details are not changing between polls.
Related
I'm making a program that will output text to speech, the user will need to be able to change the output device (for example to virtual audio cable). At the moment I'm using https://www.npmjs.com/package/say to produce the speech
e.g.
const say = require("say");
say.speak("Hello World");
but I've no clue on how I could go about choosing the audio output.
What I've found so far has been pretty useless, I'm guessing largely because I don't know the proper terminology to search for answers, regardless this is it:
I first found out about navigator.MediaDevices, then I found how I could make an audio element and change the audio device of that element via setSinkId, then I realized these things are probably(?) irrelevant since the say module seems to play sounds using powershell commands. I've even gone as far as trying to change powershell's output device in app volume device preferences(img), but that seems to do nothing.
I'm pretty much stumped right now, so I'd appreciate any help please. I'm not set on using Say as a module, it just seemed easy to use at first.
Edit:
A workaround I've settled with is making my own TTS class and using SpVoice.
I have something similar to this:
const childProcess = require('child_process');
class TTS {
constructor(channel, speed) {
this.speed = speed;
this.channel = channel;
this.baseCommand = "$speak = New-Object -ComObject SAPI.SPVoice;"
}
speak(text){
var command = this.baseCommand +
`$speak.AudioOutput = foreach ($o in $speak.GetAudioOutputs()) {if ($o.getDescription() -eq '${this.channel}') {$o; break;}}; `
+ "$speak.Speak([Console]::In.ReadToEnd());"
this.child = childProcess.spawn('powershell', [command], {shell: true})
this.child.stdin.setEncoding('ascii')
this.child.stdin.end(text);
this.child.addListener('exit', (code, signal) => {
if (code === null || signal !== null) {
console.log(new Error(`error [code: ${code}] [signal: ${signal}]`))
}
this.child = null
})
}
}
Then I can pass in an audio channel like
tts = new TTS("CABLE Input (VB-Audio Virtual Cable)", 0);
tts.speak("Hello there");
and it will output TTS in my desired channel
Some of the browsers support the built in speechSynthesis API.
Save the following code in 'test.html' file and open the file in chrome web browser for testing the speech api.
<script>
//---------------- SpeechAPI Start ------------------------
function speak(message) {
try {
var utterance= new SpeechSynthesisUtterance("");
utterance.lang='en-GB';
utterance.text=message;
window.speechSynthesis.speak(utterance);
}catch(e){
console.log('Exception in speak : ' + e)
}
}
//---------------- SpeechAPI End ------------------------
</script>
<button onclick="speak('Hello, how are you doing?');">Press to speak</button>
Is this what you are looking for ?
This method throws
java.lang.IllegalStateException: Cannot forward after response has been committed
and I am unable to spot the problem. Any help?
int noOfRows = Integer.parseInt(request.getParameter("noOfRows"));
String chkboxVal = "";
// String FormatId=null;
Vector vRow = new Vector();
Vector vRow1 = new Vector();
String GroupId = "";
String GroupDesc = "";
for (int i = 0; i < noOfRows; i++) {
if ((request.getParameter("chk_select" + i)) == null) {
chkboxVal = "notticked";
} else {
chkboxVal = request.getParameter("chk_select" + i);
if (chkboxVal.equals("ticked")) {
fwdurl = "true";
Statement st1 = con.createStatement();
GroupId = request.getParameter("GroupId" + i);
GroupDesc = request.getParameter("GroupDesc" + i);
ResultSet rs1 = st1
.executeQuery("select FileId,Description from cs2k_Files "
+ " where FileId like 'M%' and co_code = "
+ ccode);
ResultSetMetaData rsm = rs1.getMetaData();
int cCount = rsm.getColumnCount();
while (rs1.next()) {
Vector vCol1 = new Vector();
for (int j = 1; j <= cCount; j++) {
vCol1.addElement(rs1.getObject(j));
}
vRow.addElement(vCol1);
}
rs1 = st1
.executeQuery("select FileId,NotAllowed from cs2kGroupSub "
+ " where FileId like 'M%' and GroupId = '"
+ GroupId + "'" + " and co_code = " + ccode);
rsm = rs1.getMetaData();
cCount = rsm.getColumnCount();
while (rs1.next()) {
Vector vCol2 = new Vector();
for (int j = 1; j <= cCount; j++) {
vCol2.addElement(rs1.getObject(j));
}
vRow1.addElement(vCol2);
}
// throw new Exception("test");
break;
}
}
}
if (fwdurl.equals("true")) {
// throw new Exception("test");
// response.sendRedirect("cs2k_GroupCopiedUpdt.jsp") ;
request.setAttribute("GroupId", GroupId);
request.setAttribute("GroupDesc", GroupDesc);
request.setAttribute("vRow", vRow);
request.setAttribute("vRow1", vRow1);
getServletConfig().getServletContext().getRequestDispatcher(
"/GroupCopiedUpdt.jsp").forward(request, response);
}
forward/sendRedirect/sendError do NOT exit the method!
A common misunderstanding among starters is that they think that the call of a forward(), sendRedirect(), or sendError() would magically exit and "jump" out of the method block, hereby ignoring the remnant of the code. For example:
protected void doXxx() {
if (someCondition) {
sendRedirect();
}
forward(); // This is STILL invoked when someCondition is true!
}
This is thus actually not true. They do certainly not behave differently than any other Java methods (expect of System#exit() of course). When the someCondition in above example is true and you're thus calling forward() after sendRedirect() or sendError() on the same request/response, then the chance is big that you will get the exception:
java.lang.IllegalStateException: Cannot forward after response has been committed
If the if statement calls a forward() and you're afterwards calling sendRedirect() or sendError(), then below exception will be thrown:
java.lang.IllegalStateException: Cannot call sendRedirect() after the response has been committed
To fix this, you need either to add a return; statement afterwards
protected void doXxx() {
if (someCondition) {
sendRedirect();
return;
}
forward();
}
... or to introduce an else block.
protected void doXxx() {
if (someCondition) {
sendRedirect();
}
else {
forward();
}
}
To naildown the root cause in your code, just search for any line which calls a forward(), sendRedirect() or sendError() without exiting the method block or skipping the remnant of the code. This can be inside the same servlet before the particular code line, but also in any servlet or filter which was been called before the particular servlet.
In case of sendError(), if your sole purpose is to set the response status, use setStatus() instead.
Do not write any string before forward/sendRedirect/sendError
Another probable cause is that the servlet writes to the response while a forward() will be called, or has been called in the very same method.
protected void doXxx() {
out.write("<p>some html</p>");
// ...
forward(); // Fail!
}
The response buffer size defaults in most server to 2KB, so if you write more than 2KB to it, then it will be committed and forward() will fail the same way:
java.lang.IllegalStateException: Cannot forward after response has been committed
Solution is obvious, just don't write to the response in the servlet. That's the responsibility of the JSP. You just set a request attribute like so request.setAttribute("data", "some string") and then print it in JSP like so ${data}. See also our Servlets wiki page to learn how to use Servlets the right way.
Do not write any file before forward/sendRedirect/sendError
Another probable cause is that the servlet writes a file download to the response after which e.g. a forward() is called.
protected void doXxx() {
out.write(bytes);
// ...
forward(); // Fail!
}
This is technically not possible. You need to remove the forward() call. The enduser will stay on the currently opened page. If you actually intend to change the page after a file download, then you need to move the file download logic to page load of the target page. Basically: first create a temporary file on disk using the way mentioned in this answer How to save generated file temporarily in servlet based web application, then send a redirect with the file name/identifier as request param, and in the target page conditionally print based on the presence of that request param a <script>window.location='...';</script> which immediately downloads the temporary file via one of the ways mentioned in this answer Simplest way to serve static data from outside the application server in a Java web application.
Do not call forward/sendRedirect/sendError in JSP
Yet another probable cause is that the forward(), sendRedirect() or sendError() methods are invoked via Java code embedded in a JSP file in form of old fashioned way <% scriptlets %>, a practice which was officially discouraged since 2003. For example:
<!DOCTYPE html>
<html lang="en">
<head>
...
</head>
<body>
...
<% sendRedirect(); %>
...
</body>
</html>
The problem here is that JSP internally immediately writes template text (i.e. HTML code) via out.write("<!DOCTYPE html> ... etc ...") as soon as it's encountered. This is thus essentially the same problem as explained in previous section.
Solution is obvious, just don't write Java code in a JSP file. That's the responsibility of a normal Java class such as a Servlet or a Filter. See also our Servlets wiki page to learn how to use Servlets the right way.
See also:
What exactly does "Response already committed" mean? How to handle exceptions then?
Unrelated to your concrete problem, your JDBC code is leaking resources. Fix that as well. For hints, see also How often should Connection, Statement and ResultSet be closed in JDBC?
even adding a return statement brings up this exception, for which only solution is this code:
if(!response.isCommitted())
// Place another redirection
Typically you see this error after you have already done a redirect and then try to output some more data to the output stream. In the cases where I have seen this in the past, it is often one of the filters that is trying to redirect the page, and then still forwards through to the servlet. I cannot see anything immediately wrong with the servlet, so you might want to try having a look at any filters that you have in place as well.
Edit: Some more help in diagnosing the problem…
The first step to diagnosing this problem is to ascertain exactly where the exception is being thrown. We are assuming that it is being thrown by the line
getServletConfig().getServletContext()
.getRequestDispatcher("/GroupCopiedUpdt.jsp")
.forward(request, response);
But you might find that it is being thrown later in the code, where you are trying to output to the output stream after you have tried to do the forward. If it is coming from the above line, then it means that somewhere before this line you have either:
output data to the output stream, or
done another redirect beforehand.
Good luck!
You should add return statement while you are forwarding or redirecting the flow.
Example:
if forwardind,
request.getRequestDispatcher("/abs.jsp").forward(request, response);
return;
if redirecting,
response.sendRedirect(roundTripURI);
return;
This is because your servlet is trying to access a request object which is no more exist..
A servlet's forward or include statement does not stop execution of method block. It continues to the end of method block or first return statement just like any other java method.
The best way to resolve this problem just set the page (where you suppose to forward the request) dynamically according your logic. That is:
protected void doPost(request , response){
String returnPage="default.jsp";
if(condition1){
returnPage="page1.jsp";
}
if(condition2){
returnPage="page2.jsp";
}
request.getRequestDispatcher(returnPage).forward(request,response); //at last line
}
and do the forward only once at last line...
you can also fix this problem using return statement after each forward() or put each forward() in if...else block
I removed
super.service(req, res);
Then it worked fine for me
Bump...
I just had the same error. I noticed that I was invoking super.doPost(request, response); when overriding the doPost() method as well as explicitly invoking the superclass constructor
public ScheduleServlet() {
super();
// TODO Auto-generated constructor stub
}
As soon as I commented out the super.doPost(request, response); from within doPost() statement it worked perfectly...
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
//super.doPost(request, response);
// More code here...
}
Needless to say, I need to re-read on super() best practices :p
After return forward method you can simply do this:
return null;
It will break the current scope.
If you see this on a Spring based web application, make sure you have your method annotated with #ResponseBody or the controller annotated with #RestController instead of #Controller. It will also throw this exception if a method returns JSON, but has not been configured to have that as the response, Spring will instead look for a jsp page to render and throw this exception.
I have a YouTube video link which I am trying to supply parameters to:
https://www.youtube.com/embed/wU4DgHHwVCc?autoplay=1&start=20&end=25&loop=1
Everything else works perfectly except for the loop parameter. The video doesn't loop. According to Google Developer's page:
This parameter has limited support in the AS3 player and in IFrame embeds, which could load either the AS3 or HTML5 player. Currently, the loop parameter only works in the AS3 player when used in conjunction with the playlist parameter.
So even though my Chrome browser is not using AS3, as I've disabled flash, I added the playlist parameter to the URL just to see what would happen.
https://www.youtube.com/embed/wU4DgHHwVCc?autoplay=1&start=20&end=25&loop=1&playlist=wU4DgHHwVCc
Now the video does loop but both start and end parameters are ignored and the video starts at 00:00 instead of 00:20 as specified in the URL.
Why do I need to specify the playlist parameter when I am not using AS3 player?
Why does it ignore the start and end parameters on loop?
OS: Ubuntu 16.04 LTS
Chrome version: 60.0.3112.90 (64-bit)
Looking at Youtube documentation... You can modify their shown i-frame API example code like below to get a looping effect.
The trick is :
Set Start Time of video at 20 seconds
Listen for playback begin event and handle with onPlayerStateChange() function, which itself starts a timer of a 5 seconds countdown (ie: 5000 millisecs).
When timer reaches zero it triggers function handleVideo() which itself starts a new timer of 5 seconds. Then auto-seeks back to 20 second (start time) in video's timeline. The timer now creates a feedback loop.
Try my example code below in a new html page. Can also test it here.
Code was tested on Windows/Chrome.
<!DOCTYPE html>
<html>
<body>
<!-- 1. The <iframe> (and video player) will replace this <div> tag. -->
<div id="player"></div>
<script>
// 2. This code loads the IFrame Player API code asynchronously.
var tag = document.createElement('script');
tag.src = "https://www.youtube.com/iframe_api";
var firstScriptTag = document.getElementsByTagName('script')[0];
firstScriptTag.parentNode.insertBefore(tag, firstScriptTag);
// 3. This function creates an <iframe> (and YouTube player)
// after the API code downloads.
var player;
function onYouTubeIframeAPIReady()
{
player = new YT.Player('player',
{
width: '100%',
height: '800',
videoId: 'wU4DgHHwVCc',
startSeconds:20,
//endSeconds:25,
events:
{
'onReady': onPlayerReady,
'onStateChange': onPlayerStateChange
}
});
}
// 4. The API will call this function when the video player is ready.
function onPlayerReady(event)
{
event.target.seekTo(20);
event.target.playVideo();
}
// 5. The API calls this function when the player's state changes.
// The function indicates that when playing a video (state=1),
// the player should play for six seconds and then stop.
var done = false;
function onPlayerStateChange(event) {
if (event.data == YT.PlayerState.PLAYING && !done) {
setTimeout(handleVideo, 5000);
done = true;
}
}
// Custom function to LOOP
// Moves playhead back to 20 seconds
function handleVideo()
{
setTimeout(handleVideo, 5000);
player.seekTo(20);
}
</script>
</body>
</html>
It is simple. For single video to loop, you need to have it twice.
Try this out.
https://www.youtube.com/embed/wU4DgHHwVCc?playlist=wU4DgHHwVCc&autoplay=1&rel=0&loop=1
Note that "rel" must be set to "0" for any videos that might have them linked as they break the loop.
I made a page for testing HERE if you like to test.
I am playing with Node.js and I have created a simple script that uploads files from a directory to a server:
var request = require('request');
var file = require('file');
var fs = require('fs');
var path = require('path');
VERSION = '0.1'
CONFIG_FILE = path.join(__dirname, 'etc', 'sender.conf.json');
var config = JSON.parse(
fs.readFileSync(CONFIG_FILE).toString()
);
var DATA_DIR = __dirname
config['data_dir'].forEach(function(dir) {
DATA_DIR = path.join(DATA_DIR, dir)
});
console.log('sending data from root directory: ' + DATA_DIR);
file.walk(
DATA_DIR,
function(err, dir_path, dirs, files) {
if(err) {
return console.error(err);
}
sendFiles(dir_path, files);
}
);
function sendFiles(dir_path, files)
{
files
.filter(function(file) {
return file.substr(-5) === '.meta';
})
.forEach(function(file) {
var name = path.basename(file.slice(0, -5));
sendFile(dir_path, name);
})
;
}
function sendFile(dir_path, name)
{
console.log("reading file start: " + dir_path + "/" + name);
fs.readFile(
path.join(dir_path, name + '.meta'),
function(err, raw_meta) {
if(err) {
return console.error(err);
}
console.log("reading file done: " + dir_path + "/" + name);
sendData(
name,
JSON.parse(raw_meta),
fs.createReadStream(path.join(dir_path, name + '.data'))
);
}
);
console.log("reading file async: " + dir_path + "/" + name);
}
function sendData(name, meta, data_stream)
{
meta['source'] = config['data_source'];
var req = request.post(
config['sink_url'],
function(err, res, body) {
if(err) {
console.log(err);
}
else {
console.log(name);
console.log(meta);
console.log(body);
}
}
);
var form = req.form();
form.append(
'meta',
JSON.stringify(meta),
{
contentType: 'application/x-www-form-urlencoded'
}
);
form.append(
'data',
data_stream
);
}
It works fine, when run with only a few files. But when I run it on directory with lots of files, it chokes. This is because it keeps creating huge amounts of tasks for reading from a file, but never gets to actually doing the reading (because there is too many files). This can be observed on output:
sending data from root directory: .../data
reading file start: .../data/ac/ad/acigisu-adruire-sabeveab-ozaniaru-fugeef-wemathu-lubesoraf-lojoepe
reading file async: .../data/ac/ad/acigisu-adruire-sabeveab-ozaniaru-fugeef-wemathu-lubesoraf-lojoepe
reading file start: .../data/ac/ab/acodug-abueba-alizacod-ugvut-nucom
reading file async: .../data/ac/ab/acodug-abueba-alizacod-ugvut-nucom
reading file start: .../data/ac/as/acigisu-asetufvub-liwi-ru-mitdawej-vekof
reading file async: .../data/ac/as/acigisu-asetufvub-liwi-ru-mitdawej-vekof
reading file start: .../data/ac/av/ace-avhad-bop-rujan-pehwopa
reading file async: .../data/ac/av/ace-avhad-bop-rujan-pehwopa
...
For each file, there is console output "reading file start" produced immediately before call to fs.readFile, and "reading file async" that is produced immediately after the async reading has been scheduled. But there is no "reading file done" message even when I let it run for a long time, which means that reading of any file has probably never been even scheduled (those files are on order of 100s of bytes, so once scheduled, those reads would probably finish in single go).
This leads me to the following thought process. Async calls in Node.js are done because the event loop itself is single-threaded and we do not want to block it. However, once this requirement is satisfied, does it make any sense to nest further async calls into async calls that are themselves nested in async calls, etc.? Would it serve any particular purpose? Moreover, would not it be actual pessimisation of the code due to scheduling overhead that is not really needed and can be completely avoided if complete handling of single file have consisted of synchronous calls only?
Given the thought process above, my course of action would be to use solution from this question:
asynchronously push names of all files to async.queue
limit number of parallel tasks by setting queue.concurrency
provide file-upload handler that is completely synchronous, i.e. it synchronously reads contents of the file and after that is finished, it synchronously sends POST request to the server
This is my very first try to use Node.js and/or JavaScript, therefore it is quite possible I am completely wrong (note that e.g. sync-request package makes it very clear that synchronous calls are not desirable, which is in contradiction with my thought process above - the question is why). Any comments on validity of the above thought process as well as viability of the proposed solution and eventual alternatives to it would be very much appreciated.
== Update ==
There is very good article explaining all this in great detail directly in documentation of Node.js.
As for the particular problem at hand, it is indeed in the choice of file-system-walker-module. The solution is to use e.g. walk instead of file:
## -4,7 +4,7 ##
var request = require('request');
-var file = require('file');
+var walk = require('walk');
var fs = require('fs');
var path = require('path');
## -24,13 +24,19 ## config['data_dir'].forEach(function(dir) {
console.log('sending data from root directory: ' + DATA_DIR);
-file.walk(
- DATA_DIR,
- function(err, dir_path, dirs, files) {
- if(err) {
- return console.error(err);
- }
- sendFiles(dir_path, files);
+var walker = walk.walk(DATA_DIR)
+walker.on(
+ 'files',
+ function(dir_path, files, next) {
+ sendFiles(dir_path, files.map(function(stats) { return stats.name; }));
+ next();
+ }
+);
+walker.on(
+ 'errors',
+ function(dir_path, node_stats, next) {
+ console.error('file walker:', node_stats);
+ next();
}
);
== Original Post ==
After a bit more study, I will attempt to answer my own question. This answer is still only a partial solution (more complete answer from someone who has actual experience with Node.js would be very much appreciated).
The short answer to the main question above is that it indeed is not only desirable, but also almost always necessary to schedule more asynchronous functions from already asynchronous functions. The long explanation follows.
It is because of how Node.js scheduling works: "Everything runs on a different thread except our code.". There are two very important comments in the discussion below the linked blog post:
"Javascript always finishes the currently executing function first. An event will never interrupt a function." [Twitchard]
"Also note it won't just finish the current function, it will run to completion of all synchronous functions and I believe anything queued with process.nextTick... before the request callback is handled." [Tim Oxley]
There is also a note mentioning this in the documentatoin of the process.nextTick: "The next tick queue is completely drained on each pass of the event loop before additional I/O is processed. As a result, recursively setting nextTick callbacks will block any I/O from happening, just like a while(true); loop."
So, to summarize, all code of the script itself is running on single thread and single thread only. The asynchronous callbacks scheduled to be run are executed on that very same single thread and they are executed only after whole current next tick queue has been drained. Use of asynchronous callbacks provide the only point, when some other function can be scheduled to be run. If the file-upload handler would not schedule any additional asynchronous tasks as described in the question, its execution would block everything else until that whole file-upload handler will have been finished. That is not desirable.
This also explains why the actual reading of the input file never occurs ("recursively setting nextTick callbacks will block any I/O from happening" - see above). It eventually would occur after all the tasks for whole directory hierarchy traversed will have been scheduled. However, without further study, I am not able to answer the question how to limit the number of file-upload tasks scheduled (effectively size of the task queue) and block the scheduling loop until some of those tasks will have been processed (some room on the task queue has been freed). Hence this answer is still incomplete.
Is it possible to write non-blocking response.write? I've written a simple test to see if other clients can connect while one downloads a file:
var connect = require('connect');
var longString = 'a';
for (var i = 0; i < 29; i++) { // 512 MiB
longString += longString;
}
console.log(longString.length)
function download(request, response) {
response.setHeader("Content-Length", longString.length);
response.setHeader("Content-Type", "application/force-download");
response.setHeader("Content-Disposition", 'attachment; filename="file"');
response.write(longString);
response.end();
}
var app = connect().use(download);
connect.createServer(app).listen(80);
And it seems like write is blocking!
Am I doing something wrong?
Update So, it doesn't block and it blocks in the same time. It doesn't block in the sense that two files can be downloaded simultaneously. And it blocks in the sense that creating a buffer is a long operation.
Any processing done strictly in JavaScript will block. response.write(), at least as of v0.8, is no exception to this:
The first time response.write() is called, it will send the buffered header information and the first body to the client. The second time response.write() is called, Node assumes you're going to be streaming data, and sends that separately. That is, the response is buffered up to the first chunk of body.
Returns true if the entire data was flushed successfully to the kernel buffer. Returns false if all or part of the data was queued in user memory. 'drain' will be emitted when the buffer is again free.
What may save some time is to convert longString to Buffer before attempting to write() it, since the conversion will occur anyways:
var longString = 'a';
for (...) { ... }
longString = new Buffer(longString);
But, it would probably be better to stream the various chunks of longString rather than all-at-once (Note: Streams are changing in v0.10):
var longString = 'a',
chunkCount = Math.pow(2, 29),
bufferSize = Buffer.byteLength(longString),
longBuffer = new Buffer(longString);
function download(request, response) {
var current = 0;
response.setHeader("Content-Length", bufferSize * chunkCount);
response.setHeader("Content-Type", "application/force-download");
response.setHeader("Content-Disposition", 'attachment; filename="file"');
function writeChunk() {
if (current < chunkCount) {
current++;
if (response.write(longBuffer)) {
process.nextTick(writeChunk);
} else {
response.once('drain', writeChunk);
}
} else {
response.end();
}
}
writeChunk();
}
And, if the eventual goal is to stream a file from disk, this can be even easier with fs.createReadStream() and stream.pipe():
function download(request, response) {
// response.setHeader(...)
// ...
fs.createReadStream('./file-on-disk').pipe(response);
}
Nope, it does not block, I tried one from IE and other from firefox. I did IE first but still could download file from firefox first.
I tried for 1 MB (i < 20) it works the same just faster.
You should know that whatever longString you create requires memory allocation. Try to do it for i < 30 (on windows 7) and it will throw FATAL ERROR: JS Allocation failed - process out of memory.
It takes time for memory allocation/copying nothing else. Since it is a huge file, the response is time taking and your download looks like blocking. Try it yourself for smaller values (i < 20 or something)