Node.js cuts off files when serving over HTTPS - node.js

I am trying to serve some JavaScript files with Node.js and for whatever reason the files are being cut off in the middle of the transmission. The code:
httpsServer = http.createServer(function(req, res) {
var path = url.parse(req.url).pathname;
if (path[path.length - 1] == '/') {
path += 'index.html';
}
fs.readFile(root + path, function(err, data){
if (err) return send404(res);
res.writeHead(200, {
'Content-Type': getMimeType(getExtension(path)),
'Content-Length': data.length});
res.end(data);
});
}),
var privateKey = fs.readFileSync(settings.PRIVATE_KEY).toString();
var certificate = fs.readFileSync(settings.CERTIFICATE).toString();
var credentials = crypto.createCredentials({key: privateKey, cert: certificate});
httpsServer.setSecure(credentials);
httpsServer.listen(settings.HTTPS_PORT);
The files http://github.com/LearnBoost/Socket.IO/raw/master/socket.io.js and http://code.jquery.com/jquery-1.4.2.min.js. The first one is cut off at exactly 32KB and the second at exactly 16KB. This does not happen over HTTP, only HTTPS and only over a network (e.g.: not from localhost).
Any help would be really appreciated.

Instead of Content-Length: data.length you should use Content-Length: Buffer.byteLength(data, 'utf8') where Bufferis a global object (node 0.3.x) or var Buffer = require('buffer') in node 0.2.x which will save you a lot of hassle and might fix your problem with truncated answers too.

I have just seen this too. Same setup - HTTPS, latest node from the git repo.
One large file (170k) never properly completing sending. I tried to switch from async to synch but it made no difference. Only thing that fixed it so far was making the file smaller. It was a big floppy jpg so it was easy to do. Problem vanished.

It might be a disconnect between the default encoding for readFile and res.end(). readFile just loads the raw data if you do not specify an encoding whereas end() defaults to utf8. I'm not 100% sure but the raw data from the file could return a shorter length than the utf8 encoded string that end emits.
I tried recreating your error and failed so it might be you need to upgrade your version of node. I'm using the latest code from github.

This problem should be solved in the newest version of Node.js. Can you test it out on v0.4.2?

have you tried to use built-in HTTPS server?
http://nodejs.org/docs/v0.4.11/api/https.html

Related

Slowness with Node.js FS : how can I list files faster?

I just want to read files names from a dir :
const fs = require('fs');
fs.readdir("repo/_posts", (err, files) => {
files.forEach(file => {
res.write(file + "\n");
});
})
With only a 15 files, it is very slow, it takes several seconds to display the file names. What did I do wrong?
Edit : as suggested by #Darin Dimitrov in the comments, I've tried to replace res.write by console.log, then it's fast. Is res.write a bad practice in a loop or something similar?
Thanks :)
Most browsers will buffer output received from the server for a variety of reasons (including content encoding detection) and some of them may buffer more than others. If you can access the same url via a utility such as cURL and you see the expected output in a much more timely manner, then this confirms the browser buffering "issue."

How can I stream multiple remote images to a zip file and stream that to browser with ExpressJS?

I've got a small web app built in ExpressJs that allows people in our company to browse product information. A recent feature request requires that users be able to download batches of images (potentially hundreds at a time). These are stored on another server.
Ideally I think I need to to stream the batch of files to a zip file and stream that to the end user's browser as a download. All preferably without having to store the files on the server. The idea being that I want to reduce load on the server as much as possible.
Is it possible to do this or do I need to look at another approach? I've been experimenting with the 'request' module for the initial download.
If anyone can point me in the right direction or recommend any NPM modules that might help it would be very much appreciated.
Thanks.
One useful module for this is archiver, but I'm sure there are others as well.
Here's an example program that shows:
how to retrieve a list of URL's (I'm using async to handle the requests, and also to limit the # of concurrent HTTP requests to 3);
how to add the responses for those URL's to a ZIP file;
to stream the final ZIP file somewhere (in this case to stdout, but in case of Express you can pipe to the response object).
Example:
var async = require('async');
var request = require('request');
var archiver = require('archiver');
function zipURLs(urls, outStream) {
var zipArchive = archiver.create('zip');
async.eachLimit(urls, 3, function(url, done) {
var stream = request.get(url);
stream.on('error', function(err) {
return done(err);
}).on('end', function() {
return done();
});
// Use the last part of the URL as a filename within the ZIP archive.
zipArchive.append(stream, { name : url.replace(/^.*\//, '') });
}, function(err) {
if (err) throw err;
zipArchive.finalize().pipe(outStream);
});
}
zipURLs([
'http://example.com/image1.jpg',
'http://example.com/image2.jpg',
...
], process.stdout);
Do note that although this doesn't require the image files to be locally stored, it does build the ZIP file entirely in memory. Perhaps there are other ZIP modules that would allow you to work around that, although (AFAIK) the ZIP file format isn't really great in terms of streaming, as it depends on metadata being appended to the end of the file.

Getting the mime type from a request in nodeJS

Im learning nodejs but I ran into a roadblock. Im trying to setup a simple server that will serve static files. My problem is that unless I explicitly type in the extension in the url, I cannot get the file extension. The httpheader 'content-type' comes in as undefined .
Here is my code, pretty simple:
var http = require("http"),
path = require("path"),
fs = require("fs");
var server = http.createServer(function(request, response){
console.log([path.extname(request.url), request.headers['content-type']]);
var fileName = path.basename(request.url) || "index.html",
filePath = __dirname + '/public/' + fileName;
console.log(filePath);
fs.readFile( filePath, function(err,data){
if (err) throw err
response.end(data);
});
})
server.listen(3000)
Any ideas?
FYI I dont just wanna dive into connect or other, I wanna know whats going on before I drop the grind and go straight to modules.
So static web servers generally don't do any deep magic. For example, nginx has a small mapping of file extensions to mime types here: http://trac.nginx.org/nginx/browser/nginx/conf/mime.types
There's likely also some fallback logic defaulting to html. You can also use a database of file "magic numbers" as is used by the file utility to look at the beginning of the file data and guess based on that.
But there's no magic here. It's basically
go by the file extension when available
maybe go by the beginning of the file content as next option
use a default of html because normally only html resources have URLs with no extensions, whereas images, css, js, fonts, multimedia, etc almost always do use file extensions in their URIs
Also note that browsers generally have fairly robust set of checks that will intepret files correctly even when Content-Type headers are mismatched with the actual response body data.

Send file to browser from string using socket.io

I am using PDFKit and socket.io in a node.js project to generate a pdf when a user clicks a button on the front end. How do I stream or otherwise send the resulting pdf to the end-user from here? I'd rather avoid saving the file to the file system and then having to delete it later if I can... hoping to stream it somehow.
socket.on('customerRequestPDF', function(){
doc = new PDFDocument;
doc.text('Some text goes here', 100, 100);
//I could do this but would rather avoid it
doc.write('output.pdf');
doc.output(function(string) {
//ok I have the string.. now what?
});
});
A websocket isn't really the appropriate mechanism to deliver the PDF. Just use a regular HTTP request.
// assuming Express, but works similarly with the vanilla HTTP server
app.get('/pdf/:token/filename.pdf', function(req, res) {
var doc = new PDFDocument();
// ...
doc.output(function(buf) { // as of PDFKit v0.2.1 -- see edit history for older versions
res.writeHead(200, {
'Content-Type': 'application/pdf',
'Cache-Control': 'private',
'Content-Length': buf.length
});
res.end(buf);
});
});
Now a word of warning: This PDF library is broken. As of version 0.2.1, the output is a proper Buffer, but it uses the deprecated binary string encoding internally instead of Buffers. (Previous versions gave you the binary-encoded string.) From the docs:
'binary' - A way of encoding raw binary data into strings by using only the first 8 bits of each character. This encoding method is deprecated and should be avoided in favor of Buffer objects where possible. This encoding will be removed in future versions of Node.
This means that when node removes the binary string encoding, the library will stop working.

Node.js, nodetuts tutorial - example code will run, timeout after a while,(Chunked HTTP encoding)

This is a node tutorial for a beginner. Why doesn't my code work? It's identical to his and I have tried workarounds without success. Please help if you can.
http://nodetuts.com/tutorials/2-webtail-nodejs-child-processes-and-http-chunked-encoding.html#video
In this tutorial, he writes a log/text file to the webpage located at localhost:4000, previous examples work (tutorial 1) however I can't get this to do anything at all, it runs and that's all.
Could anyone get this printing a file to the webpage please. :)
Thanks!
var http = require('http');
var spawn = require('child_process').spawn;
http.createServer(function(request, response){
response.writeHead(200, {
'Content-Type' : 'text/plain'
});
var tail_child = spawn('tail', ['-f', '/var/log/system.log']);
request.connection.on('end', function(){
tail_child.kill();
});
tail_child.stdout.on('data', function(data){
console.log(data.toString());
response.write(data);
});
}).listen(4000);
I have tried the following and they are making no difference:
console.log(data.toString());
response.write(data);
response.end();
and
console.log(data.toString());
response.end(data);
Edit after MiguelSanchezGonzalez answer:
I added tail_child.stderr.pipe(process.stdout); into the right place, and this is the response I am getting:
CreateProcessW: The system cannot find the file specified.
A file certainly does exist here, and I have also tested many different paths including text files in the same folder as the script (such as '/test.txt' for example. So maybe i'm wrong, i just assumed when it said file it was talking about my file.
Edit 2:
I also checked if the path exists (crudely built from here): Fastest way to check for existence of a file in NodeJs and it does say that the file indeed exists.
This code:
var file = path.normalize = ('var/log/system.log');
fs.exists(file, function(exists){
util.debug(exists ? "yep, its there":"nope");
});
console.log(file);
outputs this:
var/log/system.log
DEBUG: yep, its there
OK, i'm so new to this and it shows, thanks everyone for helping me I have now found the cause. I was stupid and did not mention i'm on windows, I thought this was irrelevant as I had no idea this code was making a external call to another program.
Yeah I know, stupid Joe, I thought tail was just a command within node and didn't think properly. This mess makes sense now.
var tail_child = spawn('C:/cygwin/bin/tail.exe', ['-f', 'var/log/system.log']);
This fixes everything, install cygwin and hardlink that binary!
This code was meant for a linux/unix environment I guess, so we just have to make do.
I was talking with the author of nodetuts, (Pedro) and I mentioned I was on a windows box, Bam it all made sense, the code makes more sense to me now too (I was searching for ages on node documentation for -f and couldn't find much. haha)
I should have put windows in the description of my environment.
Oh well.
Silly Joe.
Hy Joseph, I recomend to you to change this line in your code to see what kind of error do you have
http.createServer(function(request, response){
response.writeHead(200, {
'Content-Type' : 'text/plain'
});
var tail_child = spawn('tail', ['-f', '/var/log/system.log']);
request.connection.on('end', function(){
tail_child.kill();
});
tail_child.stdout.on('data', function(data){
console.log(data.toString());
response.write(data);
});
/* Add this line to see the error in your terminal */
tail_child.stderr.pipe(process.stdout);
}).listen(4000);
This will show you the description of the error, maybe you don't have this file in your filesystem o you can't open it, try changing the path file to other that you know that exists and can open it.

Resources