I'm trying to create a simple app which will create ZIP on the fly containg a few files and stream them to the client. Almost works. The problem what I encountered is that the nodejs blocks when streaming is in progress. All pendig requests will be processed after finishing current streaming. But what is interesting, these pending requests will be processed concurrently! So it can do this! I don't understand, why nodejs (expressjs?) can't start processing next request during streaming one file and what I did wrong... :|
I'm using node-archiver as a stream source.
Here's my part of code:
var express = require('express');
var archiver = require('archiver');
var router = express.Router();
router.get('/job/:id', function(req, res) {
var job = req.jobs[req.params.id];
var archive = archiver('zip');
archive.pipe(res);
for (var n in job.files) {
var f = job.files[n];
archive.file(job.path + f, {name: f});
}
res.setHeader("content-type", "application/zip");
res.setHeader("Content-Disposition", 'attachment; filename="reports.zip"')
archive.finalize();
});
module.exports = router;
Any advices? Thanks!
EDIT: I've noticed another problem, completely not related with archiver. I have following basic app:
var http = require('http');
var fs = require('fs');
var server = http.createServer(function (req, res) {
var stream = fs.createReadStream('file.blob');
stream.pipe(res);
});
server.listen(31922);
Tell me, why it get stuck in this case? Result is absolutely the same as using archiver. OS what I use is SmartOS (based on Open Solaris), it's Unix. Maybe this is a problem? Any ideas?
For all those of you struggling with similar problem. I'm being dumb perhaps. The solution is simple. Basically testing method was wrong. My browser (and other tested have similar behaviour) blocks processing next request from the same host name when previous one is not finished. And in this case this is not finished request as downloading is still in progress. Thanks for your help!
It seems that Archiver depends on synchronous code, there is recent issue open on github addressing this:
https://github.com/ctalkington/node-archiver/issues/85
Given that it is synchronous, that's probably where your block is comming from.
Quote:
This module depends on file-utils which in the README says: "this is a
set of synchronous utility. As so, it should never be used on a
Node.js server. This is meant for users/command line utilities."
Related
I have the following code written with NodeJS and ExpressJS:
const express = require("express");
const fs = require("fs");
const bodyParser = require("body-parser");
const jsonParser = bodyParser.json();
const hostname = "127.0.0.1";
let port = 3001;
const app = express();
app.use(express.static(__dirname + "/answers"));
const answersPath = __dirname + "/answers/answers.json";
app.patch("/new/answer", jsonParser, function (req, res) {
try {
const questionId = req.body.questionId;
const answer = req.body.answer;
const answersJson = JSON.parse(fs.readFileSync(`${answersPath}`, "utf8"));
if (answersJson[questionId]) {
answersJson[questionId] = [...answersJson[questionId], answer];
} else {
answersJson[questionId] = [answer];
}
fs.writeFileSync(`${answersPath}`, JSON.stringify(answersJson));
res.sendStatus(200);
} catch (e) {
console.error(e);
res.sendStatus(500);
}
});
app.listen(port);
console.log(`Server running at http://${hostname}:${port}/`);
What it basically does, it has an endpoint (/new/question), on which it receives as a JSON format, a question and an answer.
If the question exists already in the answers.json file, it adds the new answer to the list of answers for that question. If not, it creates a new question with a list of the answer.
Now, I've read the following article: https://www.geeksforgeeks.org/how-to-handle-concurrency-in-node-js/
And what I understood from here, is that even though the endpoint would get called at the same time by two clients, both of the responses will be saved, one after the other - one of them will wait for the other one, i.e. the file will not get overwritten.
So my question is, is this true? NodeJS deals with concurrency on its own, or do I need to implement something to prevent this from happening?
Thank you, and sorry if this is a dumb question 😞.
Although readFileSync() and writeFileSync() might do what you want to achieve, you should avoid using synchronous functions in Node.js.
Synchronous functions will block the entire Node.js process, not just the a single Express route. This means your server will become unresponsive while reading or writing the file. This will become an issue if the file gets bigger.
Instead of using a file, you could keep the data only in memory. If you need to persist the data between server restarts, you can read it when the server starts and write it when the server stops. In this case it might be okay to use synchronous functions.
I'm currently running Node.js by Browserify for my website.
It reads the JSON file and I get the message through MQTT.
But the problem is that it seems like writefile does not work.
(Running this as node test.js in the terminal works by the way).
What is wrong with my code?
Moreover, Is this the best way to store any user data?
Thank you so much in advance.
Here's some part of my code
var fs = require("fs");
var path = require("path");
let newFile = fs.readFileSync('/home/capstone/www/html/javascript/test.json');
function testT() { //THIS WORKS FINE
let student0 = JSON.parse(newFile);
var myJSON = JSON.stringify(student0);
client.publish("send2IR", myJSON);
response.end();
};
function write2JSON() { //PROBLEM OF THIS CODE
const content = 'Some content!'
fs.writeFileSync('/home/capstone/www/html/javascript/test.json', content)
};
document.getElementById("blink").addEventListener("click", publish);
document.getElementById("write").addEventListener("click", write2JSON);
You cann't write directly for security reasons. For other hand you can use a server as API to do the filye system tasks and in the client only trigger the events.
This post is very related with your problem:
Is it possible to write data to file using only JavaScript?
I want to print a message whenever the file that I am watching has changed. I am able to do that using console.log but I couldn't figure out how to do that using response.write or similar functions.
var counter = 0;
const
http = require('http'),
fs = require('fs'),
filename = process.argv[2];
var server = http.createServer(function(request, response) {
response.writeHead(200, { 'Content-Type' : 'text/plain' });
counter = counter + 1;
response.end('Hello client ' + Math.round(counter / 2));
});
server.on('listening', function() {
var watcher = fs.watch(filename, function(){
console.log('The file ' + filename + ' has just changed.');
});
});
server.listen(8080);
Also, the reason why I have done that Math.round(counter / 2) is because counter is increasing by 2 each time a client is connected. I was wondering why this is happening and if there is a better technique to resolve this.
For you to be able to do it using response.write it would need to be part of your server request handler function.
File events can occur independently of someone sending a request, so it's handling is independent to that of handling a request. Because of this, there is no associated request for you to write to.
If you want to keep track of all the file change events and then show it to the user whenever they do send a request, consider storing the information about changes in an object outside your handler functions and when the a request takes place, read that object to see if there have been changes and write a response based on it to the user.
If you want to inform an end user that the file has change, for example in a web browser, then you have a number of options, including polling the server or using websockets.
I would recommend you take a look at Server Sent Events
It is easy to implement and there are npm module out there to make it even easier in node.js. e.g. npm sse
You can try node module chokidar
https://github.com/paulmillr/chokidar
it is a gread module for file watching
I am a newbie to javascript.
What i am trying to do is to fetch data from the data base and then transmit it on the internet.
Now i can only read one entry at a time but i want to compress all the entries together rather than compressing one entry at a time.
I can either store all of them in an array and then pass this array to zlib function. but this take up alot of time and memory.
Is it somehow possible to compress the data while transmitting it in node js with express api at the same time as it is being read, sort of like streaming servers, who on real time compress data while retrieving it from memory and then transmitting it over to the client
It's certainly possible. You can play around with this example:
var express = require('express')
, app = express()
, zlib = require('zlib')
app.get('/*', function(req, res) {
res.status(200)
var stream = zlib.createGzip()
stream.pipe(res)
var count = 0
stream.write('[')
;(function fetch_entry() {
if (count > 10) return stream.end(']')
stream.write((count ? ',' : '') + JSON.stringify({
_id: count,
some_random_garbage: Math.random(),
}))
count++
setTimeout(fetch_entry, 100)
})()
})
app.listen(1337)
console.log('run `curl http://localhost:1337/ | zcat` to see the output')
I assume you're streaming JSON, and setTimeout calls would need to be replaced with actual database calls of course. But the idea stays the same.
I'd recommend to use node.js's pipe.
Here is an example of pipe streaming with zlib (compression): it reads a file, compresses it and writes it to a new file.
var gzip = zlib.createGzip();
var fs = require('fs');
var inp = fs.createReadStream('input.txt');
var out = fs.createWriteStream('input.txt.gz');
inp.pipe(gzip).pipe(out);
You can change the input to come from your database input and change the output to be the HTTP response.
ref : http://nodejs.org/api/stream.html
ref : http://nodejs.org/api/zlib.html
I'm trying to figure out a few simple best practices when it comes to structuring a nodeJS server object. Please note that I'm coming from a LAMP background, so the whole thing is somewhat of a paradigm shift for me.
Static Content
Static content has documented examples, and works like a charm:
var http = require('http'),
url = require('url'),
fs = require('fs'),
sys = require(process.binding('natives').util ? 'util' : 'sys');
server = http.createServer(function(req, res){
var parsedURL = url.parse(req.url, true);
var path = parsedURL.pathname;
var query = parsedURL.query;
switch (path){
case '/':
res.writeHead(200, {'Content-Type': 'text/html'});
res.write('<h1>Greetings!</h1>');
res.end();
break;
default:
fs.readFile(__dirname + path, function(err, data){
if (err) return send404(res);
res.writeHead(200, {'Content-Type':'text/html'})
res.write(data, 'utf8');
res.end();
});
}
}).listen(80);
send404 = function(res){
res.writeHead(404);
res.write('404');
res.end();
};
The server listens for requests, looks up the files, and shoots the content of those files back to the client. Obviously the example I gave is quite dumb and doesn't account for files that aren't text/html, but you get the idea.
Dynamic Content
But what if we don't want to serve static content? What if we want to, for instance, have a hello world file which takes in a value from the querystring and responds differently.
My first guess is that I should create a second file using the nodeJS module setup, give it some module methods which take in information, and just concatenate a crap load of strings together.
For instance, a hello world module called hello.js:
exports.helloResponse = function( userName ) {
var h = "<html>";
h += "<head><title>Hello</title></head>";
h += "<body>Hello, " + userName +"</body>";
h += "</html>";
}
and then add the following case to the server handler:
case 'hello':
res.writeHead(200, {'Content-Type':'text/html'})
res.write(require("./hello.js").helloResponse(query["userName"]), 'utf8');
res.end();
I'm OK with the module, but I hate the fact that I have to create a giant concatenated string in javascript. Does the S.O. community have any ideas? What approaches have you taken to this situation?
Node.js is a "bare bones" platform (not unlike Python or Ruby), which makes it very flexible.
You will benefit from using Node.js with a web framework such as Express.js to handle the grunt work. Express.js uses another module, Connect.js, to provide routing, configuration, templating, static file serving, and more.
You will also want a template language. EJS, for example, gives you a PHP/JSP-style templating language, while Jade provides a Haml-esque approach.
Explore the list of frameworks and templating engines on the Node.js modules page and see which you prefer.
Node.js is not bare bones any more. Use a framework of you choice. Take a look at least at express and socketstream frameworks. Also take a look at socket.io as a replacement for AJAX.