I am a newbie to javascript.
What i am trying to do is to fetch data from the data base and then transmit it on the internet.
Now i can only read one entry at a time but i want to compress all the entries together rather than compressing one entry at a time.
I can either store all of them in an array and then pass this array to zlib function. but this take up alot of time and memory.
Is it somehow possible to compress the data while transmitting it in node js with express api at the same time as it is being read, sort of like streaming servers, who on real time compress data while retrieving it from memory and then transmitting it over to the client
It's certainly possible. You can play around with this example:
var express = require('express')
, app = express()
, zlib = require('zlib')
app.get('/*', function(req, res) {
res.status(200)
var stream = zlib.createGzip()
stream.pipe(res)
var count = 0
stream.write('[')
;(function fetch_entry() {
if (count > 10) return stream.end(']')
stream.write((count ? ',' : '') + JSON.stringify({
_id: count,
some_random_garbage: Math.random(),
}))
count++
setTimeout(fetch_entry, 100)
})()
})
app.listen(1337)
console.log('run `curl http://localhost:1337/ | zcat` to see the output')
I assume you're streaming JSON, and setTimeout calls would need to be replaced with actual database calls of course. But the idea stays the same.
I'd recommend to use node.js's pipe.
Here is an example of pipe streaming with zlib (compression): it reads a file, compresses it and writes it to a new file.
var gzip = zlib.createGzip();
var fs = require('fs');
var inp = fs.createReadStream('input.txt');
var out = fs.createWriteStream('input.txt.gz');
inp.pipe(gzip).pipe(out);
You can change the input to come from your database input and change the output to be the HTTP response.
ref : http://nodejs.org/api/stream.html
ref : http://nodejs.org/api/zlib.html
Related
I'm working on a simple local image server that provides images to a web application with some JSON. The web application has pagination that will do a get request "/images?page=X&limit&200" to an express.js server that returns the JSON files in a single array. I want to take advantage of the browser's internal caching such that if a user goes to a previous page the express.js returns an ETAG. I was wondering how this could be achieved with express.js? For this application, I really just want the computation of the ETAG to take in three parameters the page, the directory, and the limit (It doesn't need to consider the whole JSON body). Also this application is for local use only, so I want the server to do the heavy lifting since I figured it be faster than the browser. I did see https://www.npmjs.com/package/etag which seems promising, but I'm not sure how to use it with express.js
Here's a boilerplate of the express.js code I have below:
var express = require('express');
var app = express();
var fs = require('fs');
app.get('/', async (req, res) =>{
let files = [];
let directory = fs.readdirSync("mypath");
let page = parseInt(req.query.page);
let limit = parseInt(req.query.limit);
for (let i = 0; i < limit; ++i) {
files.push(new Promise((resolve) => {
fs.readFile(files[i + page * limit].name, (err, data) => {
// format the data so easy to use for UI
resolve(JSON.parse(data));
});
});
}
let results = await Promise.all(files);
// compute an etag here and attach it the results.
res.send(results);
});
app.listen(3000);
When your server sends an ETag to the client, it must also be prepared to check the ETag that the client sends back to the server in the If-None-Match header in a subsequent "conditional" request.
If it matches, the server shall respond with status 304; otherwise there is no benefit in using ETags.
var serverEtag = "<compute from page, directory and limit>";
var clientEtag = req.get("If-None-Match");
if (clientEtag === serverEtag) res.status(304).end();
else {
// Your code from above
res.set("ETag", serverEtag);
res.send(results);
}
The computation of the serverEtag could be based on the time of the last modification in the directory, so that it changes whenever any of the images in that directory changes. Importantly, this could be done without carrying out the fs.readFile statements from your code.
I'm trying to analyze a file I'll be uploading from react, I need to know if it can be uploaded based on several factors.
I found https://github.com/TooTallNate/node-wav
It works great on nodejs and I'm trying to use it on react. The sample creates a readable stream and pipes it to the wav reader.
var fs = require('fs');
var wav = require('wav');
var file = fs.createReadStream('track01.wav');
var reader = new wav.Reader();
// the "format" event gets emitted at the end of the WAVE header
reader.on('format', function (format) {
//Format of the file
console.log(format);
});
file.pipe(reader);
Using FilePond controller I'm able to get a base64 string of the file. But I can't figure out how to pass it to the reader
this is what I have so far on ReactJS:
var reader = new wav.Reader();
reader.on('format', function (format) {
//Format of file
console.log('format', format);
});
const buffer = new Buffer(base64String, 'base64')
const readable = new Readable()
readable._read = () => { }
readable.push(buffer)
readable.push(null)
readable.pipe(reader)
But I get Error: bad "chunk id": expected "RIFF" or "RIFX", got "u+Zj"
Since this file works on NodeJS with the same lib is obvious I'm doing something wrong.
EDIT:
this was a problem with my Base64 string, this method works if anyone needs to analyze a wav on the frontend
I have a paginated request that gives me a list of objects, which I later concat to get the full list of objects.
If I attempt to JSON.stringify this, it fails for large objects with range error. I was looking for a way to zlib.gzip to handle large JSON objects.
Try installing stream-json it will solve your problem, It's a great wrapper around streams and parsing a JSON.
//require the modules stream-json
const StreamArray = require('stream-json/utils/StreamArray');
// require fs if your using a file
const fs = require('fs');
const zlib = require('zlib');
// Create an instance of StreamArray
const streamArray = StreamArray.make();
fs.createReadStream('./YOUR_FILE.json.gz')
.pipe(zlib.createUnzip()) // Unzip
.pipe(streamArray.input); //Read the stream
//here you can do whatever you want with the stream,
//you can stream it to response.
streamArray.output.pipe(process.stdout);
In the example, I'm using a JSON (file) but you can use a collection and pass it to the stream.
Hope that's help.
I'm not sure how to create a blob from a wav file in node. Do I just use Buffer like so?...
var blippityBlob = new Buffer(filePathToWave);
Maybe you could take a look at BinaryJS
Quoting:
BinaryJS is a lightweight framework that utilizes websockets to send, stream, and pipe binary data bidirectionally between browser javascript and Node.js.
Server Code
var server = BinaryServer({port: 9000});
server.on('connection', function(client){
client.on('stream', function(stream, meta){
var file = fs.createWriteStream(meta.file);
stream.pipe(file);
});
});
Client Code
var client = BinaryClient('ws://localhost:9000');
client.on('open', function(stream){
var stream = client.createStream({file: 'hello.txt'});
stream.write('Hello');
stream.write('World!');
stream.end();
});
The answer lies in a combination of these two posts:
Node.js can´t create Blobs?
Convert a binary NodeJS Buffer to JavaScript ArrayBuffer
I'm trying to create a simple app which will create ZIP on the fly containg a few files and stream them to the client. Almost works. The problem what I encountered is that the nodejs blocks when streaming is in progress. All pendig requests will be processed after finishing current streaming. But what is interesting, these pending requests will be processed concurrently! So it can do this! I don't understand, why nodejs (expressjs?) can't start processing next request during streaming one file and what I did wrong... :|
I'm using node-archiver as a stream source.
Here's my part of code:
var express = require('express');
var archiver = require('archiver');
var router = express.Router();
router.get('/job/:id', function(req, res) {
var job = req.jobs[req.params.id];
var archive = archiver('zip');
archive.pipe(res);
for (var n in job.files) {
var f = job.files[n];
archive.file(job.path + f, {name: f});
}
res.setHeader("content-type", "application/zip");
res.setHeader("Content-Disposition", 'attachment; filename="reports.zip"')
archive.finalize();
});
module.exports = router;
Any advices? Thanks!
EDIT: I've noticed another problem, completely not related with archiver. I have following basic app:
var http = require('http');
var fs = require('fs');
var server = http.createServer(function (req, res) {
var stream = fs.createReadStream('file.blob');
stream.pipe(res);
});
server.listen(31922);
Tell me, why it get stuck in this case? Result is absolutely the same as using archiver. OS what I use is SmartOS (based on Open Solaris), it's Unix. Maybe this is a problem? Any ideas?
For all those of you struggling with similar problem. I'm being dumb perhaps. The solution is simple. Basically testing method was wrong. My browser (and other tested have similar behaviour) blocks processing next request from the same host name when previous one is not finished. And in this case this is not finished request as downloading is still in progress. Thanks for your help!
It seems that Archiver depends on synchronous code, there is recent issue open on github addressing this:
https://github.com/ctalkington/node-archiver/issues/85
Given that it is synchronous, that's probably where your block is comming from.
Quote:
This module depends on file-utils which in the README says: "this is a
set of synchronous utility. As so, it should never be used on a
Node.js server. This is meant for users/command line utilities."