How to gzip http request post(client) data for node.js server - node.js

I have implemented node.js server application, which accepts post data from client(long json string). Is there a way I can gzip the post data at browser end and unzip it in node.js?
I specifically want to gzip the request and not response.

check https://jsfiddle.net/gynz82tg/
decompress in nodejs just same after you get the base64 encoded request string.
var jsonStr = JSON.stringify({
name: "JiangYD"
})
$('#origin').text(jsonStr);
var zip = new JSZip();
zip.file("data", jsonStr);
var content = zip.generate();
$('#compressed').text(content);
zip = new JSZip(content, {base64:true});
$('#decompressed').text(zip.file("data").asText());
<script src="https://raw.githubusercontent.com/Stuk/jszip/master/dist/jszip.js"></script>
<div id='origin'></div>
<div id='compressed'></div>
<div id='decompressed'></div>
UPDATE
because jsZip update the API
https://jsfiddle.net/cvuqr6h4/
async function go(){
const jsonStr = JSON.stringify({
name: "JiangYD"
})
$('#origin').text(jsonStr);
let zip = new JSZip();
zip.file("data", jsonStr);
const content = await zip.generateAsync({type : "base64"});
$('#compressed').text(content);
zip = new JSZip();
await zip.loadAsync(content, {base64:true});
const decoded = await zip.file("data").async('string');
$('#decompressed').text(decoded);
}
go();

You could try this: https://github.com/sapienlab/jsonpack
Example Client Code:
<script src="jsonpack.js" />
<script>
var BIG_JSON = {.....};
var packed = jsonpack.pack(BIG_JSON);
$.post('path_to_server',packed);
</script>
Example Nodejs Code:
var jsonpack = require('jsonpack/main');
app.on('/packed_data',function(req,res){
try{
jsonpack.unpack(req.data);
}catch(e){
//not good packed data.
}
})
This is a sample code of course i don't know what framework or libraries you use, but you can see how this could be implemented.
Anyway be careful with this because zipping and unzipping data is always a heavy cpu bound task. If you have several megabytes of data you dont want to make your users from phones,tablets etc.. to make this tasks!

Related

Create a Blob for Video in NodeJs18 and Use it on Client Side

Learning NodeJs and Blob, and I Try to Create a Blob from a Video File in NodeJS. And send this in a Json file to my Client. Data will be Fetch inside GetStaticProps in NextJS.
Here is What I Created in NodeJS Server :
const fileBuffer = fs.readFileSync(filePath); // Path is something like example.com/video.mp4
const blob = new Blob([fileBuffer], { type: 'video/mp4' });
blobUrl = URL.createObjectURL(blob); // Return blob:nodedata:c3b1baf2-fba8-404f-8d3c-a1184a3a6db2
It retrun this :
blob:nodedata:c3b1baf2-fba8-404f-8d3c-a1184a3a6db2
But How can I use it in Client ? <video src="blob:nodedat..." is not Working, so I am doing something wrong for sure
Can you help me uderstand what is wrong ?

Serve clickable download URL in NodeJS

At my endpoint in my NodeJS server, after retrieving an audio file stored as a Buffer in MongoDB, I want to represent it with a URL (much like how you do with URL.createObjectURL(blob) in the frontend on the browser). I then plan to res.render() the URL in HTML through Handlebars on the client, so that the user can click on it to download it:
<a href={{url}}>Click me to download the file!</a>
In the NodeJs server, I have converted the MongoDB Buffer into a JavaScript ArrayBuffer through:
var buffer = Buffer.from(recordingFiles[0].blobFile);
var arrayBuffer = Uint8Array.from(buffer).buffer;
I am unsure where to proceed from here. I seen solutions using fs or res.download(), but they don't seem applicable to my situation. Thanks in advance for any help!
Hopefully this can help.
var blob = new Blob(BUFFER, {type: "audio mime type"});
var link = document.createElement('a');
link.href = window.URL.createObjectURL(blob);
var fileName = reportName;
link.download = fileName;
link.click();
Do you always need to preload the audio file onto the page?
If not, then I would advise you to add a separate endpoint to download the file on demand. The frontend link can send a get request to the endpoint and download the file only if the user clicked it.
Otherwise you'd always be downloading the buffer behind the scenes, even if the user didn't intend to download it. This is especially problematic on slow connections.
Frontend:
<a href={{`${baseUrl}/download/${audioId}`}}>Click me to download the file!</a>
Backend:
const stream = require('stream');
app.get('/download/:audioId', function (request, response) {
// Retrieve the tag from our URL path
const audioId = request.params.audioId;
const fileData; // TODO: Get file buffer from mongo.
const fileContents = Buffer.from(fileData, 'base64');
const readStream = new stream.PassThrough();
readStream.end(fileContents);
response.set('Content-disposition', 'attachment; filename=' + fileName);
response.set('Content-Type', '<your MIME type here>');
readStream.pipe(response);
});
A list of relevant MIME types can be found here.

XML scraping using nodeJs

I have a very huge xml file that I got by exporting all the data from tally, I am trying to use web scraping to get elements out of my code using cheerio, but I am having trouble with the formatting or something similar. Reading it with fs.readFileSync() works fine and the console.log shows complete xml file but when I write the file using the fs.writeFileSync it makes it look like this:
And my web scraping code outputs empty file:
const cheerio = require('cheerio');
const fs = require ('fs');
var xml = fs.readFileSync('Master.xml','utf8');
const htmlC = cheerio.load(xml);
var list = [];
list = htmlC('ENVELOPE').find('BODY>TALLYMESSAGE>STOCKITEM>LANGUAGENAME.LIST>NAME.LIST>NAME').each(function (index, element) {
list.push(htmlC(element).attr('data-prefix'));
})
console.log(list)
fs.writeFileSync("data.html",list,()=>{})
You might try checking to make sure that Cheerio isn't decoding all the HTML entities. Change:
const htmlC = cheerio.load(xml);
to:
const htmlC = cheerio.load(xml, { decodeEntities: false });

Converting blob from database to an image javascript

I have a database, where one of the tables has a blob field and I want to display it as an image. I can't really find any solution for this - any working npm package or a sample of code would be useful. I'm using reactjs and nodejs.
What you want to do is create an URL that you can pass to the img src of the HTML img
JS
var url = window.URL || window.webkitURL;
var imageSrc = url.createObjectURL('your blob');
document.querySelector("#myimage").src = imageSrc;
HTML
<img id="myimage"/>
Method 1 create readable stream and pipe to response
var stream = require('stream');
var bufferStream = new stream.PassThrough();
bufferStream.end(new Buffer( blob, 'binary' ));
res.writeHead(200, {
'Content-Type' : 'image/jpg'
});
// res is standered express res object
bufferStream.pipe(res)
Method 2
pass blob to base64
var bufferBase64 = new Buffer( blob, 'binary' ).toString('base64');
show image
class Example extends React.Component{
render() {
return <img src={"data:image/jpeg;" + bufferBase64} />
}
}
reference
https://stackoverflow.com/.../how-to-create-a-readstream-with-a-buffer-using-nodejs

Uploading a file and sending it to the backend with React and Node

I need to upload a file in react and send it to the Node backend.
Since I never worked with uploading and sending files before, this is a little troubling for me.
So far I found this:
// this creates a React component that can be used in other components or
// used directly on the page with React.renderComponent
var FileForm = React.createClass({
// since we are starting off without any data, there is no initial value
getInitialState: function() {
return {
data_uri: null,
};
},
// prevent form from submitting; we are going to capture the file contents
handleSubmit: function(e) {
e.preventDefault();
},
// when a file is passed to the input field, retrieve the contents as a
// base64-encoded data URI and save it to the component's state
handleFile: function(e) {
var self = this;
var reader = new FileReader();
var file = e.target.files[0];
reader.onload = function(upload) {
self.setState({
data_uri: upload.target.result,
});
}
reader.readAsDataURL(file);
},
// return the structure to display and bind the onChange, onSubmit handlers
render: function() {
// since JSX is case sensitive, be sure to use 'encType'
return (
<form onSubmit={this.handleSubmit} encType="multipart/form-data">
<input type="file" onChange={this.handleFile} />
</form>
);
},
});
Source: https://fitacular.com/blog/react/2014/06/23/react-file-upload-base64/
But now I basically just end up with some sort of string. But I need to send that file via REST to my Express backend, which needs to save that file in CouchDB.
What is the best/easiest way to accomplish that?
If you are using body-parser, know that it handles json and url encoded forms, not multipart data !
You should use an other module.
For more infos, give a look at : File uploading with Express 4.0: req.files undefined

Resources