Is there any chance to store Buffer data with a string data together on Redis.
Pseudo code:
// Data compression
var user = { name: "Xyz" }
var userString = JSON.stringify(user)
var userBuffer = new Buffer(userString, "utf8")
var compressed = zlib.gzip(userBuffer)
// Adding data onto Redis
var data = { id: 1, compressed: compressed }
var dataString = JSON.stringify(data)
redis.set("test", dataString)
Although it seems impossible I wanna ask.
Is there any chance to store compressed data with string together?
Edit:
After storing compressed data with uncompressed data in the same key, I cannot decompress the user data above.
Pseudo code:
var dataString = redis.get("test")
var data = JSON.parse(dataString)
console.log(data)
// writes { id:1, compressed: { type: Buffer, data: [31, 139, 8...] } }
var compressed = data.compressed
var user = zlib.gunzip(compressed)
// user would be undefined here
const zlib = require('zlib');
const redis = require('redis').createClient();
var user = { name: "Xyz" }
var userString = JSON.stringify(user)
var userBuffer = new Buffer(userString, "utf8")
var compressed = zlib.gzipSync(userBuffer)
var data = { id: 1, compressed: compressed }
var dataString = JSON.stringify(data)
console.log('Compressed', dataString);
redis.set('mykey', dataString, err => {
if (err) throw err;
redis.get('mykey', (err, dataString) => {
if (err) throw err;
var data = JSON.parse(dataString)
var buffer = Buffer.from(data.compressed);
var uncompressed = zlib.gunzipSync(buffer);
console.log('Uncompressed', JSON.parse(uncompressed));
redis.end(false);
})
});
The key is to convert the string-representation of the Buffer back to a proper instance before decompressing it (using Buffer.from()).
However, the JSON-representation of a Buffer isn't very concise, so you probably should look into an alternative way of storing the data in Redis (one where you don't have to stringify the buffer). Perhaps using a hash.
Related
I have readableStream with an encrypted file and I would like to decrypt it. So far I was able to create 2 readableStreams - with 1 I extracted the IV and the second was used to decrypt the rest of the data, but I would like to just pipe it into one stream - both IV extraction AND the decryption. Something like this:
function getDecryptionStream(password, initVectLength = 16) {
const cipherKey = getCipherKey(password);
const unzip = zlib.createUnzip();
return new Transform({
transform(chunk, encoding, callback) {
if (!this.initVect) {
this.initVect = chunk.subarray(0, initVectLength);
chunk = chunk.subarray(initVectLength, chunk.length);
}
const decipher = crypto.createDecipheriv(algorithm, cipherKey, this.initVect);
callback(null, chunk.pipe(decipher).pipe(unzip));
}
});
}
function decrypt({ file, newFile, passphrase }) {
const readStream = fs.createReadStream(file);
const writeStream = fs.createWriteStream(newFile);
const decryptStream = getDecryptionStream(passphrase , IV_LENGTH);
readStream
.pipe(decryptStream)
.pipe(writeStream);
}
However I cannot figure out, how to process the chunk as chunk.pipe(decipher) throws an error. TypeError: chunk.pipe is not a function as it as Buffer
is there any way to store a json object without converting it to stream? I can upload after converting it to stream. But is there any to to store object as {something}.json with out converting it to stream?
what I do now
const azureStorage = require('azure-storage');
const blobService = azureStorage.createBlobService(accountName, accountKey);
var Readable = require('stream').Readable
var msg = {
a: "something",
b: "anotherthing"
}
var stream = new Readable
stream.push(JSON.stringify(msg))
stream.push(null)
var option = {
contentSettings: {contentType: 'application/json'}
}
stream.pipe(blobService.createWriteStreamToBlockBlob('container', 'something.json', option, function onResponse(error, result) {
console.log("done")
}));
Is there a better way?
Sure, you can just send text using createblockblobfromtext like this:
blobService.createBlockBlobFromText(
'container',
'something.json',
JSON.stringify(msg)
option,
function onResponse(error, result) {
console.log("done")
});
I am using fill-pdf npm module for filling template pdf's and it creates new file which is read from the disk and returned as buffer to callback. I have two files for which i do the same operation. I want to combine the two buffers there by to form a single pdf file which i can send back to the client. I tried different methods of buffer concatenation. The buffer can be concatenated using Buffer.concat, like,
var newBuffer = Buffer.concat([result_pdf.output, result_pdf_new.output]);
The size of new buffer is also the sum of the size of the input buffers. But still when the newBuffer is sent to client as response, it shows only the file mentioned last in the array.
res.type("application/pdf");
return res.send(buffer);
Any idea ?
As mentioned by #MechaCode, the creator has ended support for HummusJS.
So I would like to give you 2 solutions.
Using node-pdftk npm module
The Following sample code uses node-pdftk npm module to combine
two pdf buffers seamlessly.
const pdftk = require('node-pdftk');
var pdfBuffer1 = fs.readFileSync("./pdf1.pdf");
var pdfBuffer2 = fs.readFileSync("./pdf2.pdf");
pdftk
.input([pdfBuffer1, pdfBuffer2])
.output()
.then(buf => {
let path = 'merged.pdf';
fs.open(path, 'w', function (err, fd) {
fs.write(fd, buf, 0, buf.length, null, function (err) {
fs.close(fd, function () {
console.log('wrote the file successfully');
});
});
});
});
The requirement for node-pdftk npm module is you need to install the
PDFtk library. Some of you may find this overhead / tedious. So I have another solution using pdf-lib library.
Using pdf-lib npm module
const PDFDocument = require('pdf-lib').PDFDocument
var pdfBuffer1 = fs.readFileSync("./pdf1.pdf");
var pdfBuffer2 = fs.readFileSync("./pdf2.pdf");
var pdfsToMerge = [pdfBuffer1, pdfBuffer2]
const mergedPdf = await PDFDocument.create();
for (const pdfBytes of pdfsToMerge) {
const pdf = await PDFDocument.load(pdfBytes);
const copiedPages = await mergedPdf.copyPages(pdf, pdf.getPageIndices());
copiedPages.forEach((page) => {
mergedPdf.addPage(page);
});
}
const buf = await mergedPdf.save(); // Uint8Array
let path = 'merged.pdf';
fs.open(path, 'w', function (err, fd) {
fs.write(fd, buf, 0, buf.length, null, function (err) {
fs.close(fd, function () {
console.log('wrote the file successfully');
});
});
});
Personally I prefer to use pdf-lib npm module.
HummusJS supports combining PDFs using its appendPDFPagesFromPDF method
Example using streams to work with buffers:
const hummus = require('hummus');
const memoryStreams = require('memory-streams');
/**
* Concatenate two PDFs in Buffers
* #param {Buffer} firstBuffer
* #param {Buffer} secondBuffer
* #returns {Buffer} - a Buffer containing the concactenated PDFs
*/
const combinePDFBuffers = (firstBuffer, secondBuffer) => {
var outStream = new memoryStreams.WritableStream();
try {
var firstPDFStream = new hummus.PDFRStreamForBuffer(firstBuffer);
var secondPDFStream = new hummus.PDFRStreamForBuffer(secondBuffer);
var pdfWriter = hummus.createWriterToModify(firstPDFStream, new hummus.PDFStreamForResponse(outStream));
pdfWriter.appendPDFPagesFromPDF(secondPDFStream);
pdfWriter.end();
var newBuffer = outStream.toBuffer();
outStream.end();
return newBuffer;
}
catch(e){
outStream.end();
throw new Error('Error during PDF combination: ' + e.message);
}
};
combinePDFBuffers(PDFBuffer1, PDFBuffer2);
Here's what we use in our Express server to merge a list of PDF blobs.
const { PDFRStreamForBuffer, createWriterToModify, PDFStreamForResponse } = require('hummus');
const { WritableStream } = require('memory-streams');
// Merge the pages of the pdfBlobs (Javascript buffers) into a single PDF blob
const mergePdfs = pdfBlobs => {
if (pdfBlobs.length === 0) throw new Error('mergePdfs called with empty list of PDF blobs');
// This optimization is not necessary, but it avoids the churn down below
if (pdfBlobs.length === 1) return pdfBlobs[0];
// Adapted from: https://stackoverflow.com/questions/36766234/nodejs-merge-two-pdf-files-into-one-using-the-buffer-obtained-by-reading-them?answertab=active#tab-top
// Hummus is useful, but with poor interfaces -- E.g. createWriterToModify shouldn't require any PDF stream
// And Hummus has many Issues: https://github.com/galkahana/HummusJS/issues
const [firstPdfRStream, ...restPdfRStreams] = pdfBlobs.map(pdfBlob => new PDFRStreamForBuffer(pdfBlob));
const outStream = new WritableStream();
const pdfWriter = createWriterToModify(firstPdfRStream, new PDFStreamForResponse(outStream));
restPdfRStreams.forEach(pdfRStream => pdfWriter.appendPDFPagesFromPDF(pdfRStream));
pdfWriter.end();
outStream.end();
return outStream.toBuffer();
};
module.exports = exports = {
mergePdfs,
};
I have a few huge csv files, what I need to store in a mongo database. Because these files are too big, I need to use stream. I pause the stream while the data writing into the database.
var fs = require('fs');
var csv = require('csv');
var mongo = require('mongodb');
var db = mongo.MongoClient.connect...
var readStream = fs.createReadStream('hugefile.csv');
readStream.on('data', function(data) {
readStream.pause();
csv.parse(data.toString(), { delimiter: ','}, function(err, output) {
db.collection(coll).insert(data, function(err) {
readStream.resume();
});
});
});
readStream.on('end', function() {
logger.info('file stored');
});
But the csv.parse drop an error, because I would need to read the files line by line to handle them as csv, and convert to json for the mongodb. Maybe I should not pause them, but use an interface. I didn't find any solution for this yet.
Any help would be appreciated!
I think you might want to create a stream of lines from your raw data stream.
Here is an example from the split package. https://www.npmjs.com/package/split
fs.createReadStream(file)
.pipe(split())
.on('data', function (line) {
//each chunk now is a seperate line!
})
Adapted to your example it might look like this
var readStream = fs.createReadStream('hugefile.csv');
var lineStream = readStream.pipe(split());
lineStream.on('data', function(data) {
//remaining code unmodified
I'm unsure if bulk() was a thing back in '15, but whosoever is trying to import items from large sources should consider using them.
var fs = require('fs');
var csv = require('fast-csv');
var mongoose = require('mongoose');
var db = mongoose.connect...
var counter = 0; // to keep count of values in the bulk()
const BULK_SIZE = 1000;
var bulkItem = Item.collection.initializeUnorderedBulkOp();
var readStream = fs.createReadStream('hugefile.csv');
const csvStream = csv.fromStream(readStream, { headers: true });
csvStream.on('data', data => {
counter++;
bulkOrder.insert(order);
if (counter === BATCH_SIZE) {
csvStream.pause();
bulkOrder.execute((err, result) => {
if (err) console.log(err);
counter = 0;
bulkItem = Item.collection.initializeUnorderedBulkOp();
csvStream.resume();
});
}
}
});
I have a Base64 string that I am converting to binary like this:
var b64string = req.body.image.substr(23);//The base64 has a imageURL
var buf = new Buffer(b64string, 'base64');
I need to insert this into MongoDB GridFS. The problem I am having is that createReadStream require a filepath where I already have the file in memory.
This is what I am trying that does not work
var grid = new gfs(db, mongo, 'files');
grid.createWriteStream(options, function (err, ws) {
fs.createReadStream(buf, {autoClose: true})
.pipe(ws)
.on('close', function (f) {
console.log(f._id)
res.send(f._id)
})
.on('error', function (err) {
console.log(err)
})
})
But as I described, it wants a path where I have buf
UPDATE ---
I was over thinking it...
This works
var b64string = req.body.image.substr(23);
var buf = new Buffer(b64string, 'base64');
var grid = new Grid(db, 'files');
grid.put(buf, {}function(err, file){})