I have a web service that takes a base 64 encoded string representing an image, creates a thumbnail of that image using the imagemagick library, then stores both of them in mongodb. I am doing this with the following code (approximately):
var buf = new Buffer(req.body.data, "base64"); //original image
im.resize({ srcData: buf, width: 256 }, function(err, stdout, stderr) {
this.thumbnail = new Buffer(stdout, "binary");
//store buf and stdout in mongo
});
You will notice that I am creating a Buffer object using the "binary" encoding, which the docs say not to do:
'binary' - A way of encoding raw binary data into strings by using
only the first 8 bits of each character. This encoding method is
deprecated and should be avoided in favor of Buffer objects where
possible. This encoding will be removed in future versions of Node.
First off I'm not sure what they are saying there. I'm trying to create a Buffer object and they seem to imply I should already have one.
Secondly, the source of the problem appears to be that the imagemagick resize method returns a string containing binary data. Doing typedef(stdout) return "string" and printing it out to the screen certainly appears to show a bunch of non-character data.
So what do I do here? I can't change how imagemagick works. Is there another way of doing what I'm trying to do?
Thats how I am doing the same with success, storing images in mongodb.
//original ---> base64
var thumbnail = new Buffer(req.body.data).toString('base64');
//you can store this string value in a mongoose model property, and save to mongodb
//base64 ---> image
var buffer = new Buffer(thumbnail, "base64");
I am not sure if storing images as base64 is the best way to do it
Please try this as your base64 might not be pre-handled:
var imgRawData =
req.body.images[0].replace(/^data:image\/png;base64,|^data:image\/jpeg;base64,|^data:image\/jpg;base64,|^data:image\/bmp;base64,/, "");
var yourBuffer = new Buffer(imgRawData, "base64");
Then, save the yourBuffer into MongoDB buffer.
Related
So, I'm trying to pass an image to a Node Lambda through API Gateway and this is automatically base64 encoded. This is fine, and my form data all comes out correct, except somehow my image is being corrupted, and I'm not sure how to decode this properly to avoid this. Here is the relevant part of my code:
const multipart = require('aws-lambda-multipart-parser');
exports.handler = async (event) => {
console.log({ event });
const buff = Buffer.from(event.body, 'base64');
// using utf-8 appears to lose some of the data
const decodedEventBody = buff.toString('ascii');
const decodedEvent = { ...event, body: decodedEventBody };
const jsonEvent = multipart.parse(decodedEvent, false);
const asset = Buffer.from(jsonEvent.file.content, 'ascii');
}
First off, it would be good to know if aws-sdk had a way of parsing the multipart form data rather than using this unsupported third party code. Next, the value of asset ends with a buffer that's exactly the same size as the original file, but some of the byte values are off. My assumption is that the way this is being encoded vs decoded is slightly different and maybe some of the characters are being interpreted differently.
Just an update in case anybody else runs into a similar problem - updated 'ascii' to 'latin1' in both places and then it started working fine.
I have a variable in my NodeJS application which contains base64 string of an image. I need to send a form to some server with POST request containing this image. The problem is I can't convert base64 string to an image without writing it to a filesystem. Here's my code:
const imagePath = path.resolve(__dirname, '../../../images/anomalies/' + Date.now() + '.png')
fs.writeFileSync(imagePath, img, { encoding: 'base64' })
setTimeout(() => {
fs.unlinkSync(imagePath)
}, 30_000)
const form = new FormData()
form.append('photo', fs.createReadStream(imagePath))
As you can see, I need to write base64 string to a file and then grab it with fs.createReadStream. Otherwise file won't upload. I tried converting it to ReadStream via stream-buffers (but server still not accepted that data) and also I tried to make a Blob from it but Node don't have blobs and all these modules on npm are either too old or don't have typings which is not great at all. I also tried Buffer.from(base64, 'base64'), which doesn't work as well.
Is there any way to create an uplodable image file from base64 encoded string without accessing filesystem in NodeJS?
You should be able to convert the Buffer object returned by fs.readFileSync to a base64 string
const base64Image = fs.readFileSync(imagePath).toString('base64')
I have a base64 encoded csv file, and I want to process it without saving to storage. How do you decode a base64 string, then assign it to a variable and then parse it using NodeJS?
There are many modules in the main npm repository. This is just one I chose, you can use another one. The module is base-x, the docs page has examples, which you should modify slightly to work with the base64 encoding:
var BASE64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
var bs64 = require('base-x')(BASE64);
var decoded = bs64.decode(youStringVariable);
// then store the decoded string or log it, or whatever
// console.log(decoded);
// myApi.store(decoded); etc.
For some reason I need to store some files (mostly images or pdfs) into my database (PG 9.2.20).
Those files are uploaded by users and when I download them back, they are corrupted.
I'm working with nodejs.
The column type I store the file is BYTEA.
This is how I store them :
const { files, fields } = await asyncBusboy(ctx.req);
const fileName = files[0].filename;
const mimeType = files[0].mimeType;
const bufferedFile = fs.readFileSync(files[0].path, { encoding: 'hex' });
const fileData = `\\x${bufferedFile}`;
//Just a basic insert into with knex.raw
const fileId = await storageModel.create(fields.name, fields.description, fileName, mimeType, fileData, ctx.user);
And this is how I retrieve my file :
const file = await storageModel.find(ctx.params.fileId, ctx.user);
ctx.body = Buffer.from(file.file_bin, 'hex');
ctx.set('Content-disposition', `attachment; filename=${file.file_name}`);
The file is corrupted, and of course, if I look closely, the uploaded file and the one I downloaded are different.
See hex screenshot, there is some additional data at the start of the downloaded one : http://imgur.com/a/kTRAB
After some more testing I can tell the problem lies into the koa part, when I put the buffer into the ctx.body. It got corrupted (???)
EDIT : I was working with Swagger UI : https://github.com/swagger-api/swagger-ui/issues/1605
You should not use bytea as a regular text string. You should pass in type Buffer directly, and have the driver escape it for you correctly.
Not sure which driver you are using, but for example...
pg-promise does it automatically, see the example
node-postgres is supposed to do it automatically, which it does mostly, but I know there were issues with arrays, recently fixed here.
massive.js - based on pg-promise since v3.0, so the same story - it just works.
I am writing a sails.js app. I am writing an API to accept a file and encrypt it.
var file = req.body('myFile');
var fileBuffer = convertToBuffer(file);
How do I convert a file to buffer?
That looks like you've got a string which represents the body of your file.
You just have to make a new buffer with it.
var fileBuffer = Buffer.from(file)
If your encoding is NOT utf8 you can specify an alternate encoding as a second optional argument.
var fileBuffer = Buffer.from(file, 'base64')
If the file is actually on disk, this is even easier, since, by default, the fs.readFile operation returns a buffer.
fs.readFile(file, function(err, buffer){})
If you're in a real old version of node Buffer.from doesn't exist and you have to use a very memory-unsafe new constructor. Please consider upgrading your node instance to support Buffer.from