I have a base64 encoded csv file, and I want to process it without saving to storage. How do you decode a base64 string, then assign it to a variable and then parse it using NodeJS?
There are many modules in the main npm repository. This is just one I chose, you can use another one. The module is base-x, the docs page has examples, which you should modify slightly to work with the base64 encoding:
var BASE64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
var bs64 = require('base-x')(BASE64);
var decoded = bs64.decode(youStringVariable);
// then store the decoded string or log it, or whatever
// console.log(decoded);
// myApi.store(decoded); etc.
Related
I'm using Firebase real-time database for my app. Daily backup is enabled for the database.
The database contains data with accents in words such as "Manutenção".
If I check this text in the Firebase console it is shown as "Manutenção".
If I export the data from the Firebase console it is shown as "Manutenção".
But if I download the backup file (.gzip) and after extraction, it is shown as "Manutenção". Notice here the encoding of accents. This encoding is according to https://string-functions.com/encodingtable.aspx?encoding=65001&decoding=10000
Why does the .gzip backup file encode the accents?
How to decode these encoded accents programmatically?
I tried to use the node module iconv but was not able to convert it.
var Iconv = require('iconv').Iconv;
var iconv = new Iconv('macintosh', 'UTF-8');
var buffer = iconv.convert('Manutenção');
console.log(buffer.toString()); // Manutenção
how can I get back "Manutenção" from "Manuten√ß√£o"?
Thanks!
Checking the threads, it seems that it is an issue with macOS
How can I convert encoding of special characters in python?
How to decode these characters? á é í
Solution
const iconv = require('iconv-lite');
let isMacRomanEncoded = (data.indexOf('¬') > -1) || (data.indexOf('√') > -1);
if(isMacRomanEncoded){
// MacRoman encoded, convert to utf-8
let buffer = iconv.encode(data, 'MacRoman');
return iconv.decode(buffer, 'utf-8');
}else{
// not MacRoman encoded, return the original
return data;
}
I have a variable in my NodeJS application which contains base64 string of an image. I need to send a form to some server with POST request containing this image. The problem is I can't convert base64 string to an image without writing it to a filesystem. Here's my code:
const imagePath = path.resolve(__dirname, '../../../images/anomalies/' + Date.now() + '.png')
fs.writeFileSync(imagePath, img, { encoding: 'base64' })
setTimeout(() => {
fs.unlinkSync(imagePath)
}, 30_000)
const form = new FormData()
form.append('photo', fs.createReadStream(imagePath))
As you can see, I need to write base64 string to a file and then grab it with fs.createReadStream. Otherwise file won't upload. I tried converting it to ReadStream via stream-buffers (but server still not accepted that data) and also I tried to make a Blob from it but Node don't have blobs and all these modules on npm are either too old or don't have typings which is not great at all. I also tried Buffer.from(base64, 'base64'), which doesn't work as well.
Is there any way to create an uplodable image file from base64 encoded string without accessing filesystem in NodeJS?
You should be able to convert the Buffer object returned by fs.readFileSync to a base64 string
const base64Image = fs.readFileSync(imagePath).toString('base64')
I used the code below to encode a file to base64.
var bitmap = fs.readFileSync(file);
return new Buffer(bitmap).toString('base64');
I figured that in the file we have issues with “” and ‘’ characters, but it’s fine with "
When we have It’s, node encodes the characters, but when I decode, I see it as
It’s
Here's the javascript I'm using to decode:
fs.writeFile(reportPath, body.buffer, {encoding: 'base64'}
So, once the file is encoded and decoded, it becomes unusable with these funky characters - It’s
Can anyone shed some light on this?
This should work.
Sample script:
const fs = require('fs')
const filepath = './testfile'
//write "it's" into the file
fs.writeFileSync(filepath,"it's")
//read the file
const file_buffer = fs.readFileSync(filepath);
//encode contents into base64
const contents_in_base64 = file_buffer.toString('base64');
//write into a new file, specifying base64 as the encoding (decodes)
fs.writeFileSync('./fileB64',contents_in_base64,{encoding:'base64'})
//file fileB64 should now contain "it's"
I suspect your original file does not have utf-8 encoding, looking at your decoding code:
fs.writeFile(reportPath, body.buffer, {encoding: 'base64'})
I am guessing your content comes from a http request of some sorts so it is possible that the content is not utf-8 encoded. Take a look at this:
https://www.w3.org/International/articles/http-charset/index if charset is not specified Content-Type text/ uses ISO-8859-1.
Here is the code that helped.
var bitmap = fs.readFileSync(file);
// Remove the non-standard characters
var tmp = bitmap.toString().replace(/[“”‘’]/g,'');
// Create a buffer from the string and return the results
return new Buffer(tmp).toString('base64');
You can provide base64 encoding to the readFileSync function itself.
const fileDataBase64 = fs.readFileSync(filePath, 'base64')
I have a web service that takes a base 64 encoded string representing an image, creates a thumbnail of that image using the imagemagick library, then stores both of them in mongodb. I am doing this with the following code (approximately):
var buf = new Buffer(req.body.data, "base64"); //original image
im.resize({ srcData: buf, width: 256 }, function(err, stdout, stderr) {
this.thumbnail = new Buffer(stdout, "binary");
//store buf and stdout in mongo
});
You will notice that I am creating a Buffer object using the "binary" encoding, which the docs say not to do:
'binary' - A way of encoding raw binary data into strings by using
only the first 8 bits of each character. This encoding method is
deprecated and should be avoided in favor of Buffer objects where
possible. This encoding will be removed in future versions of Node.
First off I'm not sure what they are saying there. I'm trying to create a Buffer object and they seem to imply I should already have one.
Secondly, the source of the problem appears to be that the imagemagick resize method returns a string containing binary data. Doing typedef(stdout) return "string" and printing it out to the screen certainly appears to show a bunch of non-character data.
So what do I do here? I can't change how imagemagick works. Is there another way of doing what I'm trying to do?
Thats how I am doing the same with success, storing images in mongodb.
//original ---> base64
var thumbnail = new Buffer(req.body.data).toString('base64');
//you can store this string value in a mongoose model property, and save to mongodb
//base64 ---> image
var buffer = new Buffer(thumbnail, "base64");
I am not sure if storing images as base64 is the best way to do it
Please try this as your base64 might not be pre-handled:
var imgRawData =
req.body.images[0].replace(/^data:image\/png;base64,|^data:image\/jpeg;base64,|^data:image\/jpg;base64,|^data:image\/bmp;base64,/, "");
var yourBuffer = new Buffer(imgRawData, "base64");
Then, save the yourBuffer into MongoDB buffer.
I'm using a node.bcrypt.js hash returning hex numbers in node.js for a password reset token.
user.reset_password_token = require('crypto').randomBytes(32).toString('hex'
);
Should I also base64 encode the token before I pass it around in urls (ie: link reset email)?
Is there any benefit to doing this?
I seem to recall base64 encoding can contain forward slashes which would mess up the path:
var token = user.reset_password_token;
//is there any benefit to doing base64 encoding?
var encoded_token = new Buffer(token).toString('base64');
var reset_link = 'http://example.com/reset/'+ encoded_token;
sendResetLink( reset_link );
You don't need a third-party library for that. You can just use base64url encoding (starting from nodejs v14.18.0)
const encoded_token = Buffer.from(token).toString('base64url');
I solved it using URLSafeBase64 nodejs LIB at https://www.npmjs.org/package/urlsafe-base64
var email =email_lines.join("\r\n").trim();
var base64EncodedEmail = URLSafeBase64.encode(new Buffer(email));
gmail.users.messages.send({userId:"me",
resource: {raw:base64EncodedEmail} }, callbackFn});
Another option is base64url library:
base64url("ladies and gentlemen we are floating in space");
// bGFkaWVzIGFuZCBnZW50bGVtZW4gd2UgYXJlIGZsb2F0aW5nIGluIHNwYWNl
base64 can indeed contain forward slashes, but base32 can't!