I'm having trouble serving binary data from node. I worked on a node module called node-speak which does TTS (text to Speech) and return a base64 encoded audio file.
So far I'm doing this to convert from base64 to Buffer/binary and then serve it:
// var src = Base64 data
var binAudio = new Buffer(src.replace("data:audio/x-wav;",""), 'base64');
Now I'm trying to serve this audio from node with the headers like so:
res.writeHead(200, {
'Content-Type': 'audio/x-wav',
'Content-Length': binAudio.length
});
And serving it like so:
res.end(binAudio, "binary");
But its not working at all. Is there something I havnt quite understood or am I doing something wrong, because this is not serving a valid audio/x-wav file.
Note: The Base64 data is valid i can serve it like so [see below] and it works fine:
// assume proper headers sent and "src" = base64 data
res.end("<!DOCTYPE html><html><body><audio src=\"" + src + "\"/></body></html>");
So why can I not serve the binary file, what am I doing wrong?
Two things are wrong.
not Conetnt-Length, it's Content-Length
res.end(binAudio, "binary"); is wrong. Use res.end(binAudio);. With "binary", it expects a string - binary is a deprecated string encoding in node, use no encoding if you already have a buffer.
Related
I created a NodeJS application which should get some data from an external API-Server. That server provides its data only as 'Content-Type: text/plain;charset=ISO-8859-1'. I have got that information through the Header-Data of the server.
Now the problem for me is that special characters like 'ä', 'ö' or 'ü' are shown as �.
I tried to convert them with Iconv to UTF-8, but then I got these things '�'...
My question is, what am I doing wrong?
For testing I use Postman. These are the steps I do to test everything:
Use Postman to trigger my NodeJS application
The App requests data from the API-Server
API-Server sends Data to NodeJS App
My App prints out the raw response-data of the API, which already has those strange characters �
The App then tries to convert them with Iconv to UTF-8, where it shows me now this '�' characters
Another strange thing:
When I connect Postman directly to the API-Server, the special characters get shown as they have too without problems. Therefore i guess my application causes the problem but I cannot see where or why...
// Javascript Code:
try {
const response = await axios.get(
URL
{
params: params,
headers: headers
}
);
var iconv = new Iconv('ISO-8859-1', 'UTF-8');
var converted = await iconv.convert(response.data);
return converted.toString('UTF-8');
} catch (error) {
throw new Error(error);
}
So after some deeper research I came up with the solution to my problem.
The cause of all trouble seems to lie within the post-process of axios or something similar. It is the step close after data is received and convertet to text and shortly before the response is generated for my nodejs-application.
What I did was to define the "responseType" of the GET-method of axios as an "ArrayBuffer". Therefore an adjustment in axios was necessary like so:
var resArBuffer = await axios.get(
URL,
{
responseType: 'arraybuffer',
params: params,
headers: headers
}
);
Since JavaScript is awesome, the ArrayBuffer provides a toString() method itself to convert the data from ArrayBuffer to String by own definitions:
var response = resArBuffer.data.toString("latin1");
Another thing worth mentioning is the fact that I used "latin1" instead of "ISO-8859-1". Don't ask me why, some sources even recommended to use "cp1252" instead, but "latin1" workend for me here.
Unfortunately that was not enough yet since I needed the text in UTF-8 format. Using "toString('utf-8')" itself was the wrong way too since it would still print the "�"-Symbols. The workaround was simple. I used "Buffer.from(...)" to convert the "latin1" defined text into a "utf-8" text:
var text = Buffer.from(response, 'utf-8').toString();
Now I get the desired UTF-8 converted text I needed. I hope this thread helps anyone else outhere since thse informations hwere spread in many different threads for me.
I am having some issues converting a payload received (req.body) to a correctly formatted base64 string.
The payload is received and looks like the below example. I know that it's encrypted coming in but I'm wondering if there is anything that could be happening node server side that makes it look like this, it seems malformed and not how it should be
body: '&R۽5�l|L�\u001b�\u0014햱����\u00020#��[cV[AD&P�\u0001��˯���\n' +
`#B軉�6Y¤�\u0010�l�\u0012"D�dʦ�nb�g���\u0017����߉�{�a\u000e�:��\u0014\u0005�4\u0018!��u\u001e��s!վ]�\u0011KɆ�<!\u001d��#a�Ӿǥ+\f�iWEź�����:^�Վߎ�NP�M�G�_x�}�b1W�t�\u000f?*�2N�s��\u0000\u0015\u001e��o��� |y.\u0004n�e��64z�eu3\u0007(��j�R�\u0001 jzO\u0012�IF\u0002��w_����%�\u001b\u0010��\u0010��5�\u0016�.1�\u0006�\f\u0014�$�|\u000e�E�5�����o�MΆA\u001a��\u0010������-ܹ��\u0003�jV�0b\u0002�\u001f��\\^"\\���\u0000��%�̓B�TfI��3��2U���[#�ۍ�'bT�]�\u0007�������\u0016 �P��x?\u0014�ly*8\u00134�NR����<��\u0012^�"#�V���!\u0010=�\u0006�"r�c�a�/L���vq�<\u0015�\u0006H��\u0014�\u001f�m�~�Ֆ�\u0011>L+����Yw���٘�\u0007ur�&�i�B4\n` +
If I convert the payload to a base64 string then i get something like this (the + and / characters do not seem right)
var encryptedBytes = Buffer.from(req.body);
var encryptedStr = encryptedBytes.toString('base64');
console.log({encryptedStr});
{ encryptedStr: 'JlLbvTXvv71sfEzvv70b77+9fxTtlrHvv73vv73vv73vv70CMCPvv73vv71bY1ZbQUQmUO+/vQHvv73vv73Lr++/ve+/ve+/vQojQui7ie+/vTZZwqTvv70Q77+9bO+/vRIiRO+/vWTKpu+/vW5i77+9Z++/ve+/ve+/vRfvv73vv73vv73vv73fie+/vXvvv71hDu+/vTrvv73vv70UBe+/vTQYIe+/ve+/vXUe77+977+9cyHVvl3vv70RS8mG77+'}
If I compare this to a base64 string I grab from the request on an iOS device for example, these seem rather different, plus this base64 string below can be decrypted successfully, which implies the issue could be within node?
{ base64data: 'ZGRIUUJhR0dMc3BTVFdQSHppS3BZUDY1UmJWSkFmbnRpekg1a29nUnlFMGtZemExU0RwS1h0VHlNd1lHMnhRcEZiMjEzNEwwYXduNllHR1p0aU1HM3YzcWlyTnlSd2RWSmNHQldMRVVMWklWaGRpNzBNWHVPNkZaSnJBUWZ6YnBJbERESzBiTEpoUGVCS3ZiU1d2NnRIcktIb'}
So my question here, is there something i need to do node server side to correctly parse or translate the req.body ready to be converted into a base64 string?
I have a variable in my NodeJS application which contains base64 string of an image. I need to send a form to some server with POST request containing this image. The problem is I can't convert base64 string to an image without writing it to a filesystem. Here's my code:
const imagePath = path.resolve(__dirname, '../../../images/anomalies/' + Date.now() + '.png')
fs.writeFileSync(imagePath, img, { encoding: 'base64' })
setTimeout(() => {
fs.unlinkSync(imagePath)
}, 30_000)
const form = new FormData()
form.append('photo', fs.createReadStream(imagePath))
As you can see, I need to write base64 string to a file and then grab it with fs.createReadStream. Otherwise file won't upload. I tried converting it to ReadStream via stream-buffers (but server still not accepted that data) and also I tried to make a Blob from it but Node don't have blobs and all these modules on npm are either too old or don't have typings which is not great at all. I also tried Buffer.from(base64, 'base64'), which doesn't work as well.
Is there any way to create an uplodable image file from base64 encoded string without accessing filesystem in NodeJS?
You should be able to convert the Buffer object returned by fs.readFileSync to a base64 string
const base64Image = fs.readFileSync(imagePath).toString('base64')
I have a base64 encoded csv file, and I want to process it without saving to storage. How do you decode a base64 string, then assign it to a variable and then parse it using NodeJS?
There are many modules in the main npm repository. This is just one I chose, you can use another one. The module is base-x, the docs page has examples, which you should modify slightly to work with the base64 encoding:
var BASE64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
var bs64 = require('base-x')(BASE64);
var decoded = bs64.decode(youStringVariable);
// then store the decoded string or log it, or whatever
// console.log(decoded);
// myApi.store(decoded); etc.
I have a simple uploading code by node.js.
var http = require('http')
var fs = require('fs')
var server = http.createServer(function(req, res){
if(req.url == '/upload') {
var a = fs.createWriteStream('a.jpg', { defaultEncoding: 'binary'})
req.on('data', function(chunk){
a.write(chunk)
})
.on('end', function()){
a.end()
res.end('okay')
})
}
else {
fs.createReadStream('./index.html').pipe(res);
// just show <form>
}
})
server.listen(5000)
when I upload some image, I cannot get exact same file.
Always files are broken.
When I try to do this using formidable, I can get a fine file.
So I studied formidable but I cannot understand how did it catch data and save.
I could find formidable use parser to calculate something about chunk from request but I did not get it all.
(It is definitely my brain issue :( ).
Anyway, what is the difference between my code and formidable?
What am I missing?
Is it a wrong way to just add all chunks from http request and save it by
fs.createWriteStream or fs.writeFile ?
What concepts am I missing?
First, req is a Readable stream. You can simply do:
req.pipe(fs.createWriteStream('a.jpg'))
for the upload part. This is copying all byte data from request stream to file.
This will work when you send raw file data as the request body:
curl --data-binary #"/home/user/Desktop/a.jpg" http://localhost:8080/upload
Because this sends request body exactly as image binary data, that gets streamed to a file on server.
But, there is another request format called multipart/form-data. This is what web browsers use with <form> to upload files.
curl -form "image=#/home/user1/Desktop/a.jpg" http://localhost:8080/upload
Here the request body contains multiple "parts", one for each file attachment or form field, separated by special "boundary" characters:
--------------------------e3f25f5319cd6624
Content-Disposition: form-data; name="image"; filename="a.jpg"
Content-Type: application/octet-stream
JPG IHDRH-ÑtEXtSoftwareAdobe.....raw file data
--------------------------e3f25f5319cd6624
Hence you will need much more complicated code to extract the file part data from it. NPM Modules like busboy and formidable do exactly that.