Saving file to AWS S3 - node.js

Question

I've been following some examples here and on Google of how to write a Nodejs Lambda function to fetch an (audio) file from a URL and save it to S3. I've arrived at this so far:
var AWS = require('aws-sdk');
var https = require('https');
var s3 = new AWS.S3();
const querystring = require('querystring');
exports.handler = function(event, context) {
const params = querystring.parse(event.body);
const audioUrl = params['audioUrl'];
https.get(audioUrl, function(res) {
var body = '';
res.on('data', function(chunk) {
// Agregates chunks
body += chunk;
});
res.on('end', function() {
// Once you received all chunks, send to S3
var params = {
Bucket: 'bucket_name',
Key: 'filename.wav',
Body: body
};
s3.putObject(params, function(err, data) {
if (err) {
console.error("ERROR: ", err, err.stack);
} else {
console.log("DATA:", data);
}
});
});
});
};
This results in file with filename filename.wav being saved to S3 bucket but the file is not a WAV audio file which is what I want. It seems that in the process of retrieving the file from audioUrl and saving it to S3 the format/encoding of the file is lost.
Any hints or solutions to this would be appreciated!
Thanks!

Answer 1

You're converting a binary file, into a string, that's why your uploaded file is corrupted. What you need to do is work with Buffer instead of string.
const chunks = [];
res.on('data', function(chunk) {
// Agregates chunks
chunks.push(chunk)
});
res.on('end', function() {
// Once you received all chunks, send to S3
var params = {
Bucket: 'bucket_name',
Key: 'filename.wav',
Body: Buffer.concat(chunks)
};
s3.putObject(params, function(err, data) {
if (err) {
console.error("ERROR: ", err, err.stack);
} else {
console.log("DATA:", data);
}
});
});
In any case, instead of buffering the chunks, is better to just pass the stream directly, and let the S3 SDK handle it.
var params = {
Bucket: 'bucket_name',
Key: 'filename.wav',
Body: res // pass the readable stream directly
};
s3.putObject(params, function(err, data) {});
The only caveat of passing a stream directly, is that the library can only work with streams that it can determine the length of. If that request sets the content-length correctly, you won't have any issue.
S3.putObject only accepts streams that it can determine the length of
streaming images doesnt work
You can bypass this limitation, by using s3.upload instead.
Note: Haven't worked with the S3 SDK in a while, but they didn't support pipe last time I used it, to do directly: res.pipe(s3.putObject())