I am trying to convert S3 video file to audio file through Lambda function. Whenever video files are uploaded into an S3 bucket I have to generate an audio file and save it back to S3 bucket by triggering the AWS Lambda function. I can convert the video file to audio in local. ( Convert video to an audio file using FFMPEG). But I am wondering, how to do this conversion part in Lambda function every time the video file is uploaded into an S3 bucket. I have no idea how to do this AWS Lambda function. Please share your suggestions.
Sample code:
var ffmpeg = require('fluent-ffmpeg');
/**
* input - string, path of input file
* output - string, path of output file
* callback - function, node-style callback fn (error, result)
*/
function convert(input, output, callback) {
ffmpeg(input)
.output(output)
.on('end', function() {
console.log('conversion ended');
callback(null);
}).on('error', function(err){
console.log('error: ', e.code, e.msg);
callback(err);
}).run();
}
convert('./df.mp4', './output.mp3', function(err){
if(!err) {
console.log('conversion complete');
//...
}
});
Thanks,
You just need to set up an event on s3 bucket - put object - to trigger lambda function (you will get access to the description of the object uploaded to that S3 bucket through the first parameter of the lambda function).
If you can convert the video file to audio on your local machine, using some external libraries, then you need to create a zip file containing your lambda function (in the root of the zip file) as well as the dependencies.
This is pretty simple in case of Node. Create a new folder, run npm init, install needed modules, create index.js file where you put your Node code. Zip all the contents of this folder (not the folder itself). When you create new lambda function, choose to upload this zip file.
If you are wondering about how to programatically communicate with AWS resources and manipulate them, then check aws-sdk which you can import as a module and use it for that purpose.
So basically what you will need to inside of your lambda function is to parse event argument (the first parameter) to obtain bucket and key of the uploaded object. Then you will call s3.getObject method to get the data. Process the data with your custom logic. Call s3.putObject to store the newly transformed data to new S3 location.
Lambda has access to its own local file system, if your code needs to store some data there. You just need to specify absolute path to the file, such as /tmp/output.mp3. To retrieve it, you can use fs module. Then, you can continue with s3.putObject.
Related
I'm currently building a Alexa application in Node with Lambda. I have the need to convert and merge several audio files. I'm currently creating an audio file using google text-to-speech (long story on the need for it) which I write to /tmp and pulling an audio file from s3 which I also write to /tmp. I'm then using sox to merge the two files (see below) and write back to S3 (currently public) which I then have hard coded to play that particular clip.
My question is if it is possible to play audio directly from the /tmp folder as opposed to having to write the file back to S3.
await lambdaAudio.sox('-m /tmp/google-formatted.mp3 /tmp/audio.mp3 /tmp/result.mp3')
// get data from resulting mp3
const data = await readFile('/tmp/result.mp3');
const base64data = new Buffer(data, 'binary');
// put file back on AWS for playing
s3.putObject({
Bucket: 'my-bucket',
Key: 'result.mp3',
Body: base64data,
ACL:'public-read'
},function (resp) {
console.log('Done');
});
return`<audio src="https://s3.amazonaws.com/my-bucket/result.mp3" />`;
I usually upload the lambda function zipping the code and modules and in general all the files that my code requires.
https://developer.amazon.com/blogs/post/Tx1UE9W1NQ0GYII/Publishing-Your-Skill-Code-to-Lambda-via-the-Command-Line-Interface
So if you zip the /tmp directory and publish it as part of your lambda code the audio file will be accessible by your lambda function
I set up my discord BOT using node.js. For my advantage, I would need to store some data on a external file, but I don't seem to be able to access it from my index.js file (the main Bot file).
I've tried having one static array in the external js/json files, but I can only retrieve undefined/empty values. Additionally, when I tried with a .txt file, once retrieved the content, I found it unable to call functions such as string.split().
Did I miss something in the package content perhaps?
Assuming the data you are storing is in UTF-8 encoding:
var fs = require('fs');
fs.readFile('path/to/file', 'utf8', function(err, contents) {
// code using file data
});
Assuming no errors contents will be a string of the data that is inside that file.
https://code-maven.com/reading-a-file-with-nodejs
I would like to make my code save binary data uploaded via CURL
Here is CURL command I use: curl --header "Content-Type:application/octet-stream" --trace-ascii debugdump.txt --data-binary #test-files/small-test.zip http://localhost:8000/chrome.
On the Node JS (receiving side) there is script that takes AWS Lambda event object and takes event.body property into this function:
handler.js
export default (async function run(event, context, callback) {
//here event.body keeps the content of small-test.zip
//logged content is encoded with \u0000 signs- take a look at second screenshot
console.log(event);
writeToDiskAndUnpackDocument(event.body);
}
function writeToDiskAndUnpackDocument(binaryFileContents) {
//here I get binary content - take a look at second screenshot
//logged content looks the same as displayed by linux command "cat small-test.zip"
console.log(binaryFileContents);
//this command writes event.body to disk, but result file is not the same as that in curl command
fs.writeFile('/tmp/document.zip', binaryFileContents,'binary');
}
The document.zip lands inside tmp folder, when I issue cat document.zip it looks different from the input file test-files/small-test.zip. I don't know why it differs. Here is the screenshot, at the top is original file, at the bottom is received file.
I am using serverless-offline to do developing of the solution.
How can I properly save this ZIP file on in Lambdas tmp (first my own linux laptop)?
Comparison of binary data in event object and in writeToDiskAndUnpackDocument argument:
I will avoid dealing with local filesystem when dealing with serverless, Why not use S3 to download your binary file?
If you still want to download a binary file and pipe it to filesystem,
request
.get('http://example.com/doodle.png')
.on('error', function(err) {
console.log(err)
})
.pipe(fs.createWriteStream('/tmp/doodle.png'))
It can download and get it to the filesystem.
Reference:
https://github.com/request/request
Hope it helps.
I've been trying to upload gzipped log files to S3 using the AWS NodeJS sdk, and occasionally find that the uploaded file in S3 is corrupted/truncated. When I download the file and decompress using gunzip in a bash terminal, I get:
01-log_2014-09-22.tsv.gz: unexpected end of file.
When I compare file sizes, the downloaded file comes up just a tiny bit short of the original file size (which unzips fine).
This doesn't happen consistently...one out of every three files or so is truncated. Reuploading can fix the problem. Uploading through the S3 Web UI also works fine.
Here's the code I'm using...
var stream = fs.createReadStream(localFilePath);
this.s3 = new AWS.S3();
this.s3.putObject({
Bucket: bucketName,
Key: folderName + filename,
ACL: "bucket-owner-full-control",
Body: stream,
},function(err) {
// stream.close();
callback(err);
});
I shouldn't have to close the stream since it defaults to autoclose, but the problem seems to occur either way.
The fact that its intermittent suggests it's some sort of a timing or buffering issue, but I can't find any controls to fiddle with that might affect that. Any suggestions?
Thanks.
I am generating a PNG on the server side of a node.js application, using ImageMagick and the gm library for node.js (GraphicsMagick for node.js).
// start with a blank image
var gmImage = gm(100, 100, "#000000ff");
// Draw the stuff on the new blank image
When I'm finished drawing stuff using the gm library, I am storing that image to the file system:
gmImage.write(imagePath, function (err) {
...
});
I am now moving to s3. I want to skip this previous step and write the image direct to s3 without using a temporary file.
Is there a way to write the gmImage to a buffer or something?
Take a look at the stream section of the API: https://github.com/aheckmann/gm#streams
You should be able to pipe stdout into s3
var gmImage = gm(100, 100, "#000000ff");
gmImage.stream(function (err, stdout, stderr) {
stdout.pipe(s3Stream);
});