I am having a list of 2 videos in a folder which I upload to AWS from my local directory when I call uploadDir function.
The videos are named dynamically to the path of their already existing file, because am re-uploading to replace the same file in the existing directory on aws
Example.
Video 1 name is: https://myBucketName.s3.amazonaws.com/Movies/video/comedy/93dcf22e84918d0efcb90ba2/Tom-41560712aa0f84f49ed130f2/Home Alone.mp4
Video 2 name is:
https://myBucketName.s3.amazonaws.com/Movies/video/action/26wjf31g64918d0efcb90bw6/Ben-71560712aa0f84f49ed130f4/Mission Impossible.mp4
But because I can't save a filename with special characters I then replace the video name with special characters in my folder directory using the function below
var str1 = videoName.replace(':','X');
var str2 = str1.replace(/[&\/\\#,<>{}]/g,'Q');
So I replaced all the ':' with 'X', and replaced all the '/' with 'Q'
So now the video name in my directory I want to upload back to AWS is looking like this
Replaced Video 1 name is: httpsXQQmyBucketName.s3.amazonaws.comQMoviesQvideoQcomedyQ93dcf22e84918d0efcb90ba2QTom-41560712aa0f84f49ed130f2QHome Alone.mp4
Replaced Video 2 name is: httpsXQQmyBucketName.s3.amazonaws.comQMoviesQvideoQactionQ26wjf31g64918d0efcb90bw6QBen-71560712aa0f84f49ed130f4QMission Impossible.mp4
So now I want to upload the videos back to their AWS path, which is originally the video name.
So for me to achieve this I believe I must first replace the original special characters, So all the 'X' will be ':', and all the 'Q' will be '/'
And the next step is for me to upload to the video name path.
How can I achieve this.
I already have a function uploading the videos to Aws bucket, but not to the directly I want the videos to be.
This is the code uploading the videos to Aws
var AWS = require('aws-sdk');
var path = require("path");
var fs = require('fs');
const uploadDir = function(s3Path, bucketName) {
let s3 = new AWS.S3();
function walkSync(currentDirPath, callback) {
fs.readdirSync(currentDirPath).forEach(function (name) {
var filePath = path.join(currentDirPath, name);
var stat = fs.statSync(filePath);
if (stat.isFile()) {
callback(filePath, stat);
} else if (stat.isDirectory()) {
walkSync(filePath, callback);
}
});
}
walkSync(s3Path, function(filePath, stat) {
let bucketPath = filePath.substring(s3Path.length+1);
let params = {Bucket: bucketName, Key: bucketPath, Body: fs.readFileSync(filePath) };
s3.putObject(params, function(err, data) {
if (err) {
console.log(err)
} else {
console.log('Successfully uploaded '+ bucketPath +' to ' + bucketName);
}
});
});
};
uploadDir(path.resolve('path-containing-videos'), 'bucket-name');
Related
I am able to download the file from s3 bucket like so:
const fileStream = s3.getObject(options).createReadStream();
const writableStream = createWriteStream(
"./files/master_driver_profile_pic/image.jpeg"
);
fileStream.pipe(fileStream).pipe(writableStream);
But the image is not getting written properly. Only a little bit of the image is visible and the rest is blank.
I think you should first createWriteStream and then createReadStream. (Check the docs)
var s3 = new AWS.S3();
var params = {Bucket: 'myBucket', Key: 'myImageFile.jpg'};
var file = require('fs').createWriteStream('/path/to/file.jpg');
s3.getObject(params).createReadStream().pipe(file);
OR
you can go without streams:
// Download file
let content = await (await s3.getObject(params).promise()).Body;
// Write file
fs.writeFile(downloadPath, content, (err) => {
if (err) { console.log(err); }
});
I have a zip file in S3 that contains several txt files and an image.
I am needing to grab that file from S3 and then unzip all of the files with their original filenames into the tmp directory in lambda using node.js
I am not super familiar with node and while the S3 part of getting the file works fine, I frankensteined the zip part together from the interwebs and was wondering if someone could help me get it right.
const zlib = require('zlib');
const fs = require('fs');
try {
const s3Object = await s3
.getObject({
Bucket: 'mybucket',
Key: `zip/${myzipfilename}`
})
.promise();
console.log("s3 zip fetched");
// write file to tmp
writeFileSync(`/tmp/${myzipfilename}`, s3Object.Body);
//unzip files
const fileContents = fs.createReadStream(`/tmp/${myzipfilename}`);
//I am quite sure this part is incorrect and is currently giving me an error
const writeStream = fs.createWriteStream(`./tmp/${filename.slice(0, -3)}`);
const unzip = zlib.createGunzip();
fileContents.pipe(unzip).pipe(writeStream);
}
End result within the lambda tmp directory would be something like:
/tmp/myoriginaltxtfilename.txt
/tmp/myoriginaltxtfilename2.txt
/tmp/myoriginaltxtfilename3.txt
/tmp/myoriginalimagefilename.png
I don't need to rezip anything.
You have a couple of issues in your code. First of all, at this line:
const writeStream = fs.createWriteStream(`./tmp/${filename.slice(0, -3)}`);
filename is not defined.
Second, you're using nodejs zlib to extract a .zip file which contains multiple files which won't work. zlib module is only for streams and buffers that represent singular resources not zip archives. You could use node-stream-zip instead.
Let's say you've successfully downloaded the zip file from S3 and saved in /tmp directory. Using node-stream-zip extracting the files from the zip file without unzipping it would look something like this:
const StreamZip = require('node-stream-zip');
const zip = new StreamZip({
file: `/tmp/${myzipfilename}`,
storeEntries: true
});
zip.on('ready', () => {
console.log('All entries read: ' + zip.entriesCount);
});
zip.on('entry', (entry) => {
if ('/' === entry.name[entry.name.length - 1]) {
console.log('[DIR]', entry.name);
return;
}
console.log('[FILE]', entry.name);
zip.stream(entry.name, (err, stream) => {
if (err) {
console.log('Error: ', err.toString());
return;
}
stream.on('error', (err) => {
console.log('[ERROR]', err);
return;
});
stream.pipe(fs.createWriteStream(`/tmp/${entry.name}`));
});
});
I want to be able to copy files within the same bucket from the root directory to a subfolder within a subfolder however excluding that subfolder by using the aws-sdk.
i.e:
I want to use this AWS-CLI command in a gulp file task:
aws s3 cp s3://bucketName s3://bucketName/last_good/YYYYMMDD --recursive --exclude "last_good/*"
I've used the copy examples used from How to copy/move all objects in Amazon S3 from one prefix to other using the AWS SDK for Node.js
I am just not sure how to specify the folder to exclude. In my above example it would be the last_good folder.
var gulp = require('gulp');
var AWS = require('aws-sdk');
var async = require('async');
var bucketName = 'bucketname';
var oldPrefix = '';
var newPrefix = 'last_good/20190817/';
var s3 = new AWS.S3({params: {Bucket: bucketName}, region: 'us-west-2'});
gulp.task('publish', function() {
CopyToLastGood();
}
function CopyToLastGood() {
var done = function(err, data) {
if (err) console.log(err);
else console.log(data);
};
s3.listObjects({Prefix: oldPrefix}, function(err, data) {
if (data.Contents.length) {
async.each(data.Contents, function(file, cb) {
var params = {
CopySource: bucketName + '/' + file.Key,
Key: file.Key.replace(oldPrefix, newPrefix)
};
s3.copyObject(params, function(copyErr, copyData){
if (copyErr) { // an error occured
console.log(err);
}
else {
console.log('Copied: ', params.Key); //successful response
cb();
}
});
}, done);
}
});
}
I expect contents of root to update last_good/20190817/ however not copying the last_good folder itself.
I've solved my solution using a delimiter option on the s3.listObjects params.
i.e:
s3.listObjects({Prefix: oldPrefix, Delimiter:'/'}
this only lists files within the root.
I'm using the aws-sdk for Nodejs and I'm trying to get an object that's located in a sub-directory.
But I don't get any files back.
The file path is: mybucket/subfoldername/file.txt
getObject = function getListObjects(bucketName, bucketPath, fileName) {
return new Promise((resolve, reject) => {
let params = {Key: fileName, Bucket: bucketName+bucketPath};
s3.getObject(params, function (error, data) {
if (error){
console.log(`Can't connect to S3. S3 Path = [${fileName}] to Bucket = [${params.Bucket}]`, error);
return reject(error);
} else {
console.log(`Found files in S3. S3 Path =[${fileName}] to Bucket = [${params.Bucket}]`);
return resolve(data.Contents);
}
});
});
};
The arguments I pass are:
bucketName: "mybucket"
bucketPath: "/subfoldername"
fileName: "file.txt"
Please advise.
The Keyargument should include the "path" string, too, as S3 bucket files are referenced by its complete path:
let params = { Key: `${bucketPath}/${fileName}`, Bucket: bucketName };
I've taken the liberty of using template quotes there (``). Also, notice how I've added a "/" separator, please check that it's not already included in bucketPath.
I want to concatenate the files uploaded on Amazon S3 server.
How can I do this.
Concatenation on local machine i can do using following code.
var fs = require('fs'),
files = fs.readdirSync('./files'),
clips = [],
stream,
currentfile,
dhh = fs.createWriteStream('./concatfile.mp3');
files.forEach(function (file) {
clips.push(file.substring(0, 6));
});
function main() {
if (!clips.length) {
dhh.end("Done");
return;
}
currentfile = './files/' + clips.shift() + '.mp3';
stream = fs.createReadStream(currentfile);
stream.pipe(dhh, {end: false});
stream.on("end", function() {
main();
});
}
main();
You can achieve what you want by breaking it into two steps:
Manipulating files on s3
Since s3 is a remote file storage, you can't run code on s3 server to do the operation locally (as #Andrey mentioned).
what you will need to do in your code is to fetch each input file, process them locally and upload the results back to s3. checkout the code examples from amazon:
var s3 = new AWS.S3();
var params = {Bucket: 'myBucket', Key: 'mp3-input1.mp3'};
var file = require('fs').createWriteStream('/path/to/input.mp3');
s3.getObject(params).createReadStream().pipe(file);
at this stage you'll run your concatenation code, and upload the results back:
var fs = require('fs');
var zlib = require('zlib');
var body = fs.createReadStream('bigfile.mp3').pipe(zlib.createGzip());
var s3obj = new AWS.S3({params: {Bucket: 'myBucket', Key: 'myKey'}});
s3obj.upload({Body: body}).
on('httpUploadProgress', function(evt) { console.log(evt); }).
send(function(err, data) { console.log(err, data) });
Merging two (or more) mp3 files
Since MP3 file include a header that specifies some information like bitrate, simply concatenating them together might introduce playback issues.
See: https://stackoverflow.com/a/5364985/1265980
what you want to use a tool to that. you can have one approach of saving your input mp3 files in tmp folder, and executing an external program like to change the bitrate, contcatenate files and fix the header.
alternatively you can use an library that allows you to use ffmpeg within node.js.
in their code example shown, you can see how their merge two files together within the node api.
ffmpeg('/path/to/part1.avi')
.input('/path/to/part2.avi')
.input('/path/to/part2.avi')
.on('error', function(err) {
console.log('An error occurred: ' + err.message);
})
.on('end', function() {
console.log('Merging finished !');
})
.mergeToFile('/path/to/merged.avi', '/path/to/tempDir');
Here's my quick take on the problem of downloading and processing S3 objects. My example is focused mostly on getting the data local and then processing it once it's all downloaded. I suggest you use one of the ffmpeg approaches mentioned above.
var RSVP = require('rsvp');
var s3 = new AWS.S3();
var bucket = '<your bucket name>';
var getFile = function(key, filePath) {
return new RSVP.Promise(function(resolve, reject) {
var file = require('fs').createWriteStream(filePath);
if(!file) {
reject('unable to open file');
}
s3.getObject({
Bucket: bucket,
Key: key
}).on('httpData', function(chunk) {
file.write(chunk);
}).on('httpDone', function() {
file.end();
resolve(filePath);
});
});
};
var tempFiles = ['<local temp filename 1>', '<local temp filename 2>'];
var keys = ['<s3 object key 1>', '<s3 object key 2>'];
var promises = [];
for(var i = 0; i < keys.length; ++i) {
var promise = getFile(keys[i], tempFiles[i]);
promises.push(promise);
}
RSVP.all(promises).then(function(data) {
//do something with your files
}).catch(function(error) {
//handle errors
});