using request.get file body is not correct - node.js

I'm trying to upload a file to s3 using request.js but the file data seems to be incorrect after I upload it. Should I be using a data property of the response object instead?
var flickrPhotoUrl = 'http://c1.staticflickr.com/3/2207/2305523141_1f98981685_z.jpg?zz=1';
request.get(flickrPhotoUrl, function(error, response, body){
if (body) {
s3.upload({
Body: body,
Bucket: 'my-uploads',
Key: 'photo.jpg',
}, function (err) {
done(err);
});
}
else
{
done('no response');
}
});
When I grab the file from s3 after the upload, it's not a recognizable image and seems to be twice as big.

request by default converts your image binary data to utf8 string. That's why the file size is larger than the actual size. Try passing encoding:null to keep the body as buffer:
request.get({encoding:null, uri: flickrPhotoUrl}, function(error, response, body){
...
})
Update:
I think you can also pass a readable stream in Body parameter. This is faster than above for large files.
var stream = request.get(flickrPhotoUrl)
s3.upload({
Body: stream,
Bucket: 'my-uploads',
Key: 'photo.jpg',
}

Related

Create a Bucket with HTML file from a Lambda (want HTML, get Octet Stream)

Geez, I thought this would be easy, but I'm stuck. I created a bucket with the AWS SDK, created a file in that bucket which I want to be an HTML file. I get an HTML file, but S3 insists that it has to be an Octet-steam. What I want is for it to render in a browser.
var params = {
Bucket: myBucket,
Key: myKey,
Body: '<html>hi</html>',
Metadata: {
'Content-Type': 'Application/html'
},
ACL: 'public-read'
};
s3.putObject(params, function(err, data) {
if (err) {
console.log(err)
} else {
console.log("Successfully uploaded data to myBucket/myKey");
}
});
Long story, but trying not to do it from the CLI
I believe your Content-Type should be text/html, not Application/html.

How to handle an image file from googleusercontent CDN in Node?

Trying to request an image from Google CDN and upload it to S3.
Using the https://github.com/request/request library and Node / Express;
A little confused how to handle payload coming back from Google CDN.
The image comes back in the body field and in encoded. Not sure how it is encoded.
Given a URL to a Google CDN:
const fileURL = https://lh4.googleusercontent.com/EWE1234rFL006WfQKuAVrsYMOiKnM6iztPtLgXM5U…3i26LoPHQwPTQME7ne3XoMriKVjUo3hrhwWw1211223
request(fileURL, (err, res, body) => {
//NOT sure how to handle the response here??
//Trying base64
fs.writeFileSync(`.tmp/file1.png`, body, {encoding: 'base64'});
//Trying Binary
fs.writeFileSync(`.tmp/file.png`, body, {encoding: 'binary'});
}
body comes back as:
�PNG↵↵IHDRv&vл� IDATx�}���<z�f���];��o]��A�N�.po�/�/R���..............
1). Request an image from googleusercontent Google CDN (image was originally pasted in a Google Doc)
2). Create an image file and write to disk on the Server.
Neither of the fs.writeFileSync seem to produce a readable image file.
Any advice on handling this would be awesome..
Pass the response as the body to your S3 upload.
var request = require('request'),
fs = require('fs'),
aws = require('aws-sdk'),
s3 = new aws.S3(),
url = 'https://lh4.googleusercontent.com/-2XOcvsAH-kc/VHvmCm1aOoI/AAAAAAABtzg/SDdN1Vg5FFs/s346/14%2B-%2B1';
# Store in a file
request(url).pipe(fs.createWriteStream('file.gif'));
request(url, {encoding: 'binary'}, function(error, response, body) {
# Another way to store in a file
fs.writeFile('file.gif', body, 'binary', function(err) {});
# Upload to S3
s3.upload({
Body: body,
Bucket: 'BucketName',
Key: 'file.gif',
}, function(err, data) {});
});

Corrupt/truncated mp4 upload to S3 bucket using NodeJS and AWS S3

I'm using fs.readFile to read in a local mp4 file of around 700KB, and AWS SDK to load the file onto my S3 bucket. The process works, but the file is corrupt, presumably truncated because the size of the resulting file on S3 is only around 500KB.
The code I'm using works for images just fine. Just not working for mp4 files. Here's my code:
fs.readFile(file_location, function (err, data) {
if (err) {
console.log('fs error': err);
} else {
var base64data = new Buffer(data, 'binary');
var params = {
Bucket: config.settings.bucket,
Key: 'file.mp4',
Body: base64data,
ContentEncoding: 'base64',
ContentType: 'video/mp4'
};
s3.putObject(params, function(err, data) {
if (err) {
console.log('Error putting object on S3: ', err);
} else {
console.log('Placed object on S3: ', object_key);
}
});
}
});
I've tried omitting the ContentEncoding and/or the ContentType properties from the params object.
EDIT:
So when I did a console.log of the length of the data that fs.readFile returned, it consistently comes up short. I didn't realize I wasn't actually handling base64 data, either. So if I use the following code where I ask to read the file in using base64, the data.length from fs.readFile appears to be correct, and an mp4 file shows up on my S3 bucket with a playhead in the browser (which I wasn't getting before), but the video won't play & it's black, so it appears to still be corrupt.
fs.readFile(file_location, 'base64', function (err, data) {
if (err) {
console.log('fs error': err);
} else {
var params = {
Bucket: config.settings.bucket,
Key: 'file.mp4',
Body: data,
ContentEncoding: 'base64',
ContentType: 'video/mp4'
};
s3.putObject(params, function(err, data) {
if (err) {
console.log('Error putting object on S3: ', err);
} else {
console.log('Placed object on S3: ', object_key);
}
});
}
});
The data variable you get back from readFile is already good to go. Just pass that directly on with Body: data. Drop the ContentEncoding param entirely as you are not sending base64 data.

How can I upload a file from an HTTP response body to S3 using putObject() from the AWS SDK (in Node.js)?

I'm trying to save a PDF file into S3 with the AWS SDK.
I'm getting the PDF through the body of a POST request (Application/PDF).
When saving the file into the local HD with fs.writeFile, the file looks ok. But when uploading it to S3, the file is corrupted (it's just a single
page PDF).
Any help or hint would be greatly appreciated!
var data = body // body from a POST request.
var fileName = "test.pdf";
fs.writeFile(fileName, data, {encoding : "binary"}, function(err, data) {
console.log('saved'); // File is OK!
});
s3.putObject({ Bucket: "bucketName", Key: fileName, Body: data }, function(err, data) {
console.log('uploaded') // File uploads incorrectly.
});
EDIT:
It works if I write and then read the file and then upload it.
fs.writeFile(fileName, data, {encoding : "binary"}, function(err, data) {
fs.readFile(fileName, function(err, fileData) {
s3.putObject({ Bucket: "bucketName", Key: fileName, Body: fileData }, function(err, data) {
console.log('uploaded') // File uploads correctly.
});
});
});
Try setting the contentType and/or ContentEncoding on your put to S3.
ContentType: 'binary', ContentEncoding: 'utf8'
See the code sample here for working example putObject makes object larger on server in Nodejs
I think it is because the data is consumed (i.e. a stream).
It would explain why after writting the data you send nothing to S3 and reading again the data you can send a valid PDF.
Try and see if it works by just sending the data directly to S3 without writting it to disk.
Yes, you forgot about callback of writeFile function, so when you started uploading to Amazon S3 your file wasn't saved completly. You shouldn't forget that node.js is asynchronous and an app won't wait when the fs.writeFile finishes it work, it simply run s3.putObject the same time.
/**
* JS library: Promise.promisify from bluebirdjs
**/
My code is as below
global.Promise = require('bluebird');
const aws = require('aws-sdk');
const aswAccessKey = {
accessKeyId: 'your-accesskey-id',
secretAccessKey: 'your-secret-access-key'
};
const fs = require('fs');
const path = require('path');
const uuidV4 = require('uuid/v4');
// Create S3 service object
// available apiVersion: '2006-03-01', '2013-04-01',
const s3 = new aws.S3(Object.assign(aswAccessKey, {
apiVersion: '2013-04-01'
}));
function putObject(bucketName, file) {
console.log('putObject into ', bucketName);
/**
* If we don't use versioned bucket, we must not pass VersionId
*/
const params = {
Bucket: bucketName,
Key: '',
Body: 'Plain text',
ACL: 'public-read',
ContentType: 'binary',
CacheControl: 'max-age=172800'
};
return Promise
.promisify(fs.readFile, {
context: fs
})(file)
.then((fileData) => {
console.log(fileData);
params.Body = fileData;
params.Key = 'g01/' + uuidV4() + '-' + path.basename(file);
return Promise
.promisify(s3.putObject, {
context: s3
})(params)
.then((data) => {
console.log('successful');
console.log(data);
})
.catch((err) => {
console.log('Error', err);
});
})
.catch(() => {
});
}

How to buffer an HTTP response using the request module?

I would like to stream the contents of an HTTP response to a variable. My goal is to get an image via request(), and store it in in MongoDB - but the image is always corrupted.
This is my code:
request('http://google.com/doodle.png', function (error, response, body) {
image = new Buffer(body, 'binary');
db.images.insert({ filename: 'google.png', imgData: image}, function (err) {
// handle errors etc.
});
})
What is the best way to use Buffer/streams in this case?
The request module buffers the response for you. In the callback, body is a string (or Buffer).
You only get a stream back from request if you don't provide a callback; request() returns a Stream.
See the docs for more detail and examples.
request assumes that the response is text, so it tries to convert the response body into a sring (regardless of the MIME type). This will corrupt binary data. If you want to get the raw bytes, specify a null encoding.
request({url:'http://google.com/doodle.png', encoding:null}, function (error, response, body) {
db.images.insert({ filename: 'google.png', imgData: body}, function (err) {
// handle errors etc.
});
});
var options = {
headers: {
'Content-Length': contentLength,
'Content-Type': 'application/octet-stream'
},
url: 'http://localhost:3000/lottery/lt',
body: formData,
encoding: null, // make response body to Buffer.
method: 'POST'
};
set encoding to null, return Buffer.
Have you tried piping this?:
request.get('http://google.com/doodle.png').pipe(request.put('{your mongo path}'))
(Though not familiar enough with Mongo to know if it supports direct inserts of binary data like this, I know CouchDB and Riak do.)
Nowadays, you can easily retreive a file in binary with Node 8, RequestJS and async await. I used the following:
const buffer = await request.get(pdf.url, { encoding: null });
The response was a Buffer containing the bytes of the pdf. Much cleaner than big option objects and old skool callbacks.

Resources