I'm a bit lost on the way a video is being send from React Native to the backend and having it working on S3. All help would be appreciated especially where I might be going wrong.
Initially, from React Native I use the Expo Camera to record a video. Once it has stopped recording, we use fetch to send the data as follows:
const startRecording = async () => {
setIsRecording(true);
const video = await camera.recordAsync({
maxDuration: 15
});
const data = new FormData();
data.append('video', {
name: 'mobile-video-upload',
uri: video.uri
});
try {
const res = await fetch('url/users/testing', {
method: 'post',
body: data
});
} catch (error) {
console.log('error uploading');
}
};
the type of Data we get back from Camera component through IOS is:
Object {
"uri": "...Camera/D3B7B5F5-6A17-4C45-A0BE-897956A9E637.mov",
}
On the backend I'm using a middleware known as multer. My route for the backend looks like this
const multer = require('multer');
const upload = multer({ dest: 'uploads/' });
router.post('/testing', upload.single('video'), async (req, res) => {
let buffer = null;
fs.readFile(req.file.path, (err, data) => {
if (err) {
console.log('There was an error reading file: ', err);
}
buffer = data;
});
const params = {
Bucket: bucket_name,
Key: 'testing124.mov',
ContentType: req.file.mimetype,
Body: buffer
};
try {
let uploadPr = await s3.putObject(params).promise();
console.log(uploadPr);
} catch (error) {
console.log('There was an err ', error);
}
The data we see in req.file is:
{
fieldname: 'video',
originalname: 'mobile-video-upload',
encoding: '7bit',
mimetype: 'video/quicktime',
destination: 'uploads/',
filename: 'aed58f2dfbcc8daa7964fb3df7d3b4f4',
path: 'uploads/aed58f2dfbcc8daa7964fb3df7d3b4f4',
size: 480422
}
What might I be doing wrong in order to have a valid video uploaded? I'm unable to view the video from s3 whether I download the file or try using the link and viewing the video.
Thank you for all the help.
If your already using multer, use this:
https://www.npmjs.com/package/multer-s3
Related
I am trying to use the google drive api for storing images and retrieving them when necessary.
Uploading to drive through javascript i working fine.
The problem is when i do files.get() request for a particular file-id the response coming is having data in some weird format.(looking into header it is gzip encoding). Can someone suggest how can i convert this to base64 so that i can display the image in the front end.
code for uploading the file:
const uploadFile = async (fileObject) => {
const bufferStream = new stream.PassThrough();
bufferStream.end(fileObject.buffer);
const { data } = await driveService.files.create({
media: {
mimeType: fileObject.mimeType,
body: bufferStream,
},
requestBody: {
name: fileObject.originalname,
parents: [folderId],
},
fields: "id,name,webViewLink",
});
return data;
};
code for get request:
try{
const id = req.body.id;
driveService.files
.get({
fileId: id,
alt: "media",
})
.then(
function (response) {
res.status(200).json(response);
},
function (reason) {
throw reason;
}
);
} catch (e) {
console.error(e);
res.status(400).send(e.message);
}
the response i am getting is something like this:
Response Json for files.get request
I tried using Arraybuffers and then toString('base64'). but it did'nt helped.
There is a option to set response type while making the request.
The following changes solved the issue:
driveService.files.get(
{
fileId: id,
alt: "media",
},
{
responseType: "arraybuffer",
encoding: null,
},
function (err, response) {
if (err) throw err;
else {
var imageType = response.headers["content-type"];
var base64 = Buffer.from(response.data, "utf8").toString("base64");
var dataURI = "data:" + imageType + ";base64," + base64;
res.status(200).json({ src: dataURI });
}
now if i place this dataURI in src of an img element it works!!
I'm trying to upload an array of photos to a server but the req.files array always shows up empty when it gets there.
req.body displays the array as expected.
The images are added through a Dropzone component. (I've tried switching this for a standard input but they both seem to pass files the same way)
<Dropzone
onDrop={onDrop}
onSubmit={uploadPhotos}
maxFiles={20}
inputContent="Drop 20 Images"
inputWithFilesContent={files => `${20 - files.length} more`}
/>
The images are applied to FormData with the name image files are appended before being sent via an Axios POST request with multipart/form-data headers set.
export const uploadPhotos = (files) => {
const formData = new FormData();
for (let i = 0; i < files.length; i += 1) {
formData.append("image[]", files[i]);
}
const config = {
headers: {
'Content-Type': `multipart/form-data`
}
}
return async (dispatch, getState) => {
try {
const response = await axios.post('/api/kite/upload',
formData, config)
.then(function(response) {
console.log(response.data);
dispatch({
type: ORDER_CHANGE,
payload: response.data
});
});
} catch (err) {
console.log(err);
} finally {
console.log('done');
}
}
}
once passed to the server only req.body seems to contain any data and req.files is empty despite using Multer middleware as the second parameter. Once passed to files.map() items are undefined undefined, presumably because req.files is an empty array.
var multer = require('multer');
var AWS = require('aws-sdk');
AWS.config.setPromisesDependency(bluebird);
const storage = multer.diskStorage({
destination: (req, file, cb) => {
cb(null, 'upload')
},
filename: (req, file, cb) => {
cb(null, file.fieldname + '-' + Date.now())
}
});
const upload = multer({
storage: storage
}).array('image');
router.post('/upload', upload, function (req, res) {
const file = req.files;
let s3bucket = new AWS.S3({
accessKeyId: IAM_USER_KEY,
secretAccessKey: IAM_USER_SECRET,
Bucket: 'BUCKETNAME'
});
s3bucket.createBucket(function () {
let Bucket_Path = 'https://console.aws.amazon.com/s3/buckets/BUCKETNAME?region=eu-west-1';
var ResponseData = [];
file.map((item) => {
// item.x are all undefined
var fileStream = fs.createReadStream(filePath);
var params = {
Bucket: Bucket_Path,
Key: item.originalname,
Body: item.buffer,
ACL: 'public-read'
};
s3bucket.upload(params, function (err, data) {
if (err) {
res.json({ "error": true, "Message": err});
} else{
ResponseData.push(data);
if(ResponseData.length == file.length){
res.json({ "error": false, "Message": "File Uploaded SuceesFully", Data: ResponseData});
}
}
});
});
});
});
My end goal is to pass the images to an Amazon S3 bucket. I don't think it impacts this since there is no data to send but I've included it incase it has somehow affecting this.
I've been through lots of other similar Stack Overflow questions and medium post and the main three resolutions to this issue seem to be included in the flow above.
Append file name to items of FormData array
Set POST request headers
Include Multer middleware in express parameter
Can anyone help me figure out why req.files is an empty array?
It might be that Dropzone isn't processing the files. Try adding this to the uploadPhotos function:
const acceptedFiles = myDropzone.getAcceptedFiles() // "myDropzone" is just the Dropzone instance
for (let i = 0; i < acceptedFiles.length; i++) {
myDropzone.processFile(acceptedFiles[i])
}
I am trying to upload a file to AWS S3 using [putObject][1] but it results in files of 0 byte size.
I do get successful response back from the putObject call.
Node.js code:
const aws = require("aws-sdk");
const s3 = new aws.S3();
module.exports = {
upload: function(req, res, next) {
console.log("Going to upload");
console.log(req.files);
let uploadFile = req.files.file;
const s3PutParams = {
Bucket: process.env.S3_BUCKET_NAME,
Key: uploadFile.name,
Body: uploadFile.data,
ACL: "public-read"
};
const s3GetParams = {
Bucket: process.env.S3_BUCKET_NAME,
Key: uploadFile.name
};
console.log(s3PutParams);
s3.putObject(s3PutParams, function(err, response) {
if (err) {
console.error(err);
} else {
console.log("Response is", response);
var url = s3.getSignedUrl("getObject", s3GetParams);
console.log("The URL is", url);
res.json({
returnedUrl: url,
publicUrl: `https://${process.env.S3_BUCKET_NAME}.s3.amazonaws.com/${uploadFile.name}`
});
}
});
}
};
Testing through POSTMAN:
Backend Console log
Can anyone help me in figuring out what is wrong?
EDIT on 11/20:
#EmmanuelNK helped in spotting the fact that Buffer.byteLength(req.files.file.data) is 0. He had the below questions:
Are you trying to write the whole buffer into memory or are you trying to stream it to s3?
Sorry if the answer is not to the point, still getting my feet wet.
Basically I want to upload an image to S3 and then later use that URL to show it on a webpage. In other words like a photobucket
how you are using upload
For now I am just testing my backend code (posted in the question) using postman. Once I get that going, will have a file upload form on the front end calling this route.
Is that helpful? Thanks in advance for your help.
If you're using express-fileupload as the file uploading middleware, and you've set the useTempFiles option to true, keep in mind that your data file buffer will be empty (check usage), which correlates to the issue you're facing. To get around this, simply read the temp. file once more to get the intended file buffer.
import fs from 'fs';
// OR
const fs = require('fs');
// in your route
let uploadFile = req.files.file;
// THIS
fs.readFile(uploadedFile.tempFilePath, (err, uploadedData) => {
if (err) { throw err; }
const s3PutParams = {
Bucket: process.env.S3_BUCKET_NAME,
Key: uploadFile.name,
Body: uploadData, // <--- THIS
ACL: "public-read"
};
const s3GetParams = {
Bucket: process.env.S3_BUCKET_NAME,
Key: uploadFile.name
};
console.log(s3PutParams);
s3.putObject(s3PutParams, function(err, response) {
if (err) {
console.error(err);
throw err;
} else {
console.log("Response is", response);
var url = s3.getSignedUrl("getObject", s3GetParams);
console.log("The URL is", url);
res.json({
returnedUrl: url,
publicUrl: `https://${process.env.S3_BUCKET_NAME}.s3.amazonaws.com/${uploadFile.name}`
});
}
});
});
Below is the Rest API to upload the video to S3,unable to play the video by downloading as players throwing invalid file format.
app.post('/insert-video', async(req, res) => {
const {
data,
name: fileName,
size: fileSize,
type: fileType
} = req.body
// ASW code start
const base64data = new Buffer(data, 'binary');
let params = {
Bucket: "uploadvidoe",
Key: fileName,
ContentType: fileType,
Body: base64data
};
try {
let uploadPromise = await new AWS.S3().putObject(params).promise();
console.log("Successfully uploaded data to bucket");
} catch (e) {
console.log("Error uploading data: ", e);
}
});
I am using Multer for file uploading in Node js, but I am not getting proper file. I am trying to upload the image.
I have to upload the imported file on the S3 bucket but the file is not found, I don't know where I am wrong.
Here is my node js code:
import "#tsed/multipartfiles";
import Path = require("path");
const rootDir = Path.resolve(__dirname);
const aws = require("aws-sdk");
import { Express } from "express";
import { Controller, Post } from "#tsed/common";
type MulterFile = Express.Multer.File;
import { MultipartFile } from "#tsed/multipartfiles";
#controller("/session")
export class TeleconsultSessionController implements interfaces.Controller {
#httpPost("/uploadDocuments")
public async uploadDocument( #MultipartFile() file: MulterFile, req: Request, res: Response) {
try {
aws.config.update({ accessKeyId: "AKIAIYKXXXXXXXXX", secretAccessKey: "iMzE0wfryXXXXXXXXXXXXXXXXXXXX" });
aws.config.update({ region: "us-east-1" });
const s3 = new aws.S3();
s3.upload({
"Bucket": "teleconsult-development",
"Key": "image",
"Body": file
}, function (err: any, data: any) {
if (err) {
console.log("Error uploading data: ", err);
} else {
console.log(data);
}
});
}
catch (err) {
if (err instanceof ValidationException) {
res.status(400);
res.send({ error: err.getMessage() });
} else {
res.status(500);
res.send({ error: err.getMessage() });
}
}
}
In response when I hit through postman I am getting the value of file as:
Object {------WebKitFormBoundaryAo1BhVpGaB8uYmsw\r\nContent-Disposition: form-data; name: ""image1"; filename="1528114981689_566_28-1-2
Uploading files with multer will keep files inside server's upload folder first depends on your multer settings.
If a file is successfully uploaded by multer, you will get the following data from req.file.
{
fieldname: 'file',
originalname: 'sample.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
destination: './uploads/',
filename: 'd1fccf5dc0125937f6fbe7a4dacfc069',
path: 'uploads/d1fccf5dc0125937f6fbe7a4dacfc069',
size: 23904
}
The path value I will be using in the example below is not a directory. Multer will change your filename withiout an extension.
You will need to read that uploaded file as fileStream and then upload it to S3 will works.
For example:
let path = req.file;
let fileStream = fs.createReadStream(file.path);
fileStream.on('error', function (err) {
if (err) { throw err; }
});
fileStream.on('open', function () {
s3.upload({
"Bucket": "yourBucketName",
"Key": file.originalname,
"Body": fileStream
}, function (err: any, data: any) {
if (err) {
console.log("Error uploading data: ", err);
} else {
console.log(data);
}
});
});
Hope this helps you.