I am receiving some text from a POST response in a REST API. I want to directly make a text file in s3 for that. All the examples I have stumble upon are using a local file and then uploading it. Is there any way to directly upload it without saving in the Local System?
You can directly pipe the req to s3.upload function as below
import express from 'express';
import {S3} from 'aws-sdk';
const s3 = new S3();
const app = express();
app.post('/data', async (req, res) => {
var params = {
Body: req,
Bucket: "yourbucketname here",
Key: "exampleobject",
};
s3.upload(params, async (err, data) => {
if(err)
log.info(err);
else
log.info("success");
});
});
const server = app.listen(8081,async () => log.info("App listening") )
The posted file will be directly uploaded to aws s3.
Related
I've created an api to upload image to amazon s3 bucket with nodejs, multer, and multer-s3 .
Its working fine in development returning me a response of image URL which is downloadable and accessible but when I host my node app on Aws Lambda (API GATEWAY) the API returns me the same response with the image URL but this time when I open the downloaded image it shows me INVALID FILE FORMATTED
Here is my Code
const uploadImage = async (req,res) => {
let myFile = req.file.originalname.split(".")
const fileType = myFile[myFile.length - 1]
const params = {
Bucket: 'test-bucket2601',
Key: `${Date.now()}.${fileType}`,
Body: req.file.buffer
}
s3.upload(params, (error, data) => {
if(error){
res.status(500).send(error)
}
res.json({data})
})
}
Route of middleware
routes.route('/upload').post(upload,uploadImage);
in post method the first argument is upload middleware
Middleware Code:
const s3 = new aws.S3({
credentials: {
accessKeyId: awsKeys?.accessKeyId,
secretAccessKey: awsKeys?.secretAccessKey
}
});
const storage = multer.memoryStorage({
destination: function(req, file, callback) {
callback(null, '')
}})
const upload = multer({storage}).single('imageUrl')
So I'm trying to make the html form:
<form action="blahblah" encblah="multipart/form-data" whatever>
Thats not the problem, I need to make that form send the blob to express
app.post('/upload/avatars', async (req, res) => {
const body = req.body;
console.log(req.file);
console.log(body);
res.send(body);
});
So I can access the blob, create a read stream, pipe it to the cloud, and bam, upload the file without downloading anything on the express server it self.
Is that possible?
If yes, please tell me how.
If no, please tell me other alternatives.
On the client we do a basic multi-part form upload. This example is setup for a single image but you could call uploadFile in sequence for each image.
//client.ts
const uploadFile = (file: File | Blob) => {
const formData = new FormData();
formData.append("image", file);
return fetch("/upload", {
method: "post",
body: formData,
});
};
const handleUpload = (event: any) => {
return event.target.files.length ? uploadFile(event.target.files[0]) : null;
};
On the server we can use multer to read the file without persisting it to disk.
//server.js
const express = require("express");
const app = express();
const multer = require("multer");
const upload = multer();
app.post(
"/upload",
upload.fields([{ name: "image", maxCount: 1 }]),
(req, res, next) => {
console.log("/upload", req.files);
if (req.files.image.length) {
const image = req.files.image[0]; // { buffer, originalname, size, ...}
// Pipe the image.buffer where you want.
res.send({ success: true, count: req.files.image.originalname });
} else {
res.send({ success: false, message: "No files sent." });
}
}
);
For larger uploads I recommend socket.io, but this method works for reasonably sized images.
it is possible, but when you have a lot of traffic it would overwhelm your express server (in case you are uploading videos or big files ) but if it's for uploading small images (profile image, etc...) you're fine. either way you can use Multer npm
I'd recommend using client-side uploading on ex: s3-bucket, etc..., which returned a link, and therefore using that link.
I am making a multer storage engine which makes stream connection between client and S3 Server.
At middle of the stream, my code examine chunks and send it to S3.
I could get a file stream from node.js server. But when I request file array upload, node inspector shows only one stream. What should I do?
Stream engine snippet
CustomStreamEngine.prototype._handleFile = function _handleFile (req, file, cb) {
// for inspect
req.files.length // 1
file;
};
request controller
var streamStorage = multer({
storage: streamEngine()
});
dev.post('/rec_test', streamStorage.array('source'), (req, res, next) => {
});
I just published this streaming multipart/form-data parser on npm as form-parser:
You should be able to do the following:
dev.post('/rec_test', async (req, res, next) => {
// Parse request
await parser(req, async ({ fieldType, fieldName, fieldContent }) => {
// Log all fields
console.log({ fieldType, fieldName, fieldContent });
// Handle 'source' file fields
if (fieldType === 'file' && fieldName === 'source[]') {
// Get file info
const { fileName, fileType, fileStream } = fieldContent;
// Upload fileStream to S3 :-)
}
});
});
Hope it's helpful.
K
I think you can add some logs to https://github.com/expressjs/multer/blob/master/lib/make-middleware.js to check.
Currently, I use axios on the client to send multi files to the server with multer. And I can see all files in the function
busboy.on('file', function (fieldname, fileStream, filename, encoding, mimetype), but there is only one file at a time, and this function will call the _handfile function of the custom storage, so that I think it is the reason for your issue.
Hope it can help you
i want to
1-choose an image from my filesystem and upload it to server/local
2- get its url back using node js service . i managed to do step 1 and now i want to get the image url instead of getting the success message in res.end
here is my code
app.post("/api/Upload", function(req, res) {
upload(req, res, function(err) {
if (err) {
return res.end("Something went wrong!");
}
return res.end("File uploaded sucessfully!.");
});
});
i'm using multer to upload the image.
You can do something like this, using AWS S3 and it returns the url of the image uploaded
const AWS = require('aws-sdk')
AWS.config.update({
accessKeyId: <AWS_ACCESS_KEY>,
secretAccessKey: <AWS_SECRET>
})
const uploadImage = file => {
const replaceFile = file.data_uri.replace(/^data:image\/\w+;base64,/, '')
const buf = new Buffer(replaceFile, 'base64')
const s3 = new AWS.S3()
s3.upload({
Bucket: <YOUR_BUCKET>,
Key: <NAME_TO_SAVE>,
Body: buf,
ACL: 'public-read'
}, (err, data) => {
if (err) throw err;
return data.Location; // this is the URL
})
}
also you can check this express generator, which has the route to upload images to AWS S3 https://www.npmjs.com/package/speedbe
I am assuming that you are saving the image on the server file system and not a Storage solution like AWS S3 or Google Cloud Storage, where you get the url after upload.
Since, you are storing it on the filesystem, you can rename the file with a unique identifier like uuid or something else.
Then you can make a GET route and request that ID in query or path parameter and then read the file having that ID as the name and send it back.
I'm using the method where the client sends a request to the server to upload a file to an s3 bucket, and then the server sends back a signed request to allow the client to do this. I'm following this tutorial -
https://devcenter.heroku.com/articles/s3-upload-node
Does anyone know how I can write an API endpoint test for this? I'm not doing the client side code since it's an iPhone app but I still want to test my endpoint in my tests.
Based on a code like that, from your link :
app.get('/sign-s3', (req, res) => {
const s3 = new aws.S3();
const fileName = req.query['file-name'];
const fileType = req.query['file-type'];
const s3Params = {
Bucket: S3_BUCKET,
Key: fileName,
Expires: 60,
ContentType: fileType,
ACL: 'public-read'
};
s3.getSignedUrl('putObject', s3Params, (err, data) => {
if(err){
console.log(err);
return res.end();
}
const returnData = {
signedRequest: data,
url: `https://${S3_BUCKET}.s3.amazonaws.com/${fileName}`
};
res.write(JSON.stringify(returnData));
res.end();
});
});
I would do a unit test, as a full integration test will depend on your aws account on your test env. For that I would mock req and s3.getSignedUrl and test that getSignedUrl is called with the correct parameters. I would also add a test, still with mock, to be sure a correct json is returned.