I am using the multer package to handle uploading files to a google cloud bucket. Initially, my multer package was working correctly but once I migrated to firebase I found that all my attempts to upload files were not resolving.
After running firebase functions:log I found I was receiving the message Error: Unexpected end of form at Multipart._final I'm not totally sure why this error may be occurring especially since it was previously working.
This question seems to have the same issue however there don't seem to be any answers which help me. If possible I would like to know what the error message itself means as well.
Below is my code for invoking my multer function
const sendFile = () =>{
if(acceptedFiles.length > 0){
for(let i = 0; i < acceptedFiles.length; i++){
let file = acceptedFiles[i]
let blob = file.slice(0,file.size)
let newFile = new File([blob], file.name, {type: "text/plain"});
let formData = new FormData();
formData.append('dataFile',newFile)
fetch('https://somePath',{
method: "Post",
body: formData
})
.then(res => res.text())
.then((x) => console.log(x))
.then(() => sendDB(acceptedFiles[i],i))
}
}else{
window.alert("No files have been selected to be uploaded")
}
}
Here is where I actually send my file to my google bucket
app.post('/uploadFile', multer.single('dataFile'),async (req,res) =>{
console.log('made it to upload')
try{
if(req.file){
console.log('file found trying to upload....')
const blob = bucket.file(req.file.originalname);
const blobstream = blob.createWriteStream();
blobstream.on('finish',() => {
res.status(200).send('Success')
})
blobstream.end(req.file.buffer);
}
}catch(error){
res.status(500).send(error);
console.log(error)
}
})
Here is how I configured my multer and google cloud storage
const multer = Multer({
storage: Multer.memoryStorage()
})
let projectId = 'someID'
let keyFilename = 'someKeyFileName'
const storage = new Storage({
projectId,
keyFilename
})
I'm more than happy to provide any additional information if it's needed.
Update:
After some more research, it seems that firebase does not support serverside file upload. Is this only when using firebase storage or will this also apply to my case where I am attempting to upload my files to a google cloud bucket? Reference to this claim in case I'm incorrect.
I'm not sure if it's your case, because I was getting the same error, but without firebase, however in my case the problem was Multer itself. Dowgraded to 1.4.3 and it started working.
See https://github.com/expressjs/multer/issues/1144
Related
I'm trying to send a blob image, but I'm getting Error: Unexpected end of form using multer with Serverless Framework.
From console.log
My understanding is I have to append it to FormData before sending it in the body, but I haven't been able to get backend to accept file without crashing
uploadImage(imageData: File) {
console.log('IMAGE DATA', imageData);
let formData = new FormData();
formData.append('file', imageData, 'file.png');
let headers = new HttpHeaders();
headers.append('Content-Type', 'multipart/form-data');
headers.append('Accept', 'application/json');
let options = { headers: headers };
const api = environment.slsLocal + '/add-image';
const req = new HttpRequest('PUT', api, formData, options);
return this.http.request(req);
}
backend
const multerMemoryStorage = multer.memoryStorage();
const multerUploadInMemory = multer({
storage: multerMemoryStorage
});
router.put(
'/add-image',
multerUploadInMemory.single('file'),
async (req, res: Response) => {
try {
if (!req.file || !req.file.buffer) {
throw new Error('File or buffer not found');
}
console.log(`Upload Successful!`);
res.send({
message: 'file uploaded'
});
} catch (e) {
console.error(`ERROR: ${e.message}`);
res.status(500).send({
message: e.message
});
}
console.log(`Upload Successful!`);
return res.status(200).json({ test: 'success' });
}
);
app.ts
import cors from 'cors';
import express from 'express';
import routers from './routes';
const app = express();
import bodyParser from 'body-parser';
app.use(cors({ maxAge: 43200 }));
app.use(
express.json({
verify: (req: any, res: express.Response, buf: Buffer) => {
req.rawBody = buf;
}
})
);
app.use('/appRoutes', routers.appRouter);
app.use(
bodyParser.urlencoded({
extended: true // also tried extended:false
})
);
export default app;
From my understanding with serverless framework I have to install
npm i serverless-apigw-binary
and add
apigwBinary:
types: #list of mime-types
- 'image/png'
to the custom section of the serverless template yaml file.
The end goal is not to save to storage like S3, but to send the image to discord.
What am I missing? I appreciate any help!
I recently encountered something similar in a react native app. I was trying to send a local file to an api but it wasn't working. turns out you need to convert the blob file into a base64 string before sending it. What I had in my app, took in a local file path, converted that into a blob, went through a blobToBase64 function, and then I called the api with that string. That ended up working for me.
I have this code snippet to help you but this is tsx so I don't know if it'll work for angular.
function blobToBase64(blob: Blob) {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onerror = reject;
reader.onload = () => {
resolve(reader.result as string);
};
reader.readAsDataURL(blob);
});
}
Hope this helps!
You can convert your Blob to a File using
new File([blob], "filename")
and then you should be able pass that file to your existing uploadImage method.
Looks like you are passing Blob instead of File based on your console.log(). So you should convert Blob to a File before calling the server. You can change your frontend code like this:
uploadImage(imageData: File) {
// Convert Blob to File
const file = new File([imageData], "file_name", { type: imageData.type });
let formData = new FormData();
formData.append('file', file, 'file.png');
const api = environment.slsLocal + '/add-image';
return this.http.put(api, formData);
}
Note: For more info about converting Blob to File, you can check this StackOverflow question.
The thing that got it working for me was this article.
There might be something different about using Express through Serverless Framework so things like mutler and express-fileupload might not work. Or could be because it's an AWS Lambda function. I don't know this for sure though. I just know I never got it working. This article was the only thing that worked for Serverless Framework + Express.
I also had to install version 0.0.3 of busboy ie npm i busboy#0.0.3. The newer version didn't work for busboy. Newer version was saying Busboy is not a constructor
Since I'm sending the file to discord and not S3 like this article does, I had to tweak the parser.event part in this part of the article for the handler.ts
export const uploadImageRoute = async (
event: any,
context: Context
): Promise<ProxyResult> => {
const parsedEvent: any = await parser(event);
await sendImageToDiscord(parsedEvent.body.file);
const response = {
statusCode: 200,
body: JSON.stringify('file sent successfully')
};
return response;
};
comes in as a Buffer which I was able to send as a file like this
const fs = require('fs-extra');
const cwd = process.cwd();
const { Webhook } = require('discord-webhook-node');
const webhook = new Webhook('<discord-webhook-url>');
export async function sendImageToDiscord(arrayBuffer) {
var buffer = Buffer.from(arrayBuffer, 'base64');
const newFileName = 'nodejs.png';
await fs.writeFile(`./${newFileName}`, buffer, 'utf-8').then(() => {
webhook.sendFile(`${cwd}/${newFileName}`);
});
}
});
I hope this helps someone!
So I'm trying to make the html form:
<form action="blahblah" encblah="multipart/form-data" whatever>
Thats not the problem, I need to make that form send the blob to express
app.post('/upload/avatars', async (req, res) => {
const body = req.body;
console.log(req.file);
console.log(body);
res.send(body);
});
So I can access the blob, create a read stream, pipe it to the cloud, and bam, upload the file without downloading anything on the express server it self.
Is that possible?
If yes, please tell me how.
If no, please tell me other alternatives.
On the client we do a basic multi-part form upload. This example is setup for a single image but you could call uploadFile in sequence for each image.
//client.ts
const uploadFile = (file: File | Blob) => {
const formData = new FormData();
formData.append("image", file);
return fetch("/upload", {
method: "post",
body: formData,
});
};
const handleUpload = (event: any) => {
return event.target.files.length ? uploadFile(event.target.files[0]) : null;
};
On the server we can use multer to read the file without persisting it to disk.
//server.js
const express = require("express");
const app = express();
const multer = require("multer");
const upload = multer();
app.post(
"/upload",
upload.fields([{ name: "image", maxCount: 1 }]),
(req, res, next) => {
console.log("/upload", req.files);
if (req.files.image.length) {
const image = req.files.image[0]; // { buffer, originalname, size, ...}
// Pipe the image.buffer where you want.
res.send({ success: true, count: req.files.image.originalname });
} else {
res.send({ success: false, message: "No files sent." });
}
}
);
For larger uploads I recommend socket.io, but this method works for reasonably sized images.
it is possible, but when you have a lot of traffic it would overwhelm your express server (in case you are uploading videos or big files ) but if it's for uploading small images (profile image, etc...) you're fine. either way you can use Multer npm
I'd recommend using client-side uploading on ex: s3-bucket, etc..., which returned a link, and therefore using that link.
Looking for help on Uploading and Retrieving Images from MongoDb using multer.
My front end is ReactNative.(Not sure if this is needed but just to be sure.)
Multer
Problem: After looking and following tutorials i'm able to encode my path to base64 and upload it to my DB but now i'm confused how to retrieve the file from my DB. I saw some tutorials about decoding it from base64 but I don't quite understand how do I go about retrieving an image and displaying it in postman. (I tried looking but haven't found anything that gives me an answer. I'm sorry if this is a duplicated question. If you could point me in a direction or give me some advice I would be really greatful.)
**POST**
route.post("/sad", upload.single("image"), (req, res, next) => {
console.log(req.file);
const img = fs.readFileSync(req.file.path);
const img_enc = img.toString('base64');
const obj = {
usrImage: {
data: new Buffer.from(img_enc, 'base64'),
contentType: "image/jpg",
},
};
console.log(obj);
const newAccout = new account(obj);
newAccout.save();
});
**RETRIEVE**
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.json(img)
//How do decode my buffer to show an image in Postman?
})
}
)
I am trying to create a userprofile where a username,password and image is saved. If you can help save an Image and then retrieve it from my accounts collection.
Hey I would advise that you start using a 3rd party for file upload like cloudinary very good way of managing files i.e images or video...
I am not that well of with multer but I can give a quick code example using Formidable does the same work as multer
Before you can start you'd need to make an account on cloudinary.com(don't worry its free)
Code below is how you could handle file upload
const Formidable = require("formidable"); //Meant for body parsing
const cloudinary = require("cloudinary").v2; // file uploader
//This below is your connection/configuration to get access to your cloudinary account so cloud_name, api_key and api_secret you'll get in your home dashboard(Cloudinary)
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
});
router.post('/api/file-upload', (req, res)=>{
const form = new Formidable.InconmingForm();
form.parse(req, (error, fields, files)=>{
const {file} = files
cloudinary.uploader.upload(file.path, {folder:"/"}, (err, res)=>{
const file_url = res.secure_url //This would be the url for your file given back by cloudinary
})
})
})
This script should upload your file and the file_url will be having the url of the file that you upload having ssl then after that you can now continue saving to mongoDB
Cloudinary docs for NodeJS
https://cloudinary.com/documentation/node_integration
Nice clear and understandable docs
Shameless plug
If you get lost you can check this video out on YouTube that I made handling file upload with cloudinary then save url given back to mongoDB
https://youtu.be/mlu-tbr2uUk
First call api find one
you will need fs module to complete following query
const fs = require('fs');
let data = await db.user.findOne({
where: {
id = req.body.id
}
})
// _________________ base 64 string data from findone query data
// |
let buff = new Buffer(data.image, 'base64');
let name = name.jpeg
let path = `tmp/${name}`; // <--- destination and file name you want to give to your file
fs.writeFileSync(path, buff);// < --this will write file to given path
fs.readFile(path, function (err, content) {// <------to send file in postman response
if (err) {
res.writeHead(400)
console.log(err);
res.end("No such image");
} else {
//specify the content type in the response will be an image
res.writeHead(200);
res.end(content);
}
});
fs.unlink(path, (err) => { // <-----to delete file from tmp directory
if (err) {
console.log(err)
}
})
Try this and switch to preview tab in postman.
I haven't tried it but maybe it helps.
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.setHeader('contentType','image/jpg').send(img)
})
})
I have a script that can call a RESTful API and retrieve CSV data from a report in chunks. I'm able to concatenate, parse, and display this data in the console. I am also able to write this CSV data to a local file and store it.
What I am trying to figure out is how to skip creating a file to store this data before uploading it to GCS and instead transfer it directly into Google Cloud Storage to save as a file. Since I am trying to make this a serverless cloud function, I am trying to stream it directly from memory into a Google Cloud Storage file.
I found this 'Streaming Transfers' documentation on google, but it only references doing this with 'gsutil' and I am struggling to find any examples or documentation on how to do this with node.js. I also tried to follow this answer on Stack overflow, but it's from 2013 and the methods seem a little out-dated. My script also isn't user-facing, so I don't need to hit any routes.
I am able to upload local files directly to my bucket using the function below, so Authentication isn't an issue. I'm just unsure how to convert a CSV blob or object in memory into a file in GCS. I haven't been able to find many examples so wasn't sure if anyone else has solved this issue in the past.
const { Storage } = require('#google-cloud/storage');
const storage = new Storage({
projectId,
keyFilename
});
function uploadCSVToGCS() {
const localFilePath = './test.csv';
const bucketName = "Test_Bucket";
const bucket = storage.bucket(bucketName);
bucket.upload(localFilePath);
};
I also found a 3rd party plugin that Google references called 'boto' that seems to do what I want, but this is for python, not node.js unfortunately.
Streaming object data to Cloud Storage is illustrated in the documentation. You will need to understand how node streams work, and make use of createWriteStream. The sample code is not exactly what you want, but you'll use the same pattern:
function sendUploadToGCS (req, res, next) {
if (!req.file) {
return next();
}
const gcsname = Date.now() + req.file.originalname;
const file = bucket.file(gcsname);
const stream = file.createWriteStream({
metadata: {
contentType: req.file.mimetype
},
resumable: false
});
stream.on('error', (err) => {
req.file.cloudStorageError = err;
next(err);
});
stream.on('finish', () => {
req.file.cloudStorageObject = gcsname;
file.makePublic().then(() => {
req.file.cloudStoragePublicUrl = getPublicUrl(gcsname);
next();
});
});
stream.end(req.file.buffer);
}
#doug-stevenson thanks for pushing me in the right direction. I was able to get it to work with the following code:
const { Storage } = require('#google-cloud/storage');
const storage = new Storage();
const bucketName = 'test_bucket';
const blobName = 'test.csv';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const request = require('request');
function pipeCSVToGCS(redirectUrl) {
request.get(redirectUrl)
.pipe(blob.createWriteStream({
metadata: {
contentType: 'text/csv'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
I am using google-cloud-storage on Node.js . Trying to find a solution to programmatically download an image from an external url then upload to GCS.
I'm using fetch API to get the image and turn it into a blob
fetch('image.jpg').then( res => {
var blob = res.blob()
})
All tutorials I've found on the web deal with form upload using multer. But nothing about my case.
Now what's the right solution to upload this blob on my GCS bucket ?
EDIT : I've also tried this, without success
bucket.upload(urlImage, function(err, file){
const blobStream = file.createWriteStream();
blobStream.on('error', (err) => {
console.log('error',err)
});
blobStream.on('finish', () => {
console.log('finished with success')
});
blobStream.end(file);
})
Try using the request module for Node.js to obtain the image from the URL and pipe the result to be written into the bucket:
var request = require('request');
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const bucket = storage.bucket('your-bucket-name');
const file = bucket.file('image-name.jpg');
request(‘image-URL’).pipe(file.createWriteStream());
It worked for me.