Converting recevied body of axios POST to Blob/ArrayBuffer - node.js

I have a React/Typescript frontend that is suppose to send a file to an Express backend via an Axios POST call. The Express backend is then suppose to forward the file to Azure storage. The Azure append method expects "a string, Blob, ArrayBuffer, ArrayBufferView, or a function returning NodeJS.ReadableStream".
However, what I receive on the backend (req.body) is not in any of those formats. I am not quite sure what is the format of the data received by the backend is in and/or how to convert it to one of the required formats. The console.log(req.body) for a very simple image gives the following
[Object: null prototype] { '�PNG\r\n\x1A\n\x00\x00\x00\rIHDR\x00\x00\x00#\x00\x00\x00"\b\x02\x00\x00\x00Z���\x00\x00\x00\x01sRGB\x00��\x1C�\x00\x00\x00\x04gAMA\x00\x00��\x0B�a\x05\x00\x00\x00\tpHYs\x00\x00\x0E�\x00\x00\x0E�\x01�o�d\x00\x00\x00\x1AIDATHK��\x01\x01\x00\x00\x00� ��nH#\x00\x00\x00��\x01\x0E\x14\x00\x01\x05?��\x00\x00\x00\x00IEND�B`�': ''}
I tried setting Content-Type to multipart/form-data (and using formData.append as is done in this post How to post a file from a form with Axios) or setting Content-Type to file.type; however, in those cases I just receive empty object ({}) on the backend for req.body.
Frontend
const onHandleChange = async (event: any) => {
const file: File = event.currentTarget.files[0];
axios
.post("/api/files", file, {
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
})
.then((uploadResponse: AxiosResponse | Promise<AxiosResponse>) =>
console.log(uploadResponse)
);
};
return (
<>
<p>Demo</p>
<form>
<input type="file" id="fileField" onChange={onHandleChange}></input>
</form>
</>
);
Backend
router.post('/', async (req, res, next) => {
try {
const datalakeServiceClient = new DataLakeServiceClient(
`https://${ACCOUNT}.dfs.core.windows.net`,
defaultCredential
);
const fileSystemClient = datalakeServiceClient.getFileSystemClient(FILESYSTEM);
const fileClient = fileSystemClient.getFileClient('simpleFile');
const content = req.body;
const length = parseInt(req.header('Content-Length'));
await fileClient.create();
await fileClient.append(content, 0, length); // This line expects Blob/ArrayBuffer, etc. and gives the error
await fileClient.flush(length);
console.log('Done');
} catch (error) {
throw new Error(error);
}
});

Related

Sending blob image from Angular to ExpressJS

I'm trying to send a blob image, but I'm getting Error: Unexpected end of form using multer with Serverless Framework.
From console.log
My understanding is I have to append it to FormData before sending it in the body, but I haven't been able to get backend to accept file without crashing
uploadImage(imageData: File) {
console.log('IMAGE DATA', imageData);
let formData = new FormData();
formData.append('file', imageData, 'file.png');
let headers = new HttpHeaders();
headers.append('Content-Type', 'multipart/form-data');
headers.append('Accept', 'application/json');
let options = { headers: headers };
const api = environment.slsLocal + '/add-image';
const req = new HttpRequest('PUT', api, formData, options);
return this.http.request(req);
}
backend
const multerMemoryStorage = multer.memoryStorage();
const multerUploadInMemory = multer({
storage: multerMemoryStorage
});
router.put(
'/add-image',
multerUploadInMemory.single('file'),
async (req, res: Response) => {
try {
if (!req.file || !req.file.buffer) {
throw new Error('File or buffer not found');
}
console.log(`Upload Successful!`);
res.send({
message: 'file uploaded'
});
} catch (e) {
console.error(`ERROR: ${e.message}`);
res.status(500).send({
message: e.message
});
}
console.log(`Upload Successful!`);
return res.status(200).json({ test: 'success' });
}
);
app.ts
import cors from 'cors';
import express from 'express';
import routers from './routes';
const app = express();
import bodyParser from 'body-parser';
app.use(cors({ maxAge: 43200 }));
app.use(
express.json({
verify: (req: any, res: express.Response, buf: Buffer) => {
req.rawBody = buf;
}
})
);
app.use('/appRoutes', routers.appRouter);
app.use(
bodyParser.urlencoded({
extended: true // also tried extended:false
})
);
export default app;
From my understanding with serverless framework I have to install
npm i serverless-apigw-binary
and add
apigwBinary:
types: #list of mime-types
- 'image/png'
to the custom section of the serverless template yaml file.
The end goal is not to save to storage like S3, but to send the image to discord.
What am I missing? I appreciate any help!
I recently encountered something similar in a react native app. I was trying to send a local file to an api but it wasn't working. turns out you need to convert the blob file into a base64 string before sending it. What I had in my app, took in a local file path, converted that into a blob, went through a blobToBase64 function, and then I called the api with that string. That ended up working for me.
I have this code snippet to help you but this is tsx so I don't know if it'll work for angular.
function blobToBase64(blob: Blob) {
return new Promise((resolve, reject) => {
const reader = new FileReader();
reader.onerror = reject;
reader.onload = () => {
resolve(reader.result as string);
};
reader.readAsDataURL(blob);
});
}
Hope this helps!
You can convert your Blob to a File using
new File([blob], "filename")
and then you should be able pass that file to your existing uploadImage method.
Looks like you are passing Blob instead of File based on your console.log(). So you should convert Blob to a File before calling the server. You can change your frontend code like this:
uploadImage(imageData: File) {
// Convert Blob to File
const file = new File([imageData], "file_name", { type: imageData.type });
let formData = new FormData();
formData.append('file', file, 'file.png');
const api = environment.slsLocal + '/add-image';
return this.http.put(api, formData);
}
Note: For more info about converting Blob to File, you can check this StackOverflow question.
The thing that got it working for me was this article.
There might be something different about using Express through Serverless Framework so things like mutler and express-fileupload might not work. Or could be because it's an AWS Lambda function. I don't know this for sure though. I just know I never got it working. This article was the only thing that worked for Serverless Framework + Express.
I also had to install version 0.0.3 of busboy ie npm i busboy#0.0.3. The newer version didn't work for busboy. Newer version was saying Busboy is not a constructor
Since I'm sending the file to discord and not S3 like this article does, I had to tweak the parser.event part in this part of the article for the handler.ts
export const uploadImageRoute = async (
event: any,
context: Context
): Promise<ProxyResult> => {
const parsedEvent: any = await parser(event);
await sendImageToDiscord(parsedEvent.body.file);
const response = {
statusCode: 200,
body: JSON.stringify('file sent successfully')
};
return response;
};
comes in as a Buffer which I was able to send as a file like this
const fs = require('fs-extra');
const cwd = process.cwd();
const { Webhook } = require('discord-webhook-node');
const webhook = new Webhook('<discord-webhook-url>');
export async function sendImageToDiscord(arrayBuffer) {
var buffer = Buffer.from(arrayBuffer, 'base64');
const newFileName = 'nodejs.png';
await fs.writeFile(`./${newFileName}`, buffer, 'utf-8').then(() => {
webhook.sendFile(`${cwd}/${newFileName}`);
});
}
});
I hope this helps someone!

Send uploaded files with multer to another API from my own typescript API using axios

I'm working on a Node Typescript Rest API with Express. I did an uploadFiles middleware using multer that store all the files uploaded on a diskStorage, and this works fine.
But then, I want to immediately send these same files to another API to process them and extract data.
I am stuck here because I can't achieve to send these files with axios in a formData.
I first tried to access my files on req.files, and then send them with axios like this :
const filesMulter = req.files as Express.Multer.File[];
for await (const cvFile of filesMulter) {
const formData = new FormData();
formData.append("file", cvFile);
const extractRes = await axios.post(
`${process.env.API_EXTRACT_BASEURL}/cv/`,
formData,
{
headers: {
"Content-Type": "multipart/form-data",
},
}
);
}
But I got this error in the append() :
Argument of type 'File' is not assignable to parameter of type 'string | Blob'.
So I tried to convert the file to a Blob, so the file is accepted in the formData.append(), but then I get this axios error :
AxiosError: Data after transformation must be a string, an ArrayBuffer, a Buffer, or a Stream
I tried to convert the file in different formats (Buffer, Stream, string...) but I get the append() error or the Axios error mentionned before...
I even tried to get again the file from the disk by using
import fs from "fs";
const getFile = async (filePath: string): Promise<Buffer> => {
return await new Promise((resolve, reject) => {
fs.readFile(filePath, (error, data) => {
if (error != null) {
reject(error);
} else {
resolve(data);
}
});
});
};
export default getFile;
And then :
const cvFileOnDisk = await getFile("/path/to/my/file/hardcoded/example.pdf");
This return a Buffer, but again I cannot add it to the formData with append()...
Any idea how I could solve this ?
Thanks a lot !

Get external api with axios and req body to url for search and ajax NodeJS express

I use node.js expess with MVC pattern and use axios for get json url. The all logic request api i wrote it in Controller , I try to create some get api with axios like this.
this my chartController.js , it's some logic for get json from url with req.body.symbol (it's just symbol of stock from user submitted input form , I want to get that symbol to my axios get url in code below in const chartChartPage )
'use strict';
const axios = require('axios');
const request = require('request');
/* For Get my Api Token Data */
const dataTokens = require('../config/apitokens');
const chartChartPage = async (req,res) => {
try {
const symbol = req.body.symbol;
const url = `https://${dataTokens.sandbox}.iexapis.com/stable/stock/${symbol}/chart/1m?token=${dataTokens.tokens}`;
const fetchData = await axios.get(url);
res.status(200).json(fetchData.data);
}
catch (error){
res.status(400).send(error.message);
}
}
module.exports = {
chartPage,
chartChartPage
}
Now, i have to added some routes in my chart.js
i think i should add router.post('/', chartChartPage) for get that symbol in input when user submitted Maybe i'm wrong.
var express = require('express');
var router = express.Router();
var {chartPage , chartChartPage} = require('../controllers/chartControllers');
router.get('/', chartPage);
router.post('/', chartChartPage);
module.exports = router;
and in my chartChartPage.js (it's just js file for my template)
and use ajax to get that data from url above (with chartChartPage) to get data for build chart stock
and try to console.log that data but that's not work in console
$(function(){
chartChartPage();
});
const chartChartPage = async () => {
await $.ajax({
url: '/chart',
type: 'POST',
dataType: 'json',
success: (response) => {
if (response != null) {
console.log(response);
}
},
error: (err) => {
console.log(err);
}
});
}
and when i submitted form i got that all json data in my page, but i want when submitted it's render to same page and work in console.log in ajax get url above. How can i fix.?
enter image description here

Upload video stored in server via API with form-data content-type

I have a bunch of videos stored in a folder on my Node.js server and a React client app using Axios to make call to several APIs.
From my React client i get videos from my server and then I want to send them via POST request with an API. The content-type of that request has to be multipart/form-data. When i upload a video with HTML5 input i can post it with the API as you can see in the following code:
function testUpload(props) {
const [upload, setUpload] = useState(null);
useEffect( () => {
if(upload != null) libraryUpload("testUpload", upload)
}, [upload])
const handleChange = (e) => {
setUpload(e.target.files[0]);
}
const libraryUpload = async (mediaName, media) => {
const formData = new FormData();
formData.append("files", media);
formData.append("name", mediaName);
const config = {
headers: {
"Content-Type": "multipart/form-data"
}
}
const response = await instanceOfApi.post("/library", formData, config);
console.log(response);
}
return (
<input type="file" name="file" id="file" onChange={(e) => handleChange(e)} />
)
}
I send my videos from my node server this way :
app.get('/video/:info', (req, res) => {
res.sendFile('assets/videos/' + req.params.info, { root: __dirname });
});
I can display them in my client but i can not send them with my POST request because the data I receive from my server are not like what I receive when I use HTML5 input form to upload a video.
How can I convert those videos from my server into valid form data like i do using input type="file"? Should I change the way i send them from my server or is there a way to make them valid from my client?

Why can't I access req.body with multer?

My app (using vue) allows users to upload files with some info about the data to my node backend. When the user submits the form, this function is triggered:
methods: {
buttonOK () {
const formData = new FormData()
formData.append('name', this.detailFirm.name)
formData.append('description', this.detailFirm.description)
formData.append('version', this.detailFirm.version)
formData.append('date', this.detailFirm.date)
formData.append('file', this.file)
for (var [key, value] of formData.entries()) {
console.log(key, value)
}
let headers = {
'Content-Type': 'multipart/form-data',
'Accept': 'multipart/form-data'
}
this.$http.put('/firmware', formData, {headers: headers})
this.visible = false
}
The log statement shows everything that it ought to, and when this request is made, the network tab in the chrome dev tools shows the post data going through, and it has all the values it should:
name: test
description: test
version: 1
date: 0555-05-05
file: (binary)
My multer middleware looks like this:
const multer = require('multer')
const mult = multer({
dest: '/firmware'
})
module.exports = function (req, res, next) {
/* --Convert multipart/form-data to useable format within express-- */
if (req.path === '/firmware') {
mult.single('file')
console.log('MULTER MIDDLEWARE')
}
next()
}
The log statement there works, leading me to believe that multer is working.
I can't seem to access this information in back end though. Here I have tried both file and formData as the file name in mult.single('').
Here is my controller function:
let firmware = {
name: req.body.name,
version: req.body.version,
description: req.body.description,
date: req.body.date,
file: req.body.file
}
firmwareRepo.create(firmware, (err, create) => {
.............
I've read some other questions, and have made a few adjustments, but I always get an empty object when I log req.body in the controller. Please advise.
https://github.com/expressjs/multer#diskstorage
Note that req.body might not have been fully populated yet. It depends on the order that the client transmits fields and files to the server.
EDIT:
Firstly, I remember I had one problem on the frontend (React), by adding headers, which are not needed (somehow by adding formdata headers u **** up everything), here is the example:
data append stuff goes here
const data = new FormData()
data.append('id', values.id)
......
return async (dispatch) => {
const respond = await fetch('/api/postdata', {
method: 'post',
headers: {
//SEE? THIS IS EMPTY
},
body: data
})
// send form to backend
dispatch(dataSend())
}
}
Second issue could be on the backend. The thing is, that you can't just simply access file info through the req.body. You need to access it through the req.file
.post('/api/post', (req, res, next)=> {
const photo = {}
const newData = {}
uploadData(req, res, (err) => {
if(err){
console.log('error')
}
else {
Object.assign(photo, {file: req.file})
Object.assign(newData, {newData: req.body})
Then pass the photo to where you want to do something with it
const addDataController = new AddDataController(req, res, next, newAdvertData, photo)
addAdvertController.postAdvert()
}
})
Basically what I did is I separated regular data with file, and passed them further to combine and conclude the form. Sorry if this won't help, you're very close anyways!
I don't know why this worked, but everything started functioning as it should when I stopped using multer as an imported middleware, like this:
module.exports = function (req, res, next) {
/* --Convert multipart/form-data to useable format within express-- */
if (req.path === '/firmware') {
mult.single('formData')
console.log('MULTER MIDDLEWARE')
}
next()
}
and instead applied it directly to the route function, like this:
router.put('/firmware', upload.single('formData'), firmware.create) // doesn't work as standalone middleware
If anyone knows why that would be the case, please let me know.

Resources