Javascript File object missing from body - node.js

I have a valid object file with data in it when I do the request on the front end.
But when I receive the body on the express backend the file object is empty.
let body = {
metadata: {
date: date,
location: location,
description: description,
file: file
}
}
const headers = {'body': JSON.stringify(body)}
axios.get('/api/xummMint', {headers}).then( (res) => {
console.log("xumm data coming");
console.log(res)
setData({xummData: res});
})
when I console.log(req.headers.body) I receive an empty file object even though the file had data on the frontend:
{"metadata":{"date":"2022-12-30","location":"asf","description":"asdf","file":{}}}
I tried using fetch and adding content-Type: application/json but still same results. Can anyone help?

Chances are that you meant to send this as a POST request with a JSON body, not as a GET request with a header called "body"
axios.post(
'/api/xummMint',
body,
{headers: {'Content-Type': 'application/json'}}
).then(res => {
console.log("xumm data coming");
console.log(res);
setData({xummData: res});
});

Related

Re-uploading a file after passing them into AWS lambdas(nodejs) + API gateway

I have a REST API created using AWS Lambdas and API gateway.
This API accommodates file uploads and I have a requirement to parse the uploaded files and then send those files individual to another source as form data.
When I send two images via postman and log the event body I get the following string. (It has two files)
----------------------------269453880547064499146449
Content-Disposition: form-data; name="file"; filename="smiling.png"
Content-Type: image/png
�PNG
IHDR��asBIT|d�^IDAT8���=KBa��IB����"�-P\�6!Bz�D�c�hk�K�hAV`Km���K�PKm��Oý�R�-�O��<�#ZkzT���4B��nMȑ0����##�A�- ����7w��"�fY��J�C�)�Z`D3�a��E�h���<F�7w����d�ɉ7���Y�?f�+Y�&9�B����P����`%�d:�T
�m�h�K�`����zT;�e �mc�$=A�q���&#Y��4O=W����P#�T���*��V�t`a��H�UD��6��Һ���W[��, ��u�Ea�.c�-��S�z���Q�`���S�~y��xݡ�O�]IEND�B`�
----------------------------269453880547064499146449
Content-Disposition: form-data; name="file"; filename="smiling.png"
Content-Type: image/png
�PNG
IHDR��asBIT|d�^IDAT8���=KBa��IB����"�-P\�6!Bz�D�c�hk�K�hAV`Km���K�PKm��Oý�R�-�O��<�#ZkzT���4B��nMȑ0����##�A�- ����7w��"�fY��J�C�)�Z`D3�a��E�h���<F�7w����d�ɉ7���Y�?f�+Y�&9�B����P����`%�d:�T
�m�h�K�`����zT;�e �mc�$=A�q���&#Y��4O=W����P#�T���*��V�t`a��H�UD��6��Һ���W[��, ��u�Ea�.c�-��S�z���Q�`���S�~y��xݡ�O�]IEND�B`�
----------------------------269453880547064499146449--
I used lambda-multipart-parser to parse this body and extract the meta data.
import parser from 'lambda-multipart-parser'
....
const result = await parser.parse(event)
The result above gives the files with the following type.
{
filename: string
content: Buffer
contentType: string
encoding: string
fieldname: string
}
The issue I have is that I have to send these files individually to another server as multipart/formdata.
Approaches I've tried so far.
Approach 1
Used regex to split the event body by the boundary the web boundary in the example is ----------------------------269453880547064499146449.
const splitFileBody = event.body?.split(fileBoundary)
After splitting it I am left with a string.
// In this example I am only trying to send one file
formData.append('file', splitFileBody[0])
Outcome : The above approach gives me me a 400 HTTP status code along with the following error
body of your POST request is not well-formed multipart/form-data
Approach 2
Tried creating an object of type File in lambda using the parsed file as follows
const fileC = new File([file.content], file.fieldname, { type: file.contentType, lastModified: Number(new Date()) })
console.log('fileC', fileC)
const formData = new FormData()
formData.append('file', fileC)
Outcome : Lambda logs give File is not defined error.
Approach 3
Passed in the parsed information into form.append directly without new File
form.append('file', file.content, {
filename: file.filename,
contentType: file.contentType
});
Outcome: The above approach gives me me a 400 HTTP status code along with the following error
body of your POST request is not well-formed multipart/form-data
Are my approaches correct. If not what should I do differently. If the approaches above are fine, which of them is better and how to I avoid the errors?
Additional Info
I am using form-data package because I kept on getting FormData is not defined during the lambda runtime.
import FormData from 'form-data'
....
formData.append('file', file.content, {
filename: file.filename
})
I am using axios to make the post request
const res = await axios.post(URL, formData, {
headers: {
...formData.getHeaders(),
'content-length': file.content.length
}
})
The parsed file content can be directly appended to formData from form-data as follows
const result = await parser.parse(event)
const formData = new FormData();
for (const file of result.files) {
formData.append('file', file.content, {
filename: file.filename,
contentType: file.contentType
});
}
Additionally the content-length should have the the value formData.getLengthSync() as follows. The reason is that formData might contain other fields appended to it and file.content.length only contains the size of the file instead of including the other data appended.
await axios.post(URL, formData, {
headers: {
...formData.getHeaders(),
'Content-Length': formData.getLengthSync()
}
})

Walmart Seller API "Bulk Item Setup" doesn't work

I tried to use Walmart API v4.2 to publish some items. I used "Bulk Item Setup" API method to create some feed. I used some types of ways to did it:
Send binary file (in the request body, for-data) with header "multipart/form-data" (this way was described in the Walmart API docs)
Send stringified object in the request body with header 'Content-Type': 'application/json',
Walmart API correctly returns me feedId.
But all of these ways didn't work! After feed creating, I saw "Submitted" status at the Walmart Seller Center. But this status was changed after few minutes to "Error". At the error column I see "ERROR TYPE: Data Error" with a description "Malformed data. Please check data file to ensure it is formatted properly.java.lang.NullPointerException".
I use my backend NodeJs app to do it. I use Axios for making a request.
My code example:
async createFeed(wdpId, wdpSecret, accessToken, feedsData) {
try {
const string = JSON.stringify(feedsData);
const file = Buffer.from(string);
const formData = new FormData();
formData.append('file', file);
const baseToken = WalmartService.getBaseAuthToken(wdpId, wdpSecret);
const options = {
params: {
feedType: 'MP_WFS_ITEM',
},
headers: {
Authorization: baseToken,
'WM_SEC.ACCESS_TOKEN': accessToken,
'WM_QOS.CORRELATION_ID': uuidv4(),
'WM_SVC.NAME': 'Walmart Marketplace',
Accept: 'application/json',
'Content-Type': 'application/json',
...formData.getHeaders(),
},
};
return (
axios
.post(`${process.env.WALMART_API_BASEURL}/feeds`, formData, options)
.then((response) => {
return response.data;
})
.catch((error) => {
console.error(error.message);
throw new BadRequestException('Walmart error, ', error.message);
})
);
} catch (error) {
throw new BadRequestException('Can not create listing');
}
}
It is difficult to identify the exact issue based on the information you provided. Few things that you might want to check
If you are appending/attaching a file (as I see it in the code), use Content-Type header as "multipart/form-data. Also, make sure the file name has a .json extension if you are sending data as a json string. If you don't use this, it might default to xml and you will get the same error as what you see.
Try invoking the API using a rest client like Postman and verify if that call is successful.
If you do want to send the data as HTTP body (instead of a file), that should work too with Content-Type as application/json. This has not been documented on their developer portal, but it works.

How to PUT file to Tika-server in NodeJs

The Scenario
I am running a VueJs client, a NodeJs Restify API Server, and a Tika-server out of the official Docker Image. A user makes a POST call with formData containing a PDF file to be parsed. The API server receives the POST call and I save the PDF on the server. The API server should PUT the file to the unpack/all endpoint on the Tika-server and receive a zip containing a text file, a metadata file, and the set of images in the PDF. I would then process the zip and pass some data back to the client.
The Problem
I create a buffer containing the file to be parsed using let parsingData = fs.createReadStream(requestFilename); or let parsingData = fs.readFileSync(requestFilename);, set the axios data field to parsingData, then make my request. When I get the response from the Tika-server, it seems the Tika-server has treated the request as empty; within the zip, there are no images, the TEXT file is empty, the METADATA.
When I make the following request to the Tika-server via CURL curl -T pdf_w_images_and_text.pdf http://localhost:9998/unpack/all -H "X-Tika-PDFExtractInlineImages: true" -H "X-Tika-PDFExtractUniqueInlineImagesOnly: true"> tika-response.zip, I get a response zip file containing accurate text, metadata, stripped images.
The Code
let parsingData = fs.createReadStream('pdf_w_images_and_text.pdf');
axios({
method: 'PUT',
url: 'http://localhost:9998/unpack/all',
data: parsingData,
responseType: 'arraybuffer',
headers: {
'X-Tika-PDFExtractInlineImages': 'true',
'X-Tika-PDFExtractUniqueInlineImagesOnly': 'true'
},
})
.then((response) => {
console.log('Tika-server response recieved');
const outputFilename = __dirname+'\\output.zip';
console.log('Attempting to convert Tika-server response data to ' + outputFilename);
fs.writeFileSync(outputFilename, response.data);
if (fs.existsSync(outputFilename)) {
console.log('Tika-server response data saved at ' + outputFilename);
}
})
.catch(function (error) {
console.error(error);
});
The Question
How do I encode and attach my file to my PUT request in NodeJs such that the Tika-server treats it as it does when I make the request through CURL?
Axios is sending the request with a content type of application/x-www-form-urlencoded and therefore the file content isn't being detected and parsed.
You can change this by passing either the known content type of the file, or a content type of application/octet-stream to allow Apache Tika Server to auto-detect.
Below is a sample based on your question's code that illustrates this:
#!/usr/bin/env node
const fs = require('fs')
const axios = require('axios')
let parsingData = fs.createReadStream('test.pdf');
axios({
method: 'PUT',
url: 'http://localhost:9998/unpack/all',
data: parsingData,
responseType: 'arraybuffer',
headers: {
'X-Tika-PDFExtractInlineImages': 'true',
'X-Tika-PDFExtractUniqueInlineImagesOnly': 'true',
'Content-Type': 'application/octet-stream'
},
})
.then((response) => {
console.log('Tika-server response recieved');
const outputFilename = __dirname+'/output.zip';
console.log('Attempting to convert Tika-server response data to ' + outputFilename);
fs.writeFileSync(outputFilename, response.data);
if (fs.existsSync(outputFilename)) {
console.log('Tika-server response data saved at ' + outputFilename);
}
})
.catch(function (error) {
console.error(error);
});

form-data request body is empty

I'm trying to send an image and some data from an API to another. The image is stored in memory with multer. But when I want to send it, the body is just empty. I tried the same request with postman and it worked perfectly.
postman test
postman test image
server test
server test image
Here is some code. I removed some of it so you can read it better
export const saveImage = async ({ image, name, folder, options }: { image: any, name?: any, folder: string, options?: any }) => {
try {
const fd = new FormData();
fd.append("image", image.buffer, image.originalname);
if(options) {
fd.append("options[resize][height]", options?.resize?.height);
fd.append("options[resize][width]", options?.resize?.width);
}
if(name) fd.append("name", name);
fd.append("folder", folder);
fd.append("servideId", IMAGES_ID);
fd.append("serviceSecret", IMAGES_SECRET);
console.log(fd)
const formHeaders = fd.getHeaders();
const request = await axios.post(`${IMAGES_URL}/api/images`, {
headers: formHeaders,
body: fd
});
return request.data.id;
} catch (error) {
const { response } = error;
console.log(response.request.data)
if(error?.response?.data?.error) {
throw { statusCode: error.response.status, message: error.response.data.error }
}
console.error("Images API", error);
throw new InternalError("Something gone wrong");
}
}
When I log the FormData, I can see in _streams, the data that I'm sending, but the Images API receives an empty body.
FormData screenshot
If you need more information tell me, please! Thank you
The axios API for the post method is: axios.post(url[, data[, config]]). The second argument must always be the data you send along.
In your case axios thinks { headers: formHeaders, body: fd } is the body and the request ends up being application/json. To send a file with data using axios in Node.js, do the following:
const response = await axios.post(`${IMAGES_URL}/api/images`, fd, {
headers: {
...formHeaders,
'X-Custom-Header': 'lala', // optional
},
});
Your question inspired me to turn this answer into an article — Send a File With Axios in Node.js. It covers a few common pitfalls and you'll learn how to send files that are stored as a Buffer or coming from a Stream.
With Axios, you can directly use the form data without having to deal with headers.
axios.post("/api/images", fd)
If you wish to modify headers at some point in the future, you should pass the formData to the `data` field instead of `body`.
axios.post("/api/images", { headers: formHeaders, data: fd })
Correction in comments.
It can also be done using the Axios API syntax.
axios({method: 'post', url: 'url', data: fd, headers: {} })
In the backend, multer will add your files to req.file instead of req.body, if you have properly configured it to do so.

How to post multipart/form-data with node.js superagent

I am trying to send the content-type in my superagent post request to multipart/form-data.
var myagent = superagent.agent();
myagent
.post('http://localhost/endpoint')
.set('api_key', apikey)
.set('Content-Type', 'multipart/form-data')
.send(fields)
.end(function(error, response){
if(error) {
console.log("Error: " + error);
}
});
The error I get is:
TypeError: Argument must be a string
If I remove the:
.set('Content-Type', 'multipart/form-data')
I don't get any error but my back end is receiving the request as content-type: application/json
How can I force the content type to be multipart/form-data so that I can access req.files()?
First, you do not mention either of the following:
.set('Content-Type', 'multipart/form-data')
OR
.type('form')
Second, you do not use the .send, you use .field(name, value).
Examples
Let's say you wanted to send a form-data request with the following:
two text fields: name and phone
one file: photo
So your request will be something like this:
superagent
.post( 'https://example.com/api/foo.bar' )
.set('Authorization', '...')
.accept('application/json')
.field('name', 'My name')
.field('phone', 'My phone')
.attach('photo', 'path/to/photo.gif')
.then((result) => {
// process the result here
})
.catch((err) => {
throw err;
});
And, let's say you wanted to send JSON as a value of one of your fields, then you'd do this.
try {
await superagent
.post( 'https://example.com/api/dog.crow' )
.accept('application/json')
.field('data', JSON.stringify({ name: 'value' }))
}
catch ( ex ) {
// .catch() stuff
}
// .then() stuff...
Try
.type('form')
instead of
.set('Content-Type', 'multipart/form-data')
See http://visionmedia.github.io/superagent/#setting-the-content-type
It is not clear what is in the fields variable that you are sending, but here is some information that may help you determine where your problem lies.
To begin with, if you are actually trying to build a multi-part request, this is the official documentation for doing so: http://visionmedia.github.com/superagent/#multipart-requests
as for the error that you got...
The reason is that during the process of preparing the request, SuperAgent checks the data to be sent to see if it is a string. If it is not, it attempts to serialize the data based on the value of the 'Content-Type', as seen below:
exports.serialize = {
'application/x-www-form-urlencoded': qs.stringify,
'application/json': JSON.stringify
};
which is used here:
// body
if ('HEAD' != method && !req._headerSent) {
// serialize stuff
if ('string' != typeof data) {
var serialize = exports.serialize[req.getHeader('Content-Type')];
if (serialize) data = serialize(data);
}
// content-length
if (data && !req.getHeader('Content-Length')) {
this.set('Content-Length', Buffer.byteLength(data));
}
}
this means that to set a form 'Content-Type' manually you would use
.set('Content-Type', 'application/x-www-form-urlencoded')
or
.type('form') as risyasin mentioned
any other 'Content-Type' will not be serialized, and Buffer.byteLength(data) will subsequently throw the TypeError: Argument must be a string exception if the value of your fields variable is not a string.
Here is what worked for me. I had a single field form, that was uploading a file. I turned the form into a HTML5 FormData element and then did it as follows:
var frm = new FormData(document.getElementById('formId'));
var url = 'url/here';
superagent.post(url)
.attach('fieldInFormName', frm.get('fieldInFormName'))
.end( function (error, response) {
//handle response
});
Please note, I tried various ways of setting the 'Content-Type' manually in superagent, and it never worked correctly because of the multipart identifier needed in the Content-Type.

Resources