How to parse Multi-part form data in an Azure Function App with HTTP Trigger? (NodeJS) - node.js

I want to write a NodeJS HTTP endpoint using Azure Functions.
This endpoint will be a POST endpoint which takes files and upload these to blob storage.
However, NodeJS multipart form data parsers are all in the form of httpserver or expressJS middleware.
Is there any available tools that can parse the multipart form data after it has all been received from the Function Application's wrapper?
Thanks!

To answer original question:
However, NodeJS multipart form data parsers are all in the form of
httpserver or expressJS middleware.
Is there any available tools that can parse the multipart form data
after it has all been received from the Function Application's
wrapper?
Even 2 years later after you asked this question state of multipart form data parsers is not great, like you noticed majority of them assume req object which is a stream and tutorials/demos show how to parse multipart/form-data with express or httpServer.
However there is a parse-multipart npm package which can process req.body from azure function and return you array of objects with code similar to following:
const multipart = require("parse-multipart");
module.exports = function (context, request) {
context.log('JavaScript HTTP trigger function processed a request.');
// encode body to base64 string
const bodyBuffer = Buffer.from(request.body);
const boundary = multipart.getBoundary(request.headers['content-type']);
// parse the body
const parts = multipart.Parse(bodyBuffer, boundary);
context.res = { body : { name : parts[0].filename, type: parts[0].type, data: parts[0].data.length}};
context.done();
};
(original source: https://www.builtwithcloud.com/multipart-form-data-processing-via-httptrigger-using-nodejs-azure-functions/)
One area where I noticed parse-multipart can struggle is parsing forms with text fields. A slightly improved version which handles it better is called multipart-formdata:
require('multipart-formdata').parse(req.body, boundary)
//returns [{field, name, data, filename, type}, ...] where data is buffer you can use to save files

As Azure Functions has wrapped http server object in Node.js, and exposes a simple req and context with several functionalities, refer to https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-node#exporting-a-function for details.
And mostly, Azure Functions is designed for triggers and webhooks requests, you can refer to https://learn.microsoft.com/en-us/azure/azure-functions/functions-compare-logic-apps-ms-flow-webjobs for the detailed comparison.
Meanwhile, you can try the answer of Image upload to server in node.js without using express to parse the request body content to file content, and upload to Azure Storage leveraging Azure Storage SDK for node.js, you can install custom node modules via KUDU console. Refer to https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-node#node-version--package-management for more info.
And I suggest you can try to leverage Azure API App in node.js to approach your requiremnet. As it is an expressjs based project, which will be more easier to handle upload files.
Any further concern, please feel free to let me know.

I have good experiences with the new azure-function-multipart package. Example code might look like this:
const {
default: parseMultipartFormData,
} = require("#anzp/azure-function-multipart");
module.exports = async function (context, req) {
const { fields, files } = await parseMultipartFormData(req);
console.log("fields", fields)
console.log("files", files)
};
See docs for more details.

You can try to use this adapter for functions and express, it may allow you to successfully use the multi-part middleware you want: https://github.com/yvele/azure-function-express
As a less desirable option, you can parse the body yourself, all the multi-part data will be available in req.body and will look something like this:
------WebKitFormBoundarymQMaH4AksAbC8HRW
Content-Disposition: form-data; name="key"
value
------WebKitFormBoundarymQMaH4AksAbC8HRW
Content-Disposition: form-data; name=""
------WebKitFormBoundarymQMaH4AksAbC8HRW--
I do think it's a good idea to support httpserver / express better in order to enable this extensibility.

Related

how to read a file from form-Data in an AWS Lambda function

I am trying to send a PDF or text file to an AWS Lambda function (Node JS). I'd like to be able to process this file in the lambda function. I know often the best practice is to use a trigger function from an s3 bucket. However, I'd like to be able to send this lambda function a file from formdata, extract information from the file and then return the extracted info back.
I have been able to first encode the file in 64 bit binary data and send it to AWS lambda via a JSON, but often when I try to decode the file (especially a PDF) in the Lambda Function it is corrupted or empty.
Image files seem to work well for this type of encoding, but been unsuccessful with PDFs. Any help greatly appreciated. Rather then encode in base 64 is there a way I can obtain the file from a formdata? My code is below:
export const handler = async(event) => {
console.log("event", event)
var converted = atob(event.body) // RATHER HOW WOULD I READ A FILE FROM FORMDATA
const response = {
"statusCode": 200,
"headers": {"Content-Type":"text/html", //"application/pdf", // "multipart/form-data", //
"Access-Control-Allow-Origin":"*",
"Access-Control-Allow-Headers":"Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token"
},
"body": event,
"isBase64Encoded": true
}
return response;
};
thanks so much
I assume that you are using an API gateway to trigger the lambda function. In that case, you have to enable multipart/form-data in API gateway as mentioned in the documentation. The gist of the documentation is as follows:
In the Settings pane, choose Add Binary Media Type in the
Binary Media Types section. Type a required media type, for example, image/png, in the input text field.
Add Content-Type and Accept to the request headers for your proxy method.
Add those same headers to the integration request headers.
Deploy the API
PS: If you are using Lambda Proxy integration ({proxy+}, just steps 1 and 4 are enough.

simply way to handle post data/image in node js through Postman Form-Data

Is there a way to post form-data through postman using Nodejs. I have seen many platforms but the author is using the front-end to post data which manipulates the back-end code. I want to understand the code
Yes you can send the data using form-data in post man.
You can see the formdata under the body section in this form data you have to change the type from text to file
after changing to file type you can browse your local machine. to choose the image
You can further parse the files in node js using formidable. You can install this package and you can configure to your needs. It can handle upto 1000 files
Coming to code part in node js.
First configure the formidable
const form = new formidable.IncomingForm({
multiples: true,
keepExtensions: true,
});
After that you have to parse the fieldvalue and file
form.parse(req);
Then you can seperate the value and files uisng
form.on("file", (field, file) => {}) //For file
form.on("field", (fieldName, fieldValue) => {}) //for fieldvalue
Hope this one helps for you!!
Sure thing. Set request type to POST and then define your data in the request body
check this out
https://learning.postman.com/docs/sending-requests/requests/#sending-body-data

NodeJs with Express not parsing form data from node-fetch

I'm creating two APIs with NodeJS, Express, and TypeScript. One API will take a request with a content type of multipart/form-data, and will use the body in the request to make another request to a second API.
I'm using Postman, so the chain of request looks something like this
Postman -> First API -> Second API
I use node-fetch to make a request from the first API to the second one. The body of the request is a FormData, which contains some files and key-value pairs.
const form = new FormData();
// File for profile picture
const profilePictureBuffer = await (await fetch(user.profilePicture)).buffer();
form.append('profilePicture', profilePictureBuffer);
// File for ID Card
const idCardBuffer = await (await fetch(user.idCardUrl)).buffer();
form.append('idCard', idCardBuffer);
// This part iterats over the obsect of 'user',
// which contains other key-value pairs
Object.entries(user).forEach((data) => {
form.append(data[0], data[1]);
});
// Make POST request to second API
const pinterUser = await fetch(secondAPIURL, {
method: 'post',
body: form,
headers: form.getHeaders()
});
I ran both of the APIs on localhost so that I can monitor the logs for any bugs. As I make a request from Postman to the first API, then the first API make another request to the second API, I got the following error log in the terminal for the second API
TypeError: Cannot read property '0' of undefined
After some investigation, I found out that, in the second API, the req.body and req.files are empty objects. This means that Express did not parse the incoming request. Note that I've also already a multer middleware to handle the files in the request.
Furthermore, I have added the following lines of code in my server.ts file for the second API
/** Parse the body of the request */
router.use(express.urlencoded({ extended: true }));
router.use(express.json());
However, when I tried making the request from Postman, it returns a successful response.
I'm not really sure what's going on here. I've tried looking for some answer regarding this similar issue, but most of them suggest adding urlencoded and json, or using some other library to handle parsing form data.
In my case, the first suggestion doesn't solve my problem since I already added them from the start, and the latter is what I'm trying to avoid.
I wonder if anybody could point out what I was missing here? Thanks in advance

Response from Webhook using NodeJS Client

I built a custom webhook as a fulfillment endpoint for a Dialogflow intent. It works fine when I respond with raw JSON, ie: {'fullfillmentText':'hi'},
but does not seem to work using the "actions-on-google" library.
The code from their website implies this should work:
app.intent('myintent', (conv) => {
conv.close('See you later!');
});
But it does not. Google Home just says my app isn't responding. It might be that as it stands my function (using Fn Project) has to return JSON and if I return JSON as a response that it isn't expecting it fails. Maybe someone can shed some light?
Edit 1:
I'm using a custom webhook using the Fn Project open source functions as a service. Demonstrating how to use the project is my purpose here so I don't want to use inline editor or Google Cloud Functions or firebase or any other default option.
Here's the rest of the code
const fdk = require('#fnproject/fdk');
const request = require('request');
const dialogflow = require('actions-on-google');
const app = dialogflow({
debug: true,
});
fdk.handle(function (input) {
app.intent('myintent', (conv) => {
conv.ask('I could not understand. Can you say that again?');
});
return {'fulfillmentText': 'response from webhook'}
});
Although you are creating the app object, which does the Intent handler processing and such, and registering a handler with it via app.intent(), you're not doing anything to "export" it so app's methods are called when the webhook is triggered. When called, it gets the body of the request and will format the JSON for the response.
If, for example, you were using Firebase functions, you would connect app to be handled through the functions with something like
exports.fulfillment = functions.https.onRequest(app);
But you're not. You're using a different framework.
The library comes with a number of frameworks that are supported out of the box, but the Fn Project isn't one of them. In theory, you can create your own Framework object which will do this for you (the "Frameworks" section of this article discusses it briefly, but doesn't go into details about how to do so).
As you surmise, it may be easiest for you to just read the JSON request and generate the JSON response yourself without using the actions-on-google library. Or you can look into a library such as multivocal to see if it would be easier to leverage its multi-framework support.

Building a simple RESTful API with Mongo and Node

So I'm working with a project that recently switched over to Node from Rails, where one of my favorite features was how easy it was to create a simple REST API, like so:
localhost:3000/materials/ Gets a JSON document of all objects
inside materials
localhost:3000/materials/:id Gets a JSON output of the object with
that id, e.g. /materials/123123 gives me item 123123
localhost:3000/materials/ Gets a JSON document of all objects
inside materials
And so on. I'm using Mongo. Is there a way of doing this in Node, or is there a guide or a package I should install that can do this?
Check out LoopBack from StrongLoop Suite, it has a built-in RESTful api explorer that interacts with Mongo using the mongodb connector. http://docs.strongloop.com/loopback/#using-the-api-explorer and http://strongloop.com/strongloop-suite/loopback/
For example, you could create the "materials" model with just the following commands:
slc lb model materials and then the restful api will get auto-generated for you at localhost:3000/explorer.
You can use Restify, to build RESTful Web Services in Node.js
A good tutorial at: https://www.openshift.com/blogs/day-27-restify-build-correct-rest-web-services-in-nodejs
Express seems like what you want. You can specify routes very simply as follows:
app.get('/:collection', function(request, response) {
// the value of :collection is stored in request.params
var coll = request.params.collection;
var search = request.query; // a hash containing the querystring arguments
// do stuff with request body, query parameters, etc.
response.send(data); // send the response
});
app.get('/:collection/:item', function(request, response) {
var coll = request.params.collection;
var item = request.params.item;
// do stuff
res.json(data); // can also send a JSON response
});
Take a look at restgoose. It is a Typescript library that makes simple REST API development super simple.
https://xurei.github.io/restgoose/

Resources