I am writing a handler as AWS Lambda Function which is supposed to wrap the content into the docx file. I am using the Content-Disposition in response headers in order to achieve the objective. Below is the code that I have written so far:
//Process Array Buffer and extract the plain content out of it
export const extractContent = async (data: Buffer) => {
return new Promise((resolve, reject) =>
textract.fromBufferWithMime(
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
data,
{ preserveLineBreaks: true },
(err, content) => {
if (err) {
reject(err);
} else {
resolve(content);
}
}
)
);
};
export const handlerName = async (
event: APIGatewayProxyEvent
): Promise<APIGatewayProxyResult> => {
if (event.body === null) {
return ErrorResponse;
}
const result = await class.func(JSON.parse(event.body)); //returns an array buffer object
if (result instanceof Error) {
return ErrorResponse;
}
const content = await extractContent(result.data)
.then(res => res)
.catch(err => err);
const headers = {
"Access-Control-Allow-Origin": "*",
"Strict-Transport-Security": "'max-age=31536000'",
"Access-Control-Expose-Headers": "Content-Disposition",
"Content-Disposition": attachment; filename=fn.docx; filename*=UTF-8''fn.docx,
"Content-Type": "application/vnd.ms-word.document"
};
return {
body: content,
headers,
statusCode: 200,
};
};
Now only string can be returned in APIGatewayResponse; hence I am not able to return byte object | stream | buffer instead.
I am able to download the docx file but it is not getting opened in MS-Word; I am always getting a detailed error that either file is corrupt or some part is invalid or missing.
I have already tried the base64 encoded string but still not able to get what is desired. Please suggest a solution to it.
I writing the handler in typescript Node using the SLS framework.
Related
I am currently trying to develop a google cloud function to parse multipart files (excel format or csv) in order to populate the firestore database.
I am using busboy in a helper function to parse the file, convert it to json and return it to the main function.
Everything goes well until I am trying to return the parsed data. I thought the most logic way of doing was to return the data from the busboy 'finish' event but it seems not to return the data as once back in the main function it is undefined. I first thought of some issue related to asynchronous code execution but when I tried to only print the data in the busboy finish event it worked properly.
I've tried to find some related content online but unfortunately didnt success. Here is my helper function :
// Takes a multipart request and sends back redable data
const processRequest = (req) => {
const busboy = Busboy({headers: req.headers});
formats = ['application/vnd.ms-excel', 'text/csv', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'];
var finalData;
// fieldname is the request key name of the file
// file is the stream
// fname is the name of the fileq
busboy.on('file', (fieldname, file, fname) => {
// Checks if file is right format
if(!formats.includes(fname.mimeType)) throw new FileFormatError('File must be excel or csv');
bytes = [];
// Checks that the request key is the right one
if(fieldname == 'file') {
// Data is the actual bytes, adds it to the buffer each time received
file.on('data', (data) => {
bytes.push(data);
});
// Concatenates the bytes into a buffer and reads data given mimetype
file.on('end', async () => {
buffer = Buffer.concat(bytes);
if(fname.mimeType === 'application/vnd.ms-excel' ||
fname.mimeType === 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet') {
workbook = XLSX.read(buffer, {type: 'buffer'});
json = excelToJson(workbook);
console.log(json);
finalData = json;
}
if (fname.mimeType === 'text/csv') {
var csv = [];
const stream = Readable.from(buffer.toString());
stream.pipe(CSV.parse({delimiter: ','}))
.on('error', (err) => {
console.log('csv parsing error');
console.log(err.message);
}).on('data', (row) => {
csv.push(row);
}).on('end', () => {
console.log('csv file properly processed');
console.log(csv);
// CSV PARSING LOGIC TO COME, JUST TESTING RIGHT NOW
finalData = csv;
});
}
});
}
});
busboy.on('finish', () => {
console.log('busboy finished');
return finalData;
// WHEN ONLY PRINTED THE DATA IS PRESENT AND DISPLAYS PROPERLY HERE
})
// Processes request body bytes
busboy.end(req.rawBody);
}
There must be something I am misunderstanding but as of yet I cannot point out what.
Thanks in advance for your time :)
You're not waiting for your CSV parsing to actually finish.
It would be better to refactor your async code to use async/await.
Since you're using libraries that might only support callback-style async, you'll need to do some new Promise wrapping yourself.
Understandably, I haven't tested the below code, but something like this...
/**
* Parse the given buffer as a CSV, return a promise of rows
*/
function parseCSV(buffer) {
return new Promise((resolve, reject) => {
const csv = [];
const stream = Readable.from(buffer.toString());
stream
.pipe("text/csv".parse({ delimiter: "," }))
.on("error", reject)
.on("data", (row) => csv.push(row))
.on("end", () => resolve(csv));
});
}
/**
* Parse the given buffer as a spreadsheet, return a promise
*/
async function parseSpreadsheet(mimeType, buffer) {
if (
mimeType === "application/vnd.ms-excel" ||
mimeType === "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
) {
const workbook = XLSX.read(buffer, { type: "buffer" });
return excelToJson(workbook);
}
if (mimeType === "text/csv") {
return parseCSV(buffer);
}
throw new Error(`Unknown mime type ${mimeType}`);
}
/**
* Get the bytes of the field `fieldName` in the request.
*/
function getFileFromRequest(req, fieldName) {
return new Promise((resolve, reject) => {
const busboy = Busboy({ headers: req.headers });
busboy.on("file", (name, file, info) => {
// Only process the field we care about
if (name != fieldName) {
return;
}
const bytes = [];
file.on("data", (data) => bytes.push(data));
file.on("end", () =>
resolve({
info,
buffer: Buffer.concat(bytes),
}),
);
file.on("error", reject);
});
busboy.end(req.rawBody);
});
}
async function parseRequest(req) {
// (1) Get the file as a buffer
const { info, buffer } = await getFileFromRequest(req, "file");
// (2) Try parsing it as a spreadsheet
const data = await parseSpreadsheet(info.mimeType, buffer);
// (3) Do something with the data?
return data;
}
Uploading file and get the formdata in nodejs server as seen below:
now all I need is post this data to remote API,
As you see in the image, all props are fine except the uploaded file. What should I do?
nodejsAPI:
async (req, res) => {
var form = new formidable.IncomingForm();
var params = {}
form.parse(req, async (err, fields, files) => {
Object.keys(fields).forEach(function(name) {
params[name] = fields[name]
});
params['MyFile'] = files['MyFile']
const result = await AdManagementService.createAdvertisement(params)
});
}
createAdvertisement action:
const createAdvertisement = async ({
Type,
MyFile,
}) => {
try {
const response = await axios.post(
`/ad/upload-file?Type=${Type}`,
{
data:MyFile,//if this line removed, it works fine..
headers: {
'Content-Type': 'multipart/form-data',
},
}
)
console.log(response)
return response.data
} catch (error) {
return error
}
}
it works fine if data:MyFile is removed, but I need send the file as well, what should I do?
It returns 400
I have the following code that takes base64 string, and send it to API that accepts binary. I am getting empty response from the api call
let base64String = event.base64String;
// pass the base64 string into buffer
let buffer = new Buffer(base64String, 'base64');
// TODO check file type
processImage(buffer)
.then(result => {
console.log("result are " + result);
callback(result);
}).catch(error => callback(error));
let processImage = function (buffer) {
// get the file extension
return new Promise((resolve, reject) => {
var options = {
method: 'POST',
url: 'https://<UR - not visible for privacy>',
headers:
{
'Content-Type': 'application/octet-stream'
},
body: buffer.toString('binary')
};
request(options, function (error, response, body) {
if (error) reject(error);
console.log(body);
resolve(body);
});
}
Equivalent in postman is that I simply specify binary in body and attach file, again post call but URL is removed from pic, the header is content-type: application/octet-stream, it works in postman but not in node.js
I am using an AWS Node Lambda to resize an image and send that image back in binary format, and I am not sure if I am handling this right or not. Basically, I want to send the binary data back in my response in a way that it can just be loaded without any front end changes. I have a perfectly good version working that returns base64 data, but how would I modify this to return binary instead?
Here is my resizing function:
function imageSizer(url, args) {
return new Promise((resolve, reject) => {
bufferRequest.get(url, args, function (err, res, body) {
if (err) {
console.log(err);
reject(err);
}
const originalFormat = url.includes('png') ? 'png' : 'jpeg';
let newSize = (args.exact === true)
? sharp(body).resize(args.width, args.height)
: sharp(body).resize(args.width, args.height).max();
newSize.toFormat(originalFormat)
.toBuffer()
.then((outputBuffer) => {
const newImage = "data:" + res.headers["content-type"] + ";base64," + new Buffer(outputBuffer).toString('base64');
resolve(newImage);
})
.catch((error) => {
console.log(error);
reject(error);
})
});
})
}
Here is my handler:
function handler(event, context) {
imageSizer(event.url, event.queryStringParameters);
.then((result) => {
context.succeed({
statusCode: 200,
headers: { 'Content-Type': 'image/jpeg'},
body: result
});
})
.catch((error) => {
context.succeed({
statusCode: 502,
headers: { 'Content-Type': 'application/json' },
body: `Error sizing image ${error}`
});
})
}
As mentioned, this works fine for base64 but I am unsure what changes to make to const newImage = to send back binary data the browser can use to load the image.
I went with this solution:
Change:
.then((outputBuffer) => {
const newImage = "data:" + res.headers["content-type"] + ";base64," + new Buffer(outputBuffer).toString('base64');
resolve(newImage);
})
To:
.then((outputBuffer) => {
resolve(outputBuffer);
})
This returns the buffer in binary format, without having to do anything ourselves.
I'm glad to see AWS now supports multipart/form-data on AWS Lambda, but now that the raw data is in my lambda function how do I process it?
I see multiparty is a good multipart library in Node for multipart processing, but its constructor expects a request, not a raw string.
The input message I am receiving on my Lambda function (after the body mapping template has been applied) is:
{ "rawBody": "--ce0741b2-93d4-4865-a7d6-20ca51fe2689\r\nContent-Disposition: form-data; name=\"Content-Type\"\r\n\r\nmultipart/mixed; boundary=\"------------020601070403020003080006\"\r\n--ce0741b2-93d4-4865-a7d6-20ca51fe2689\r\nContent-Disposition: form-data; name=\"Date\"\r\n\r\nFri, 26 Apr 2013 11:50:29 -0700\r\n--ce0741b2-93d4-4865-a7d6-20ca51fe2689\r\nContent-Disposition: form-data; name=\"From\"\r\n\r\nBob <bob#mg.mydomain.io>\r\n--ce0741b2-93d4-4865-a7d6-20ca51fe2689\r\nContent-Disposition: form-data; name=\"In-Reply-To\"\r...
etc and some file data.
The body mapping template I'm using is
{
"rawBody" : "$util.escapeJavaScript($input.body).replaceAll("\\'", "'")"
}
How can I parse this data to acecss the fields and files posted to my Lambda function?
busboy doesn't work for me in the "file" case. It didn't throw an exception so I couldn't handle exception in lambda at all.
I'm using aws-lambda-multipart-parser lib wasn't hard like so. It just parses data from event.body and returns data as Buffer or text.
Usage:
const multipart = require('aws-lambda-multipart-parser');
const result = multipart.parse(event, spotText) // spotText === true response file will be Buffer and spotText === false: String
Response data:
{
"file": {
"type": "file",
"filename": "lorem.txt",
"contentType": "text/plain",
"content": {
"type": "Buffer",
"data": [ ... byte array ... ]
} or String
},
"field": "value"
}
This worked for me - using busboy
credits owed to Parse multipart/form-data from Buffer in Node.js which I copied most of this from.
const busboy = require('busboy');
const headers = {
'Content-Type': 'application/json',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Methods': 'OPTIONS, POST',
'Access-Control-Allow-Headers': 'Content-Type'
};
function handler(event, context) {
var contentType = event.headers['Content-Type'] || event.headers['content-type'];
var bb = new busboy({ headers: { 'content-type': contentType }});
bb.on('file', function (fieldname, file, filename, encoding, mimetype) {
console.log('File [%s]: filename=%j; encoding=%j; mimetype=%j', fieldname, filename, encoding, mimetype);
file
.on('data', data => console.log('File [%s] got %d bytes', fieldname, data.length))
.on('end', () => console.log('File [%s] Finished', fieldname));
})
.on('field', (fieldname, val) =>console.log('Field [%s]: value: %j', fieldname, val))
.on('finish', () => {
console.log('Done parsing form!');
context.succeed({ statusCode: 200, body: 'all done', headers });
})
.on('error', err => {
console.log('failed', err);
context.fail({ statusCode: 500, body: err, headers });
});
bb.end(event.body);
}
module.exports = { handler };
Building on #AvnerSo :s answer, here's a simpler version of a function that gets the request body and headers as parameters and returns a promise of an object containing the form fields and values (skipping files):
const parseForm = (body, headers) => new Promise((resolve, reject) => {
const contentType = headers['Content-Type'] || headers['content-type'];
const bb = new busboy({ headers: { 'content-type': contentType }});
var data = {};
bb.on('field', (fieldname, val) => {
data[fieldname] = val;
}).on('finish', () => {
resolve(data);
}).on('error', err => {
reject(err);
});
bb.end(body);
});
If you want to get a ready to use object, here is the function I use. It returns a promise of it and handle errors:
import Busboy from 'busboy';
import YError from 'yerror';
import getRawBody from 'raw-body';
const getBody = (content, headers) =>
new Promise((resolve, reject) => {
const filePromises = [];
const data = {};
const parser = new Busboy({
headers,
},
});
parser.on('field', (name, value) => {
data[name] = value;
});
parser.on('file', (name, file, filename, encoding, mimetype) => {
data[name] = {
filename,
encoding,
mimetype,
};
filePromises.push(
getRawBody(file).then(rawFile => (data[name].content = rawFile))
);
});
parser.on('error', err => reject(YError.wrap(err)));
parser.on('finish', () =>
resolve(Promise.all(filePromises).then(() => data))
);
parser.write(content);
parser.end();
})