Decompressing data received as binary data in lambda - incorrect header check - node.js

Question

I want to send compressed data (gzip) to some URL that will trigger a (proxy) lambda function, that will decompress the data.
The lambda function (NodeJS 8):
let zlib = require('zlib');
exports.handler = async (event) => {
let decompressedData = zlib.gunzipSync(event['body'])
return {
"statusCode": 200,
"body": decompressedData.toString()
};
};
I trigger it with a curl command to the URL (through API gateway), for some file that I compressed example.gz with gzip:
curl -X POST --data-binary #example.gz https://URL...
As a result, I get:
{"message": "Internal server error"}
And the error is (logs in Cloudwatch):
"errorMessage": "incorrect header check",
"errorType": "Error",
"stackTrace": [
"Gunzip.zlibOnError (zlib.js:153:15)",
"Gunzip._processChunk (zlib.js:411:30)",
"zlibBufferSync (zlib.js:144:38)",
"Object.gunzipSync (zlib.js:590:14)",
"exports.handler (/var/task/test_index.js:5:33)"
]
When I looked at the event['body'] itself, I see the exact data as I see in example.gz. Perhaps I need some special header? I just want to pass the data as is.

Answer 1

as Michael - sqlbot said, By default, API Gateway can't pass binary data into a Lambda function.
What worked for me:
I added the header Content-Type: application/octet-stream in the curl command, and in the API gateway settings, on Binary Media Types I added application/octet-stream.
This way, the data is passed in base64, and afterwards I just converted the date that is in base64 to a buffer:
let data = Buffer.from(event['body'], "base64")
And afterwards just decompress it.
For more information read here

Answer 2

1/ First you need to build your gzip correctly, ensure that gzip file header is not present : curl command a gzipped POST body to an apache server
Wrong way :
echo '{ "mydummy" : "json" }' > body
gzip body
hexdump -C body.gz
00000000 1f 8b 08 08 20 08 30 59 00 03 62 6f 64 79 00 ab |.... .0Y..body..|
00000010 56 50 ca ad 4c 29 cd cd ad 54 52 b0 52 50 ca 2a |VP..L)...TR.RP.*|
00000020 ce cf 53 52 a8 e5 02 00 a6 6a 24 99 17 00 00 00 |..SR.....j$.....|
00000030
Good way :
echo '{ "mydummy" : "json" }' | gzip > body.gz
hexdump -C body.gz
00000000 1f 8b 08 00 08 0a 30 59 00 03 ab 56 50 ca ad 4c |......0Y...VP..L|
00000010 29 cd cd ad 54 52 b0 52 50 ca 2a ce cf 53 52 a8 |)...TR.RP.*..SR.|
00000020 e5 02 00 a6 6a 24 99 17 00 00 00 |....j$.....|
0000002b
2/ In curl don't forget to specify the content-encoding with
-H "Content-Encoding: gzip"
3/ In addition if you use express+compress you don't need to call zlib
curl -X POST "http://example.org/api/a" -H "Content-Encoding: gzip" -H "Content-Type: application/json" --data-binary #body.gz
router.post("/api/a", function(req, res){
console.log(req.body); // { mydummy: 'json' }
});