AWS http api gateway + lambda (node/express) Internal Server Error - node.js

Question

I get internal server error when I have a long running query.
Actually, I have to fetch historic data through an API, which sometime can take longer than 30 seconds. It depends on the query how complex it is. It can take 1 min also.
Not sure but guessing, API gateway timeout is set to 30 seconds (and I cann't increase it) and my query execution time is more then 30 seconds. So I get internal server error I believe.
HOW can I say above statement ?
because If I run the same query locally, I mean in node/express locally by running npm run start, it works fine even if takes1 mins, response will always come back.
But when I deploy node/express code to lambda function, it throws error if any query takes longer period to execute.
I have following setup of node/express
const express = require("express");
const serverless = require("serverless-http");
const app = express();
app.use(cors());
app.use((req, res, next) => {
res.setHeader('Connection', 'keep-alive'); // I added this line as suggested in some post but not helping
res.setHeader('Keep-Alive', 'timeout=30'); // I added this line as suggested in some post but not helping
res.setHeader("Access-Control-Allow-Headers", "X-Requested-With,content-type");
res.setHeader("Access-Control-Allow-Origin", "*");
res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS, PUT, PATCH, DELETE");
res.setHeader("Access-Control-Allow-Credentials", true);
next();
});
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
app.use(`api-end-point/user`, userRoute);
....
if (process.env.NODE_ENV !== "lambda") {
PORT = process.env.PORT || 7000;
const server = app.listen(PORT, () => {
console.log(`node-express server running in ${process.env.NODE_ENV} mode on ${PORT}`);
});
server.timeout = 0;
}else {
module.exports.handler = serverless(app); // this is for lambda function
}
I deploy this code to AWS lambda function.
HTTP API gateway is configured with two routes /ANY, /{proxy+}
TIMEOUT
API gateway is set to default 30 seconds. [I can not increase this time as not allowed by AWS]
Lambda is set to 10 **mins**
CORS
I really have no idea how can I fix this problem ?
How can I increase API gateway timeout or How can I keep connection alive ?

Answer 1

You cannot increase the API Gateway timeout to greater than 30 seconds, as has already been mentioned.
The only solution I know of at this time is to run your Lambda asynchronously, but this cannot be done in an Http API. But if you're willing to change it to a REST API, then this can be done with a combination of turning on Lambda Proxy Integration in the REST API and invoking the Lambda asynchronously utilizing an invoke header X-Amz-Invocation-Type. This will allow your Lambda to run asynchronously (up to 15 minutes) with an API call.

Answer 2

https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html
Since timeout cannot be incresed, you might change using single HTTP request to
https://en.wikipedia.org/wiki/Post/Redirect/Get Pattern
Client POST query
Server response an url for the result
Client GET the url multiple times -- it will be 200 OK when the result ready
or WebSocket
The document says Idle Connection Timeout for WebSocket is up to 10 minutes

Answer 3

Using Lambda means subscribing to patterns from the Serverless catalog/philosophy. Which means using async whenever possible.
As far as I understand your Lambda needs receives a request, does another call to something (not specified) which takes 30~60s.
The API Gateway has a hardcoded timeout of 29s (hard limit).
To solve this problem the application would need to be re-architectured:
Trigger the Lambda asynchronously using X-Amz-Invocation-Type Event from the Frontend.
The Lambda calls the history API and stores the result in some storage (DynamoDB, S3, RDS, ...).
The frontend queries the backend from the frontend until the data is available (or use WebSockets)
This way the historic API call can take up to 15m and the calls can be cached in the storage to speed up further calls. If it needs more than 15m then I would ask the historic API to re-architecture.