Respond to Koa request immediately, continue middleware chain - node.js

I am writing the middleware for API endpoints in my app that respond to webhooks from other applications, and am relatively new to Koa, so am not completely familiar with its patterns.
I would like to structure my middleware as follows:
exports.updateReceived = async (ctx, next) => {
// Respond to server issuing the webhook
ctx.res.body = "ok";
ctx.res.statusCode = 200;
// Grab what we need from the request
const { headers, state, request } = ctx;
const { body } = request;
// Do some async work
const { example } = await doSomeAsyncWork(ctx);
// Prepare a database query
const query = { aValue: anId };
// Run the DB query
const result = await Thing.findOne(query);
// Add data to the request
state.thing = result;
// Move on...
return next();
};
However, this does not appear to be working, as an error in any of my async methods can cause the route to error out.
My goal is for this endpoint to always respond "yep, ok" (immediately), meaning it is simply up to the application to handle any error states.
I have researched this fairly well, and have come across this pattern:
app.use(async ctx => {
db.fetch() // Assuming a Promise is returned
.then(() => { ... })
.catch(err => {
log(err)
})
// status 200 will be returned regardless of if db.fetch() resolves or rejects.
ctx.status = 200
})
However, this does not meet my needs as the middleware makes no use of next, so it is not really a useful pattern, so far as I can tell.
Could someone tell me what I am overlooking?

next() invokes the downstream middleware chain and returns a promise that resolves after all downstream middleware/handlers have finished.
That means you can simply implement your own upstream error handler that catches any errors and always ensures a 200 OK response.
const Koa = require('koa')
const app = new Koa()
app.use(async (ctx, next) => {
next().catch((err) => {
// Print error for our own records / debugging
console.error(err)
// But ensure that outgoing response is always a smile
ctx.status = 200
ctx.body = ':)'
})
})
app.use(async (ctx) => {
// Do your webhook / biz logic here, but for demonstration
// let's make it always throw an error. Thus upstream next()
// will be a rejected promise.
throw new Error('this middleware will always bubble up a rejected promise')
})
app.listen(3000, () => {
console.log('listening on 3000')
})

Note: We are not awaiting next(), so we can end the request immediately. However the next handler in the chain will still have the opportunity to process the logic
app.use((ctx, next) => {
next()
ctx.status = 200
})
app.use( async ctx =>{
db.fetch()
.then(() => { ... })
.catch(err => log(err))
}
}

Just to divert the solution in a different side, You could consider adding your work to some kind of MessageQueue and then let another process do that task for you. Basically asynchrously but you will still be important. This kind of pattern suits for your requirement.
There are many messaging system availble like AWS SQS which you could consider. This way your api will be very light weight and it will do thing which it needs to and send a command to your messaging system to do extra stuff. You are basically separting your core logic and the doing things in background which scales very nicely as well.

Related

Add additional functionality to end method on Node/Express Response object?

My team and I are trying to mutate the response.end method in our Express middleware in order to have extra functionality be called just before the server responds back to the client.
Here is our attempt:
return (req: Request, res: Response, next: NextFunction): NextFunction => {
// reassign res.end in order to allow logger functionality before
// a response is sent back the client
const temp = res.end;
res.end = () => {
// instantiates PostQuery object with passed in query data from limiter middleware
const postQuery = new PostQuery(gateURI, projectID, res.locals.graphqlGate);
// our logger middleware functionality
try {
await postQuery.post();
} catch (err) {
if (err) console.log(err);
}
// our temp variable holding node's res.end definition
return temp.call(this);
};
return next();
};
Our test server throws this error when we include this function in our middleware chain:
TypeError: Cannot read properties of undefined (reading 'finished')
at end (node:_http_outgoing:856:19)
at /Users/jon/Documents/Solo Projects/OSP/graphQL-gate-logger/src/index.ts:65:25
index.ts:65 points to return temp.call(this)
We have also tried return temp() , as well as binding temp to the res object, and receive the same error in every instance.
Is there some other way we can reach this goal or do we have to start back at the drawing board?
If you don't have to execute your code BEFORE the response has been sent, but can instead do it right afterwards, then you can use the finish event on the res stream.
app.use((req, res, next) => {
res.on('finish', () => {
console.log(`got finish event for ${req.url}`);
// do your business here after a response has been sent
});
next();
});
There are also a couple problems with your existing override middleware. First off, you aren't preserving arguments that can be optionally send to res.end(). Second, res.end() is supposed to return res which makes it chainable. You aren't doing that. You have assigned it an async function which returns a promise, not res.
Though I think it would be much better to use the finish event as illustrated above and not have to override any methods, this would fix some of the problems with your override:
return (req: Request, res: Response, next: NextFunction): NextFunction => {
// reassign res.end in order to allow logger functionality before
// a response is sent back the client
const origEnd = res.end;
res.end = function(...args) {
// instantiates PostQuery object with passed in query data from limiter middleware
const postQuery = new PostQuery(gateURI, projectID, res.locals.graphqlGate);
// our logger middleware functionality
postQuery.post().catch(err => {
console.log(err);
}).finally(() => {
return origEnd.call(this, ...args);
});
return res;
};
return next();
};

Why can't I make express route synchronous

I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.

How can I respond the request from an authorization middleware in a Next.js api route?

Let's say I have the following Next.js api route.
/api/protected-api
This api will get a authorization: "Bearer: TOKEN" header for authorization purposes.
import { NextApiHandler } from "next";
const apiHandler: NextApiHandler = async (req, res) => {
await runAuthMiddleware(req,res);
// THE REST OF THE API LOGIC SHOULD ONLY RUN IF AUTHORIZATION SUCCEEDS
// IN THEORY, runAuthMiddleware WOULD RESPOND WITH 403 IF AUTHORIZATION FAILS
return res.json(data);
};
What I mean by the code above is:
If authorization fails, I would like to respond a 403 from the runAuthMiddleware function, and don't even bother running the rest of the code in the apiHandler.
Is this even possible? Is this an anti-pattern?
Should I got with something like this instead?
const apiHandler: NextApiHandler = async (req, res) => {
const authSuccess = await runAuthMiddleware(req,res);
if (authSuccess)
return res.json(data);
else
return res.status(403).send("Forbidden");
};
UPDATE:
It seems that there's no easy way of doing it. This package offers a possible solution: next-api-middleware
There is an easy way to do it if that is the only thing that you need, just make your own higher order function which checks auth:
const withAuth = (f) => async (req, res) => {
const isLogged = await checkIfUserIsLogged(req)
if (isLogged) {
return f(req, res)
} else {
return res.status(403).send("Forbidden");
}
}
const apiHandler = withAuth(async (req, res) => {
// This code now only runs of the user is logged
return res.json(data);
});
For more complex scenarios I would recommend to use https://github.com/hoangvvo/next-connect or something like that. But if you only need one middleware then it is completely fine to write your own.

Using https for REST requests from inside expressjs applicaiton

From inside my expressJS application I have to verify that a cookie token is valid with a back-end server. So the relevant code involved in this is as follows:
app.get('*', (req, res, next) => {
console.log('GET: ' + req.path);
// ...
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
// build request
const restReq = https.request(authServerOptions, result => {
console.log('back-end response' + result.statusCode);
result.on('data', data => {
next(); // token is good now proceed.
});
result.on('error', error => {
res.redirect('somewhere'); // token is bad or timeout
});
});
restReq.write(token);
restReq.end();
}
So the main get function sets the REST request in motion and then just returns without calling next() or anything.
Questions:
Is this the right code for doing this? What happens if the callbacks are never called?
Is the application blocked from processing other requests until the back-end server returns or times out?
If so is there some way of freeing up the thread to process more requests?
Thanks in advance for any help. I haven't found many examples for this code pattern so if there is one a link would be appreciated.
Yes, I think the general idea of your implementation is correct.
I would also suggest, as done in the comments, to use a client such as axios to handle the request in a less verbose and more comprehensive manner, which would leave your code looking something like this:
const axios = require('axios');
app.get('*', (req, res, next) => {
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
axios.post(url, payload, opts)
.then(response => next())
.catch(error => {
console.error(error);
res.redirect('somewhere');
});
});
A bit more to the point, but functionally almost equivalent to your implementation. The one thing you are missing is the onerror callback for your request object, which currently may fail and never return a response as you correctly suspected. You should add:
restReq.on('error', error => {
console.error(error);
res.redirect('somewhere');
});
On the same vein, it would probably be more fitting to call next on result end, instead of doing so while reading response data:
result.on('end', () => {
next();
});
Then you'd be covered to guarantee that a callback would be invoked.
Neither implementation blocks the processing of future requests, as the call to the token validation service is done asynchronously in both cases.

How do you programmatically exit a node.js/express route on an event?

I want to be able to exit execution of a post route when an event is sent from the client-side. I'm using socket.io but I'm not sure it can do what I want. I am using the uploads route to process a file, but if the user deletes the file, I want the app.post execution to end, similar to either a res.end() or return statement.
My app in the front-end receives a file from the user and immediately is sent to the post route for processing. If the user deletes the file and uploads a new one, the previous post route is still going. I want to make sure the previous one was terminated, cancelled, etc.
I'm currently using socket.io to communicate front-end to back-end.
How can I achieve this?
app.post('/uploads', async (req, res) => {
// async func1
// async func2
// if we receive an event from the front end while processing here, how can I exit the post route?
// async func3
});
You can add UUID for each request you make and return it to the front-end. The request will be resolved with the 202 ACCEPTED status code meaning the request was accepted and being handled but the HTTP request will be resolved.
Now you can implement a resourceManagerServeic that will allow APIs (http or ws) to change the state of a resource (like canceling it).
app.post('/uploads', async (req, res) => {
const resourceUuid = resourceManagerServeic.createResource();
res.status(202); // ACCEPTED
res.send({ uuid: resourceUuid });
// start besnise logic
await function1();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function2();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
await function3();
if(resourceManagerServeic.isCanceled(resourceUuid)) {
// cleanup
return; // stop request handling
}
});
app.del('/uploads/:resourceUuid', async (req, res) => {
resourceManagerServeic.cancle(req.params.resourceUuid);
res.end() // handle response
});
I guess that your are using Express. Take a look at express-async-handler
You can invoke it
const asyncHandler = require('express-async-handler')
app.post('/upload', asyncHandler(async(req, res) => {
await firstfunc()
await secondfunc()
}))

Resources