Add additional functionality to end method on Node/Express Response object? - node.js

My team and I are trying to mutate the response.end method in our Express middleware in order to have extra functionality be called just before the server responds back to the client.
Here is our attempt:
return (req: Request, res: Response, next: NextFunction): NextFunction => {
// reassign res.end in order to allow logger functionality before
// a response is sent back the client
const temp = res.end;
res.end = () => {
// instantiates PostQuery object with passed in query data from limiter middleware
const postQuery = new PostQuery(gateURI, projectID, res.locals.graphqlGate);
// our logger middleware functionality
try {
await postQuery.post();
} catch (err) {
if (err) console.log(err);
}
// our temp variable holding node's res.end definition
return temp.call(this);
};
return next();
};
Our test server throws this error when we include this function in our middleware chain:
TypeError: Cannot read properties of undefined (reading 'finished')
at end (node:_http_outgoing:856:19)
at /Users/jon/Documents/Solo Projects/OSP/graphQL-gate-logger/src/index.ts:65:25
index.ts:65 points to return temp.call(this)
We have also tried return temp() , as well as binding temp to the res object, and receive the same error in every instance.
Is there some other way we can reach this goal or do we have to start back at the drawing board?

If you don't have to execute your code BEFORE the response has been sent, but can instead do it right afterwards, then you can use the finish event on the res stream.
app.use((req, res, next) => {
res.on('finish', () => {
console.log(`got finish event for ${req.url}`);
// do your business here after a response has been sent
});
next();
});
There are also a couple problems with your existing override middleware. First off, you aren't preserving arguments that can be optionally send to res.end(). Second, res.end() is supposed to return res which makes it chainable. You aren't doing that. You have assigned it an async function which returns a promise, not res.
Though I think it would be much better to use the finish event as illustrated above and not have to override any methods, this would fix some of the problems with your override:
return (req: Request, res: Response, next: NextFunction): NextFunction => {
// reassign res.end in order to allow logger functionality before
// a response is sent back the client
const origEnd = res.end;
res.end = function(...args) {
// instantiates PostQuery object with passed in query data from limiter middleware
const postQuery = new PostQuery(gateURI, projectID, res.locals.graphqlGate);
// our logger middleware functionality
postQuery.post().catch(err => {
console.log(err);
}).finally(() => {
return origEnd.call(this, ...args);
});
return res;
};
return next();
};

Related

Stripe Webhook 400 error. Raw request body issues

I've been following Stripes documentation on setting up webhooks and testing locally through their Stripe CLI. I can successfully create events, but I receive a response of 400 POST.
2021-12-14 23:18:53 --> payment_intent.created [evt_3K6ox6LqfpEEJtVG0yhTBKDf]
2021-12-14 23:18:53 <-- [400] POST http://localhost:4242/webhook [evt_3K6ox6LqfpEEJtVG0yhTBKDf]
No matter what I do, I always receive a "No signatures found matching the expected signature for payload. Are you passing the raw request body you received from Stripe?" error. Looking through previous questions relating to this, the issue revolves around passing raw data.
I have doublechecked that my webhook secret matches that of the what the CLI gives me.
Here is the latest attempt:
// Use JSON parser for all non-webhook routes
app.use((req, res, next) => {
if (req.originalUrl === '/webhook') {
next();
} else {
express.json()(req, res, next);
}
});
// Stripe requires the raw body to construct the event
app.post('/webhook', express.raw({type: 'application/json'}), (req, res) => {
const sig = req.headers['stripe-signature'];
const rawBody = req.body
let event;
try {
event = stripe.webhooks.constructEvent( rawBody, sig, webhookSecret);
} catch (err) {
// On error, log and return the error message
console.log(`❌ Error message: ${err.message}`);
return res.status(400).send(`Webhook Error: ${err.message}`);
}
// Successfully constructed event
console.log('✅ Success:', event.id);
// Return a response to acknowledge receipt of the event
res.json({received: true});
});
It's a common issue and could depend on your setup and framework. If you are using Node, I recommend going through this Github issue and see if workarounds there work for you.
Lastly you can write into Stripe Support and their dev could help figure out what went wrong.
For 2022, change the app.use(express.json()) to this.
app.use(express.json({
limit: '5mb',
verify: (req, res, buf) => {
req.rawBody = buf.toString();
}
}));
Then you have to change the webhook code to use the rawBody variable that was added correctly and remove the middleware express.raw({type:'application/json'}) since that won't be needed once you have the rawBody variable added to the request anyways.
app.post('/webhook', (request, response) => {
const sig = request.headers['stripe-signature'];
let event;
try {
event = stripe.webhooks.constructEvent(request.rawBody, sig, endpointSecret); //#JA - Had to modify this to take the rawBody since this is what was needed.
} catch (err) {
response.status(400).send(`Webhook Error: ${err.message}`);
return;
}
// Handle the event
switch (event.type) {
case 'payment_intent.succeeded':
const paymentIntent = event.data.object;
// Then define and call a function to handle the event payment_intent.succeeded
break;
// ... handle other event types
default:
console.log(`Unhandled event type ${event.type}`);
}
// Return a 200 response to acknowledge receipt of the event
response.send();
});
The nice part about this fix is you can continue to use express.json() middleware to continue to parse the json from your API's as normal and none of your code should break :).

Respond to Koa request immediately, continue middleware chain

I am writing the middleware for API endpoints in my app that respond to webhooks from other applications, and am relatively new to Koa, so am not completely familiar with its patterns.
I would like to structure my middleware as follows:
exports.updateReceived = async (ctx, next) => {
// Respond to server issuing the webhook
ctx.res.body = "ok";
ctx.res.statusCode = 200;
// Grab what we need from the request
const { headers, state, request } = ctx;
const { body } = request;
// Do some async work
const { example } = await doSomeAsyncWork(ctx);
// Prepare a database query
const query = { aValue: anId };
// Run the DB query
const result = await Thing.findOne(query);
// Add data to the request
state.thing = result;
// Move on...
return next();
};
However, this does not appear to be working, as an error in any of my async methods can cause the route to error out.
My goal is for this endpoint to always respond "yep, ok" (immediately), meaning it is simply up to the application to handle any error states.
I have researched this fairly well, and have come across this pattern:
app.use(async ctx => {
db.fetch() // Assuming a Promise is returned
.then(() => { ... })
.catch(err => {
log(err)
})
// status 200 will be returned regardless of if db.fetch() resolves or rejects.
ctx.status = 200
})
However, this does not meet my needs as the middleware makes no use of next, so it is not really a useful pattern, so far as I can tell.
Could someone tell me what I am overlooking?
next() invokes the downstream middleware chain and returns a promise that resolves after all downstream middleware/handlers have finished.
That means you can simply implement your own upstream error handler that catches any errors and always ensures a 200 OK response.
const Koa = require('koa')
const app = new Koa()
app.use(async (ctx, next) => {
next().catch((err) => {
// Print error for our own records / debugging
console.error(err)
// But ensure that outgoing response is always a smile
ctx.status = 200
ctx.body = ':)'
})
})
app.use(async (ctx) => {
// Do your webhook / biz logic here, but for demonstration
// let's make it always throw an error. Thus upstream next()
// will be a rejected promise.
throw new Error('this middleware will always bubble up a rejected promise')
})
app.listen(3000, () => {
console.log('listening on 3000')
})
Note: We are not awaiting next(), so we can end the request immediately. However the next handler in the chain will still have the opportunity to process the logic
app.use((ctx, next) => {
next()
ctx.status = 200
})
app.use( async ctx =>{
db.fetch()
.then(() => { ... })
.catch(err => log(err))
}
}
Just to divert the solution in a different side, You could consider adding your work to some kind of MessageQueue and then let another process do that task for you. Basically asynchrously but you will still be important. This kind of pattern suits for your requirement.
There are many messaging system availble like AWS SQS which you could consider. This way your api will be very light weight and it will do thing which it needs to and send a command to your messaging system to do extra stuff. You are basically separting your core logic and the doing things in background which scales very nicely as well.

Nodejs global variable scope issue

I'm quite new to Nodejs. In the following code I am getting json data from an API.
let data_json = ''; // global variable
app.get('/', (req, res) => {
request('http://my-api.com/data-export.json', (error, response, body) => {
data_json = JSON.parse(body);
console.log( data_json ); // data prints successfully
});
console.log(data_json, 'Data Test - outside request code'); // no data is printed
})
data_json is my global variable and I assign the data returned by the request function. Within that function the json data prints just fine. But I try printing the same data outside the request function and nothing prints out.
What mistake am I making?
Instead of waiting for request to resolve (get data from your API), Node.js will execute the code outside, and it will print nothing because there is still nothing at the moment of execution, and only after node gets data from your api (which will take a few milliseconds) will it execute the code inside the request. This is because nodejs is asynchronous and non-blocking language, meaning it will not block or halt the code until your api returns data, it will just keep going and finish later when it gets the response.
It's a good practice to do all of the data manipulation you want inside the callback function, unfortunately you can't rely on on the structure you have.
Here's an example of your code, just commented out the order of operations:
let data_json = ''; // global variable
app.get('/', (req, res) => {
//NodeJS STARTS executing this code
request('http://my-api.com/data-export.json', (error, response, body) => {
//NodeJS executes this code last, after the data is loaded from the server
data_json = JSON.parse(body);
console.log( data_json );
//You should do all of your data_json manipluation here
//Eg saving stuff to the database, processing data, just usual logic ya know
});
//NodeJS executes this code 2nd, before your server responds with data
//Because it doesn't want to block the entire code until it gets a response
console.log(data_json, 'Data Test - outside request code');
})
So let's say you want to make another request with the data from the first request - you will have to do something like this:
request('https://your-api.com/export-data.json', (err, res, body) => {
request('https://your-api.com/2nd-endpoint.json', (err, res, body) => {
//Process data and repeat
})
})
As you can see, that pattern can become very messy very quickly - this is called a callback hell, so to avoid having a lot of nested requests, there is a syntactic sugar to make this code look far more fancy and maintainable, it's called Async/Await pattern. Here's how it works:
let data_json = ''
app.get('/', async (req,res) => {
try{
let response = await request('https://your-api.com/endpoint')
data_json = response.body
} catch(error) {
//Handle error how you see fit
}
console.log(data_json) //It will work
})
This code does the same thing as the one you have, but the difference is that you can make as many await request(...) as you want one after another, and no nesting.
The only difference is that you have to declare that your function is asynchronous async (req, res) => {...} and that all of the let var = await request(...) need to be nested inside try-catch block. This is so you can catch your errors. You can have all of your requests inside catch block if you think that's necessary.
Hopefully this helped a bit :)
The console.log occurs before your request, check out ways to get asynchronous data: callback, promises or async-await. Nodejs APIs are async(most of them) so outer console.log will be executed before request API call completes.
let data_json = ''; // global variable
app.get('/', (req, res) => {
let pr = new Promise(function(resolve, reject) {
request('http://my-api.com/data-export.json', (error, response, body) => {
if (error) {
reject(error)
} else {
data_json = JSON.parse(body);
console.log(data_json); // data prints successfully
resolve(data_json)
}
});
})
pr.then(function(data) {
// data also will have data_json
// handle response here
console.log(data_json); // data prints successfully
}).catch(function(err) {
// handle error here
})
})
If you don't want to create a promise wrapper, you can use request-promise-native (uses native Promises) created by the Request module team.
Learn callbacks, promises and of course async-await.

Will emitting request and response handler in Express.js cause problem?

I have a doubt in my design pattern on my Express app,
so i wrap my controller in try and catch, and the catch method is emitting (req, res) handler from controller and later will be handled by a function that send response back to the client.
the code is more or less like this :
const errorExceptionHandler = fn => (req, res, next) => {
fn(req, res, next).catch((err) => {
emitter.emit('onControllerError', {
err: err,
req: req,
res: res,
next: next
})
})
}
the code above emtting req, res, and next, the default parameters that express provided.
emitter.on('onControllerError', params => {
const err = params.err
const req = params.req
const res = params.res
const next = params.next
if (!res.headerSent) {
res.send({
status: 500,
url: process.env.DEBUG ? req.url : undefined,
message: process.env.DEBUG ? err.message : "Something went wrong!"
})
}
})
and above is how the 'onControllerError' event is handled, my concern is, will this cause trouble later if the traffic goes up? or will it send a wrong response to the client?
Increased traffic wouldn't matter here as each request is still handled independently, plus all the necessary data is being passed directly to the event handler.
So no, based on your code I can't think of any reason why it would start to fail.

node http-proxy: async modification of request body

I need to modify the request body asynchronously. Something along the lines of this:
proxy.on('proxyReq', function(proxyReq, req, res, options) {
if(req.body) {
new Promise(function(resolve){
setTimeout(function() { // wait for the db to return
'use strict';
req.body.text += 'test';
let bodyData = JSON.stringify(req.body);
proxyReq.setHeader('Content-Type','application/json');
proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
// stream the content
proxyReq.write(bodyData);
resolve();
},1);
});
}
});
When I run this I get the error saying cannot modfiy headers once they have been set. Which makes sense.
How can I halt the sending of the request until I'm ready? I've looked at removing various listeners from proxyReq without success..
By looking at the source code #-) it seems like it's not really possible because the proxyReq event is sent and then the code moves on.
If it would instead wait for a promise, it would be possible (if you'd return that promise as well).
A minimal fork on this lib could be for example:
// Enable developers to modify the proxyReq before headers are sent
proxyReq.on('socket', function(socket) {
if(server) { server.emit('proxyReq', proxyReq, req, res, options); }
});
(proxyReq.proxyWait || Promise.resolve())
.then( ... // rest of the code inside the callback
And then
proxy.on('proxyReq', function(proxyReq, req, res, options) {
if(req.body) {
proxyReq.proxyWait = new Promise(function(resolve){
setTimeout(function() { ...
But depending on your use case, there might be other solutions as well. For example, consider if it's really necessary that you use this proxy library. It You could alternatively use http directly, where you have all the control on the events and callbacks.
You can set selfHandleResponse: true inside the HttpProxy.createProxyServer. This then allows (and forces) you to handle the proxyRes manually!
const proxy = HttpProxy.createProxyServer({selfHandleResponse: true});
proxy.on('proxyRes', async (proxyReq, req, res, options) => {
if (proxyReq.statusCode === 404) {
req.logger.debug('Proxy Request Returned 404');
const something = await doSomething(proxyReq);
return res.json(something);
}
return x;// return original proxy response
});
I came here looking for the solution to a slightly different problem: Modifying the request headers (not body) before proxying.
I post this here in case that it is helpful to others. And maybe the code can be adapted to also modify the request body.
const http = require('http');
const httpProxy = require('http-proxy');
var proxy = httpProxy.createProxyServer({});
var server = http.createServer(function(req, res) {
console.log(`${req.url} - sleeping 1s...`);
setTimeout(() => {
console.log(`${req.url} - processing request`);
req.headers['x-example-req-async'] = '456';
proxy.web(req, res, {
target: 'http://127.0.0.1:80'
});
}, 1000);
});
server.listen(5050);

Resources