Express server for both serving pages and endpoint - node.js

I want to have a Node web server serving pages and also set as an endpoint listening to webhooks. The example for the first comes from Rocket Rides, with the relevant code being:
const express = require('express');
// ...
const app = express();
// ...
// CRUD routes for the pilot signup and dashboard
app.use('/pilots', require('./routes/pilots/pilots'));
app.use('/pilots/stripe', require('./routes/pilots/stripe'));
// ...
// Index page for Rocket Rides
app.get('/', (req, res) => {
res.render('index');
});
// ...
// Start the server on the correct port
const server = app.listen(process.env.PORT || config.port, () => {
console.log('🚀 Rocket Rides server started:', config.publicDomain);
});
For the second, I use this tutorial with the following relevant code:
// Match the raw body to content type application/json
app.post('/webhook', bodyParser.raw({type: 'application/json'}), (request, response) => {
console.log("called!");
let event;
try {
event = JSON.parse(request.body);
} catch (err) {
response.status(400).send(`Webhook Error: ${err.message}`);
}
// Handle the event
switch (event.type) {
case 'payment_intent.succeeded':
const paymentIntentSucceeded = event.data.object;
break;
case 'payment_method.attached':
const paymentMethod = event.data.object;
break;
// ... handle other event types
default:
// Unexpected event type
return response.status(400).end();
}
// Return a response to acknowledge receipt of the event
response.json({received: true});
});
app.listen(8000, () => console.log('Webhooks running on port 8000'));
With both parts, the server does not handle the webhook request:
Webhooks running on port 8000
POST /webhook 404 590.525 ms - 1415
and the sender receives a 404.
When I comment out most of the code in the first part, the webhook request is handled properly:
Webhooks running on port 8000
called!
and the sender receives a 200.
I believe one of the routes from the web server is masking the route for the endpoint. I tried looking for one with this thread:
app._router.stack.forEach(function(middleware){
if(middleware.route){ // routes registered directly on the app
routes.push(middleware.route);
} else if(middleware.name === 'router'){ // router middleware
middleware.handle.stack.forEach(function(handler){
route = handler.route;
route && routes.push(route);
});
}
});
console.log(routes);
and the only relevant one was GET /.
If I include the code for the endpoint before the code for the router, the webhook is handled properly.
How can I find which route is masking the webhook endpoint?

Put the more specific route definitions first like this:
app.use('/pilots/stripe', require('./routes/pilots/stripe'));
app.use('/pilots', require('./routes/pilots/pilots'));
And, the more general route definitions later. That makes sure the more specific routes aren't gobbled up by the more general handlers.
Keep in mind that with app.use(), something like app.use('/pilots') will match any route that starts with /pilots which would include all your /pilots/stripe routes. So, you want to make sure and put the app.use('/pilots/stripe', ...) before the app.use('/pilots', ...).
Another observation. In your /webhook handler, you need to return after you send an error status so the rest of your request handler doesn't continue to run.
// Match the raw body to content type application/json
app.post('/webhook', bodyParser.raw({type: 'application/json'}), (request, response) => {
console.log("called!");
let event;
try {
event = JSON.parse(request.body);
} catch (err) {
response.status(400).send(`Webhook Error: ${err.message}`);
return; // <===== Add this
}
....
}
This appears to be a bug in the actual stripe documentation.
If I include the code for the endpoint before the code for the router, the webhook is handled properly.
I would guess that you have bodyparser middleware elsewhere in your server. If that middleware is BEFORE this route, then this route won't get to use its bodyParser.raw() and get the data the way it wants and it will not work properly. This is because whichever bodyParser middleware runs first reads the body and parses it and puts it wherever that middleware is configured to put it. Once the body is read, it's gone from the stream so any other middleware that comes along and also tries to read the body data from the stream will find the stream empty.
So, this route just has to be before any other body parsing middleware that might handle JSON.
If you provided a link to your full code, we could take a look and see where this is happening.

Related

Express routes alongside socket.io ones

I'm trying to handle regular http requests alongside socket.io.
The app.ts file has some REST Route handles, and error handler for all routes that don't exist.
app.use("/users", userRouter);
app.use("/products", productRouter);
app.all("*", async (req: Request, res: Response) => {
throw new NotFoundError("route");
});
Suppose I want to add one route (namespace) that will handle the socket.io connections.
With my current implementation I keep getting an error that the route doesn't exist (from the app.all) because express thinks it's a regular http client.
What is the right way to implement regular http requests alongside socket.io ones?
I know that in order to implement namespace I need to use
const io = new Server(httpServer, { cors: { origin: "*" } });
io.of("/socket").on("connection", () => {
// Do something
});
But localhost::PORT/socket is treated as if it is a regular request

Socket.io only emits outside of io.on('connection') function inside express route

After several hours of messing around with socket.io, I finally got it working, but it doesn't work as the docs describe. At this point, I'm just wondering what is going on here..
server.js
const server = require('http').createServer(app);
const io = require('socket.io')(server);
const PORT = process.env.PORT || 5000;
server.listen(PORT);
// this works as expected
// io.on('connection', socket => {
// console.log('connect', socket.id);
// socket.emit('test', 'emit from server');
// });
// Make io accessible to our router
app.use(function (req, res, next) {
req.io = io;
next();
});
app.use('/api/posts', require('./routes/api/posts'));
/api/posts.js
router.post('/', async (req, res) => {
const user = await User.findById(req.user.id)
// this does not work
// req.io.on('connection', socket => {
// socket.emit('123', "foo: bar");
// });
// but this does...
req.io.emit(user.id, "foo: bar");
})
client
const myComponent = ({
auth: { user },
notifications: {
notifications,
activeNotifications,
loading: notification_loading,
}}) =>{
const [socketData, setSocketData] = useState(false);
useEffect(() => {
if (!notification_loading) {
const socket = io();
socket.on('connect', () => {
console.log('LINE2: 67 _connect ', 'connect');
});
socket.on(user.id, function (data) {
setSocketData(data);
});
}
}, [notification_loading]);
useEffect(() => {
socketData &&
pollNotifications(
{
newestIndex: notifications[0] ? notifications[0].date : 0,
},
socketData
);
setSocketData(false);
return () => {};
}, [socketData]);
return (<span> blah blah </span>)
}
I stumbled onto some posts of people using it with just io.emit, so I tried it and it worked, but does anyone know why?
You have to think through the lifetime of a browser page and the sequence of events with a socket.io connection to understand why there's a problem with the version of the code that doesn't work.
Let's assume you have a web page that makes a socket.io connection when the page is first loaded into the browser.
If, sometime after that page is loaded, that page makes an Ajax call to do a POST to the / route, then doing req.io.on('connection', socket => { ....}) will install an event handler for future connection events that will happen for other pages and, in fact, those event handlers will accumulate with lots of duplicates. The page that has make the Ajax call is already connected to socket.io so that page itself will not generate another connect message and thus you will never see the results of that event handler in the page that made the ajax call.
If, sometime after that page is loaded, that page makes a form POST to the / route, then it will indeed install yet another handler for the connect event just like before and again, they will accumulate with lots of duplicate event handlers over time. Your POST handler would presumably send a response and that would cause the browser to load a NEW web page. When that web page loads and then makes its socket.io connection, you probably would get that 123 event sent to it, but you still can't code it this way because of the build-up of event handlers.
What you need to remember is that something like this:
// this does not work
req.io.on('connection', socket => {
socket.emit('123', "foo: bar");
});
is a global handler for all socket.io connect events from all clients. It does NOT pertain to just the client that is making the current POST request. Thus, you never want to install global event handlers within routes because every single time that route is hit by any client, you will add yet another duplicate event handler and they will pile up indefinitely.
There are likely other ways to solve whatever problem you're trying to solve, but frankly, we'd need to know what the actual problem is in order to suggest a better way to solve it. Here you are only describing the problem with your solution and not describing what the actual problem is. Since, this is the wrong approach, we need to go back several steps to what the fundamental problem is so we can suggest a different approach.
In many cases, what people are trying to do is to "emit to a socket.io connection that corresponds to the client that is making the current http request (if there is one)". That is usually done by getting the socket.id from an http cookie and then using that socket.id to find the right socket to emit to. This is generally only possible if the http request comes from an Ajax call from an already established web page. If the http request comes from the browser attempting to fetch and load a new web page, then that page does not yet have an established socket.io connection so you can't yet emit to it.

How to send request information with Sentry's new unified SDK?

The legacy Sentry Node SDK (raven) allowed sending HTTP request information along with the error, by passing the request object inside the options object (2nd argument):
Raven.captureException(someError, { req: req });
Line of code extracted from the documentation: https://docs.sentry.io/clients/node/usage/#raven-recording-breadcrumbs
With that we could get additional context about the error, such as what was the device being used.
Is there any way we can pass the request object with the new SDK? The context section of the new SDK explains how to send data such as user identification, custom tags, etc by using scopes, but no request option is available in there. Does this mean that the request information should now be manually sent inside tags and/or extra objects?
As #MarkusUnterwaditzer shared, the Express integration worked for me: https://docs.sentry.io/platforms/node/express/ The key parts are adding the Sentry.Handlers.requestHandler() middleware and then either the errorHandler middleware or doing Sentry.captureException(error) yourself (with no need to pass in {req: req}).
Sample code:
const express = require('express');
const app = express();
const Sentry = require('#sentry/node');
Sentry.init({ dsn: 'https://8f0620a3bfea4f2ca26aefb074851e23#sentry.io/280382' });
// The request handler must be the first middleware on the app
app.use(Sentry.Handlers.requestHandler());
app.get('/', function mainHandler(req, res) {
throw new Error('Broke!');
});
// The error handler must be before any other error middleware
app.use(Sentry.Handlers.errorHandler());
// Optional fallthrough error handler
app.use(function onError(err, req, res, next) {
// The error id is attached to `res.sentry` to be returned
// and optionally displayed to the user for support.
res.statusCode = 500;
res.end(res.sentry + '\n');
});
app.listen(3000);

Error: Can't set headers after they are sent. error received after refreshing a get request

The following code works as expected the first time it executes. The second time through it fails with the following error:
Error: Can't set headers after they are sent.
Basically, i'm trying to return a message from a series of published mqtt topics. I've tried several ways to do this but am at a loss as to this behavior. The code below shows an attempt using promises as detailed in this MDN article. I also tried using async / await (i can post this code as well if needed).
Concept
a get request arrives from a webpage, this triggers a mqtt message to be published to a local broker. a separate endpoint is listening for this topic. In this case a raspberry pi with a connected webcam. The topic is received the rpi does it's thing and publishes a return message with a url as it's data. the initial mqtt publisher is listening for this response and fires res.json() in response.
This works the first time, when i refresh the webpage it fails with Can't headers after they are sent. How does one reset res? It appears that this is the case.
punchitRoute.js
const express = require('express');
const punchitRoute = express.Router();
const mqtt = require('mqtt');
const client = mqtt.connect('mqtt://192.168.88.54:1883');
client.subscribe('returned', function(){
console.log('subscribed');
})
async function img(){
return new Promise(resolve =>{ client.on('message', function(topic, url){
const message = {
url: url.toString(),
name: "garfield",
date: Date.now()
}
return message;
})
})
}
punchitRoute.get('/', async function(req, res){
const mess = await img();
console.log(mess);
res.json(mess);
});
module.exports = punchitRoute;
server.js
const express = require('express');
const app = express();
const bodyParser = require('body-parser');
const punchit = require('./routes/punchitRoute');
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended:false }));
app.get("/", (req, res) => {
res.json({
name: "Express application",
message: "welcome to punchit"
})
});
app.use('/api/punchit', punchit);
const PORT = process.env.PORT || 6040;
app.listen(PORT);
console.log('server started on: ', PORT);
My guess is that i am missing a basic fact of life regarding requests using express.
Thanks in advance for any guidance.
Thanks jfriend00 for pointing me in the right direction. First the code posted was completely incorrect. I was super confused with this and subsequently renamed the file punchitRoute file to keep working on it in a different approach, but forgot to rename the required file in server.js! That was a bonehead mistake.
The 2nd comment about event handler for client.on('message', ...) gave me the hint i needed to look into the event loop.
The idea here was to snap an image from a webcam using mqtt from a get request. The get request comes into the express server, triggers a subscription to a topic called 'returned' and publishes to a topic called 'sunrise' with the a message of 'click'.
There is a separate endpoint on a local network and listening to a local mqtt broker (both shared with the express server) that is listening for the 'sunrise' topic. This endpoint broadcast's 'returned' after the image has been saved into an S3 bucket and send the url of the image as the message back to the server running the get request. The mqtt functions work well, the issue I had was with the response event not terminating at the conclusion of the response. For this too work, several things had to change.
moved the res.send into a function called after client.on (this didn't completely work, it allowed the send to happen but still returned the error)
added event emitters to see which event were being triggered over and over again (this gave me some visibility into what was happening, but still didn't remove the error)
researched additional mqtt libraries (ended up staying with async-mqtt)
added client.unsubscribe for both topics after the res.send function (this had no effect)
finally found this answer which helped me to realize this whole time i could've just called client.once!
here's the code for the get request:
punchitRoute.get('/', async function(req, res){
//adding event listner 'send'
mqEvent.on('send', function(){
console.log('added send');
});
let sendit = function(url){
'send', //emmitting 'send'
res.status(200).send(
'<img src='+url.toString()+' />'+
'<h1>PUNCHED!</h1>'+
'<p>'+
'name: garfield <br/> time: '+moment(Date.now()).tz("America/Louisville").format('MM-DD-YYYY'));
}
client.subscribe('returned', function(){
console.log('subscribed');
});
client.publish('sunrise', 'click');
//changed this to .once and no more errors
await client.once('message', async function callout(topic, url){
try {
mqEvent.on('message', function(){
console.log('added message');
});
sendit(url.toString());
// console.log(res);
client.unsubscribe('returned', ()=>{
console.log('unsubscribed');
});
client.unsubscribe('sunrise', ()=>{
console.log('unsubscribed publish');
});
//clearing out the send event listeners
mqEvent.removeAllListeners('send');
console.log(mqEvent.listenerCount('send'));
//clearing out the message event listeners
mqEvent.removeAllListeners('message');
} catch(e){
console.log(e);
console.log('error');
}
})
});
Hopefully this can someone else faced with a similar issue.

Node Express Middleware how to send the res, req object

I am unable to send the res (request object) between functions. The following code is executed by my app.js (main express middleware):
//app.js calls File.js
//File1.js
var file2 = require('./File2.js);
export.modules = function (req,res,next) {
file2(data) {
res.send(data); //<-- this is not working
}
}
//File2.js
export.modules = function(data){
data = 'test';
}
Also I do not understand when to use next() or when to use res.end().
Its really hard to understand from you code snippets, so i will address your second question regarding next vs send
You use next inside your middlewares, which means you dont want yet to respond to your client with data, but you want to proccess the data from another middleware down the line, when you reach your final middleware you need to use res.send();
note that you cannot use res.send multiple times, so you must call it when you finished your processing and want to respond the data to the user.
you must use middleware with express as following:
var app = express();
app.use(function(req,res, next){
// some proccessing
req.proccessData = "12312312";
next();
})
app.use(function(req,res, next){
// here you respond the data to the client
res.send(req.proccessData);
})
You can also use this with routes(get, post and etc...) Just add next as third param to the route when you want to send data to next stage

Resources