I am using this (contentful-export) library in my express app like so
const app = require('express');
...
app.get('/export', (req, rex, next) => {
const contentfulExport = require('contentful-export');
const options = {
...
}
contentfulExport(options).then((result) => {
res.send(result);
});
})
now this does work, but the method takes a bit of time and sends status / progress messages to the node console, but I would like to keep the user updated also.. is there a way I can send the node console progress messages to the client??
This is my first time using node / express any help would be appreciated, I'm not sure if this already has an answer since im not entirely sure what to call it?
Looking of the documentation for contentful-export I don't think this is possible. The way this usually works in Node is that you have an object (contentfulExport in this case), you call a method on this object and the same object is also an EventEmitter. This way you'd get a hook to react to fired events.
// pseudo code
someLibrary.on('someEvent', (event) => { /* do something */ })
someLibrary.doLongRunningTask()
.then(/* ... */)
This is not documented for contentful-export so I assume that there is no way to hook into the log messages that are sent to the console.
Your question has another tricky angle though. In the code you shared you include a single endpoint (/export). If you would like to display updates or show some progress you'd probably need a second endpoint giving information about the progress of your long running task (which you can not access with contentful-export though).
The way this is usually handled is that you kick of a long running task via a certain HTTP endpoint and then use another endpoint that serves infos via polling or or a web socket connection.
Sorry that I can't give a proper solution but due to the limitation of contentful-export I don't think there is a clean/easy way to show progress of the exported data.
Hope that helps. :)
Related
I have an express service where a lot of my endpoints look like the following.
var express = require('express');
var router = express.Router();
router.get('/updateThisValue', (req, res) => {
let value = req['query'].value
try {
database.update(value, function() { // callback function on update
console.log(undefinedValue.undefined) // crash the server
})
} catch (err) {
// Log the error and other stuff
}
})
Looking at this code, if a database update is called, something will crash in the callback function and since there is no try-catch, the whole server should crash as well.
Right now, I am okay with the server crashing. The problem is that I would like to be able to log some debugging information in the callback before it crashes, namely the express endpoint hit and the parameters passed to it.
One solution is that I could just log every single endpoint hit, but the only problem with that is that the server starts to have very large logs. So we don't log endpoints hit by default. The frustration however is that recently an endpoint hit the server that did cause it crash and I'm still unsure what the actual endpoint hit and parameters passed to it where.
Basically, I want the server to still crash but at least I would be able to trace the endpoint hit and its parameters
Could this help: https://expressjs.com/en/guide/error-handling.html
Normally you would see the exception in the console or in the logs of your process manager (like pm2).
I am running into a problem while using socket.io to do some event handling. For some reason, the following code snippet does not handle the event 'update', or any event for that matter. Let me explain the situation.
I have created a file named updates.js to create a socket.io socket variable named socket_8888 that is bound to port 8888. I then use module.exports to make that socket variable available to any other file that imports updates.js using require('updates.js'). I structured my application this way because I need to emit events from several different files.
In app.js:
var updates = require('updates.js');
setTimeout(function() {
updates.regular.on("update", function () {
console.log("Updated.");
})
}, 1000);
setTimeout(
function () {
console.log(updates.regular.eventNames()); // Verifying that there is actually a listener bound to the socket -> prints ['update']
updates.regular.emit("update", 100)
}, 1500);
In updates.js:
var io = require("socket.io");
var socket_8888 = io(8888);
var updates = {
regular: socket_8888
};
module.exports = updates;
However, a few simple tests have uncovered that events are not being handled, and I really cannot figure out why. The word "Updated" should print a second and a half after I run the application using "node www", but it does not.
The reason I started doing this simple testing was because I am trying to revive an old website of mine, but after a couple years, API updates have rendered a lot of my code useless. So I am trying to rebuild. I am not trying to send events between different files on the server. I am only testing the events locally because the events were not firing to the browser client. For this reason, I decided to investigate using this simple test, and it turns out the events can not even be emitted/listened to on the actual server, let alone be handled on a client that is on a whole different network.
I have already verified that the listener is actually binding to the socket. However, I do not know how to check whether or not the socket is actually emitting the event "update".
I have written the listener to bind only after one second because attempting to bind the moment the application starts does not give Express enough time to set everything up. Otherwise, the socket would still be undefined.
I do not get any error messages. The code just does not work as I expected.
I would really appreciate it if the community can tell me why the event 'update' is not being handled.
To include update module (update.js)
Try this
It work's Perfectly
module.exports = updates
var updates = require('./updates');
I'm using react-boilerplate (with react-router, sagas, express.js) for my React app and on top of it I've added SSR logic so that once it receives an HTTP request it renders react components to string based on URL and sends HTML string back to the client.
While react rendering is happening on the server side, it also makes fetch request through sagas to some APIs (up to 5 endpoints based on the URL) to get data for components before it actually renders the component to string.
Everything is working great if I make only several request to the Node server at the same time, but once I simulate load of 100+ concurrent requests and it starts processing it then at some point it crashes with no indication of any exception.
What I've noticed while I was trying to debug the app is that once 100+ incoming requests begin to be processed by the Node server it sends requests to APIs at the same time but receives no actual response until it stops stacking those requests.
The code that's used for rendering on the server side:
async function renderHtmlDocument({ store, renderProps, sagasDone, assets, webpackDllNames }) {
// 1st render phase - triggers the sagas
renderAppToString(store, renderProps);
// send signal to sagas that we're done
store.dispatch(END);
// wait for all tasks to finish
await sagasDone();
// capture the state after the first render
const state = store.getState().toJS();
// prepare style sheet to collect generated css
const styleSheet = new ServerStyleSheet();
// 2nd render phase - the sagas triggered in the first phase are resolved by now
const appMarkup = renderAppToString(store, renderProps, styleSheet);
// capture the generated css
const css = styleSheet.getStyleElement();
const doc = renderToStaticMarkup(
<HtmlDocument
appMarkup={appMarkup}
lang={state.language.locale}
state={state}
head={Helmet.rewind()}
assets={assets}
css={css}
webpackDllNames={webpackDllNames}
/>
);
return `<!DOCTYPE html>\n${doc}`;
}
// The code that's executed by express.js for each request
function renderAppToStringAtLocation(url, { webpackDllNames = [], assets, lang }, callback) {
const memHistory = createMemoryHistory(url);
const store = createStore({}, memHistory);
syncHistoryWithStore(memHistory, store);
const routes = createRoutes(store);
const sagasDone = monitorSagas(store);
store.dispatch(changeLocale(lang));
match({ routes, location: url }, (error, redirectLocation, renderProps) => {
if (error) {
callback({ error });
} else if (renderProps) {
renderHtmlDocument({ store, renderProps, sagasDone, assets, webpackDllNames })
.then((html) => {
callback({ html });
})
.catch((e) => callback({ error: e }));
} else {
callback({ error: new Error('Unknown error') });
}
});
}
So my assumption is that something is going wrong once it receives too many HTTP requests which in turn generates even more requests to API endpoints to render react components.
I've noticed that it blocks event loop for 300ms after renderAppToString() for every client request, so once there are 100 concurrent requests it blocks it for about 10 seconds. I'm not sure if that's a normal or bad thing though.
Is it worth trying to limit simultaneous requests to Node server?
I couldn't find much information on the topic of SSR + Node crashes. So I'd appreciate any suggestions as to where to look at to identify the problem or for possible solutions if anyone has experienced similar issue in the past.
In the above image, I am doing ReactDOM.hydrate(...) I can also load my initial and required state and send it down in hydrate.
I have written the middleware file and I am using this file to decide based on what URL i should send which file in response.
Above is my middleware file, I have created the HTML string of the whichever file was requested based on URL. Then I add this HTML string and return it using res.render of express.
Above image is where I compare the requested URL path with the dictionary of path-file associations. Once it is found (i.e. URL matches) I use ReactDOMserver render to string to convert it into HTML. This html can be used to send with handle bar file using res.render as discussed above.
This way I have managed to do SSR on my most web apps built using MERN.io stack.
Hope my answer helped you and Please write comment for discussions
1. Run express in a cluster
A single instance of Node.js runs in a single thread. To take
advantage of multi-core systems, the user will sometimes want to
launch a cluster of Node.js processes to handle the load.
As Node is single threaded the problem may also be in a file lower down the stack were you are initialising express.
There are a number of best practices when running a node app that are not generally mentioned in react threads.
A simple solution to improve performance on a server running multiple cores is to use the built in node cluster module
https://nodejs.org/api/cluster.html
This will start multiple instance of your app on each core of your server giving you a significant performance improvement (if you have a multicore server) for concurrent requests
See for more information on express performance
https://expressjs.com/en/advanced/best-practice-performance.html
You may also want to throttle you incoming connections as when the thread starts context switching response times drop rapidly this can be done by adding something like NGINX / HA Proxy in front of your application
2. Wait for the store to be hydrated before calling render to string
You don't want to have to render you layout until your store has finished updating as other comments note this is a blocks the thread while rendering.
Below is the example taken from the saga repo which shows how to run the sagas with out the need to render the template until they have all resolved
store.runSaga(rootSaga).done.then(() => {
console.log('sagas complete')
res.status(200).send(
layout(
renderToString(rootComp),
JSON.stringify(store.getState())
)
)
}).catch((e) => {
console.log(e.message)
res.status(500).send(e.message)
})
https://github.com/redux-saga/redux-saga/blob/master/examples/real-world/server.js
3. Make sure node environment is set correctly
Also ensure you are correctly using NODE_ENV=production when bundling / running your code as both express and react optimise for this
The calls to renderToString() are synchronous, so they are blocking the thread while they are running. So its no surprise that when you have 100+ concurrent requests that you have an extremely blocked up queue hanging for ~10 seconds.
Edit: It was pointed out that React v16 natively supports streaming, but you need to use the renderToNodeStream() method for streaming the HTML to the client. It should return the exact same string as renderToString() but streams it instead, so you don't have to wait for the full HTML to be rendered before you start sending data to the client.
I currently have a frontend-only app that fetches 5-6 different JSON feeds, grabs some necessary data from each of them, and then renders a page based on said data. I'd like to move the data fetching / processing part of the app to a server-side node application which outputs one simple JSON file which the frontend app can fetch and easily render.
There are two noteworthy complications for this project:
1) The new backend app will have to live on a different server than its frontend counterpart
2) Some of the feeds change fairly often, so I'll need the backend processing to constantly check for changes (every 5-10 seconds). Currently with the frontend-only app, the browser fetches the latest versions of the feeds on load. I'd like to replicate this behavior as closely as possible
My thought process for solving this took me in two directions:
The first is to setup an express application that uses setTimeout to constantly check for new data to process. This data is then sent as a response to a simple GET request:
const express = require('express');
let app = express();
let processedData = {};
const getData = () => {...} // returns a promise that fetches and processes data
/* use an immediately invoked function with setTimeout to fetch the data
* when the program starts and then once every 5 seconds after that */
(function refreshData() {
getData.then((data) => {
processedData = data;
});
setTimeout(refreshData, 5000);
})();
app.get('/', (req, res) => {
res.send(processedData);
});
app.listen(port, () => {
console.log(`Started on port ${port}`);
});
I would then run a simple get request from the client (after properly adjusting CORS headers) to get the JSON object.
My questions about this approach are pretty generic: Is this even a good solution to this problem? Will this drive up hosting costs based on processing / client GET requests? Is setTimeout a good way to have a task run repeatedly on the server?
The other solution I'm considering would deal with setting up an AWS Lambda that writes the resulting JSON to an s3 bucket. It looks like the minimum interval for scheduling an AWS Lambda function is 1 minute, however. I imagine I could set up 3 or 4 identical Lambda functions and offset them by 10-15 seconds, however that seems so hacky that it makes me physically uncomfortable.
Any suggestions / pointers / solutions would be greatly appreciated. I am not yet a super experienced backend developer, so please ELI5 wherever you deem fit.
A few pointers.
Use crontasks for periodic processing of data. This is far preferable especially if you are formatting a lot of data.
Don't setup multiple Lambda functions for the same task. It's going to be messy to maintain all those functions.
After processing / fetching the feed, you can store the JSON file in your own server or S3. Note that if it's S3, then you are paying and waiting for a network operation. You can read the file from your express app and just send the response back to your clients.
Depending on the file size and your load in the server you might want to add a caching server so that you can cache the response until new JSON data is available.
Sorry if this is a basic question. I'm just starting my 3rd week of doing Node.js programming! I looked around and didn't see an answer to this, specifically. Maybe it's just assumed when answering questions about child_process.spawn/fork by those who know this stuff better than I do.
I have a Node/Express app where I want to take in an HTTP request, save a bit of data to Mongo, return success/error, but...at the same time kick off a process to take some of the data and do a lookup against a web API. I want to save that data back to Mongo, but there's no need to have that communicated back to the HTTP client. (I'll probably log the success/error of that call somewhere.)
How do I kick off that 2nd task to run independent of the main request and not cause the response to wait for it to complete?
The 2nd task will also be written in Node.js. I'd like it to just be another function in the same file, if possible.
Thanks in advance!
I don't see why you would need spawning another process just for that. In node you are not limited to the http request lifecycle to run stuff like other frameworks. This should do it:
function yourHandler(req, res, next) {
dataAccess.writeToMongo(someData, function(err, res) {
var status = err ? 500 : 200;
// write back to response already!
res.status(status);
res.end();
// do not completely terminate yet
// kick off web api call
apiClient.doSomething();
});
}