In Nodejs I have developed a small Client application that sends multiple “multipart/form-data” to my Server application using POST requests.
Each form to be sent is composed by a file (loaded from the Client hard-disk) and a string information. Basically I have the following situation :
Form 1: (File 1, String 1)
Form 2: (File 2, String 2)
Form 3: (File 3, String 3)
Etc..
To make the POST requests I’m using the “form-data” library ( https://www.npmjs.com/package/form-data ).
The problem that I’m facing is that all the POST requests are sent after the end of the execution of my Client application, but I would like to be able to send each POST request separately.
Here is a part of the code that I’m using :
function FormSubmit(item)
{
var FileStream = fs.createReadStream(item.path);
// Create an "Upload" Form and set all form parameters.
let form = new FormData();
form.append('Text1', 'test');
form.append('file', FileStream);
// Form Submit.
form.submit('http://localhost:5000/upload', function(err, res) {
if (err) {
console.log(err);
}
if (res!= undefined)
res.resume();
else
console.log('Res undefined: ', res);
});
}
I’m calling the “FormSubmit” function multiple times, and I was expecting to receive the POST request on the Server application every time after executing the command “form.submit”, but in reality I receive the POST requests all together after the entire application execution finish.
In particular the Server receives the requests on the command “self.emit('connect');” inside the function “afterConnect” in the “net.js” file in the core module.
It seems that it has nothing to do with timings, because even if i put a breakpoint and wait for some minutes after the first execution of the "FormSubmit" function, i don't receive anything on the Server application.
Probably it is not something related to the "form-data" library, because i get the same behaviour using "request", etc..
I guess it is something related to NodeJs itself or about how i wrote the Client application.
I am new to NodeJs so any help/advice would be appreciated.
Thanks.
Related
I am trying to make an API that will send back a real-time JSON. I am using NodeJS with ExpressJS, and Socket.io, and the problem is that res.send can not be sent more than one time; And, I really don't know how to send my (real-time) data without asking the refresh of my page.
Basically, I made a timer that changes the value every second.
I also tried to send a file, but I can't use this method, because my iOS app is asking a JSON data without HTML code
setInterval( function() {
var msg = Math.random().toString();
io.emit('message', msg);
console.log(msg);
res.send(msg);
}, 1000);
Maybe, there is another framework than Express than I could use and could refresh my data automatically? The console.log line works well and my data is updated every 1000ms.
Thank you in advance
I am building a scraper app with nodejs and I'd like it to scrape a certain site 2 times a day.
now, there's a problem though.
what I am used to doing is that from client side, someone makes a request and the app scrapes data and shows the result.
but what If I want the app to just do the scraping 2 times a day, without the need for client to make a request to server.
how does one do that?
Basically, it's a site where the user puts in keywords they are searching for.
the app searches for that keyword everyday and it notifies the user when the keyword shows up on the page.
so, how does one do that without having the user to search for the keyword everyday?
Seems like we can use cron jobs for scheduling, and the scraping will happen twice a day or any times I choose, but the thing is how do I send the data from the scraping to client side?
Or how do I notify the site user that the keyword was found and he can come to the site and look at it?
but what If I want the app to just do the scraping 2 times a day, without the need for client to make a request to server. how does one do that?
You use a task scheduler, such as Cron.
how do I notify the site user that the keyword was found and he can come to the site and look at it?
There are lots of options.
Email
SMS
Twitter messages
Notications + Service Workers
etc
The request npm module would allow you to do that. The following (server) app queries from an external API every 10 seconds:
const request = require('request');
function doRequest() {
request('http://www.randomtext.me/api/', function (error, response, body) {
console.log('error:', error);
console.log('statusCode:', response && response.statusCode);
console.log('body:', body);
// do whatever you need to do with you result
// and notify the user (... not clear what channel you want to use)
// could be done with sockets, email, ... or text messages (twillio) ...
});
}
setInterval(doRequest, 10000); // <-- adapt your intervall here
So this is an easy example for server to server requests ... hope that helps.
I'm using react-boilerplate (with react-router, sagas, express.js) for my React app and on top of it I've added SSR logic so that once it receives an HTTP request it renders react components to string based on URL and sends HTML string back to the client.
While react rendering is happening on the server side, it also makes fetch request through sagas to some APIs (up to 5 endpoints based on the URL) to get data for components before it actually renders the component to string.
Everything is working great if I make only several request to the Node server at the same time, but once I simulate load of 100+ concurrent requests and it starts processing it then at some point it crashes with no indication of any exception.
What I've noticed while I was trying to debug the app is that once 100+ incoming requests begin to be processed by the Node server it sends requests to APIs at the same time but receives no actual response until it stops stacking those requests.
The code that's used for rendering on the server side:
async function renderHtmlDocument({ store, renderProps, sagasDone, assets, webpackDllNames }) {
// 1st render phase - triggers the sagas
renderAppToString(store, renderProps);
// send signal to sagas that we're done
store.dispatch(END);
// wait for all tasks to finish
await sagasDone();
// capture the state after the first render
const state = store.getState().toJS();
// prepare style sheet to collect generated css
const styleSheet = new ServerStyleSheet();
// 2nd render phase - the sagas triggered in the first phase are resolved by now
const appMarkup = renderAppToString(store, renderProps, styleSheet);
// capture the generated css
const css = styleSheet.getStyleElement();
const doc = renderToStaticMarkup(
<HtmlDocument
appMarkup={appMarkup}
lang={state.language.locale}
state={state}
head={Helmet.rewind()}
assets={assets}
css={css}
webpackDllNames={webpackDllNames}
/>
);
return `<!DOCTYPE html>\n${doc}`;
}
// The code that's executed by express.js for each request
function renderAppToStringAtLocation(url, { webpackDllNames = [], assets, lang }, callback) {
const memHistory = createMemoryHistory(url);
const store = createStore({}, memHistory);
syncHistoryWithStore(memHistory, store);
const routes = createRoutes(store);
const sagasDone = monitorSagas(store);
store.dispatch(changeLocale(lang));
match({ routes, location: url }, (error, redirectLocation, renderProps) => {
if (error) {
callback({ error });
} else if (renderProps) {
renderHtmlDocument({ store, renderProps, sagasDone, assets, webpackDllNames })
.then((html) => {
callback({ html });
})
.catch((e) => callback({ error: e }));
} else {
callback({ error: new Error('Unknown error') });
}
});
}
So my assumption is that something is going wrong once it receives too many HTTP requests which in turn generates even more requests to API endpoints to render react components.
I've noticed that it blocks event loop for 300ms after renderAppToString() for every client request, so once there are 100 concurrent requests it blocks it for about 10 seconds. I'm not sure if that's a normal or bad thing though.
Is it worth trying to limit simultaneous requests to Node server?
I couldn't find much information on the topic of SSR + Node crashes. So I'd appreciate any suggestions as to where to look at to identify the problem or for possible solutions if anyone has experienced similar issue in the past.
In the above image, I am doing ReactDOM.hydrate(...) I can also load my initial and required state and send it down in hydrate.
I have written the middleware file and I am using this file to decide based on what URL i should send which file in response.
Above is my middleware file, I have created the HTML string of the whichever file was requested based on URL. Then I add this HTML string and return it using res.render of express.
Above image is where I compare the requested URL path with the dictionary of path-file associations. Once it is found (i.e. URL matches) I use ReactDOMserver render to string to convert it into HTML. This html can be used to send with handle bar file using res.render as discussed above.
This way I have managed to do SSR on my most web apps built using MERN.io stack.
Hope my answer helped you and Please write comment for discussions
1. Run express in a cluster
A single instance of Node.js runs in a single thread. To take
advantage of multi-core systems, the user will sometimes want to
launch a cluster of Node.js processes to handle the load.
As Node is single threaded the problem may also be in a file lower down the stack were you are initialising express.
There are a number of best practices when running a node app that are not generally mentioned in react threads.
A simple solution to improve performance on a server running multiple cores is to use the built in node cluster module
https://nodejs.org/api/cluster.html
This will start multiple instance of your app on each core of your server giving you a significant performance improvement (if you have a multicore server) for concurrent requests
See for more information on express performance
https://expressjs.com/en/advanced/best-practice-performance.html
You may also want to throttle you incoming connections as when the thread starts context switching response times drop rapidly this can be done by adding something like NGINX / HA Proxy in front of your application
2. Wait for the store to be hydrated before calling render to string
You don't want to have to render you layout until your store has finished updating as other comments note this is a blocks the thread while rendering.
Below is the example taken from the saga repo which shows how to run the sagas with out the need to render the template until they have all resolved
store.runSaga(rootSaga).done.then(() => {
console.log('sagas complete')
res.status(200).send(
layout(
renderToString(rootComp),
JSON.stringify(store.getState())
)
)
}).catch((e) => {
console.log(e.message)
res.status(500).send(e.message)
})
https://github.com/redux-saga/redux-saga/blob/master/examples/real-world/server.js
3. Make sure node environment is set correctly
Also ensure you are correctly using NODE_ENV=production when bundling / running your code as both express and react optimise for this
The calls to renderToString() are synchronous, so they are blocking the thread while they are running. So its no surprise that when you have 100+ concurrent requests that you have an extremely blocked up queue hanging for ~10 seconds.
Edit: It was pointed out that React v16 natively supports streaming, but you need to use the renderToNodeStream() method for streaming the HTML to the client. It should return the exact same string as renderToString() but streams it instead, so you don't have to wait for the full HTML to be rendered before you start sending data to the client.
I have a very limited knowledge about node and nob-blocking IO so forgive me if my question is too naive.
In order to return needed information in response body, I need to
Make a call to 3rd party API
Wait for response
Add some modifications and return JSON response with the information I got from API.
My question is.. how can I wait for response? Or is it possible to send the information to the client only when I received response from API (as far as I know, connection should be bidirectional in this case which means I won't be able to do so using HTTP).
And yet another question. If one request waits for response from API, does this mean than other users will be forced to wait too (since node is single-threaded) until I increase numbers of threads/processes from 1 to N?
You pass a callback to the function which calls the service. If the service is a database, for example:
db.connect(host, callback);
And somewhere else in the code:
var callback = function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
};
Or you can use anonymous functions, so:
db.connect(host, function(err, dbObject) {
// The connection was made, it's safe to handle the code here
console.log(dbObject.status);
res.json(jsonObject, 200)
});
Between the call and the callback, node handles other clients / connections freely, "non-blocking".
This type of situation is exactly what node was designed to solve. Once you receive the request from your client, you can make a http request, which should take a callback parameter. This will call your callback function when the request is done, but node can do other work (including serving other clients) while you are waiting for the response. Once the request is done, you can have your code return the response to the client that is still waiting.
The amount of memory and CPU used by the node process will increase as additional clients connect to it, but only one process is needed to handle many simultaneous clients.
Node focuses on doing slow I/O asynchronously, so that the application code can start a task, and then have code start executing again after the I/O has completed.
An typical example might make it clear. We make a call to the FB API. When we get a response, we modify it and then send JSON to the user.
var express = require('express');
var fb = require('facebook-js');
app.get('/user', function(req, res){
fb.apiCall('GET', '/me/', {access_token: access_token}, function(error, response, body){ // access FB API
// when FB responds this part of the code will execute
if (error){
throw new Error('Error getting user information');
}
body.platform = 'Facebook' // modify the Facebook response, available as JSON in body
res.json(body); // send the response to client
});
});
So my problem is that I need to push some data from an HTTP GET Request to a view and I've no clue where to start in Node.js.
What I got til now, is:
var1 = 0;
var2 = 0;
some async HTTP GET Request -> {
success: {
var1 = 1;
}
}
some async HTTP GET Request -> {
success: {
var2 = 2;
}
}
res.render('index/asd', {var1: var1, var2: var2}) --> //Output var1/var2 = 0
Can I push data after async request without to wait for it? Something like res.update(blabla)?
Can I push data after async request without to wait for it? Something like res.update(blabla)?
No. You can't. If you are obtaining some of your data asynchronously, then you need to wait until you have actually obtained all the data before calling res.render().
So, imagine you wanted to get data from two external web sites to use in res.render() operation.
const rp = require('request-promise');
app.get((req, res) => {
Promise.all([rp(someURL1), rp(someUrl2)]).then([r1, r2] => {
// got all data now in r1 and r2
res.render('index/asd', {r1, r2});
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
There is no such thing as res.update() or any way to send more data on the same response object after you've done res.render(). Once res.render() is done and has sent its data, the http connection is finished and closed. You can't send more data on it.
It would be possible to call res.render() for a template that does not need your data and then after that web page is loaded in the browser, it could make an ajax call to fetch some other data from the server which could then be inserted into the page by the javascript in the web page.
It is not possible to send data to the previous response after you've sent the original response.
It is theoretically possible to use res.write() to send data on a connection, keep the connection open and then send more data later on that same connection using res.write() again, but that isn't going to render your page in the browser and then update it later - that's now how browsers work.
It is possible to send data to a web page using a webSocket or socket.io connection (if the web page creates such a connection after being loaded into the browser), but I don't see how that applies here.
I think you have two practical options here:
Wait to call res.render() until you have all the data.
Call res.render() on a template that does not need your data. Then have the web page call back to the server with an ajax call to retrieve data and when it gets that data, it can insert it into the page.
If, long after your web page has been rendered and displayed in the browser, you then want to push some updated data to the web page, you can use a webSocket or socket.io connection. Using that the server can then send updates to the web page as they occur on the server and some receiving Javascript in the web page would manually modify the current web page display based on the newly received data. You would typically send JSON data and let the Javascript in the web page turn it into HTML in the page.
The way this would work is as follows:
Upon initial page request, server uses res.render() to render the initial page and that is sent to the browser.
Browser receives the HTML page, displays the HTML and runs the Javsacript.
Javascript in the page, makes a webSocket or socket.io connection back to the server.
Server receives the webSocket or socket.io connection and hangs onto it.
Sometime in the future, server has new data to send to the page and formulates a message (usually JSON) and sends that over the webSocket or socket.io connection.
Javascript in the web page receives incoming message from the server, examines the content of the message and decides what to do with that data - perhaps inserting some new data into the currently displayed page using DOM manipulations in the browser.