How to inform client of current state of execution - node.js

I wish to have a progress bar on the client-side built using AngularJS. This progress bar will inform the end-user of the current state of execution of a query on the server.
The server, in this case, is ExpressJS.
So, using AngularJS I will make a request to the server such as:
$http.post('/data/aPath')
.success(function (result){
//Update the progress here.
});
What I wish to know is how can I send responses without ending them so that AngularJS can receive them as shown above and I can update the progress bar? That is, on the Node.js side,
app.post('/data/aPath', function (request, response){
//What should I do here to update the client on the
//current execution state
//Something on the lines of
//response.write("finished fetching user details, moving on to
//updating records"
});
so that the client can then update the progress bar?

I haven't done this myself, to be honest, but I believe one way to approach this would be like this:
On your client-side, modify your $http.post function
function getStatusFromProcessing() {
$http.post('/data/aPath').success(function (result){
// Pseudocode:
if(result.status is something) { // ex: (result >= 20)
// update progressbar with new value
}
if(result.status is somethingElse) { // ex: (result >= 40)
// update progressbar with new value
}
// All updating of progressbar done for now
if(result.status !== 100) { // Progress is not done
return getStatusFromProcessing(); // run the function again
} else {
return; // All done, move on
}
});
}
Now, the server side:
app.post('/data/aPath', function (request, response){
// You need some kind of function/service, which returns
// the current state of the processing.
var currentStatus = getStatusFromRunningProcess();
// getStatusFromRunningProcess() should return a value between (0 - 100)
return response.json({ status: currentStatus});
});
A few notes:
Code is not tested (obviously)
Should you choose to do something like this, I think the processing time should be somewhat substantial in time
Maybe add a small timeout in your client code inside if(result.status !== 100) of a couple of (hundred) milliseconds to avoid spamming the http request. But that is fine-tuning :)
Update, alternative solution using sockets
If you don't want to make several requests to the server from the client, you can switch it around; the server sends a message to the client when the process is updated. This requires more likely less bandwidth and requests done to the server. This method is possible using sockets.
When using sockets, the socket.io framework is very popular. There are also lots of tutorials online.
In short:
The server sends a message when the status of the processing is updated
The client receives this information and updates the progress bar
Here is a SO post regarding sockets and a progress bar. This is about file uploads though, but the concept is the same.

Related

Progress bar for express / react communicating with backend

I want to make a progress bar kind of telling where the user where in process of fetching the API my backend is. But it seems like every time I send a response it stops the request, how can I avoid this and what should I google to learn more since I didn't find anything online.
React:
const {data, error, isError, isLoading } = useQuery('posts', fetchPosts)
if(isLoading){<p>Loadinng..</p>}
return({data&&<p>{data}</p>})
Express:
app.get("api/v1/testData", async (req, res) => {
try {
const info = req.query.info
const sortByThis = req.query.sortBy;
if (info) {
let yourMessage = "Getting Data";
res.status(200).send(yourMessage);
const valueArray = await fetchData(info);
yourMessage = "Data retrived, now sorting";
res.status(200).send(yourMessage);
const sortedArray = valueArray.filter((item) => item.value === sortByThis);
yourMessage = "Sorting Done now creating geojson";
res.status(200).send(yourMessage);
createGeoJson(sortedArray)
res.status(200).send(geojson);
}
else { res.status(400) }
} catch (err) { console.log(err) res.status(500).send }
}
You can only send one response to a request in HTTP.
In case you want to have status updates using HTTP, the client needs to poll the server i.e. request status updates from the server. Keep in mind though that every request needs to be processed on the server side and will take resources away which are then not available for other (more important) requests from other clients. So don't poll too frequently.
If you want to support long running operations using HTTP have a look at the following API design pattern.
Alternatively you could also use a WebSockets connection to push updates from the server to the client. I assume your computation on the backend will not be minutes long and you want to update the client in real-time, so probably WebSockets will be the best option for you. A WebSocket connection has, once established, considerably less overhead than sending huge HTTP requests/ responses between client and server.
Have a look at this thread which dicusses abovementioned and other possibilites.

Node.js server side connection to Socket.io

I have a Node.js application with a frontend app and a backend app, the backend will manage the list and "push" an update to the frontend app, the call to the frontend app will trigger a list update so that all clients receive the correct list data.
The problem is on the backend side, when I press the button, I perform an AJAX call, and that AJAX call will perform the following code (trimmed some operations out of it):
Lists.findOne({_id: active_settings.active_id}, function(error, lists_result) {
var song_list = new Array();
for (i=0; i < lists_result.songs.length; i++) {
song_list.push(lists_result.songs[i].ref);
}
Song.find({
'_id': {$in: song_list}
}, function(error, songs){
// DO STUFF WITH THE SONGS
// UPDATE SETTINGS (code trimmed)
active_settings.save(function(error, updated_settings) {
list = {
settings: updated_settings,
};
var io = require('socket.io-client');
var socket = io.connect(config.app_url);
socket.on('connect', function () {
socket.emit('update_list', {key: config.socket_key});
});
response.json({
status: true,
list: list
});
response.end();
}
});
});
However the response.end never seems to work, the call keeps hanging, further more, the list doesn't always get refreshed so there is an issue with the socket.emit code. And the socket connection stays open I assume because the response isn't ended?
I have never done this server side before so any help would be much appreciated. (the active_settings etc exists)
I see some issues that might or might not be causing your problems:
list isn't properly scoped, since you don't prefix it with var; essentially, you're creating a global variable which might get overwritten when there are multiple requests being handled;
response.json() calls .end() itself; it doesn't hurt to call response.end() again yourself, but not necessary;
since you're not closing the socket(.io) connection anywhere, it will probably always stay open;
it sounds more appropriate to not set up a new socket.io connection for each request, but just once at your app startup and just re-use that;

Node.js listen to session variable change and trigger server-sent event

I am writing a webapp, using express.js.
My webapp achieves the following
User posts 100 json objects
Each json object is processed via a service call
Once the service call is completed, a session variable is incremented
On incrementation of the session variable, a server side event must be sent to the client to update the progress bar
How do i achieve listening on a session variable change to trigger a server-sent event?
Listening to a variable change is not the only solution I seek?
I need to achieve sending a server-sent event once a JSON object is processed.
Any appropriate suggestion is welcome
Edit (based on Alberto Zaccagni's comment)
My code looks like this:
function processRecords(cmRecords,requestObject,responseObject)
{
for (var index = 0; index < cmRecords.length; index++)
{
post_options.body = cmRecords[index];
request.post(post_options,function(err,res,body)
{
if(requestObject.session.processedcount)
requestObject.session.processedcount = requestObject.session.processedcount + 1;
else
requestObject.session.processedcount = 1;
if(err)
{
appLog.error('Error Occured %j',err);
}
else
{
appLog.debug('CMResponse: %j',body);
}
var percentage = (requestObject.session.processedcount / requestObject.session.totalCount) * 100;
responseObject.set('Content-Type','text/event-stream');
responseObject.json({'event':'progress','data':percentage});
});
};
}
When the first record is updated and a server side event is triggered using the responseObject (express response object)
When the second record is updated and I try triggering a server side event using the same responseObject. I get an error saying cannot set header to a response that has already been sent
It's hard to know exactly what the situation is without seeing the routes/actions you have in your main application...
However, I believe the issue you are running into is that you are trying to send two sets of headers to the client (browser), which is not allowed. The reason this is not allowed is because the browser does not allow you to change the content type of a response after you have sent the initial response...as it uses that as an indicator of how to process the response you are sending it. You can't change either of these (or any other headers) after you have sent them to a client once (one request -> one response -> one set of headers back to the client). This prevents your server from appearing schizophrenic (by switching from a "200 Ok" response to a "400 Bad Request," for example).
In this case, on the initial request, you are telling the client "Hey, this was a valid request and here is my response (via the status of 200 which is either set elsewhere or being assumed by ExpressJS), and please keep the communication channel open so I can send you updates (by setting your content type to text/event-stream)".
As far as how to "fix" this, there are many options. When I've done this, I've used the pub/sub feature of redis to act as the "pipe" that connects everything up. So, the flow has been like this:
Some client sends a request to /your-event-stream-url
In this request, you set up your Redis subscriber. Anything that comes in on this subscription can be handled however you want. In your case, you want to "send some data down the pipe to the client in a JSON object with at least a data attribute." After you have set up this client, you just return a response of "200 Ok" and set the content type to "text/event-stream." Redis will take care of the rest.
Then, another request is made to another URL endpoint which accomplishes the task of "posting a JSON object" by hitting /your-endpoint-that-processes-json. (Note: obviously this request may be made by the same user/browser...but the application doesn't know/care about that)
In this action, you do the processing of their JSON data, increment your counters, or do whatever...and return a 200 response. However, one of the things you'd do in this action is "publish" a message on the Redis channel your subscribers from step #1 are listening to so the clients get the updates. Technically, this action does not need to return anything to the client, assuming the user will have some type of feedback based on the 200-status code or on the server-sent event that is sent down the pipe...
A tangible example I can give you is this gist, which is part of this article. Note that the article is a couple years old at this point so some of the code may have to be tweaked a bit. Also note this is not guaranteed to be anything more than an example (ie: it has not been "load tested" or anything like that). However, it may help you get started.
I came up with a solution please let me know if this is the right way to do stuff ?
Will this solution work across sessions ?
Server side Code
var events = require('events');
var progressEmitter = new events.EventEmitter();
exports.cleanseMatch = function(req, res)
{
console.log('cleanseMatch Inovked');
var progressTrigger = new events.EventEmitter;
var id = '';
var i = 1;
id = setInterval(function(){
req.session.percentage = (i/10)*100;
i++;
console.log('PCT is: ' + req.session.percentage);
progressEmitter.emit('progress',req.session.percentage)
if(i == 11) {
req.session.percentage = 100;
clearInterval(id);
res.json({'data':'test'});
}
},1000);
}
exports.progress = function(req,res)
{
console.log('progress Inovked');
// console.log('PCT is: ' + req.session.percentage);
res.writeHead(200, {'Content-Type': 'text/event-stream'});
progressEmitter.on('progress',function(percentage){
console.log('progress event fired for : ' + percentage);
res.write("event: progress\n");
res.write("data: "+percentage+"\n\n");
});
}
Client Side Code
var source = new EventSource('progress');
source.addEventListener('progress', function(e) {
var percentage = JSON.parse(e.data);
//update progress bar in client
App.updateProgressBar(percentage);
}, false);

How do you do "remember me" functionality in AngularJS?

I have a simple app I'm building using Play + AngularJS that requires authentication before most routes can be accessed. The login flow includes a "remember me" feature that stores a session ID in to the browser local storage and gets mapped to a valid authorized database session entry on the server side any time a user returns to the app.
The problem I'm having is that I do the session checking (extract cookie & compare against server) in the run() function of the module:
.run(function ($rootScope, $http, $cookieStore, $location) {
// <snip>
// check if there is already a session?
var sessionId = window.localStorage["session.id"];
if (sessionId == null) {
sessionId = $cookieStore.get("session.id");
}
if (sessionId != null) {
$http.get("/sessions/" + sessionId)
.success(function (data) {
$http.defaults.headers.common['X-Session-ID'] = data.id;
$cookieStore.put("session.id", data.id);
$rootScope.user = data.user;
})
.error(function () {
// remove the cookie, since it's dead
$cookieStore.remove("session.id");
window.localStorage.removeItem("session.id");
$location.path("/login");
});
} else {
if ($location.path() != "/login" && $location.path() != "/signup") {
$location.path("/login");
}
}
});
The problem is that this function executes an AJAX call and I don't know if the session is valid until it completes. However, the controller that loads (via the route selected by $routeProvider) can fire away another AJAX call that often kicks off before the other one finishes, resulting in a race condition and the initial request getting a 401 response code.
So my question is: how can I force run (with its associated $http call) to complete before any other part of the app runs? I have tried using $q/promise here and it doesn't seem to make a difference (perhaps run functions don't honor promises). I've been advisor to use resolve feature in $routeProvider but I don't know exactly what to do and I'm not super execited about having to put that in for every route anyway.
I assume this is a pretty common use case and it gets solved every day. Hopefully someone can give me some direction with my code, or share their approaches for "remember me" and AngularJS.
You need to manual bootstrap your app after you get session from server.It's easy if you use jQuery for example you can do, or even without jQuery you can use injector to access $http before bootstrapping
$.get(server,function(){
//success , set variable.
}).fail(function (){
//failed :( redirect to login or set session to false etc... null
})
.always(function(){
//alwyas bootstrap in both case and set result as a constant or variable Angular.module('app').variable('session',sessionResult);
});
I'm on phone right now, but this should give u the idea

fetch data from multiple table by sending only one request

I am using node.js for server side development and backbone.js for client side development. i want to fetch data from multiple table(more than 3) by sending only one request to node.js. but i cant merge all that result with each other beacuse of asynchronous execution of node.js. i have done this but it sending a lots of get request to node js for getting data from all the tables and because of these performance of my site is become slower. please help me if anyone having any idea.
I would create a method which aggregates the results from each of the requests and sends the response back. Basically each of your three async db calls would pass their data to the same method. That method would check to see if it had all of the data it needed to complete the request, and if it did, send the response.
Here is a pseudo code example:
function handleRequest(req, res) {
var results = {};
db.getUsers(function(data) {
aggregate('users', data);
});
db.getPosts(function(data) {
aggregate('posts', data);
});
db.getComments(function(data) {
aggregate('comments', data);
});
function aggregate(name, data) {
results[name] = data;
if(results.users && results.posts && results.comments) {
res.send(results);
}
}
}
This is simplified greatly, you should also of course check for errors and timeouts to the db calls, but this will allow you to wait for all the async commands to complete before sending the data.

Resources