Pushing async Data to View in NodeJS - node.js

So my problem is that I need to push some data from an HTTP GET Request to a view and I've no clue where to start in Node.js.
What I got til now, is:
var1 = 0;
var2 = 0;
some async HTTP GET Request -> {
success: {
var1 = 1;
}
}
some async HTTP GET Request -> {
success: {
var2 = 2;
}
}
res.render('index/asd', {var1: var1, var2: var2}) --> //Output var1/var2 = 0
Can I push data after async request without to wait for it? Something like res.update(blabla)?

Can I push data after async request without to wait for it? Something like res.update(blabla)?
No. You can't. If you are obtaining some of your data asynchronously, then you need to wait until you have actually obtained all the data before calling res.render().
So, imagine you wanted to get data from two external web sites to use in res.render() operation.
const rp = require('request-promise');
app.get((req, res) => {
Promise.all([rp(someURL1), rp(someUrl2)]).then([r1, r2] => {
// got all data now in r1 and r2
res.render('index/asd', {r1, r2});
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
There is no such thing as res.update() or any way to send more data on the same response object after you've done res.render(). Once res.render() is done and has sent its data, the http connection is finished and closed. You can't send more data on it.
It would be possible to call res.render() for a template that does not need your data and then after that web page is loaded in the browser, it could make an ajax call to fetch some other data from the server which could then be inserted into the page by the javascript in the web page.
It is not possible to send data to the previous response after you've sent the original response.
It is theoretically possible to use res.write() to send data on a connection, keep the connection open and then send more data later on that same connection using res.write() again, but that isn't going to render your page in the browser and then update it later - that's now how browsers work.
It is possible to send data to a web page using a webSocket or socket.io connection (if the web page creates such a connection after being loaded into the browser), but I don't see how that applies here.
I think you have two practical options here:
Wait to call res.render() until you have all the data.
Call res.render() on a template that does not need your data. Then have the web page call back to the server with an ajax call to retrieve data and when it gets that data, it can insert it into the page.
If, long after your web page has been rendered and displayed in the browser, you then want to push some updated data to the web page, you can use a webSocket or socket.io connection. Using that the server can then send updates to the web page as they occur on the server and some receiving Javascript in the web page would manually modify the current web page display based on the newly received data. You would typically send JSON data and let the Javascript in the web page turn it into HTML in the page.
The way this would work is as follows:
Upon initial page request, server uses res.render() to render the initial page and that is sent to the browser.
Browser receives the HTML page, displays the HTML and runs the Javsacript.
Javascript in the page, makes a webSocket or socket.io connection back to the server.
Server receives the webSocket or socket.io connection and hangs onto it.
Sometime in the future, server has new data to send to the page and formulates a message (usually JSON) and sends that over the webSocket or socket.io connection.
Javascript in the web page receives incoming message from the server, examines the content of the message and decides what to do with that data - perhaps inserting some new data into the currently displayed page using DOM manipulations in the browser.

Related

How to get the data in nodejs console to html

I am using smartapi provided by angelbroking.
I want to make a stock ticker which can display realtime price of stocks like this one
https://www.tickertape.in/screener?utm_source=gads&utm_medium=search&utm_campaign=screener&gclid=Cj0KCQiA8ICOBhDmARIsAEGI6o1xfYgsbvDEB6c2OFTEYRp9e5UDnJxgCyBJJphdKTduZ_EOHCAchpoaAp-WEALw_wcB
I am able to connect to websocket using the sdk provided in documentation but I don't know how to display that data in my html page.
Please suggest if you know how to get the json data from nodejs console to html.
The nodejs code is
let { SmartAPI, WebSocket } = require("smartapi-javascript");
let web_socket = new WebSocket({
client_code: "P529774",
feed_token: "0973308957"
});
web_socket.connect()
.then(() => {
web_socket.runScript("nse_cm|2885", "cn") // SCRIPT: nse_cm|2885, mcx_fo|222900 TASK: mw|sfi|dp
web_socket.runScript("nse_cm|2885", "mw")
/*setTimeout(function() {
web_socket.close()
}, 60000)*/
})
web_socket.on('tick', receiveTick)
function receiveTick(data) {
console.log("receiveTick:::::", data)
}
The response I get is similar to this :
[{"ak":"ok","task":"mw","msg":"mw"}]
[{"lo":"1797.55","ts":"ACC-EQ","tp":null,"ltp":"1800.05","ltq":"27","bs":"16","tk":"22","ltt":"31\/08\/2017 11:32:01",
"lcl":null,"tsq":"76435","cng":"-11.15","bp":"1800.00","bq":"510","mc":"34012.01277(Crs)","isdc":"18.77872
(Crs)","name":"sf","tbq":"76497","oi":null,"yh":"1801.25","e":"nse_cm","sp":"1800.90","op":"1814.00","c": "1811.20",
"to":"145093696.35","ut":"31-Aug-2017 11:32:01","h":"1817.55","v":"80391","nc":"- 00.62","ap":"1804.85","yl":"1800.00","ucl":null,"toi":"16654000" }]
The github repo for smartapi nodejs
https://github.com/angelbroking-github/smartapi-javascript
The API Docs
https://smartapi.angelbroking.com/docs/Introduction
There are many ways, here's two:
Cache the last message + HTTP polling
This is not the most efficient solution, but perhaps the simplest. Each time your recieveTick() callback hits, you could save the response message in a global object / collection (cache it). Better yet, you could pre-process the message and therefore just cache whatever info you actually care about in that global collection and save bandwidth on the connection between your frontend HTML and backend.
Then, add an HTTP endpoint to your backend that serves up the last info relevant to a given ticker. You could use Express.js or some other simple HTTP server library. That way when your frontend calls
http://<backend_host>:<backend_port>/tickers/<ticker>
Your backend will read from the cached data and serve up the needed data.
Create your own websocket and forward the data
This is a better solution, specially if your data providers API has a quick (subsecond) refresh rate. Create your own websocket server that will make a websocket connection with your frontend. Then, when you get a message from the data providers websocket, simply processes it in whatever way you would like (to get it into the format your frontend wants) then forward it to the frontend by using your websocket server. This will also be done within the recieveTick() function.
There are many websocket tools for nodejs. For help with the websocket stuff check this out https://ably.com/blog/web-app-websockets-nodejs
Also just a quick note, in your question you said "...how to get the json data from nodejs console to html". This kind of suggests that you would like to write the data to the console, and then read it from the console to html. This isn't the way you should think about it. The console was one destination, and the html is another, both originating from the websocket callback.

Nodejs how to separate multiple "multipartform-data" POST requests

In Nodejs I have developed a small Client application that sends multiple “multipart/form-data” to my Server application using POST requests.
Each form to be sent is composed by a file (loaded from the Client hard-disk) and a string information. Basically I have the following situation :
Form 1: (File 1, String 1)
Form 2: (File 2, String 2)
Form 3: (File 3, String 3)
Etc..
To make the POST requests I’m using the “form-data” library ( https://www.npmjs.com/package/form-data ).
The problem that I’m facing is that all the POST requests are sent after the end of the execution of my Client application, but I would like to be able to send each POST request separately.
Here is a part of the code that I’m using :
function FormSubmit(item)
{
var FileStream = fs.createReadStream(item.path);
// Create an "Upload" Form and set all form parameters.
let form = new FormData();
form.append('Text1', 'test');
form.append('file', FileStream);
// Form Submit.
form.submit('http://localhost:5000/upload', function(err, res) {
if (err) {
console.log(err);
}
if (res!= undefined)
res.resume();
else
console.log('Res undefined: ', res);
});
}
I’m calling the “FormSubmit” function multiple times, and I was expecting to receive the POST request on the Server application every time after executing the command “form.submit”, but in reality I receive the POST requests all together after the entire application execution finish.
In particular the Server receives the requests on the command “self.emit('connect');” inside the function “afterConnect” in the “net.js” file in the core module.
It seems that it has nothing to do with timings, because even if i put a breakpoint and wait for some minutes after the first execution of the "FormSubmit" function, i don't receive anything on the Server application.
Probably it is not something related to the "form-data" library, because i get the same behaviour using "request", etc..
I guess it is something related to NodeJs itself or about how i wrote the Client application.
I am new to NodeJs so any help/advice would be appreciated.
Thanks.

What is the proper way to emit an event with socket.io?

I want to emit an event to the client when a long fucntion comes to an end.
This will show a hidden div with a link - on the client side.
This is the approach i tested:
//server.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
require('./app/routes.js')(app, io);
//routes.js
app.post('/pst', function(req, res) {
var url = req.body.convo;
res.render('processing.ejs');
myAsyncFunction(url).then(result => {
console.log('Async Function completed');
socket.emit('dlReady', { description: 'Your file is ready!'});
//do some other stuff here
}).catch(err => {
console.log(err);
res.render('error.ejs');
})
});
I get this
ERROR: ReferenceError: socket is not defined
If i change the socket.emit() line to this:
io.on('connection', function(socket) {
socket.emit('dlReady', { description: 'Your file is ready!'});
});
Then i don't receive an error, but nothing happens at the client.
This is the client code:
<script>
document.querySelector('.container2').style.display = "none";
var socket = io();
socket.on('dlReady', function(data) { //When you receive dlReady event from socket.io, show the link part
document.querySelector('.container1').style.display = "none";
document.querySelector('.container2').style.display = "block";
});
</script>
This whole concept is likely a bit flawed. Let me state some facts about this environment that you must fully understand before you can follow what needs to happen:
When the browser does a POST, there's an existing page in the browser that issues the post.
If that POST is issued from a form post (not a post from Javascript in the page), then when you send back the response with res.render(), the browser will close down the previous page and render the new page.
Any socket.io connection from the previous page will be closed. If the new page from the res.render() has Javascript in it, when that Javascript runs, it may or may not create a new socket.io connection to your server. In any case, that won't happen until some time AFTER the res.render() is called as the browser has to receive the new page, parse it, then run the Javascript in it which has to then connect socket.io to your server again.
Remember that servers handle lots of clients. They are a one-to-many environment. So, you could easily have hundreds or thousands of clients that all have a socket.io connection to your server. So, your server can never assume there is ONE socket.io connection and sending to that one connection will go to a particular page. The server must keep track of N socket.io connections.
If the server ever wants to emit to a particular page, it has to create a means of figuring out which exact socket.io connect belongs to the page that it is trying to emit to, get that particular socket and call socket.emit() only on that particular socket. The server can never do this by creating some server-wide variable named socket and using that. A multi-user server can never do that.
The usual way to "track" a given client as it returns time after time to a server is by setting a unique cookie when the client first connects to your server. From then on, every connection from that client to your server (until the cookie expires or is somehow deleted by the browser) whether the client is connection for an http request or is making a socket.io connection (which also starts with an http request) will present the cookie and you can then tell which client it is from that cookie.
So, my understanding of your problem is that you'd like to get a form POST from the client, return back to the client a rendered processing.ejs and then sometime later, you'd like to communicate with that rendered page in the client via socket.io. To do that, the following steps must occur.
Whenever the client makes the POST to your server, you must make sure there is a unique cookie sent back to that client. If the cookie already exists, you can leave it. If it does not exist, you must create a new one. This can be done manually, or you can use express-session to do it for you. I'd suggest using express-session because it will make the following steps easier and I will outline steps assuming you are using express-session.
Your processing.ejs page must have Javascript in it that makes a socket.io connection to your server and registers a message listener for your "dlready" message that your server will emit.
You will need a top-level io.on('connection', ...) on your server that puts the socket into the session object. Because the client can connect from multiple tabs, if you don't want that to cause trouble, you probably have to maintain an array of sockets in the session object.
You will need a socket.on('disconnect', ...) handler on your server that can remove a socket from the session object it's been stored in when it disconnects.
In your app.post() handler, when you are ready to send the dlready message, you will have to find the appropriate socket for that browser in the session object for that page and emit to that socket(s). If there are none because the page you rendered has not yet connected, you will have to wait for it to connect (this is tricky to do efficiently).
If the POST request comes in from Javascript in the page rather than from a form post, then things are slightly simpler because the browser won't close the current page and start a new page and thus the current socket.io connection will stay connected. You could still completely change the page visuals using client-side Javascript if you wanted. I would recommend this option.

Socket.io authenticating as a client

I made a nodeJS script to automate a few actions on a website - which is not mine!
To have a bit more control over what is going on, I would like to listen to the events on the website's socket.io stream.
Works in NODE so far:
Logging into the website and receiving their cookies as a string for further requests
Sending requests with the cookies from the login (do the actual actions)
Open a websocket connection and listen to the public (!) events
Doesn't work in NODE yet:
Read "private" events that are only being sent to a specific user (me)
I inspected a XHR request that is happening in chrome when clicking a specific button on this website. After this request has been sent, the websocket connection on chrome emits events about the status of my action. Of course, these events are only being sent to the user who performed this action.
Doing the exact same request in node (with the cookies from the website login) gives the right response (success), but the socket stream i opened before, only shows some public events - nothing about my actions.
As seen here, it logs in, displays the website's cookies, opens a socket stream. Then it sends a XHR POST request with the displayed cookies in the headers. The response says "success", but the socket.io events popping up once a second are only the public ones (userCount).
http://i.imgur.com/ZUrA2el.jpg
After sending the request, there should be events like "step_calc" popping up, displaying the status of my action.
My script
After receiving the website's login cookies as a string, I am running this:
const io = require('socket.io-client');
const request = require('request');
main()
function main() {
var socket = io(socketURL, {});
socket.on('connect', function () {
setTimeout(function(){
performAction(); // Send XHR to server
console.log(" > Sending XHR request...")
}, 1500)
});
socket.on('step_calc', function (data) { // Personal event about my action
console.log(" >>> Event = step_calc: " + data)
});
socket.on('login_time', function (data) { // Personal event being displayed every few seconds IF LOGGED IN (chrome) console.log(" >>> Event = step_calc: " + data)
});
socket.on('userCount', function (data) { // Public event
console.log(" >>> Event = userCount: " + data)
});
socket.on('disconnect', function () {
console.log(" > [Disconnected]");
});
}
1500ms after being connected to the socket, it would send the XHR request that should make the server emit information to the socket - performAction().
When I check the chrome console:
step_calc follows to a successfull XHR request (account specific)
login_time is being displayed every 2 seconds, but only if i am logged in (account specific)
userCount is being displayed all the time - to everybody
 
I checked the socket.io-client's API guide and found out about socketIDs. But it only says, how to get this id after connecting to the server...
https://github.com/socketio/socket.io-client/blob/master/docs/API.md#socket
... and yes ... when opening the website, the first thing chrome does, is send a GET request to the website, with data like this:
EIO=3&transport=polling&t=1493058868222-0
The response contains some kind of "sid".
{"sid":"g_mqoOS__________bHb","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}
 
Well...
Now that I have gathered all of this information, how can I use it?
How can I make the socket connection be "connected" to the cookies that I got from the login (which I am using to send requests to the website)?
 
I really hope that my question is kind of understandable. Any help is appreciated, I have already put a lot of time into trying to make it work by myself.
Thanks a lot!
Edit:
I tried to add the same cookies from the handshake in chrome. One weird thing is, that the first XHR it does when i open the website (which seems to be the handshake), already contains a cookie named "io", which is then replaced by a new one. If I check the chrome console>application>cookies, I can't see this cookie at all. Where does it come from?
Left side: The request under the XHR tab on chrome
Right side: This is being displayed under the Websocket tab
http://i.imgur.com/VkRouQf.jpg
Are those two different requests or is it the same one in some way?
Does this information help somehow help to solve my problem?
From what I can see, you're not passing the session cookies to the socket.io-client constructor, which would probably mean that the socket connection isn't being authenticated.
Try this:
var socket = io(socketURL, {
extraHeaders: {
Cookie : '...'
}
});
Documented here.

Sending an http response outside of the route function?

So, I have a route function like the following:
var http = require('http').createServer(start);
function start(req, res){
//routing stuff
}
and below that,I have a socket.io event listener:
io.sockets.on('connection', function(socket){
socket.on('event', function(data){
//perform an http response
}
}
When the socket event 'event' is called, I would like to perform an http response like the following:
res.writeHead(200, {'Content-disposition': 'attachment; filename=file.zip'})
res.writeHead(200, { 'Content-Type': 'application/zip' });
var filestream = fs.createReadStream('file.zip');
filestream.on('data', function(chunk) {
res.write(chunk);
});
filestream.on('end', function() {
res.end();
});
This last part, when performed within the routing function works just fine, but unfortunately when it is called from the socket event, it of course does not work, because it has no reference to the 'req' or 'res' objects. How would I go about doing this? Thanks.
Hmmm... interesting problem:
It's not impossible to do something like what you're trying to do, the flow would be something like this:
Receive http request, don't respond, keep res object saved somewhere.
Receive websocket request, do your auth/"link" it to the res object saved earlier.
Respond with file via res.
BUT it's not very pretty for a few reasons:
You need to keep res objects saved, if your server restarts a whole bunch of response objects are lost.
You need to figure out how to link websocket clients to http request clients. You could do something with cookies/localstorage to do this, I think.
Scaling to another server will become a lot harder / will you proxy clients to always be served by the same server somehow? Otherwise the linking will get harder.
I would propose a different solution for you: You want to do some client/server steps using websockets before you let someone download a file?
This question has a solution to do downloads via websocket: receive file via websocket and initiate download dialog
Sounds like it won't work on older browsers / IE, but a nice option.
Also mentions downloading via hidden iframe
Check here whether this solution is cross-browser enough for you: http://caniuse.com/#feat=datauri
Another option would be to generate a unique URL for the download, and only append it to the browser's window (either as a hidden iframe download, or as a simple download button) once you've done your logic via websocket. This option would be more available cross-browser and easier to code.

Resources