I made a nodeJS script to automate a few actions on a website - which is not mine!
To have a bit more control over what is going on, I would like to listen to the events on the website's socket.io stream.
Works in NODE so far:
Logging into the website and receiving their cookies as a string for further requests
Sending requests with the cookies from the login (do the actual actions)
Open a websocket connection and listen to the public (!) events
Doesn't work in NODE yet:
Read "private" events that are only being sent to a specific user (me)
I inspected a XHR request that is happening in chrome when clicking a specific button on this website. After this request has been sent, the websocket connection on chrome emits events about the status of my action. Of course, these events are only being sent to the user who performed this action.
Doing the exact same request in node (with the cookies from the website login) gives the right response (success), but the socket stream i opened before, only shows some public events - nothing about my actions.
As seen here, it logs in, displays the website's cookies, opens a socket stream. Then it sends a XHR POST request with the displayed cookies in the headers. The response says "success", but the socket.io events popping up once a second are only the public ones (userCount).
http://i.imgur.com/ZUrA2el.jpg
After sending the request, there should be events like "step_calc" popping up, displaying the status of my action.
My script
After receiving the website's login cookies as a string, I am running this:
const io = require('socket.io-client');
const request = require('request');
main()
function main() {
var socket = io(socketURL, {});
socket.on('connect', function () {
setTimeout(function(){
performAction(); // Send XHR to server
console.log(" > Sending XHR request...")
}, 1500)
});
socket.on('step_calc', function (data) { // Personal event about my action
console.log(" >>> Event = step_calc: " + data)
});
socket.on('login_time', function (data) { // Personal event being displayed every few seconds IF LOGGED IN (chrome) console.log(" >>> Event = step_calc: " + data)
});
socket.on('userCount', function (data) { // Public event
console.log(" >>> Event = userCount: " + data)
});
socket.on('disconnect', function () {
console.log(" > [Disconnected]");
});
}
1500ms after being connected to the socket, it would send the XHR request that should make the server emit information to the socket - performAction().
When I check the chrome console:
step_calc follows to a successfull XHR request (account specific)
login_time is being displayed every 2 seconds, but only if i am logged in (account specific)
userCount is being displayed all the time - to everybody
I checked the socket.io-client's API guide and found out about socketIDs. But it only says, how to get this id after connecting to the server...
https://github.com/socketio/socket.io-client/blob/master/docs/API.md#socket
... and yes ... when opening the website, the first thing chrome does, is send a GET request to the website, with data like this:
EIO=3&transport=polling&t=1493058868222-0
The response contains some kind of "sid".
{"sid":"g_mqoOS__________bHb","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}
Well...
Now that I have gathered all of this information, how can I use it?
How can I make the socket connection be "connected" to the cookies that I got from the login (which I am using to send requests to the website)?
I really hope that my question is kind of understandable. Any help is appreciated, I have already put a lot of time into trying to make it work by myself.
Thanks a lot!
Edit:
I tried to add the same cookies from the handshake in chrome. One weird thing is, that the first XHR it does when i open the website (which seems to be the handshake), already contains a cookie named "io", which is then replaced by a new one. If I check the chrome console>application>cookies, I can't see this cookie at all. Where does it come from?
Left side: The request under the XHR tab on chrome
Right side: This is being displayed under the Websocket tab
http://i.imgur.com/VkRouQf.jpg
Are those two different requests or is it the same one in some way?
Does this information help somehow help to solve my problem?
From what I can see, you're not passing the session cookies to the socket.io-client constructor, which would probably mean that the socket connection isn't being authenticated.
Try this:
var socket = io(socketURL, {
extraHeaders: {
Cookie : '...'
}
});
Documented here.
Related
I'm new to backend development so have just made my first server using NodeJS and the http module. This is my code so far:
const http = require("http");
let count = 0;
const server = http.createServer((req, res) => {
res.write(count.toString());
//tells the server that all of the headers and body have been sent, so the message is complete
res.end();
count += 1;
});
server.listen(3000);
I understand how almost all of this code works. However, whenever I refresh the page on my local environment (send a new request) I would expect the displayed response to increment by 1, however, it increments by 2. The only reason I can think this would happen is that the request event listener is being fired twice on each page reload, however, I cannot find anything to help me with this issue so any help would greatly be appreciated.
If i understand what this website says https://www.w3schools.com/nodejs/met_http_createserver.asp
The requestListener is called every time a request is sent to the server.
You can check in your Navigator console (in the Network sub-tab) the different requests sent to your server.
Maybe there are multiple requests sent to your server on page reload.
I hope this helped, otherwise, you can try to log in your requestListener the req object to know what triggers it, and where does it come from.
I need the socket connection to establish before I can send data from the route to the user (otherwise it is not possible).
In app.js file I have socket connection logic:
app.use(function(req, res, next)
{
req.sio = sio;
next();
});
sio.on('connection',
function(soc)
{
console.log('socket connected');
soc.on('disconnect', function(reason)
{
console.log('socket disconnected');
});
// and more about socket connection here...
});
In index.js file I have route.post logic:
router.post('/route1', function(req, res, next) // user is moved from index.js to route1.js if he fills the form
{
var fromInput = req.body.form_name;
console.log('DATA passed from INDEX.JS: ' + formInput);
if ((formInput !== '') && (formInput !== null) && (formInput !== undefined))
{
function render()
{
//// first we render the page, so the javascript (with socket.io notes) can be read it and then the browser know that socket connection should be established
return new Promise(function(resolve, reject)
{
resolve(res.render('route1'));
});
}
// I need to pass some data AFTER the socked connection is established - cause I move to a different page path - using a **router.post** and cause of that socket connection is disconnected - so I need to wait till its usable again. For simplicity let suppose socket connection is established after 2 seconds (it is a simple check for req.soc.connected):
var soc = false;
setTimeout(function()
{
soc = true; // after 2 sec soc is true (connection is established)
}, 2000);
// Now I want to create an interval that will monitor IF socket connection is established every 100ms (so checking won't happen to often - it is not "resource hungry"). If socket connection is not ready the function should call it self (recursion) if the socket connection is established it (function) should fire a promise.
var arr = [];
arr.push(exe(100, data));
function exe(delay, d)
{
d = data;
return new Promise(function(resolve)
{
if (d === false)
{
setTimeout(function()
{
console.log('wait another ' + delay + ' [ms] - ' + d);
return resolve(exe(delay, d));
}, delay);
}
else
{
console.log('socket connected!');
return resolve(d);
}
});
}
render().then(function()
{
return Promise.all(arr).then(function(arr)
{
console.log(arr);
});
}).then(function()
{
console.log('ALL DONE!');
});
}
});
Comment are in code. If something isn't clear let me know.
#jfriend00
1 - true,
2 - true,
3 - I call render() immediately - so page is loaded and client make a socket connection, then the rest of the code should execute and send the data.
yes I did use POST with a form. There could be socket connection between the server and index page - not a problem I can create one, but I dunno what for.
"or there could be a socket.io connection created in the response to the POST when the browser renders and processes that." I'm trying that one :) I have data in this router.post I want to sent with help of sockets - but first I need to make a connection.
as I understand it... user did use form, so path is changed (socket connection is broken), then I'm in router.post I render the page FIRST - so the browser can read it's JS and make a socket connection, BUT you want to say that my response is not finished? So the browser say - ok you want me to render a page, but what now - cause we are NOT finish yet?!
So I will never establish a socket connection, cause I did not properly response? And cause of this I will not be able to send the data (later code in router.post) cause socket connection is never established cause I did not response properly? Cause my tests show me otherwise - it is working just fine.
you are right - code should works now.
till socket connection is established.
yea, good catch. I will make some kind of database - redis with express session I guess.
So again step by step.
User did fill the form so he is redirect from index.js to route1.js (so it does not make a difference if there is a socket connection BEFORE filling the form or not cause the connection is lost). We are in process of redirecting him (router.post) so I thought I will render the route1 page immediately, so the JS from it can be read by browser, and socket connection can be established (which take time - and IF its possible). So I wait with the data I want to sent to the user (in router.post for example... the form input or whatever) TILL the connection is established, and the send it to the user, with help of socket.io.
The thing is that socket io connection is lost when you change page (path). So I thought (and it could be wrong cause I'm newb) then I wait till it is established, and then send the data. I hope it does make sense.
This structure can never work. Here's what it looks like you're trying to do:
Express server receives a POST request.
Then, you try to wait for a socket.io connection to appear before you process the POST and send a response.
Finally, when you think you've found a socket.io connection, you then call your render() function to "presumably" send a response.
Without really understanding what you're' trying to accompilsh, there are a number of things wrong with the current code:
A POST request comes from either an Ajax call or a form POST. There is no socket.io connection associated directly with either one of those. There could have been a socket.io connection when the page loaded BEFORE the POST request was sent or there could be a socket.io connection created in the response to the POST when the browser renders and processes that.
Even if there was a socket.io connection created when the browser processes the POST response, you're trying to wait for the socket.io connection BEFORE you send the response so you're waiting for something that won't happen until you're done waiting (essentially a deadlock - A won't finish until B finishes, but B can't start until A finishes).
This structure render().then(waitUntil(100, d)) isn't correct. You MUST pass .then() a function reference. You are passing it a promise (the return value form calling waitUntil(...)). This is the least of your problems though because the overall structure of what you're trying to do is wrong.
The whole implementation of waitUntil() is confused and I can't even tell what it's trying to actually wait for.
This is a server that can field lots of connections from lots of clients. You can't just wait for the "next" socket.io connection and assume that connection is from the client you just got a request for. The only way to associate a socket.io connection with an http request is to use some identifying characteristic in both (usually a cookie) and then in the http request, you get the cookie and look up the cookie to see if you currently have a socket.io connection that matches that cookie. This is something that express-socket.io-session helps with.
Unfortunately, you don't describe what you're really trying to accomplish here so I can't point you to a good solution. All, I can really say here is that this scheme will not work. If you want further help with the actual problem, please edit your question to include the problem description in words (not your coding issues). Show the exact sequence of events you want to happen and explain what you're trying to accomplish and why.
I want to emit an event to the client when a long fucntion comes to an end.
This will show a hidden div with a link - on the client side.
This is the approach i tested:
//server.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
require('./app/routes.js')(app, io);
//routes.js
app.post('/pst', function(req, res) {
var url = req.body.convo;
res.render('processing.ejs');
myAsyncFunction(url).then(result => {
console.log('Async Function completed');
socket.emit('dlReady', { description: 'Your file is ready!'});
//do some other stuff here
}).catch(err => {
console.log(err);
res.render('error.ejs');
})
});
I get this
ERROR: ReferenceError: socket is not defined
If i change the socket.emit() line to this:
io.on('connection', function(socket) {
socket.emit('dlReady', { description: 'Your file is ready!'});
});
Then i don't receive an error, but nothing happens at the client.
This is the client code:
<script>
document.querySelector('.container2').style.display = "none";
var socket = io();
socket.on('dlReady', function(data) { //When you receive dlReady event from socket.io, show the link part
document.querySelector('.container1').style.display = "none";
document.querySelector('.container2').style.display = "block";
});
</script>
This whole concept is likely a bit flawed. Let me state some facts about this environment that you must fully understand before you can follow what needs to happen:
When the browser does a POST, there's an existing page in the browser that issues the post.
If that POST is issued from a form post (not a post from Javascript in the page), then when you send back the response with res.render(), the browser will close down the previous page and render the new page.
Any socket.io connection from the previous page will be closed. If the new page from the res.render() has Javascript in it, when that Javascript runs, it may or may not create a new socket.io connection to your server. In any case, that won't happen until some time AFTER the res.render() is called as the browser has to receive the new page, parse it, then run the Javascript in it which has to then connect socket.io to your server again.
Remember that servers handle lots of clients. They are a one-to-many environment. So, you could easily have hundreds or thousands of clients that all have a socket.io connection to your server. So, your server can never assume there is ONE socket.io connection and sending to that one connection will go to a particular page. The server must keep track of N socket.io connections.
If the server ever wants to emit to a particular page, it has to create a means of figuring out which exact socket.io connect belongs to the page that it is trying to emit to, get that particular socket and call socket.emit() only on that particular socket. The server can never do this by creating some server-wide variable named socket and using that. A multi-user server can never do that.
The usual way to "track" a given client as it returns time after time to a server is by setting a unique cookie when the client first connects to your server. From then on, every connection from that client to your server (until the cookie expires or is somehow deleted by the browser) whether the client is connection for an http request or is making a socket.io connection (which also starts with an http request) will present the cookie and you can then tell which client it is from that cookie.
So, my understanding of your problem is that you'd like to get a form POST from the client, return back to the client a rendered processing.ejs and then sometime later, you'd like to communicate with that rendered page in the client via socket.io. To do that, the following steps must occur.
Whenever the client makes the POST to your server, you must make sure there is a unique cookie sent back to that client. If the cookie already exists, you can leave it. If it does not exist, you must create a new one. This can be done manually, or you can use express-session to do it for you. I'd suggest using express-session because it will make the following steps easier and I will outline steps assuming you are using express-session.
Your processing.ejs page must have Javascript in it that makes a socket.io connection to your server and registers a message listener for your "dlready" message that your server will emit.
You will need a top-level io.on('connection', ...) on your server that puts the socket into the session object. Because the client can connect from multiple tabs, if you don't want that to cause trouble, you probably have to maintain an array of sockets in the session object.
You will need a socket.on('disconnect', ...) handler on your server that can remove a socket from the session object it's been stored in when it disconnects.
In your app.post() handler, when you are ready to send the dlready message, you will have to find the appropriate socket for that browser in the session object for that page and emit to that socket(s). If there are none because the page you rendered has not yet connected, you will have to wait for it to connect (this is tricky to do efficiently).
If the POST request comes in from Javascript in the page rather than from a form post, then things are slightly simpler because the browser won't close the current page and start a new page and thus the current socket.io connection will stay connected. You could still completely change the page visuals using client-side Javascript if you wanted. I would recommend this option.
I use NodeJS with socket.io for my chat application. When client refreshes/closes the window or navigates to different URL I need client to emit "disconnect" event to NodeJS server. All works nice with excpetion of IE8/9. When refresh happens "disconnect" event is not send to server so server is not aware that this particular client is no longer connected.
I managed to use:
window.onbeforeunload = function(e) {
socket.disconnect();
};
and this takes care of clicking back/forward button and then the server is informed about client disconnecting. Refreshing the page or closing the tab however doesn't send disconnect to server. It seems that refreshing the page is too quick so the socket.disconnect() has no chance of executing. It seems so as if I do alert like below, client pauses for alert window and server receives disconnect message.
window.onbeforeunload = function(e) {
socket.disconnect();
alert("hey watchout!");
};
Now, is there any way to make IE8/9 send disconnect event to NodeJS when page is refreshed?
You can send information back and forth and find the time of the last reply, compare it to now and if the difference is greater than a number, disconnect the client, then reconnect when the client starts sending again.
You can use something like:
On client:
socket.on('ping',function(){socket.emit('pong',(new Date()).getTime());});
On server:
socketReferences = new Array();
io.on('connection',function(socket){
socketReferences.push(socket);
socket.on('pong',function(data){
if(new Date()).getTime()>5000){
// disconnect code here
}
});
});
setInterval(function(){
for(socket in socketReferences)
socketReferences[socket].emit('ping','');
},700);
I need to send data to nodejs server via socket.io when the user closes the browser tab .
I tried doing :
var data={};
window.onbeforeunload = function() {
// i have a object to be sent
data.data1='abcd';
data.data2=1234;
socket.emit("senddata",data);
}
This code works when the user navigates around clicking links on the site but doesnot work when the user closes the browser tab
I also tried configuring the socket io on server side as below .. thinking the error may be due to socket connection being closed before emitting data:
var io = require('socket.io').listen(app.listen(port));
io.configure(function () {
io.set('close timeout',12000);
});
It also didnt work most of the time.
I also tried this on client side:
var socket = require('socket.io').listen(80, {
"sync disconnect on unload":false
});
It also did not work
I had tried receiving data like this
var io = require('socket.io').listen(app.listen(port));
io.sockets.on('connection', function (socket) {
socket.on('senddata', function (data) {
// data processing
});
});
please ..help me with this problem..thanks in advance..
When user connects - register on server side the time it happened.
Socket.IO has heart beating and pinging functionality. So just subscribe to disconnect event on server side and once it happens - means client disconnected - and you have on server time when client had connection. So that way you have timespan of client session.
Do not trust client to tell you any time or important data, as client can simply 'lie' - which leads to hacks/cheats/bugs on your server-side.
There is no reliable way to send data just before disconnect from client-side at all. There is nothing in Socket.IO for that, nor in just one of transports (WebSockets). As well as there is too many different scenarios of disconnection (power off, force close, ethernet cable out, wifi lose, etc).