Web-Scraping in Node.Js passing in Cookies/Headers - node.js

Question

Not sure if this is really a stackoverflow-ey question, but here goes:
I'm trying to web-scrape a page that requires you to be logged in to view the extra data. I already have an account, and am trying to replicate the POST of the login page to log me in and get a cookie to use for the rest of the pages.
If I chrome debug the POST, the request headers are:
POST /authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3Db9lB42PxZ6AZU-cmP-zOOLjaPHsif9z5yI7mVfxlLtiv00R_4O-FDtsh-GmFMYvZa7-mw6WdJMGWd1owC2SABiQOKJXdGvPC7XahPStVpsdoVypeGl3Rk-oDSNU7V4700LRV2D9URjtmIpCfAwE1WjLeUXJOZZ2GjQzF_UdBz9BtdlHR7hdR4iUMmebLKpZU2Y-vGuocOhT1D3G_gxcJ0aE9jw_PhPuFe3IGnpno86XKEsQlcviK5aYj2vyhXXTs HTTP/1.1
Host: login.thenuel.com
Connection: keep-alive
Content-Length: 177
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://login.thenuel.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3Db9lB42PxZ6AZU-cmP-zOOLjaPHsif9z5yI7mVfxlLtiv00R_4O-FDtsh-GmFMYvZa7-mw6WdJMGWd1owC2SABiQOKJXdGvPC7XahPStVpsdoVypeGl3Rk-oDSNU7V4700LRV2D9URjtmIpCfAwE1WjLeUXJOZZ2GjQzF_UdBz9BtdlHR7hdR4iUMmebLKpZU2Y-vGuocOhT1D3G_gxcJ0aE9jw_PhPuFe3IGnpno86XKEsQlcviK5aYj2vyhXXTs
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d; _ga=GA1.2.592576149.1458220828; _gat=1; __RequestVerificationToken=-t6-ReUs7KPoo2ioYs4h3OQ-2VLJYE5IRq5GcEGBG5YKGe84VXNdJO4taMK4CCV_HXbFJI_ZflWZqALWjrA29pLZcjWakodi19rtT0sJwiQ1; ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba
and I also need to pass in some form data:
__RequestVerificationToken=tTi9aRzgeb0zA1z3QMZ1iWbGuC4ajR9Ke2VctCLnUlaTKFg1m-70WSOEsZf3PLEUgRdr4n1rEPVvwmfvN6RwrGMKvqnQvjP_gWsAAxAHPY1&UserName=email%40email.com&Password=asdas
I know this is a bit complicated, but I am completely stuck, so i'll post my code and try to run through it as cleanly as I can:
'I am using Superagent-cache and Cheerio'
var url = "https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3DiNj9r1juVrmUK5DyLfXnjq6bM6Fci5E1seI-faOadJQsfBKC9PQJJA-wve3TrusBfhrcjNk8C932FDA_vgQIyrlg36K6ucoC3HZkAO-Yn-mRXmaVqZcdKPRvgwYr55UkeETK4ZsjyuOXNixzk0Z3AslC2ZVN2dqiqoPfpoYtz_n-xgtJlvN5WwRt6cEAvzSwhHkFX4UPUF_1OalC8J4aYO-FHfUjTp8Bv4xBe7w0j0exmjcsMIjpmnp4qbN3qz7u";
request
.get(url)
.end(function(err, res) {
if (err) {
console.log(err);
} else {
// This is for getting the response headers from the GET page
console.log("~~~GET~~~~~~~~~~~~~~~~~~~~~~~~")
$ = cheerio.load(res.headers);
console.log(res.headers);
var secondRequestVToken = res.headers['set-cookie'][0]
console.log("~~~Second RequestVerificationToken: " + secondRequestVToken);
var firstARRAffinity = "ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d; _ga=GA1.2.592576149.1458220828; _gat=1;"
var secondARRAffinity = res.headers['set-cookie'][1]
console.log("~~~Second ARRAfinnity: " + secondARRAffinity);
// This is the getting of the post Request data from the login page
console.log("~~~POST~~~~~~~~~~~~~~~~~~~~~~~~");
$ = cheerio.load(res.text)
var postUrl = "https://login.thenuel.com" + $('body > div > div > div.content-pane.login > form').attr("action");
console.log("~~~URL POST: " + postUrl);
var firstRequestVToken = $('body > div > div > div.content-pane.login > form > input[type="hidden"]:nth-child(1)').attr("value");
console.log("~~~First POST RequestVerificationToken: " + firstRequestVToken);
var formData= {
UserName:'email#email.com',
Password:'Password',
__RequestVerificationToken:firstRequestVToken
}
console.log(formData);
console.log("~~~COOKIE STUFF~~~")
secondRequestVToken = secondRequestVToken.slice(0, -7);
console.log(secondRequestVToken);
secondARRAffinity = secondARRAffinity.slice(0, -31)
console.log(secondARRAffinity);
var finalCookieString = firstARRAffinity + " "
+ secondRequestVToken + " " + secondARRAffinity;
console.log("~~~Final Cookie String: " + finalCookieString);
request
.post(postUrl)
.send(formData)
.set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.set("Accept-Encoding", "gzip, deflate")
.set("Accept-Language", "en-US,en;q=0.8")
.set("Cache-Control", "max-age=0")
.set("Connection", "keep-alive")
.set("Content-Length", 177)
.set("Content-Type", "application/x-www-form-urlencoded")
.set("Cookie", finalCookieString)
.set("Host", "login.thenuel.com")
.set("Origin", "https://login.thenuel.com")
.set("Referer", postUrl)
.set("Upgrade-Insecure-Requests", 1)
.set("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36")
.end(function(err, res) {
// Do respone here
if (err) {
console.log("~~~Hello failure world!");
console.log(err);
} else {
console.log("Hello success world!");
console.log(res);
}
})
So I am getting a GET of the login page to grab the 'RequestVerificationToken' and the second ARRAfinnity string - both to add to the cookie string I send to the POST.
I am then getting the POST Url from the Login Form Action, and the 'RequestVerificationToken' that gets put in the form data to send along with my username and password.
I then remove some of the guff from the tokens and stuff to make it the same as the request my browser sends off to that POST. I then make the POST call.
Here is my log with the console.logs:
Server Running...
~~~GET~~~~~~~~~~~~~~~~~~~~~~~~
{ 'cache-control': 'private',
'content-length': '1777',
'content-type': 'text/html; charset=utf-8',
'content-encoding': 'gzip',
vary: 'Accept-Encoding',
server: 'Microsoft-IIS/8.0',
'set-cookie':
[ '__RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSR
ZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; path=/',
'ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Path=/;Domain=login
.thenuel.com' ],
'x-aspnetmvc-version': '5.2',
'x-aspnet-version': '4.0.30319',
date: 'Wed, 30 Mar 2016 13:48:55 GMT',
connection: 'close',
prev: null,
next: null,
root:
{ type: 'root',
name: 'root',
attribs: {},
children: [ [Circular] ],
next: null,
prev: null,
parent: null },
parent: null }
~~~Second RequestVerificationToken: __RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8E
teTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; path=/
~~~Second ARRAfinnity: ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Pa
th=/;Domain=login.thenuel.com
~~~POST~~~~~~~~~~~~~~~~~~~~~~~~
~~~URL POST: https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcod
e%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%
252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3DiNj9r1juVrmUK5DyLfXnjq6bM6Fci5E1seI-faOadJQsf
BKC9PQJJA-wve3TrusBfhrcjNk8C932FDA_vgQIyrlg36K6ucoC3HZkAO-Yn-mRXmaVqZcdKPRvgwYr55UkeETK4ZsjyuOXNixzk0Z
3AslC2ZVN2dqiqoPfpoYtz_n-xgtJlvN5WwRt6cEAvzSwhHkFX4UPUF_1OalC8J4aYO-FHfUjTp8Bv4xBe7w0j0exmjcsMIjpmnp4q
bN3qz7u
~~~First POST RequestVerificationToken: bqsa4HwniG2PeJVTBKWjg2ux0S4zUJ-Y1U4C_YF93za33dPnNortSOTeHZyyWW
dT_WECqgr44IbJ_FjUSfu9_N3ITdjnJuJhuMvBp_dYQ4c1
{ UserName: 'email#email.com',
Password: 'Password',
__RequestVerificationToken: 'bqsa4HwniG2PeJVTBKWjg2ux0S4zUJ-Y1U4C_YF93za33dPnNortSOTeHZyyWWdT_WECqgr
44IbJ_FjUSfu9_N3ITdjnJuJhuMvBp_dYQ4c1' }
~~~COOKIE STUFF~~~
__RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlV
xzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1;
ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;
~~~Final Cookie String: ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d;
_ga=GA1.2.592576149.1458220828; _gat=1; __RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6I
Eg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; ARRAffinity=e310baf6f2079f1
b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;
Hello success world!
{ body: {},
text: '<!doctype html>\r\n<html>\r\n<head>\r\n <link rel="stylesheet" type="text/css" href="//nu
el-ui-playgroundofwonders.azurewebsites.net/Content/nuel-reset.css" async />\r\n <link rel="stylesh
eet" type="text/css" href="//nuel-ui-playgroundofwonders.azurewebsites.net/Content/nuel-base.css" asyn
c />\r\n <title>Sorry :( - The NUEL</title>\r\n</head>\r\n<body>\r\n <div class="stretch" style=
"width: 23rem; padding: 1.3rem 1rem; background-color: #ffffff; box-shadow: #e0e0e0 0px 0px 1px; margi
n: 3rem auto 0px;">\r\n <h1 class="non-content">Something\'s up!</h1>\r\n <p style="line
-height: 1.3rem;">An error occured during your last request, if you were logging in - try clearing you
r cookies and then retrying. You can <a href="https://support.google.com/chrome/answer/95647?hl=en-GB"
target="_blank">find out how to do that here</a>.</p>\r\n <p style="line-height: 1.3rem;">If t
hat doesn\'t work then just drop us an email at issues#thenuel.com and we\'ll get back to you ASAP.</p
>\r\n <a class="button minimal primary small" href="https://thenuel.com/">Go to thenuel.com</a>
\r\n </div>\r\n</body>\r\n</html>',
headers:
{ 'content-length': '765',
'content-type': 'text/html',
'content-encoding': 'gzip',
'last-modified': 'Wed, 28 Oct 2015 23:22:44 GMT',
'accept-ranges': 'bytes',
etag: '"1c315c8cd711d11:0"',
vary: 'Accept-Encoding',
server: 'Microsoft-IIS/8.0',
'set-cookie': [ 'ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Pat
h=/;Domain=login.thenuel.com' ],
date: 'Wed, 30 Mar 2016 13:48:57 GMT' },
statusCode: 200,
status: 200,
ok: true }
I would've hoped that it would have returned a cookie for being logged
The Verification tokens change each time. The correct verification token is being sent to the form and the correct one is being sent to the cookie string.
I know this is super complicated, but I'm completely stumped.
The website is login.thenuel.com.
Any help is appreciated, but I think this might just be a keep on trying sort of thing! And if there's any other info I need to provide just ask - and it might be easier to just try and do it yourself and see if you can do it!
Thanks