Implementing http partial requests with actix - rust

I've made a server which provides larger videos via HTTP-Get requests. The following code snipped is used on the server side:
use actix_web::{web, App, HttpResponse, HttpServer, Responder, HttpRequest};
use std::str::FromStr;
//Some random data
lazy_static::lazy_static! {
static ref DATA : Vec<u8> = std::fs::read("data/movie1/bigfile.mp4").unwrap();
}
fn get_all_data() -> &'static[u8] {
&DATA
}
fn get_part_data(start: u64, end: u64) -> &'static[u8] {
&DATA[start as usize.. end as usize]
}
fn data_len() -> u64 {
DATA.len() as u64
}
//Extracts the begin and end from the range header string
fn parse_range(s: &str) -> (u64, Option<u64>) {
assert_eq!(&s[0..6], "bytes=");
let split = (&s[6..]).split("-").map(|s| s.to_string()).collect::<Vec<String>>();
assert_eq!(split.len(), 2);
(u64::from_str(&split[0]).unwrap(), match u64::from_str(&split[1]) {
Err(_) => None,
Ok(a) => Some(a)
})
}
async fn body_request(request: HttpRequest) -> impl Responder {
println!("Request received: {:#?}", request);
let range_requested = request.headers().contains_key("Range");
if range_requested {
println!("Range requested...");
let range_str = request.headers().get("Range").unwrap().to_str().unwrap();
let range = parse_range(range_str);
let start = range.0;
let end = match range.1 {
None => data_len(),
Some(s) => s
};
println!("Original Range: {:#?}, Interpreted range: {} - {}", range_str, start, end);
let response = HttpResponse::PartialContent()
.header("Content-Range", format!("{} {}-{}/{}", "bytes", start, end, data_len()))
.header("Content-Type", "video/mp4")
.body(get_part_data(start, end));
println!("Response: {:#?}", response);
response
} else {
println!("Whole document requested");
HttpResponse::Ok()
.header("Content-Type", "video/mp4")
.body(get_all_data())
}
}
#[actix_rt::test]
async fn part_test() -> std::io::Result<()> {
HttpServer::new(|| {
App::new()
.route("/", web::get().to(body_request))
})
.bind(("0.0.0.0", 8080))?
.run()
.await
}
This implementation should be able to resume cancelled downloads. However, the part request is not working with the wget utility, I have already read the Mozilla entry on how to respond to partial requests, but I cannot figure out why wget is retrying/not working.
I did the following:
Start the server with cargo test part_test -- --nocapture
Start wget with wget -c -v localhost:8080 and interrupt it with ctr-c
Restart wget with the same command (the -c option should resume the operation)
I expected the wget to just resume it normally, however it is not satisfied with the server's partial content for some reason. I would like to know why and how I can fix that.
wget output:
user#student-net-etx-0499 rusttest % wget -c -v localhost:8080
--2021-12-22 15:25:43-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 300017432 (286M) [video/mp4]
Saving to: ‘index.html’
index.html 44%[============================> ] 128,02M 157MB/s
^C
user#student-net-etx-0499 rusttest % wget -c -v localhost:8080
--2021-12-22 15:25:47-- http://localhost:8080/
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 206 Partial Content
Retrying.
--2021-12-22 15:25:48-- (try: 2) http://localhost:8080/
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 206 Partial Content
Retrying.
^C
Server output:
Request received:
HttpRequest HTTP/1.1 GET:/
headers:
"accept": "*/*"
"accept-encoding": "identity"
"host": "localhost:8080"
"user-agent": "Wget/1.21.2"
"connection": "Keep-Alive"
Whole document requested
Request received:
HttpRequest HTTP/1.1 GET:/
headers:
"accept": "*/*"
"range": "bytes=158907840-"
"accept-encoding": "identity"
"host": "localhost:8080"
"user-agent": "Wget/1.21.2"
"connection": "Keep-Alive"
Range requested...
Original Range: "bytes=158907840-", Interpreted range: 158907840 - 300017432
Response:
Response HTTP/1.1 206 Partial Content
headers:
"content-type": "video/mp4"
"content-range": "bytes 158907840-300017432/300017432"
body: Sized(141109592)
Request received:
HttpRequest HTTP/1.1 GET:/
headers:
"accept": "*/*"
"range": "bytes=158907840-"
"accept-encoding": "identity"
"host": "localhost:8080"
"user-agent": "Wget/1.21.2"
"connection": "Keep-Alive"
Range requested...
Original Range: "bytes=158907840-", Interpreted range: 158907840 - 300017432
Response:
Response HTTP/1.1 206 Partial Content
headers:
"content-type": "video/mp4"
"content-range": "bytes 158907840-300017432/300017432"
body: Sized(141109592)
#cargo.toml
[package]
name = "rusttest"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
actix-web="*"
actix-rt="1.1.1"
futures="*"
serde_json = "*"
serde = { version = "*", features = ["derive"] }
qr2term = "0.2.2"
async-std= { version= "*", features = ["unstable"]}
async_flag="*"
lazy_static="*"

Related

Error establishing connection to Binance via Tungstenite Websocket API

I am trying to establish a connection to the binance websocket server with the code:
use tungstenite::{connect, Message};
use url::Url;
fn main() {
let (mut socket, response) =
connect(Url::parse("wss://stream.binance.com:9443/ws/BNBBTC#aggTrade").unwrap()).expect("Can't connect");
println!("Connected to the server");
println!("Response HTTP code: {}", response.status());
println!("Response contains the following headers:");
for (ref header, _value) in response.headers() {
println!("* {}", header);
}
}
with Cargo.toml containing:
[dependencies]
tungstenite = "0.17.3"
url = "2.3.1"
I am getting the error:
thread 'main' panicked at 'Can't connect: Http(Response { status: 400, version: HTTP/1.1, headers: {"server": "awselb/2.0", "date": "Sat, 22 Oct 2022 20:23:40 GMT", "content-type": "text/html", "content-length": "220", "connection": "close"}, body: None })', src/main.rs:6:90
stack backtrace:
0: rust_begin_unwind
at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/panicking.rs:142:14
2: core::result::unwrap_failed
at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/result.rs:1814:5
3: core::result::Result<T,E>::expect
at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/result.rs:1064:23
4: untitled12::main
at ./src/main.rs:6:9
5: core::ops::function::FnOnce::call_once
at /rustc/a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
What is the reason behind this error? What am I missing? It seems like a bad request error, however my request is supposed to be well-formed according to the binance websocket API
By default, tungstenite does not have TLS enabled, so I had to enable in Cargo.toml:
tungstenite = { version = "0.17.3", features = ["native-tls"] }

CORS problem while making a fetch call from UI app to Servant server

I've found a lot of CORS-related questions, but none of them solved my problem. As I'm quite new to web-dev, I'd appreciate any help with getting my project infrastructure up and running.
My application is made of two parts:
Server part
A server made with Haskell, Servant. The most important endpoint I've been trying to connect to is:
type Unprotected =
"login"
:> ReqBody '[JSON] Credentials
:> Verb 'POST 204 '[JSON] (Headers '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie] NoContent)
It's based on the project described here: https://github.com/haskell-servant/servant-auth
I've already found a SO question about CORS: CORS header ‘Access-Control-Allow-Origin’ missing in servant, but that didn't help.
During development I launch the server on port 8081.
UI Part
I have also a UI project, that's hosted independently on port 8833. It uses WebPack. I've also found some SO posts regarding CORS, advising to use:
devServer: {
headers: {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Credentials": "true",
"Access-Control-Allow-Methods": "GET, POST, PUT, DELETE, PATCH, OPTIONS",
"Access-Control-Allow-Headers": "X-Requested-With, content-type, Authorization"
}
That didn't help either.
Problem description:
When I make a "fetch" call in the UI project, trying to communicate with the Servant endpoint I get:
Access to fetch at 'http://localhost:8081/login' from origin 'http://localhost:8833'
has been blocked by CORS policy: Response to preflight request doesn't pass access control
check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an
opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource
with CORS disabled.
The way I use fetch:
let data = {
credentialsUserName: this.state.login,
credentialsPassword: this.state.password
};
let opts : RequestInit = {
method: "POST",
headers: new Headers({ "Content-Type": "application/json" }),
body: JSON.stringify(data)
};
fetch("http://localhost:8081/login", opts).then((value) => {
console.log(value);
// do something later
});
Questions
What shall I do in order to make the fetch work fine with the servant endpoint. Also, is there anything wrong with my application architecture ? Maybe there's a better way to design it.
Last thing: The entire code of my application is quite long, so I decided to put only to most important parts here. Feel free to request more if needed.
Edit 1
After sending the fetch request I can see two entries in "Network" tab in my browser.
The first one with the following headers:
content-type: application/json
Referer: http://localhost:8833/Login
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36
and json data I included in its body. It's status is "(failed) net::ERR_FAILED".
The other's status is 400 and contains more headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Access-Control-Request-Headers: content-type
Access-Control-Request-Method: POST
Connection: keep-alive
Host: localhost:8081
Origin: http://localhost:8833
Referer: http://localhost:8833/Login
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36
Interestingly, In "General" tab I can see:
Request URL: http://localhost:8081/login
Request Method: OPTIONS
Status Code: 400 Bad Request
Remote Address: 127.0.0.1:8081
Referrer Policy: no-referrer-when-downgrade
Request Method "OPTIONS" looks mysterious. Why is it there ?
As a quick guess I tried adding "OPTIONS" to CORS definition on Haskell side, but no luck this time either:
port :: Int
port = 8081
corsPolicy :: Middleware
corsPolicy = cors (const $ Just policy)
where
policy = simpleCorsResourcePolicy
{ corsMethods = [ "GET", "POST", "PUT", "OPTIONS" ]}
main :: IO ()
main = do
migrateDB
jwtConfig <- getJwtConfig
putStrLn $ "Serving endpoint " ++ (show port)
run port $ corsPolicy $ serveWithContext proxy (context cookieConfig jwtConfig) (appAPI cookieConfig jwtConfig)
EDIT 2
Thanks for the help - I've managed to configure Servant, here it is:
corsPolicy :: Middleware
corsPolicy = cors (const $ Just policy)
where
policy = simpleCorsResourcePolicy
{
corsMethods = [ "GET", "POST", "PUT", "OPTIONS" ],
corsOrigins = Just (["http://localhost:8833"], True),
corsRequestHeaders = [ "authorization", "content-type" ]
}
The remaining part of Main as in my previous listing.
By default Haskell CORS doesn't accept any headers. Not even "Content-Type", which I intend to send.

How to change axios content from "text/html" to application/json type in Nestjs

I discovered that axios is returning a string instead of a valid json.
headers:Object {date: "Tue, 02 Jun 2020 08:44:06 GMT", server: "Apache", connection: "close", …}
connection:"close"
content-type:"text/html; charset=UTF-8"
date:"Tue, 02 Jun 2020 08:44:06 GMT"
server:"Apache"
transfer-encoding:"chunked"
how do I change the content-type to application/json in NestJs application?
I tried this but didnt not work
const meterInfo = await this.httpService.get(url, { headers: { "Content-Type": "application/json" } }).toPromise();
Here is the invalid json returned.
"{"status":"00","message":"OK","access_token":"2347682423567","customer":{"name":"John Doe","address":"Mr. John Doe 34 Tokai, leaflet. 7999.","util":"Demo Utility","minimumAmount":"13897"},"response_hash":"c43c9d74480f340f55156f6r5c56487v8w"}"
Instead of sending a Content-Type header, you should send an Accept header with the same MIME type. This tells the server what you are expecting to receive, and if Content Negotiation is set up properly, it will allow you to get a JSON back instead of that weird string.
this.httpService.get(
url,
{
headers: {
'Accept': 'application/json',
},
},
).toPromise();
If that doesn't work, you'll need to provide your own serializer to take the string from that wonky format to JSON, or get in touch with the server admins and see if they can provide you better documentation about how to consume their API.

How to return an error to caller from an actix-web handler with client?

I created a server with actix_web that will connect through GET to another service using actix client and return body on success or error on error. I have been able to return the body but have no clue about how to return the error.
This is my handler:
fn echo_client(client: web::Data<Client>) -> impl Future<Item = HttpResponse, Error = Error> {
client
.get("127.0.0.1:9596/echo/javier") // <- Create request builder
.header("User-Agent", "Actix-web")
//.finish().unwrap()
.send() // <- Send http request
.map_err(|_| ())
//.map_err(Error::from)
.and_then(|response| {
response
.body()
.and_then(|body| {
println!("{:?}", body);
Ok(HttpResponse::Ok().body(body))
})
.map_err(|error| Err(error.error_response()))
})
}
There are three things that may fail:
Failing connection.
Non 200-status code.
Abrupt stop in body stream.
To handle 1, do not map_err to ():
.map_err(|err| match err {
SendRequestError::Connect(error) => {
ErrorBadGateway(format!("Unable to connect to httpbin: {}", error))
}
error => ErrorInternalServerError(error),
})
SendRequestError lists the errors that can occur when doing client requests.
To handle 2, make sure you use the status code from the client response:
.and_then(|response| Ok(HttpResponse::build(response.status()).streaming(response))))
actix-web handles 3 I believe.
Complete example, handling headers too:
use actix_web::client::{Client, SendRequestError};
use actix_web::error::{ErrorBadGateway, ErrorInternalServerError};
use actix_web::{web, App, Error, HttpResponse, HttpServer};
use futures::future::Future;
fn main() {
HttpServer::new(|| App::new().data(Client::new()).route("/", web::to(handler)))
.bind("127.0.0.1:8000")
.expect("Cannot bind to port 8000")
.run()
.expect("Unable to run server");
}
fn handler(client: web::Data<Client>) -> Box<Future<Item = HttpResponse, Error = Error>> {
Box::new(
client
.get("https://httpbin.org/get")
.no_decompress()
.send()
.map_err(|err| match err {
SendRequestError::Connect(error) => {
ErrorBadGateway(format!("Unable to connect to httpbin: {}", error))
}
error => ErrorInternalServerError(error),
})
.and_then(|response| {
let mut result = HttpResponse::build(response.status());
let headers = response
.headers()
.iter()
.filter(|(h, _)| *h != "connection" && *h != "content-length");
for (header_name, header_value) in headers {
result.header(header_name.clone(), header_value.clone());
}
Ok(result.streaming(response))
}),
)
}
Which actually failed:
$ curl -v localhost:8000
* Rebuilt URL to: localhost:8000/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 502 Bad Gateway
< content-length: 50
< content-type: text/plain
< date: Sun, 07 Jul 2019 21:01:39 GMT
<
* Connection #0 to host localhost left intact
Unable to connect to httpbin: SSL is not supported
Add ssl as a feature in Cargo.toml to fix the connection error:
actix-web = { version = "1.0", features=["ssl"] }
Then try the request again:
$ curl -v localhost:8000
* Rebuilt URL to: localhost:8000/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< transfer-encoding: chunked
< x-frame-options: DENY
< date: Sun, 07 Jul 2019 21:07:18 GMT
< content-type: application/json
< access-control-allow-origin: *
< access-control-allow-credentials: true
< server: nginx
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: no-referrer-when-downgrade
<
{
"args": {},
"headers": {
"Date": "Sun, 07 Jul 2019 21:07:18 GMT",
"Host": "httpbin.org"
},
"origin": "212.251.175.90, 212.251.175.90",
"url": "https://httpbin.org/get"
}

Web-Scraping in Node.Js passing in Cookies/Headers

Not sure if this is really a stackoverflow-ey question, but here goes:
I'm trying to web-scrape a page that requires you to be logged in to view the extra data. I already have an account, and am trying to replicate the POST of the login page to log me in and get a cookie to use for the rest of the pages.
If I chrome debug the POST, the request headers are:
POST /authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3Db9lB42PxZ6AZU-cmP-zOOLjaPHsif9z5yI7mVfxlLtiv00R_4O-FDtsh-GmFMYvZa7-mw6WdJMGWd1owC2SABiQOKJXdGvPC7XahPStVpsdoVypeGl3Rk-oDSNU7V4700LRV2D9URjtmIpCfAwE1WjLeUXJOZZ2GjQzF_UdBz9BtdlHR7hdR4iUMmebLKpZU2Y-vGuocOhT1D3G_gxcJ0aE9jw_PhPuFe3IGnpno86XKEsQlcviK5aYj2vyhXXTs HTTP/1.1
Host: login.thenuel.com
Connection: keep-alive
Content-Length: 177
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://login.thenuel.com
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3Db9lB42PxZ6AZU-cmP-zOOLjaPHsif9z5yI7mVfxlLtiv00R_4O-FDtsh-GmFMYvZa7-mw6WdJMGWd1owC2SABiQOKJXdGvPC7XahPStVpsdoVypeGl3Rk-oDSNU7V4700LRV2D9URjtmIpCfAwE1WjLeUXJOZZ2GjQzF_UdBz9BtdlHR7hdR4iUMmebLKpZU2Y-vGuocOhT1D3G_gxcJ0aE9jw_PhPuFe3IGnpno86XKEsQlcviK5aYj2vyhXXTs
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d; _ga=GA1.2.592576149.1458220828; _gat=1; __RequestVerificationToken=-t6-ReUs7KPoo2ioYs4h3OQ-2VLJYE5IRq5GcEGBG5YKGe84VXNdJO4taMK4CCV_HXbFJI_ZflWZqALWjrA29pLZcjWakodi19rtT0sJwiQ1; ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba
and I also need to pass in some form data:
__RequestVerificationToken=tTi9aRzgeb0zA1z3QMZ1iWbGuC4ajR9Ke2VctCLnUlaTKFg1m-70WSOEsZf3PLEUgRdr4n1rEPVvwmfvN6RwrGMKvqnQvjP_gWsAAxAHPY1&UserName=email%40email.com&Password=asdas
I know this is a bit complicated, but I am completely stuck, so i'll post my code and try to run through it as cleanly as I can:
'I am using Superagent-cache and Cheerio'
var url = "https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcode%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3DiNj9r1juVrmUK5DyLfXnjq6bM6Fci5E1seI-faOadJQsfBKC9PQJJA-wve3TrusBfhrcjNk8C932FDA_vgQIyrlg36K6ucoC3HZkAO-Yn-mRXmaVqZcdKPRvgwYr55UkeETK4ZsjyuOXNixzk0Z3AslC2ZVN2dqiqoPfpoYtz_n-xgtJlvN5WwRt6cEAvzSwhHkFX4UPUF_1OalC8J4aYO-FHfUjTp8Bv4xBe7w0j0exmjcsMIjpmnp4qbN3qz7u";
request
.get(url)
.end(function(err, res) {
if (err) {
console.log(err);
} else {
// This is for getting the response headers from the GET page
console.log("~~~GET~~~~~~~~~~~~~~~~~~~~~~~~")
$ = cheerio.load(res.headers);
console.log(res.headers);
var secondRequestVToken = res.headers['set-cookie'][0]
console.log("~~~Second RequestVerificationToken: " + secondRequestVToken);
var firstARRAffinity = "ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d; _ga=GA1.2.592576149.1458220828; _gat=1;"
var secondARRAffinity = res.headers['set-cookie'][1]
console.log("~~~Second ARRAfinnity: " + secondARRAffinity);
// This is the getting of the post Request data from the login page
console.log("~~~POST~~~~~~~~~~~~~~~~~~~~~~~~");
$ = cheerio.load(res.text)
var postUrl = "https://login.thenuel.com" + $('body > div > div > div.content-pane.login > form').attr("action");
console.log("~~~URL POST: " + postUrl);
var firstRequestVToken = $('body > div > div > div.content-pane.login > form > input[type="hidden"]:nth-child(1)').attr("value");
console.log("~~~First POST RequestVerificationToken: " + firstRequestVToken);
var formData= {
UserName:'email#email.com',
Password:'Password',
__RequestVerificationToken:firstRequestVToken
}
console.log(formData);
console.log("~~~COOKIE STUFF~~~")
secondRequestVToken = secondRequestVToken.slice(0, -7);
console.log(secondRequestVToken);
secondARRAffinity = secondARRAffinity.slice(0, -31)
console.log(secondARRAffinity);
var finalCookieString = firstARRAffinity + " "
+ secondRequestVToken + " " + secondARRAffinity;
console.log("~~~Final Cookie String: " + finalCookieString);
request
.post(postUrl)
.send(formData)
.set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.set("Accept-Encoding", "gzip, deflate")
.set("Accept-Language", "en-US,en;q=0.8")
.set("Cache-Control", "max-age=0")
.set("Connection", "keep-alive")
.set("Content-Length", 177)
.set("Content-Type", "application/x-www-form-urlencoded")
.set("Cookie", finalCookieString)
.set("Host", "login.thenuel.com")
.set("Origin", "https://login.thenuel.com")
.set("Referer", postUrl)
.set("Upgrade-Insecure-Requests", 1)
.set("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36")
.end(function(err, res) {
// Do respone here
if (err) {
console.log("~~~Hello failure world!");
console.log(err);
} else {
console.log("Hello success world!");
console.log(res);
}
})
So I am getting a GET of the login page to grab the 'RequestVerificationToken' and the second ARRAfinnity string - both to add to the cookie string I send to the POST.
I am then getting the POST Url from the Login Form Action, and the 'RequestVerificationToken' that gets put in the form data to send along with my username and password.
I then remove some of the guff from the tokens and stuff to make it the same as the request my browser sends off to that POST. I then make the POST call.
Here is my log with the console.logs:
Server Running...
~~~GET~~~~~~~~~~~~~~~~~~~~~~~~
{ 'cache-control': 'private',
'content-length': '1777',
'content-type': 'text/html; charset=utf-8',
'content-encoding': 'gzip',
vary: 'Accept-Encoding',
server: 'Microsoft-IIS/8.0',
'set-cookie':
[ '__RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSR
ZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; path=/',
'ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Path=/;Domain=login
.thenuel.com' ],
'x-aspnetmvc-version': '5.2',
'x-aspnet-version': '4.0.30319',
date: 'Wed, 30 Mar 2016 13:48:55 GMT',
connection: 'close',
prev: null,
next: null,
root:
{ type: 'root',
name: 'root',
attribs: {},
children: [ [Circular] ],
next: null,
prev: null,
parent: null },
parent: null }
~~~Second RequestVerificationToken: __RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8E
teTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; path=/
~~~Second ARRAfinnity: ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Pa
th=/;Domain=login.thenuel.com
~~~POST~~~~~~~~~~~~~~~~~~~~~~~~
~~~URL POST: https://login.thenuel.com/authenticate/login?ReturnUrl=%2Fauthorize%3Fresponse_type%3Dcod
e%26client_id%3Dc82cb4c9-7cfa-4483-938b-2d3c61efabea%26redirect_uri%3Dhttps%253A%252F%252Fthenuel.com%
252Fsignin-nuel%26scope%3Didentity%2520offline%26state%3DiNj9r1juVrmUK5DyLfXnjq6bM6Fci5E1seI-faOadJQsf
BKC9PQJJA-wve3TrusBfhrcjNk8C932FDA_vgQIyrlg36K6ucoC3HZkAO-Yn-mRXmaVqZcdKPRvgwYr55UkeETK4ZsjyuOXNixzk0Z
3AslC2ZVN2dqiqoPfpoYtz_n-xgtJlvN5WwRt6cEAvzSwhHkFX4UPUF_1OalC8J4aYO-FHfUjTp8Bv4xBe7w0j0exmjcsMIjpmnp4q
bN3qz7u
~~~First POST RequestVerificationToken: bqsa4HwniG2PeJVTBKWjg2ux0S4zUJ-Y1U4C_YF93za33dPnNortSOTeHZyyWW
dT_WECqgr44IbJ_FjUSfu9_N3ITdjnJuJhuMvBp_dYQ4c1
{ UserName: 'email#email.com',
Password: 'Password',
__RequestVerificationToken: 'bqsa4HwniG2PeJVTBKWjg2ux0S4zUJ-Y1U4C_YF93za33dPnNortSOTeHZyyWWdT_WECqgr
44IbJ_FjUSfu9_N3ITdjnJuJhuMvBp_dYQ4c1' }
~~~COOKIE STUFF~~~
__RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6IEg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlV
xzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1;
ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;
~~~Final Cookie String: ARRAffinity=f02c8a40711ffa249ac8dcf17e82c47021b4939e86cdd8caa8a1729b4a81838d;
_ga=GA1.2.592576149.1458220828; _gat=1; __RequestVerificationToken=SL9MYCWkPdY3dI66vBq7BKt4wxfzmNQCO6I
Eg8EteTdCIe-BCiKbBNCIbWtb3jD9ZbNSRZmUIlVxzICnKGX5PpPOsQvp5me7NJoc4BHu1Ew1; ARRAffinity=e310baf6f2079f1
b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;
Hello success world!
{ body: {},
text: '<!doctype html>\r\n<html>\r\n<head>\r\n <link rel="stylesheet" type="text/css" href="//nu
el-ui-playgroundofwonders.azurewebsites.net/Content/nuel-reset.css" async />\r\n <link rel="stylesh
eet" type="text/css" href="//nuel-ui-playgroundofwonders.azurewebsites.net/Content/nuel-base.css" asyn
c />\r\n <title>Sorry :( - The NUEL</title>\r\n</head>\r\n<body>\r\n <div class="stretch" style=
"width: 23rem; padding: 1.3rem 1rem; background-color: #ffffff; box-shadow: #e0e0e0 0px 0px 1px; margi
n: 3rem auto 0px;">\r\n <h1 class="non-content">Something\'s up!</h1>\r\n <p style="line
-height: 1.3rem;">An error occured during your last request, if you were logging in - try clearing you
r cookies and then retrying. You can <a href="https://support.google.com/chrome/answer/95647?hl=en-GB"
target="_blank">find out how to do that here</a>.</p>\r\n <p style="line-height: 1.3rem;">If t
hat doesn\'t work then just drop us an email at issues#thenuel.com and we\'ll get back to you ASAP.</p
>\r\n <a class="button minimal primary small" href="https://thenuel.com/">Go to thenuel.com</a>
\r\n </div>\r\n</body>\r\n</html>',
headers:
{ 'content-length': '765',
'content-type': 'text/html',
'content-encoding': 'gzip',
'last-modified': 'Wed, 28 Oct 2015 23:22:44 GMT',
'accept-ranges': 'bytes',
etag: '"1c315c8cd711d11:0"',
vary: 'Accept-Encoding',
server: 'Microsoft-IIS/8.0',
'set-cookie': [ 'ARRAffinity=e310baf6f2079f1b7c40c521ea7e13fd41184f9683f30fea9f5312b081e077ba;Pat
h=/;Domain=login.thenuel.com' ],
date: 'Wed, 30 Mar 2016 13:48:57 GMT' },
statusCode: 200,
status: 200,
ok: true }
I would've hoped that it would have returned a cookie for being logged
The Verification tokens change each time. The correct verification token is being sent to the form and the correct one is being sent to the cookie string.
I know this is super complicated, but I'm completely stumped.
The website is login.thenuel.com.
Any help is appreciated, but I think this might just be a keep on trying sort of thing! And if there's any other info I need to provide just ask - and it might be easier to just try and do it yourself and see if you can do it!
Thanks

Resources