I have been bashing my head against the wall for the last 2 days with the following problem.
This is the scenario: When I make a GET request by browsing to a particular website, this website sends a cookie called PHPSESSION="xyz" it then prompts the user to enter a password and subsequently makes a post request to the same URL sending this particular cookie and a hidden form element alongside for verification and upon success sends a pdf.
I can successfully replicate this in Postman.
I make a get request - it sets the cookie - I have password filled into my form-data responds body and manually add the secret string that is added to the form for verification -> send... and I get the pdf - so far so good.
However, I would like to automate this process so that I don't have to painstakingly extract the value of the hidden form by hand but use node.js to make these requests so I wrote the following code:
// making the get request to the URL above
// extract the cookie PHPSESSION value
const sessionString = String(response.headers.get('set-cookie')).substring(10,36)
// parse the body
const htmlBody = await response.text()
let doc = new DOMParser().parseFromString(htmlBody)
// extract the verification token from the form
const formToken = await doc.getElementById('verification__token').getAttribute('value')
let formData = new FormData();
formData.append('verification[char_1]',0)
formData.append('verification[char_2]',6)
formData.append('verification[char_3]',4)
formData.append('verification[char_4]',5)
formData.append('verification[char_5]',8)
formData.append('verification[char_6]',1)
formData.append('verification[char_7]',7)
formData.append('verification[char_8]',6)
formData.append('verification[_token]',formToken)
const obj = {
headers:{
"Cookie" : `PHPSESSID=${sessionString};`,
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "PostmanRuntime/7.29.2",
"Accept-Encoding": "gzip, deflate, br",
"credentials": "include"
},
method: "POST",
body: formData
}
const postResponse = await fetch("https://url...",obj)
const r = await postResponse.text()
Unfortunately, the post requests fails in node.js - the website is simply redirecting me to back to the form in which I have to type in the password.
I am suspecting it has something to do with the headers / cookie but I simply don't know.
Does anyone spots an obvious mistake?
Thank you
Solved... after sacrificing the entire weekend to this lovely task.
If anyone comes across a similar problem here is the solution - or at lest what helped me.
https://reqbin.com/curl
https://curlconverter.com
So basically make your request work with curl and then port it.
In my case that looked like this:
const x = await fetch('https://yourURL', {
method: 'POST',
headers: {
'Cookie': 'PHPSESSID=lfjdd2uba1bmecr064rt7chvu3; Path=/; Secure; HttpOnly;',
'Content-Type': 'application/x-www-form-urlencoded'
},
body: 'verification[_token]=5d5e4d8783daf952d5.UZ661yMyOUtJSQeG1Td7cUtxWqnI2Oaot-xMQevly4o.acH9hXR9SRkwGm30kE9WIggDNpqdl6Ln2rQnOIG9pcEp1tOiYnNLJggZcA&verification[char_1]=0&verification[char_2]=6&verification[char_3]=4&verification[char_4]=5&verification[char_5]=8&verification[char_6]=1&verification[char_7]=7&verification[char_8]=6'
});
Related
I have a Chrome Packaged App that uses oauth2 to authenticate to YouTube. I'm using YouTube to determine the user's channel (via the channel endpoint).
It works for the first user authenticated. But if I switch to a different user, the same YouTube call returns the previous user's data (i.e. channel).
Here are the steps I'm going through.
I get my auth token via a call to getAuthToken:
chrome.identity.getAuthToken({ interactive: true, scopes: ['https://www.googleapis.com/auth/youtube'] })
I get their channel information. I make a call to the channels endpoint like so:
const url = 'https://www.googleapis.com/youtube/v3/channels?part=snippet&mine=true';
const headers = { Authorization: `Bearer ${token}` };
const config = {
url,
method: 'get',
headers,
responseType: 'json',
};
return axios(config);
This works! The result of the call gives me the channel information.
The user removes their account. I use the same sequence of calls that the Google Demo uses (like this example):
a. call chrome.identiy.removeCachedAuthToken({ token }) with the token
b. call https://accounts.google.com/o/oauth2/revoke?token=${token} to revoke it
Everything is cleared out, I think.
If I look at chrome://identity-internals/ I still see the token, but the Token Status is set to Not Found
The issue:
I repeat from step 1, but I chose a different user.
I confirm that I get a new token that is different than the one I had previously
The call to the YouTube channels api returns the previous user's channel.
It turns out it was a caching issue with youtube.
I had to add 'Cache-Control': 'no-cache' to my headers.
Here is the full headers line:
const headers = {
Authorization: `Bearer ${token}`,
'Cache-Control': 'no-cache',
};
I'm currently downloading files using the request package using the following code:
request
.get({
url,
headers: {
Authorization: "base64"
}
})
.pipe(fs.createWriteStream('test.xlsx'))
This works nice and all when the authentication is a simple username/password or doesn't have one at all, but once 2 factor authentication comes in, it's going to be a real hassle since this method doesn't keep track of your cookies or login state (or however it's tracked).
So how would I get the buffer/data during puppeteer's run time then pipe it into another filestream (note I will need to do this recursively for several files).
I think you can construct the cookies header from puppeteer like this:
cookies = await page.cookies();
cookie_str = "";
for(var i = 0; i < cookies.length; i+=1){
a = cookies[i];
cookie_str += a.name + "=" + a.value + ";";
}
and then use request with a cookie header:
request.get({
url: download_link,
headers: {
"cookie": cookie_str,
}
}).pipe(fs.createWriteStream("ofname"))
I am having difficulty sending a url encoded string to the Stormpath /oauth/token API endpoint. The string is meant to look like this:
grant_type=password&username=<username>&password=<password>
Using Postman I was successful in hitting the endpoint and retrieving the data I want by providing a string similar to the one above in the request body by selecting the raw / text option. But when I generate the code snippet it looks like this:
var request = require("request");
var options = { method: 'POST',
url: 'https://<My DNS label>.apps.stormpath.io/oauth/token',
headers:
{ 'postman-token': '<token>',
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
host: '<My DNS label>.apps.stormpath.io',
accept: 'application/json' },
form: false };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
Where did that string go? I would like some help in understanding the disconnect between knowing I sent a url encoded string to the API endpoint using Postman and not seeing it in the code generated by Postman. Because now I don't know how to reproduce a successful call to the endpoint in my actual app.
To me it seems like I should simply provide a body to the request, but the response comes out to be {"error":"invalid_request","message":"invalid_request"}. I have also tried appending the url encoded string to the url but that returns a 404 error.
I'm just now getting back into using an API and am not very experienced doing so.
The form data needs to be posted as an object, here is an example:
request.post('http://service.com/upload', {form:{key:'value'}})
Taken from this documentation:
https://github.com/request/request#forms
Hope this helps!
I have been trying to make this post request to the github api for the last couple of days, but unfortunately the response is coming back as "bad message"
here is the piece of code we are sending in the post request using https request in node -
This is the post data
var issueData = JSON.stringify({
"title":title,
"body":comment
});
This is the options file
var options = {
host: 'api.github.com',
path: '/repos/sohilpandya/katasohil/issues?access_token='+sessions.token,
headers: {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0',
},
method: 'POST'
};
This is the https request
var requestaddIssue = https.request(options, function(responseFromIssues){
responseFromIssues.setEncoding('utf8');
responseFromIssues.on('data', function(chunk){
console.log('>>>>chunk>>>>>',chunk);
issueBody += chunk;
});
responseFromIssues.on('end',function(issueBody){
console.log(issueBody);
});
});
requestaddIssue.write(issueData);
requestaddIssue.end();
I have tried another approach where the authentication token for the user is in the header as
'Authentication': 'OAuth '+ sessions.token (where we are storing token inside sessions)
But the chunk response always seems to come back with the following in the console log.
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3/issues/#create-an-issue"
}
I have tried the same in apigee and it seems to work ok and returns to correct response. Hoping someone can find the minor error in the code above that is causing this bad message error.
Except the issueBody variable is not defined in the snippets you posted, the code is correct. I tried it using a personal access token.
The error you get appears because you need to add a scope with power to open issues.
I tried the repo and public_repo scopes and they are both working. Note that repo has access to private repositories. Here you can see the list of scopes.
If you're using OAuth, then you you should have an url looking like this:
https://github.com/login/oauth/authorize?client_id=<client-id>&scope=public_repo&redirect_uri=<redirect-uri>
PhantomJs's webserver does not support multipart requests, so I'm trying to send a single-part request from NodeJs.
Unfortunatly the nodejs example looks to be multipart. is there any way of doing this with NodeJs?
http://nodejs.org/api/http.html#http_http_request_options_callback
edit:
in the nodejs docs it mentions:
Sending a 'Content-length' header will disable the default chunked encoding.
but unfortunatly it's still multi-part, just not multi-multipart :P
edit2: for showing code, it's a bit hard to show a distilled example, but here goes:
node.js code (it's Typescript code):
```
//send our POST body (our clientRequest)
var postBody = "hello";
var options : __node_d_ts.IRequestOptions = {
host: host,
port: port,
path: "/",
method: "POST",
headers: {
"Content-Type": "application/json",
"Content-length": postBody.length
}
};
//logger.assert(false);
var clientRequest = http.request(options,(response: http.ServerResponse) => {
//callback stuff here
});
clientRequest.on("error", (err) => {
thisObj.abort("error", "error,request error", err);
});
//clientRequest.write();
clientRequest.end(postBody);
```
when i read the results from PhantomJS, the post/postRaw fields are null.
when I use a tool like the Chrome "Advanced REST Client" extension to send a POST body, phantomjs gets it no problem.
i don't have a network sniffer, but as described here, it says phantomjs doesnt work with multipart so I think that's a good guesss: How can I send POST data to a phantomjs script
EDIT3:
indeed, here's the request phantomjs gets from my chrome extension (valid post)
//cookie, userAgent, and Origin headers removed for brevity
{"headers":{"Accept":"*/*","Accept-Encoding":"gzip,deflate,sdch","Accept-Language":"en-US,en;q=0.8,ko;q=0.6","Connection":"keep-alive","Content-Length":"5","Content-Type":"application/json","DNT":"1","Host":"localhost:41338", "httpVersion":"1.1","method":"POST","post":"hello","url":"/"}
and here's the request phantomjs gets from the nodejs code i show above:
//full request, nothing omitted!
{"headers":{"Connection":"keep-alive","Content-Type":"application/json","Content-length":"5","Host":"10.0.10.15:41338"},"httpVersion":"1.1","method":"POST","url":"/"}