I am trying to scrap this website https://www.erobertparker.com/entrance.aspx it requires authentication I am using request module to get authenticated like this,
request({
url:"https://www.erobertparker.com/login.aspx",
method:"POST",
form:{UNENTRY:"username",PWENTRY:"password"}
},
function(error,response,body){
})
but i am unable to get authenticated what i am doing wrong can someone please guide me I am new to web scraping world :).
It's using an asp.net session cookie. You possibly need to store all cookies in a jar and then send them back on the next request.
Hi I solved this using a jar parameter in the request:
var j = request.jar();
var request = request.defaults({ jar : j }) //it will make the session default for every request
//...
request({
url:"https://www.erobertparker.com/login.aspx",
method:"POST",
form:{UNENTRY:"username",PWENTRY:"password"}
},
function(error,response,body){
//Do your logic here or even another request like
request({
url:"<ANOTHER LINK>",
method:"GET",
}, function(error, response, body){
//Some logic
});
});
You can also check the documentation of the request module:
https://github.com/request/request#examples
Related
Full disclaimer: I have never worked with Microsofts NTLM before.
I've tried around 30 different ways to access the 7Pace timetracker API on our local TFS instance. I know it works if i access the URI directly in Chrome, it prompts me for my AD login, and swiftly serves me all the data requested. Same for Postman, except there is an authentication tab for NTLM ahead of time.
Postman suggests this for Node.js using request:
var request = require("request");
var options = {
method: 'GET',
url: 'http://TFSURL/odata/TimeExport%28StartDate=%272018-11-14%27,EndDate=%272018-11-14%27%20,PopulateTopParentColumns=null,GroupTimeByDateByUser=null,IncludeBillable=null%29',
headers: {
'cache-control': 'no-cache',
Authorization: 'NTLM NOTTHEREALTOKENKJASDKLHWKLLASBEDBSDAOBAW'
}
};
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
This returns nothing. Notice the Authorization header. I've tested multiple different variants of similar. My next guess was to request through Chrome, then sniff it with Tellerik Fiddler and try to replicate the headers. I've done this as well, but to no avail. I end up with a very similar result to that above, except chrome uses negotiate:
Any ideas on how to go about this ? Maybe other debugging options?
You will need to through the 3 steps of authentication for NTLM. It's not that easy if you want to do it manually as the NTLM spec is not really open.
There's Node.js module you can use: https://www.npmjs.com/package/httpntlm (disclaimer: I created it)
To GET your url, you would need the following:
var httpntlm = require('httpntlm');
httpntlm.get({
url: "http://tfs2:8090/api/FlexPOS%20APS/odata/TimeExport%28StartDate=%272018-11-14%27,EndDate=%272018-11-14%27%20,PopulateTopParentColumns=null,GroupTimeByDateByUser=null,IncludeBillable=null%29",
username: 'your username',
password: 'your password',
workstation: 'anything',
domain: ''
}, function (err, res){
if(err) return err;
console.log(res.headers);
console.log(res.body);
});
I am currently learning Node.js and Express and wanted to build a simple app that queries the Spotify Web API without using an external library such as spotify-web-api-node. My code is available at https://pastebin.com/Jwe8sckJ
My "/callback" route looks like this;
app.get("/callback", function(req, res){
res.send("OK!")
var authCode = req.query.code
var options = { method: 'POST',
url: 'https://accounts.spotify.com/api/token',
headers: { 'content-type': 'application/x-www-form-urlencoded' },
form:
{ grant_type: 'authorization_code',
code: authCode,
redirect_uri: 'https://example.com/callback',
client_id: clientId,
client_secret: clientSecret } };
request(options, function (error, response, body) {
if (error) throw new Error(error);
var accessToken = JSON.parse(body).access_token;
var refreshToken = JSON.parse(body).refresh_token
console.log("Access Token: " + accessToken);
console.log("Refresh Token: " + refreshToken);
});
});
So far this is all working fine, I am able to get an Access Token and a Refresh Token. The part that I am confused about is how do I now use these tokens outside of the "/callback" route? Or maybe another way of asking would be how do I save the variables globally. If I do that, would it have any impact on new sessions to script?
The most common solution would be to store these tokens in a database, i.e. MongoDB or MySQL and retrieve them when needed. You'll probably want to use sessions to set the received tokens per client.
You could even save the tokens in the session, but please note that sessions are volatile.
If I got you right then you want to save access token for later use. You can declare global variable and can use it later.
global.spotify_access_token = '';
Declare this variable at the start of any file. Now you can use this variable to save access token in callback.like below :
spotify_access_token = JSON.parse(body).access_token;
Hope this will help you.
Thanks.
I use fetch() to send a post request for logon,
after server validation, I use req.session.account = account[0]; to save the account information to the session and return a redirect URL,
But after the redirect, the account information in the session is lost, why is that?
If you would have some example code, that would help us track down the problem. My first idea would be to make sure express-session is properly configured:
var app = express();
var session = require('express-session');
app.use(session({
secret: 'ibfiyegfiyegfe' // Secret is a required option.
}));
For futher configuration options, see the express-session repository.
EDIT: Based on the additional information, I'm guessing you have express-session configured to also use cookies for managing the sessions. What this means, is that for every HTTP request Express sends a response back that includes a cookie. jQuery based AJAX calls will ignore the response cookie however, which causes consecutive requests to Express to look as if you never logged in. To allow saving the cookie when performing AJAX requests with jQuery, use the xhrFields field to enable withCredentials:
$.ajax({
url: "http://....",
type: "POST",
xhrFields: {
withCredentials: true
},
data: {username: username, password: password},
success: function(responseBody) {
console.log("success!");
},
error: function(responseBody) {
console.log("error!");
}
});
Hope this helps.
Sorry to everyone, I don't making the question clear.
I use the fetch() method send a request to logon.
fetch('/logon', {
method: 'post',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
account: account,
password: password
})
}).then(function(response){
return response.json();
}).then(function(json){
if(json.success == 1){
window.location.href = json.redirecturl;
}else{
indexDispatcher.dispatch({
actionType: 'logonFaild',
errorInfo: json.msg
});
}
});
And the server's response is just simple JSON:
if (err) {
consoloe.log(err);
} else {
req.session.account = account[0]; //save account to session
var redirecturl = '/team/' + team[0].id;
console.log("account添加到session了");
res.json({
success: 1,
redirecturl: redirecturl
});
}
But when the client get the redirecturl and redirect, the account data is lost,so it will occur a TypeError: Cannot read property 'id' of undefined(id is saved in req.session.account).
But when I use jquery $.ajax relpace fetch, it works well. I don't know the reason.
I had the same experience like you hundreds of times which I finally found the problem. You don't need to worry if cookies enable or disable.
You just have to re-declare parts of your scripts (variables-the data which you want to display) in a if else condition.
Just redeclare the same thing using !isset and then else with the same variables mentioned twice in the if else condition, if you understand what i mean.
It will call back the same variables over and over. And make sure start session is above the page. no need to worry about headers or white lines, it has actually nothing to do with it.
I've tried it a million times. You don't need to bother using ajax etc. PHP can do almost everything. cheerio.
I have a simple NodeJS app which authenticates with remote Java CMS. On successful authentication the CMS returns a cookie.
However, for the life of me I can't figure how to access / get this cookie value. (I need the cookie in order to make requests to the CMS's API, once authenticated).
I get the cookie returned in a curl command ok but can't access it in NodeJS.
Curl Command:
curl -v -k --data "username=admin&password=password111&realm=cms101"
https://test.abeo.ie/gatekeeper/rs/authenticate/login
Now here is my NodeJS app's code:
//disable self-signed ssl cert rejection
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
var http = require('http');
var request = require('request');
var Cookies = require('cookies');
//enable cookies
request.defaults({jar: true});
request({
url: 'https://test.abeo.ie/gatekeeper/rs/authenticate/login', //URL to hit
qs: {username: 'admin', password: 'password111', realm: 'cms101'}, //Query string data
method: 'POST', //Specify the method
headers: { //We can define headers too
'Content-Type': 'application/x-www-form-urlencoded'
}
},
function(error, response, body){
if(error) {
console.log(error);
} else {
//response.cookie comes out as 'undefined'
console.log(response.statusCode, body, response.cookie);
}
})
So my question is: How do I read the cookie value from the response as the response.cookie value prints out as undefined.
Thanks,
Mike
** If this question seems to vague... Just say so and I will clear it up.
I think I have just answered my question :)
I had the syntax wrong and instead of
console.log(response.statusCode, body, response.cookie);
I needed
console.log(response.statusCode, body, response.headers['set-cookie']);
Just working on parsing the thing now :)
hye,
i am building an app with angular.js and node.js (Express.js) on the server side.
for some reason i am having a problem handling a delete request. no body is getting to the server side.
this is my angular.js resource code:
$scope.deleteProject = function(projectName){
var postData = {username: 'name', projectName: projectName};
Project.deleteProject.delete({}, postData,
function(res){
alert('Project Deleted');
},
function(err){
alert(err.data);
});
}
on the server side i have this:
var deleteProject = function(req, res){
console.log(req.body);
console.log(req.params);
if (req.body.projectName){
//do something
return res.send(200);
}
else
return res.send(400, 'no project name was specified');
}
now for some reason there is no body at all!! it is empty.
i have defined the route as app.delete.
if i change the route in node.js to post and in angular.js to save it works fine.
what am i missing here (banging my head).
thanks.
As per this stack overflow question and the $http service source code, a DELETE request using $http does not allow for data to be sent in the body of the request. The spec for a DELETE request is somewhat vague on whether or not a request body should be allowed, but Angular does not support it.
The only methods that allow for request bodies are POST, PUT, and PATCH. So the problem is not anywhere in your code, its in Angular's $http service.
My suggestion would be to use the generic $http(...) function and pass in the proper method:
$http({
method: 'DELETE',
url: '/some/url',
data: {...},
headers: {'Content-Type': 'application/json;charset=utf-8'}
})
Angular by default sends the Content-Type as text/plain for DELETE requests. Just add this to the headers:
var config = {
method: "DELETE"
url: yourUrl
data: yourData
headers: {"Content-Type": "application/json;charset=utf-8"}
};
$http(config);
If you want to add them to every single DELETE request add this to the app.config method in your main controller:
$httpProvider.defaults.headers.delete = { "Content-Type": "application/json;charset=utf-8" };
If you want to use the $resource object instead of $http you need to add hasBody and headers as follow:
delete: {
method: 'DELETE',
hasBody: true,
headers: {"Content-Type": "application/json;charset=UTF-8"}
}
Worked for me
Just ran into this problem. You'll have to use url params to send an id with delete.
in express:
app.delete('/api/user/:userId', user.remove);
and add to the url in angular:
$http({url: 'whatever/api/'+obj.id, method: 'DELETE'}) ...
The following works for me:
$httpProvider.defaults.headers.common['X-Requested-With'] = 'XMLHttpRequest';
$httpProvider.defaults.headers.common['Content-Type'] = 'application/json;charset=utf-8';
XMLHttpRequest is optional but useful if you are sending ajax.
https://docs.angularjs.org/api/ng/provider/$httpProvider for more information.
This worked for me.
$httpProvider.defaults.headers.delete = { "Content-Type": "application/json;charset=utf-8" };
And then
$http.delete(url, { data: data })