Related
I am deploying a MERN app on AWS EC2. Frontend is rendered perfectly on aws. While calling API(axios.post) empty body is received in backend when called through react frontend. But, when called through postman, the body is received correctly.
I am sharing the details of the request body as received on backend(I printed this for debugging purposes):
POSTMAN method:
Method: POST
Path: /api/auth
Body: { email_id: 'comiiii#gmail.com',
password: 'comiiii#123',
user_type: 'type_com' }
Headers: { host: '64.1.75.248',
'x-real-ip': '113.103.59.237',
'x-forwarded-for': '113.103.59.237',
'x-forwarded-host': 'ec2-64-1-75-248.ap-south-1.compute.amazonaws.com',
'content-type': 'application/json;charset=UTF-8',
connection: 'close',
'content-length': '94',
'user-agent': 'PostmanRuntime/7.26.8',
accept: '*/*',
'postman-token': 'a6ef6f75-716b-4843-a773-8b46d1f28427',
'accept-encoding': 'gzip, deflate, br' }
REACT FRONTEND:
Method: POST
Path: /api/auth
Body: {}
Headers: { host: '64.1.75.248',
'x-real-ip': '213.235.108.12',
'x-forwarded-for': '213.235.108.12',
'x-forwarded-host': 'ec2-64-1-75-248.ap-south-1.compute.amazonaws.com',
'content-type': 'application/json;charset=UTF-8',
connection: 'close',
accept: '*/*',
'access-control-request-method': 'POST',
'access-control-request-headers': 'access-control-allow-origin,content-type',
origin: 'http://ec2-64-1-75-248.ap-south-1.compute.amazonaws.com',
'user-agent':
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.146 Safari/537.36',
'sec-fetch-mode': 'cors',
referer: 'http://ec2-64-1-75-248.ap-south-1.compute.amazonaws.com/',
'accept-encoding': 'gzip, deflate',
'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8' }
I saw the difference between the two and realised that the headers which are different are part of forbidden ones. (Refer : https://developer.mozilla.org/en-US/docs/Glossary/Forbidden_header_name)
How to solve this?
I have a problem with my api call using NodeJS. I have no problem with postman, but when I run it with Node, server response is 401.
Here's the nodejs code:
var request = require("request");
var options = { method: 'GET',
url: 'https://api.cardmarket.com/ws/v2.0/output.json/products/find',
qs:
{ search: 'Salamangrande%25Loup%25Du%25Soleil',
idGame: '3',
idLanguage: '2' },
headers:
{ 'cache-control': 'no-cache',
Connection: 'keep-alive',
Cookie: 'PHPSESSID=m399v2el9635i3jq0e4did8f5k',
'Accept-Encoding': 'gzip, deflate',
Host: 'api.cardmarket.com',
'Postman-Token': '9b19a85d-8888-4f31-8d24-fe309a55bf76,4d8f4741-4b85-4d5e-85dc-8ac43ea5f56c',
'Cache-Control': 'no-cache',
Accept: '*/*',
'User-Agent': 'PostmanRuntime/7.19.0',
Authorization: 'OAuth realm="https%3A%2F%2Fapi.cardmarket.com%2Fws%2Fv2.0%2Foutput.json%2Fproducts%2Ffind",oauth_consumer_key="xxxxxxxxx",oauth_token="xxxxxxxxx",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1579186959",oauth_nonce="vWUKBoTTkle",oauth_version="1.0",oauth_signature="ZD2TwzLVHqOjLm3u%2BWv%2FsQ8mdfs%3D"' } };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(response.statusCode);
});
So basically here's my (perfectly working) .
Any idea what's wrong with the syntax or how to properly implement the oauth signature in the headers ?
HTTP 401 error means "unauthorized".
I don't think OAuth authorization is meant to be passed through the headers.
It is supposed to be something like :
var request = require("request");
url='https://api.cardmarket.com/ws/v2.0/output.json/products/find';
oauth = {
consumer_key: CONSUMER_KEY,
consumer_secret: CONSUMER_SECRET,
token: AUTH_TOKEN,
token_secret: TOKEN_SECRET,
signature_method : 'RSA-SHA1',
};
qs = {
search: 'Salamangrande%25Loup%25Du%25Soleil',
idGame: '3',
idLanguage: '2'
};
request.get({url:url, oauth:oauth, qs:qs, json:true}, function (e, r, product) {
console.log(product)
})
Check the request docs for more explanation on how to deal with OAuth in request.
(I strongly suggest using Axios instead of request, but that's up to you)
I'm getting a bad request from postman when I make a request using raw json on Postman. The POST request accepts the following payload:
{"BusinessID": 34222222}
When I select form-data the request is successful.
I also copied the Python code from postman and executed it but I get a bad request.
Screenshot
The same applies when I try to run the code below.
query=99858585
url = "https://xxxxxxxxxxxxx.co.io/permit/api/checkBusiness"
payload = "{\"BusinessID\": query}"
headers = {
'Content-Type': "application/json",
'User-Agent': "PostmanRuntime/7.20.1",
'Accept': "*/*",
'Cache-Control': "no-cache",
'Postman-Token': "1bda2a65-6ed4-45d2-8029-5e70efa336ec,c1a967af-856e-4ab0-ad88-83b0b5c496df",
'Host': "xxxxxxxxxxxxxxxxx.co.io",
'Accept-Encoding': "gzip, deflate",
'Content-Length': "23",
'Cookie': "ci_session=hnv0nsiksienscujctqmdm5mp2b0dp4q",
'Connection': "keep-alive",
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.json())
When I select form-data the request is successful
Headers after selecting form-data*
I have set up an angular 8 app, that connects to an express API.
I'm running it locally, to test.
My front end app connects to http://localhost:4200/ and backend to http://localhost:3000/
I've set up an express route to connect to https://api.podbean.com/v1/podcasts?access_token=baee9cb65384a814e704adc626dc969bb019f84d
which works fine, returning all podcasts
But the debugToken endpoint never works via the express route, if I use https://api.podbean.com/v1/oauth/debugToken?access_token=baee9cb65384a814e704adc626dc969bb019f84d
Using postman with basic auth clientId = '7faf9a7ad38a01c7d900c' client_secret = 'a7a3825f02be39c57ff44' it works ok, but never when connecting via localhost
I'm using GET
It must be connecting because I get an object returned, although it's an error
In Angular:
debug() {
const httpOptions = {
headers: new HttpHeaders({
'Content-Type': 'application/json',
Authorization: 'Basic ' + btoa('7faf9a7ad38a01c7d900c:a7a3825f02be39c57ff44')
})
};
console.log(httpOptions);
return this.http.get(`${this.configUrl}/debug/`, httpOptions);
}
Express:
router.get('/debug', function (req, res, next) {
var options = {
url: `https://api.podbean.com/v1/oauth/debugToken?access_token=${accessToken}`
}
request(options, function (err, response, body) {
console.log( req.headers);
if(err){
return res.status(500).json({
title: 'An error has occured',
error: err
})
}
res.json(JSON.parse(body));
next();
})
});
When I log the request headers in the express/node side
{host: 'localhost:3000',
connection: 'keep-alive',
pragma: 'no-cache',
'cache-control': 'no-cache',
'sec-fetch-mode': 'cors',
origin: 'http://localhost:4200',
authorization: 'Basic N2ZhZjlhN2FkMzhhMDFjN2Q5MDBjOmE3YTM4MjVmMDJiZTM5YzU3ZmY0NA==',
'content-type': 'application/json',
accept: 'application/json, text/plain, */*',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
dnt: '1',
'sec-fetch-site': 'same-site',
referer: 'http://localhost:4200/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-GB,en;q=0.9,en-US;q=0.8,it;q=0.7,es;q=0.6' }
Returned object:
{"error":"invalid_token","error_description":""}
Which tells me I'm connecting, just not correctly
Looks like you just confused endpoints. You are sending basic auth from Angular page to your Express endpoint,
which doesn't make much sense, because it's https://api.podbean.com who requires authorization, not your Express server.
Try adding basic auth credentials to the request which goes from your Express server to api.podbean.com
router.get('/debug', function (req, res, next) {
var options = {
url: `https://api.podbean.com/v1/oauth/debugToken?access_token=${accessToken}`,
headers: {
'Authorization': 'Basic ' + new Buffer("7faf9a7ad38a01c7d900c:a7a3825f02be39c57ff44").toString('base64')
}
}
request(options, function (err, response, body) {
...
I want to get the flights table for every city from a page but befor that i have to set search fields and hit the button and befor that i have to import captcha at the first page. The web site designed with .Net. I import the captcha by my self but i want to do the rest with nodejs.
This is what i'm doing. first, import the captcha and then from loaded page and by looking at the firebug, set request header and request url and then send using http.request with GET method to scrape the page with nodejs again. after that i get viewstate of the page and again usign firebug, set another request header and "post data" and request url and send them using http.request to scrape the final page. This information that i set using firebug, are fixed; i mean, urls, header options and post data. what i have to change is city name in post data.
But i get a page with empty table. if is possible, What should i do? (sorry for my bad english :) )
This is the url (in persian): http://sepehr.iranhrc.ir.
Thanks in advance.
var http = require('follow-redirects').http;
var querystring = require('querystring');
var cheerio = require('cheerio');
var dataToAttach = {
'ScriptManager1': 'uplFlightSearch|btnSubmit37756070715319',
'__ASYNCPOST': true,
'__EVENTARGUMENT': '',
'__EVENTTARGET': '',
'__LASTFOCUS': '',
'__VIEWSTATE': '',
'__VIEWSTATEGENERATOR': 'E4CF65F9',
'btnSubmit37756070715319': '?????',
'dplFrom': 'THR',
'dplReservationRouteType': 'RoundTrip_FixedDate',
'dplTo': '0',
'dplFlightAdults': '1',
'dplFlightChilds': '0',
'dplFlightInfants': '0',
'txtCountUp': '00:26',
'txtDepartureDate': '1394/04/02',
'txtReturningDate': '1394/04/04'
};
var flightsSearchPageROs = {
hostname: 'sepehr.iranhrc.ir',
path: '/Systems/FA/Reservation/Flight_NewReservation_Search.aspx?qry=sbV7wBdq4B7yEk1yv0OpvmofQkdkbWH49Wjk6UIMGiw95zDjDGo0/ssWJH8Wjv1D',
method: 'GET',
headers:{
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
'Cookie': 'ASP.NET_SessionId=2iexj4pfxld4mdilfwttka2q;',
'Content-Type': 'text/html; charset=utf-8',
'Host': 'sepehr.iranhrc.ir',
'Referer': 'sepehr.iranhrc.ir'
}
};
var resultsPageROs = {
hostname: 'sepehr.iranhrc.ir',
path: '/Systems/FA/Reservation/Flight_NewReservation_Search.aspx?action=display&rnd=2378726045210585',
method: 'POST',
headers:{
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Cookie': 'ASP.NET_SessionId=o1ipad335qahuaahc25ngalr;',
'Content-Length': '',
'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
'Referer': 'http://sepehr.iranhrc.ir/Systems/FA/Reservation/Flight_NewReservation_Search.aspx',
'Host': 'sepehr.iranhrc.ir',
'Pragma': 'no-cache',
'X-MicrosoftAjax': 'Delta=true',
'X-Requested-With': 'XMLHttpRequest'
}
};
var flightsSearchPage = http.request(flightsSearchPageROs, function(response{
var dataStream = '';
var htmlCode = '';
var date = '';
response.on('data', function(chunk){
dataStream += chunk;
});
response.on('end', function(){
htmlCode = cheerio.load(dataStream);
secondDataToAttach.__VIEWSTATE = htmlCode("__VIEWSTATE").val();
resultsPageROs.headers['Content-Length'] = querystring.stringify(dataToAttach).length;
resultsPageRequest();
});
});
flightsSearchPage.on('error', function(e){console.log("Error0: " + e.message);});
flightsSearchPage.end();
function resultsPageRequest(){
var changingCitiesBoxResponse = http.request(resultsPageROs, function(response){
response.setEncoding('utf8');
var dataStream = '';
var htmlCode = '';
response.on('data', function(chunk){
dataStream += chunk;
});
response.on('end', function(){
htmlCode = cheerio.load(dataStream);
console.log(htmlCode.html());
});
console.log('STATUS: ' + response.statusCode);
});
changingCitiesBoxResponse.on('error', function(e){console.log("Error1: " + e.message);});
changingCitiesBoxResponse.end(querystring.stringify(dataToAttach));
}
EDIT
One thing that i forgot to mention it was that, this is done in PHP with Curl and i looked at the code in curl part and done exactly like that with node and http.request. PHP returns correct answer but mine not.
And I used phantom as well.This is the code:
var url = "http://sepehr.iranhrc.ir/Systems/FA/Reservation/Flight_NewReservation_Search.aspx?action=display&rnd=4565721642440773";
var settings = {
operation: "POST",
encoding: "utf8",
weak: false,
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Cookie': 'ASP.NET_SessionId=2iexj4pfxld4mdilfwttka2q;',
'Content-Length': '',
'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
'Referer': 'http://sepehr.iranhrc.ir/Systems/FA/Reservation/Flight_NewReservation_Search.aspx?qry=sbV7wBdq4B7yEk1yv0OpvmofQkdkbWH49Wjk6UIMGiw95zDjDGo0/ssWJH8Wjv1D',
'Host': 'sepehr.iranhrc.ir',
'Pragma': 'no-cache',
'X-MicrosoftAjax': 'Delta=true',
'X-Requested-With': 'XMLHttpRequest'
},
data: querystring.stringify(secondDataToAttach)
};
phantom.create(function (ph){
ph.createPage(function (page){
page.open(url2, settings, function(status){
console.log(status);
page.evaluate(function(){ return document.body.innerHTML; }, function(result){
console.log('Content ' + result);
ph.exit();
});
});
});
}, {dnodeOpts: {weak: false}
});
But i didn't get the answer.
And yes, it uses AJAX to send the request. According to Firebug:
The first request is sending a post data and the second one, i think is a redirect that gives the link to the result page(i used that link but no luck) and the third is the result page with the flight details that i can't get.
May be i used them wrong. How can i use these Three request to get flights table?
I think what's happening is that the page you want the results from is loading them through a separate AJAX request, and a regular get request isn't going to get that. You'll need to either figure out what that separate request is and do it yourself, or scrape the page with something like PhantomJS that can execute the javascript for the page.
First though, go to the page on your browser regularly and watch the network tab and locate that request that loads in the data.