I have gone through most of the posts related to 403 error and tried all of the options. However I am still not able to fix, I keep getting 403 error.
I have tried various combinations of the headers but no luck
Here is my code :
import requests
headers = {
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'cache-control': 'max-age=0',
'referer': 'https://www.whatismyip.com/52.242.97.97/',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
}
response = requests.get('https://www.whatismyip.com/40.70.224.149/', headers=headers)
print(response.status_code)
Related
I am trying to access the json response on a link. It is working in my normal browser but not working when I try to access it with python requests and is sending a Cloudflare page as a response rather than json.
So far, I have tried:
Copying the headers from my browser and passing them to my request
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36',
'Accept': 'application/json, text/plain, */*',
'Accept-Language': 'en-US,en;q=0.9',
'Cache-Control': 'max-age=0',
'Referer': 'https://9gag.com/',
'Cookie': '____ri=5012; ____lo=US;'
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers)
print(req)
Sending cookies seperately
cookies = {
'____ri': '5012', '____lo': 'US',
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers, cookies=cookies)
print(req)
Both of these approaches are returning <Response [403]>
I am trying to make a request to "https://www.walmart.com/search/?page=1&query=" using request function or using scrapy module but getting the response code 444.
See below my snippet:
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36',
'Accept': 'application/json', 'Content-Type': 'application/json'
}
res = requests.get('https://www.walmart.com/', headers=headers)
cookie = res.cookies
res1 = requests.get('https://www.walmart.com/search/?page=1&query=',headers=res.headers,cookies=cookie)
But I m getting the res1.status_code as 444. Would appreciate any help here.
This is how you should reuse request elements:
import requests
with requests.Session() as connection:
connection.headers.update(
{
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/89.0.4389.86 YaBrowser/21.3.0.740 "
"Yowser/2.5 Safari/537.36",
}
)
_ = connection.get("https://www.walmart.com/")
response = connection.get('https://www.walmart.com/search/?page=1&query=')
print(response.status_code)
Output (status code):
200
I get a response like
config: {
url: '',
method: 'post',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
'accept-language': 'de-DE,de;q=0.8',
cookie: '**WANT THIS AS A STRING**',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36'
},
},
now I would like to get the cookie as a string to post it in the next request.
let cookies = response.config.headers.cookie;
I am setting up an OAuth2 flow that needs to send some extra variables to the callback. When I check req.session in the authorize call, the values for the variables are properly set. The next step is to hit the "callback" route.
router.post('/api/personas/authorize', async (req, res) => {
const source = await personaController.getSourceInformation(req.body.source)
req.session.exchangeURL = source.token_exchange_url;
req.session.exchangeFields = source.exchange_fields;
console.log('REQ', req.session)
req.session.save()
const url = constructAuthURL(source)
res.send(url);
})
In the console.log('REQ') call, both the exchangeURL and exchangeFields variables are set properly.
Then, we hit the callback route:
router.get('/callback', async (req, res) => {
console.log(req.session)
}
Those exchange variables are not set. The sessionIDs are the same between the calls and the sid cookie is being sent in both requests. Additionally, in both requests, there are some custom variables that are retained between the calls. It appears that just the variables set at /authorize are not persisted.
Any help would be greatly appreciated and I am happy to provide more info!
EDIT:
Here are the headers from the request that hits 'api/personas/authorize':
{
host: 'localhost:3000',
connection: 'keep-alive',
'content-length': '15',
accept: 'application/json, text/plain, */*',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36',
'content-type': 'application/json;charset=UTF-8',
origin: 'http://localhost:3000',
'sec-fetch-site': 'same-origin',
'sec-fetch-mode': 'cors',
'sec-fetch-dest': 'empty',
referer: 'http://localhost:3000/personas',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
cookie: 'connect.sid=s%3AK01XQwFSwwA_q-D8OxALcx--asG23hsB.%2B8j%2BgLX6Eg%2FHhyh3K5wv%2FqpM6Vmp89xX5Kh8%2FFhLMJg'
}
And here are the headers from the request that hits '/callback':
headers: {
host: 'localhost:3000',
connection: 'keep-alive',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36',
accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'cross-site',
'sec-fetch-mode': 'navigate',
'sec-fetch-dest': 'document',
referer: 'https://{XXXX}.salesforce.com/',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
cookie: 'connect.sid=s%3AK01XQwFSwwA_q-D8OxALcx--asG23hsB.%2B8j%2BgLX6Eg%2FHhyh3K5wv%2FqpM6Vmp89xX5Kh8%2FFhLMJg'
}
I am using the ws module for node.js and I am having a issue, for some unknown reason it isn't connecting to the right server. I am not sure what is causing it to happen. If you need any bits of code just ask. Here is a video of what is happening. https://youtu.be/_Ji50RzyGh4
Here is the connect code:
connect() {
if (this.server == 'none' || this.server == '' || this.server == null || this.server == undefined) return;
this.ws = new WebSocket(this.server, {
headers: {
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4',
'Cache-Control': 'no-cache',
Cookie: '__cfduid=df2db54a70c595c3c2b5a0d067d1650481506608598',
Origin: origin,
Referer: origin,
Pragma: 'no-cache',
'Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'
},
agent: proxyAgents[this.id]
});
this.binaryType = 'nodebuffer';
this.ws.onopen = this.onopen.bind(this);
this.ws.onclose = this.onclose.bind(this);
this.ws.onerror = this.onerror.bind(this);
}