I am trying to access the json response on a link. It is working in my normal browser but not working when I try to access it with python requests and is sending a Cloudflare page as a response rather than json.
So far, I have tried:
Copying the headers from my browser and passing them to my request
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36',
'Accept': 'application/json, text/plain, */*',
'Accept-Language': 'en-US,en;q=0.9',
'Cache-Control': 'max-age=0',
'Referer': 'https://9gag.com/',
'Cookie': '____ri=5012; ____lo=US;'
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers)
print(req)
Sending cookies seperately
cookies = {
'____ri': '5012', '____lo': 'US',
}
req = requests.get("https://9gag.com/v1/group-posts/group/default/type/hot", headers=headers, cookies=cookies)
print(req)
Both of these approaches are returning <Response [403]>
Related
I'm trying to make a requests post with axios, sending that postdata from a checkbox and submit button, but I don't know how to do this correctly with axios, I would appreciate your help
`
const URI = "https://www.guadeloupe.gouv.fr/booking/create/12828/0"
const data = "condition=on&nextButton=Effectuer+une+demande+de+rendez-vous"
const headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Accept-Encoding': 'gzip, deflate, br',
'Origin': 'https://www.guadeloupe.gouv.fr'
}
const resp = await axios.post(URI,data,headers)
with insomnia = INSONMIA POST REQUEST SCREEN SHUT
CHECKBOX = PHOTO CHECKBOX SUBMIT
I can't test with real REST server but I can suggest this code.
It's base on your image and code.
const resp = await axios.post(URI,
new URLSearchParams({
'condition': 'condition',
'nextButton': 'Effectuer une demande de rendez vous'
}),
{
headers:
{
'Content-Type': 'application/x-www-form-urlencoded'
}
})
I am trying to make a request to "https://www.walmart.com/search/?page=1&query=" using request function or using scrapy module but getting the response code 444.
See below my snippet:
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36',
'Accept': 'application/json', 'Content-Type': 'application/json'
}
res = requests.get('https://www.walmart.com/', headers=headers)
cookie = res.cookies
res1 = requests.get('https://www.walmart.com/search/?page=1&query=',headers=res.headers,cookies=cookie)
But I m getting the res1.status_code as 444. Would appreciate any help here.
This is how you should reuse request elements:
import requests
with requests.Session() as connection:
connection.headers.update(
{
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/89.0.4389.86 YaBrowser/21.3.0.740 "
"Yowser/2.5 Safari/537.36",
}
)
_ = connection.get("https://www.walmart.com/")
response = connection.get('https://www.walmart.com/search/?page=1&query=')
print(response.status_code)
Output (status code):
200
I have gone through most of the posts related to 403 error and tried all of the options. However I am still not able to fix, I keep getting 403 error.
I have tried various combinations of the headers but no luck
Here is my code :
import requests
headers = {
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9',
'cache-control': 'max-age=0',
'referer': 'https://www.whatismyip.com/52.242.97.97/',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
}
response = requests.get('https://www.whatismyip.com/40.70.224.149/', headers=headers)
print(response.status_code)
I get a response like
config: {
url: '',
method: 'post',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
'accept-language': 'de-DE,de;q=0.8',
cookie: '**WANT THIS AS A STRING**',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.72 Safari/537.36'
},
},
now I would like to get the cookie as a string to post it in the next request.
let cookies = response.config.headers.cookie;
I am trying to login using puppeteer at https://kith.com/account/login?return_url=%2Faccount
When I login and solve the captcha with audio, it detects me as a bot, so I am trying to change the request headers to see if that helps but cannot find anything on how to change them.
I found this, but it only shows 1 header:
await page.setRequestInterception(true)
page.on('request', (request) => {
const headers = request.headers();
headers['X-Just-Must-Be-Request-In-All-Requests'] = '1';
request.continue({
headers
});
});
You are able to set multiple HTTP headers with the dedicated puppeteer method: page.setExtraHTTPHeaders as well.
E.g.:
await page.setExtraHTTPHeaders({
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36',
'upgrade-insecure-requests': '1',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9,en;q=0.8'
})
await page.goto('...')
header is array you can add many as you want
page.on('request', (request) => {
const headers = request.headers();
headers['X-Just-Must-Be-Request-In-All-Requests'] = '1';
headers['foo'] = 'bar';
headers['foo2'] = 'bar2';
request.continue({
headers
});
});