requests post - and retry mechanism problem - python-3.x

Question

I have following code in my webscraper:
postbody = {'Submit': {}, 'czas_kon2': '', 'czas_pocz2': '', 'num_pacz': '', 'typ': 'wsz'}
post = requests.post(spolka, data=postbody)
data = post.text
I am executing it over 400 webpages in a loop, to scrape data using multiprocessing (8 processes).
data is supposed to contain whole html page for further xml processing.
But out of 400 pages I get 2 that does not return meaningful content. I suspect it is because of heavy load I create. I tried time.sleep(1), time.sleep(10) but no luck here.
How could I ensure that data or post variable always contain whole page, like for 398 working ones?
I tried simple while loop for retry... but it is far from perfect (I was able to get 1 out of remaining 2 pages) afrer one extra attempt.
while len(data) < 1024:
postbody = {'Submit': {}, 'czas_kon2': '', 'czas_pocz2': '', 'num_pacz': '', 'typ': 'wsz'}
post = requests.post(spolka, data=postbody)
data = post.text

Answer 1

I think you should add a request headers.
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0'}
postbody = {'Submit': {}, 'czas_kon2': '', 'czas_pocz2': '', 'num_pacz': '', 'typ': 'wsz'}
post = requests.post(spolka, data=postbody, headers=headers)
and more headers example:
headers = {
'Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
'Host': 'www.google.com',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'X-Requested-With': 'XMLHttpRequest',
'Cookies': '',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0'
}