Irregular behaivour of URL Link - Multiple pages webscraping - python-3.x

Question

I am writing a Python code to get data from a sharepoint website using Beautiful Soup.
Each page has 10 rows of details. So I should be collecting all the links up to the last page and then get the entire list of data I need.
Issues
When I am trying to open the page2 urllink Using Python code, it is still opening the page1 (base url) link.
When I open the base url from browser (page1) link and from there using next button, I am able to navigate to page2. But the same when I open a new tab and directly copy paste the page2 link, it refreshes and opens and page1 (base url) link.
Code:
import requests
from requests_ntlm import HttpNtlmAuth
session = requests.Session()
session.auth = HttpNtlmAuth('username','password')
r = session.get("UrlLinkOfPage2")
print(r.status_code)
print(r.content)

Answer 1

The issue with some websites is that they are expecting some special headers to be sent, so open the site home page in your browser (login if required), open the network tab in developer tools and inspect the request your browser make to access the second page. Then copy all the headers it is sending and make a dictionary in your python code containing those headers like:
my_headers = {
'some-header': value,
'another-header': another-value
}
then use requests library to send those headers when requesting the page like:
response = session.get(second_page_url, headers=my_headers)