logging into a twitter using python3 and requests - python-3.x

Question

I have a project that I am working on, and the requirements are to login to a website using a username and password. I have to do it in python, and then be able to access a part of the site only accessible to people who are logged in. I have tried a few variations of coding to do this, and haven't been able to successfully log in yet. Here is my coding:
the function to login to it:
def session2(url):
#r = requests.get(url)
#ckies = []
#print("here are the cookies for twitter:\n")
#for cky in r.cookies:
# print(cky.name, cky.value)
# ckies.append(cky)
s = requests.Session()
session = s.get(url, verify=False)
print("\nheaders from site\n")
print(session.headers)
tree = html.fromstring(session.text)
# extract the auth token needed to login along with username and password
auth_token = list(set(tree.xpath("//input[#name='authenticity_token']/#value")))[0]
uname = "username"
pword = "password"
username = 'session[username_or_email]'
password = 'session[password]'
# payload = {name of username variable : string you want, name of password variable:
# string you want, name of auth token: string gotten from session
payload = dict(username = uname, password = pword , authenticity_token = auth_token)
header = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'}
#do post request
# might have to change headers to be a header for chrome
response = s.post(
url,
data = payload,
#headers = dict(referer = url)
headers = header
)
print("\nheaders post\n")
print(response.request.headers)
session = s.get("http://www.twitter.com/username/followers", verify=False)
print("\nheaders get\n")
print(session.headers)
print("\nhtml doc\n")
print(session.text)
return session
code to call it:
url = "http://www.twitter.com/login"
sessions = session2(url)
the username on the site looks like this when you inspect it:
<input class="js-username-field email-input js-initial-focus" type="text" name="session[username_or_email]" autocomplete="on" value="" placeholder="Phone, email or username">
and the password section/token section look like this:
<input class="js-password-field" type="password" name="session[password]" placeholder="Password">
<input type="hidden" value="ef25cb09a8c7fe16c54e3df099e206e605b1170a" name="authenticity_token">
I know the auth token changes, which is why i have it get it from the function. When I try to run this, it just goes to the main page rather than the page i need.
One problem I think is that when I print out the header that I send in the post, it says:
{'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Accept': '/', 'User-Agent': 'python-requests/2.9.1'}
which I thought I changed to chrome's header, but it doesn't seem to stick.
Also, I know there is a way if I use Oauth, but I'm not allowed to use that, i have to do it based on being able to login like I'm using a browser.
Can you tell me if there is anything wrong with what I've done, as well as any hints on how to fix it? I've tried other stack overflow problems using requests and logging in, but those didn't work either.
EDIT: ok, i did a response.request.headers, and it came out with the right header, i think, so i don't think that is the problem
header it prints:
{'Accept': '*/*', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36', 'Cookie': '_twitter_sess=some huge amount of number/letters; guest_id=v1%3A147509653977967101', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate'}

Answer 1

This will log you in:
import requests
from bs4 import BeautifulSoup
username = "uname"
password = "pass"
# login url
post = "https://twitter.com/sessions"
url = "https://twitter.com"
data = {"session[username_or_email]": username,
"session[password]": password,
"scribe_log": "",
"redirect_after_login": "/",
"remember_me": "1"}
with requests.Session() as s:
r = s.get(url)
# get auth token
soup = BeautifulSoup(r.content, "lxml")
AUTH_TOKEN = soup.select_one("input[name=authenticity_token]")["value"]
# update data, post and you are logged in.
data["authenticity_token"] = AUTH_TOKEN
r = s.post(post, data=data)
print(r.content)
You can see if we run it using my own account, we get my name from my profile:
In [30]: post = "https://twitter.com/sessions"
In [31]: url = "https://twitter.com"
In [32]: data = {"session[username_or_email]": username,
....: "session[password]": password,
....: "scribe_log": "",
....: "redirect_after_login": "/",
....: "remember_me": "1"}
In [33]: with requests.Session() as s:
....: r = s.get(url)
....: soup = BeautifulSoup(r.content, "lxml")
....: AUTH_TOKEN = soup.select_one("input[name=authenticity_token]")["value"]
....: data["authenticity_token"] = AUTH_TOKEN
....: r = s.post(post, data=data)
....: soup = BeautifulSoup(r.content, "lxml")
....: print(soup.select_one("b.fullname"))
....:
<b class="fullname">Padraic Cunningham</b>
Just be aware each time you login, you will the We noticed a recent login for your account ... email.