My goal is to log-in to Gmail, serialize the cookies, exit the browser, then open a new browser, load the saved cookies, and check my email without needing to enter my log-in details. Pretty straight-forward, and I've been able to do this with almost every website I've tried. However, Gmail forces me to sign in again, each time.
Here's my code:
from splinter import Browser
import selenium
import pickle
def export_cookies(browser, the_name):
yummy = browser.cookies.all(verbose=True)
location = 'cookies/' + the_name
pickle_save(yummy, location)
print(the_name, "saved", len(yummy))
def pickle_save(obj, location):
file_name = location
file_object = open(file_name, 'wb')
pickle.dump(obj, file_object)
file_object.close()
def pickle_load_account(cookie_file_name):
try:
return pickle.load(open(cookie_file_name, "rb"))
except FileNotFoundError:
return 0
def browser_add_cookies(browser, cookies):
if len(cookies) > 0:
for cookie in cookies:
browser.cookies.add({cookie['name']: cookie['value']})
print("-----", len(cookies), " cookies added, reloading")
browser.visit('https://mail.google.com/mail/u/0/#inbox')
else:
print("No cookies to load. Error.")
browser = Browser('firefox')
browser.visit('https://mail.google.com/mail/u/0/#inbox')
cookie_file = "cookies/name"
load_cookies = pickle_load_account(cookie_file)
browser_add_cookies(browser, load_cookies)
browser.visit('https://mail.google.com/mail/u/0/#inbox')
input("Save cookies?")
export_cookies(browser, "name")
This code assumes the cookies were already saved, but then re-saves them in the end, so a second run (if you sign in manually the first time).
My guess is that Gmail somehow uses a more advanced method of cookie recognition?
You are replicating cookies on for one domain. You need to replicate for below domains as well
accounts.google.com
mail.google.com
And may be even more, see the screenshot below on fresh login
Example code for a browser wrapper to make set cookies easier:
class SplinterBrowserWrapper:
def __init__(self, splinter_browser):
b = self.browser = splinter_browser
b.visit('https://' + ROOT_DOMAIN)
def add_cookies(self, cookies: dict):
self.browser.cookies.add(cookies)
for cookie in self.list_cookies():
cookie['domain'] = ROOT_DOMAIN
self.add_single_cookie_dict(cookie)
self.browser.reload()
def add_single_cookie_dict(self, single_cookie_dict):
self.browser.driver.add_cookie(single_cookie_dict)
def get_cookies(self):
return self.browser.cookies.all()
def list_cookies(self):
return self.browser.driver.get_cookies()
routine:
visit mail.google.com
mail.google.com ask accounts.google.com: this guy login or not?
accounts answer: NO!
mail: OK, just another wandering guest...
guest is not happy, tries to set cookies via splinter:
visit mail.google.com and set cookies
cookies are set, but bound with sub-domain mail.google.com (run browser.driver.get_cookies() and check the returned list)
sub-domain accounts.google.com still no cookies
mail ask accounts the same question and get the same NO answer
emmm...
guest knows the trick now, guest tries again:
visit accounts.google.com and set cookies
visit mail.google.com
mail ask accounts, get answer YES
mail login
N~I~C~E~
another routine, another website called bilibili:
visit www.bilibili.com
www ask account.bilibili.com
account has no logged in cookies, account redirect browser to passport.bilibili.com
passport ask user for username & password
passport get correct auth, set cookies, and redirect back to home page as login
splinter chanllege bilibili, round 1:
visit www.bilibili.com, set cookies, reload page, not login
visit account.bilibili.com, be redirected to passport.bilibili.com
set cookies on passport
but passport not want cookies, passport NEED username & password
passport stay put, refuse to redirect to account or www
account still no cookies
visit account again, redirected to passport again, dead loop
splinter challenge round 2:
visit www.bilibili.com, set cookies, cookies are bound with www.bilibili.com
dig into underlying selenium driver, list all cookies with browser.driver.get_cookies(), every single cookie in this list is a dict, with keys name, value, path, domain, secure, httpOnly
iterate this cookies list, for every cookie dict, change value of domain to the parent domain bilibili.com, with no sub-domain
set these manipulated cookies back into selenium drive one by one, via browser.driver.add_cookie
now account see these cookies bound with its parent domain bilibili.com
account is obedient, account accpet these cookies, account now answer YES when anyone ask if user is login
reload the page and user login
Related
I started to use SQLModel that is created by the same person as FastAPI, but I cannot seem to find how to combine authentication/authorization to get the logged-in user and then get current user from that with SQLModel.
I can authenticate a user by just checking it against a database as below, but then how can I keep this session alive so that I can do other stuff with it such as get current user?
I am purely testing with localhost:8000/docs, so I thought maybe I need to create some Jinja2 templates to test it out in a browser, but not sure?
def login_test(input_email: str, input_password: str, session: Session = Depends(get_session)):
# Check if user exists or not
if not check_user_exists(input_email=input_email):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST,
detail="User does not exist, please create an account")
# Check if password is ok
if not check_hashed_password(input_password, session.query(User).filter(User.email == input_email).first().password):
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST,
detail="Password is incorrect")
# Login based on email and password if both ok
return {"Message": "Login successful"}```
I'm trying to download some files from behind a SSO (Single Sign-On) site. It seems to be SAML authenticated, that's where I'm stuck. Once authenticated I'll be able to perform API requests that return JSON, so no need to interpret/scrape.
Not really sure how to deal with that in mechanicalsoup (and relatively unfamiliar with web-programming in general), help would be much appreciated.
Here's what I've got so far:
import mechanicalsoup
from getpass import getpass
import json
login_url = ...
br = mechanicalsoup.StatefulBrowser()
response = br.open(login_url)
if verbose: print(response)
# provide the username + password
br.select_form('form[id="loginForm"]')
print(br.get_current_form().print_summary()) # Just to see what's there.
br['UserName'] = input('Email: ')
br['Password'] = getpass()
response = br.submit_selected().text
if verbose: print(response)
At this point I get a page telling me javascript is disabled and that I must click submit to continue. So I do:
br.select_form()
response = br.submit_selected().text
if verbose: print(response)
That's where I get a complaint about state information being lost.
Output:
<h2>State information lost</h2>
State information lost, and no way to restart the request<h3>Suggestions for resolving this problem:</h3><ul><li>Go back to the previous page and try again.</li><li>Close the web browser, and try again.</li></ul><h3>This error may be caused by:</h3><ul><li>Using the back and forward buttons in the web browser.</li><li>Opened the web browser with tabs saved from the previous session.</li><li>Cookies may be disabled in the web browser.</li></ul>
The only hits I've found on scraping behind SAML logins are all going with a selenium approach (and sometimes dropping down to requests).
Is this possible with mechanicalsoup?
My situation turned out to require Javascript for login. My original question about getting into SAML auth was not the true environment. So this question has not truly been answered.
Thanks to #Daniel Hemberger for helping me figure that out in the comments.
In this situation MechanicalSoup is not the correct tool (due to Javascript) and I ended up using selenium to get through authenication then using requests.
I've been having this issue for a while now. I'm trying to add a Sign in through Steam button, which upon login, not only retrieves the user's ID, but also validates the signature. Steam uses OpenID 2.0.
I have followed the documentation here. I have followed these steps carefully, spending the better part of my day on trying to figure this out. My code is this:
let s = data['openid.signed'].split(',');
let x = Buffer.from(s.map(x => `${x}:${data['openid.' + x]}`).join('\n') + '\n', 'utf8');
let c = crypto.createHash('sha1').update(x).digest('base64');
console.log(x.toString('utf8')); // This is the key:value string
console.log(c); // This is the final result; the generated signature
Where data is the response given from the OpenID provider.
Logging x (key:value pair string) gives the expected output of:
signed:signed,op_endpoint,claimed_id,identity,return_to,response_nonce,assoc_handle
op_endpoint:https://steamcommunity.com/openid/login
claimed_id:https://steamcommunity.com/openid/id/765611981[1234567]
identity:https://steamcommunity.com/openid/id/765611981[1234567]
return_to:http://127.0.0.1:8000/resolve
response_nonce:2018-12-01T17:53: [some_hash]=
assoc_handle:1234567890
However, my generated hash c does not match the given signature, openid.sig. Note that I use a \n at the end of the above key:value pair string, as that is how I interpreted the documentation.
Note. The reason why I need authentication is that I want to connect the Steam account to an account on my website, and being logged in via Steam gives you full access to your account on my website, meaning that it's of utter importance that a user cannot simply enter another users id and get access to their account (replay attack). Because of this, I need to somehow validate the signature.
I have never worked with OpenID before, so please excuse any foolish mistakes of mine. I highly recommend reading the documentation that is linked above, so that you can verify what I am doing is right.
Kinds regards,
Initial Request
Make your Steam login button link to
https://steamcommunity.com/openid/login?openid.ns=http://specs.openid.net/auth/2.0&openid.claimed_id=http://specs.openid.net/auth/2.0/identifier_select&openid.identity=http://specs.openid.net/auth/2.0/identifier_select&openid.return_to=https://mywebsite.com&openid.realm=https://mywebsite.com&openid.mode=checkid_setup
and replace the openid.return_to and openid.realm query string parameters.
openid.return_to: This is the URL that Steam will redirect to upon successful login with appended query string parameters.
openid.realm The URL Steam will ask the user to trust. It will appear as a message like this when the user is on the Steam login page: Sign into {openid.realm} using your Steam account. Note that {openid.realm} is not affiliated with Steam or Valve.
Handling the response
Upon successful login, Steam will redirect to a URL like
https://mywebsite.com/?openid.ns=http://specs.openid.net/auth/2.0&openid.mode=id_res&openid.op_endpoint=https://steamcommunity.com/openid/login&openid.claimed_id=https://steamcommunity.com/openid/id/76561198002516729&openid.identity=https://steamcommunity.com/openid/id/76561198002516729&openid.return_to=https:/%mywebsite.com&openid.response_nonce=2020-08-27T04:44:16Zs4DPZce8qc+iPCe8JgQKB0BiIDI=&openid.assoc_handle=1234567890&openid.signed=signed,op_endpoint,claimed_id,identity,return_to,response_nonce,assoc_handle&openid.sig=W0u5DRbtHE1GG0ZKXjerUZDUGmc=
To verify the user, make a call from your backend to https://steamcommunity.com/openid/login copying every query string parameter from that response with one exception: replace &openid.mode=id_res with &openid.mode=check_authentication. So the final call will be to this URL:
https://steamcommunity.com/openid/login?openid.ns=http://specs.openid.net/auth/2.0&openid.mode=check_authentication&openid.op_endpoint=https://steamcommunity.com/openid/login&openid.claimed_id=https://steamcommunity.com/openid/id/76561198002516729&openid.identity=https://steamcommunity.com/openid/id/76561198002516729&openid.return_to=https://mywebsite.com&openid.response_nonce=2020-08-28T04:44:16Zs4DPZce8qc+iPCe8JgQKB0BiIDI=&openid.assoc_handle=1234567890&openid.signed=signed,op_endpoint,claimed_id,identity,return_to,response_nonce,assoc_handle&openid.sig=W0u5DRbtHE1GG0ZKXjerUZDUGmc=
Steam will return a text/plain response like this:
ns:http://specs.openid.net/auth/2.0
is_valid:true
If true the user is valid, false invalid. Note this call will only return true once and subsequent calls with the same parameters will always return false. From here, you can decide how to maintain the user being logged in (such as creating a unique cookie) and return a redirect response to something like your site's homepage, last page before they clicked the Steam login button, or user detail page, etc...
I using flask-login for authorizing in my site.
If I login once, flask do not offer login again, but I want users to be logged in every time they log on to the site.
P.S.: I try to use remember=False in login_user() function, but it didn't help.
What can be the correction that need to be done?
I now see case for fresh loging like password change etc It is provided for by the use of fresh_login_required
flask_login.login_fresh()
This returns True if the current login is fresh. So your views protected with login required you can do something like
if not login_fresh():
#redirect to your login page with a request to reauthenticate
or (and am using change-password just as an example you can use it on any and all views that require fresh login everytime)
from flask.ext.login import fresh_login_required
#app.route("/change-password")
#fresh_login_required
def change_password():
#do stuff here
If the user is not authenticated, LoginManager.unauthorized() is called as normal. If they are authenticated, but their session is not fresh, it will call LoginManager.needs_refresh() instead. (In that case, you will need to provide a LoginManager.refresh_view.) which you can do as below as per the docs
login_manager.refresh_view = "accounts.reauthenticate"
login_manager.needs_refresh_message = (
u"To protect your account, please reauthenticate to access this page."
)
login_manager.needs_refresh_message_category = "info"
If what you are looking at is logging someone out after lets say 5 minutes of inactivity for which this question and this question gives you a very good answer, so you would do it like this
from datetime import timedelta
from flask import session, app
#make the session permanent and set expiry period
session.permanent = True
app.permanent_session_lifetime = timedelta(minutes=5)
#everytime a user visits, modify the session so that you know they are still active
#app.before_request
def func():
session.modified = True
You can make the lifetime very small for a start for testing purposes
I am using SocialAuth libraries in my JSF application for providing 'login with google/facebook'. As shown below it requires me to stores the SocialAuthManager object ('manager') in the session and then redirect to 'google/facebook' URL
//Create an instance of SocialAuthManager and set config
SocialAuthManager manager = new SocialAuthManager();
manager.setSocialAuthConfig(config);
// URL of YOUR application which will be called after authentication
String successUrl= "http://opensource.brickred.com/socialauthdemo/socialAuthSuccessAction.do";
// get Provider URL to which you should redirect for authentication.
// id can have values "facebook", "twitter", "yahoo" etc. or the OpenID URL
String url = manager.getAuthenticationUrl(id, successUrl);
// Store in session
session.setAttribute("authManager", manager);
Then get the 'manager' from session on succssfull/failure redirection from facebook/redirect as shown below:
// get the social auth manager from session
SocialAuthManager manager = (SocialAuthManager)session.getAttribute("authManager");
// call connect method of manager which returns the provider object.
// Pass request parameter map while calling connect method.
AuthProvider provider = manager.connect(SocialAuthUtil.getRequestParametersMap(request));
// get profile
Profile p = provider.getUserProfile();
The problem is if I am already logged in to facebook or google in a one of the 'tab' of the browser then this works perfectly OK. But if I am not logged in already then session becomes NULL and consequently 'manager' as well.
In other words if redirection from 'my application to facebook to my application' happens then it fails. If I am already logged in to facebook then redirection does not happens and it works.
Can someone help?
NOTE: this works perfectly well in case of IE but does not work in case of Chrome & Mozila
the reason for this behavior is that you are calling the redirected page from different domain so when page redirection happens your session data is lost.
please have a look at this link
http://31stdimension.blogspot.in/2012/04/how-to-connect-facebook-using-jsfjava.html