I am wokirng on windows and with python3.
I used requests module to get to a web page with this code-
requests.get('https://github.com/')
Then i got the SSLError [SSL: CERTIFICATE VERIFY FAILED ] error. Then i turned off the switch for SSL certificates check with verify=False.
import requests
requests.get('https://github.com/', verify=False)
And now it is browsing the site but returns a warning.
Warning (from warnings module):
File "C:\Python27\lib\site-packages\urllib3\connectionpool.py", line 847
InsecureRequestWarning)
InsecureRequestWarning: Unverified HTTPS request is being made. Adding
certificate verification is strongly advised. See:
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
<Response [200]>
Now i am working with a script and i don't want to show this warning to user.I got a way to hide warning but i am not sure how to use it.
warnings.simplefilter("ignore")
But this ignores all warnings and i just want to hide this 'InsecureRequestWarning' particularly. Please guide me how to do this.
So i got the solution to hide the warning. Here is what i did.
import warnings
from requests.packages.urllib3.exceptions import InsecureRequestWarning
warnings.simplefilter('ignore',InsecureRequestWarning)
requests.get('https://github.com/',verify=False)
And this thing solved my problem.
Related
I'm trying to download some files from behind a SSO (Single Sign-On) site. It seems to be SAML authenticated, that's where I'm stuck. Once authenticated I'll be able to perform API requests that return JSON, so no need to interpret/scrape.
Not really sure how to deal with that in mechanicalsoup (and relatively unfamiliar with web-programming in general), help would be much appreciated.
Here's what I've got so far:
import mechanicalsoup
from getpass import getpass
import json
login_url = ...
br = mechanicalsoup.StatefulBrowser()
response = br.open(login_url)
if verbose: print(response)
# provide the username + password
br.select_form('form[id="loginForm"]')
print(br.get_current_form().print_summary()) # Just to see what's there.
br['UserName'] = input('Email: ')
br['Password'] = getpass()
response = br.submit_selected().text
if verbose: print(response)
At this point I get a page telling me javascript is disabled and that I must click submit to continue. So I do:
br.select_form()
response = br.submit_selected().text
if verbose: print(response)
That's where I get a complaint about state information being lost.
Output:
<h2>State information lost</h2>
State information lost, and no way to restart the request<h3>Suggestions for resolving this problem:</h3><ul><li>Go back to the previous page and try again.</li><li>Close the web browser, and try again.</li></ul><h3>This error may be caused by:</h3><ul><li>Using the back and forward buttons in the web browser.</li><li>Opened the web browser with tabs saved from the previous session.</li><li>Cookies may be disabled in the web browser.</li></ul>
The only hits I've found on scraping behind SAML logins are all going with a selenium approach (and sometimes dropping down to requests).
Is this possible with mechanicalsoup?
My situation turned out to require Javascript for login. My original question about getting into SAML auth was not the true environment. So this question has not truly been answered.
Thanks to #Daniel Hemberger for helping me figure that out in the comments.
In this situation MechanicalSoup is not the correct tool (due to Javascript) and I ended up using selenium to get through authenication then using requests.
I am trying to connect to a sharepoint site, however I get the AttributeError: 'NoneType' object has no attribute 'text'.
my code is as follows
import sharepy
s = sharepy.connect("https://example.sharepoint.com/",\
username='username', password='password')
Any idea why I get this error? I am allowed in the site, but not on the https://example.sharepoint.com/ server url, and also not allowed through python. I have tried several other ways in, using urllib.request and also requests. There my issue lies with having the HTTPError: Forbidden. The code run there is:
from sharepoint import SharePointSite, basic_auth_opener
server_url = 'https://example.sharepoint.com/'
opener = basic_auth_opener(server_url, "username",
"password")
site = SharePointSite(server_url, opener)
for sp_lists in site.lists:
print(sp_lists.id)
Seems like I have an issue with permission on the server or could it be something different?
sharepy shows this error if SharePoint returns an error message during the authentication. It is an error in the library because it doesn't know how to handle all error codes.
In your case the site url is wrong - "https://example.sharepoint.com" instead of "https://example.sharepoint.com/".
it might possible that you are missing header or other, please follow below link for more ideas on HTTP Error 403: Forbidden
https://mediatemple.net/community/products/dv/204644980/why-am-i-seeing-a-403-forbidden-error-message
from selenium import webdriver
opt = webdriver.chrome.options.Options()
opt.add_argument("--headless")
opt.add_argument("--disable-gpu")
driver = webdriver.Chrome(chrome_options=opt);
driver.get("https://steamcommunity.com/")
I'm trying to run headless chrome, with Selenium. But keep getting the following error:
[0331/134342.207:INFO:CONSOLE(0)] "The SSL certificate used to load resources from https://steamcommunity-a.akamaihd.net will be distrusted in the future. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information.", source: https://steamcommunity.com/ (0)
I have tried adding the --ignore-certificate-errors flag, but still get the same error.
I'm not completely sure this is the resolution you are looking for, but you can simply use the http URL http://steamcommunity.com/ without the SSL s suffix. This will (I hope) prevent presenting this notification.
Hi I'm Using Robot Framework.
I'm trying to send a Get Request but I get the following warning
C:\Python27\lib\site-packages\urllib3-1.21.1-py2.7.egg\urllib3\connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
Has anyone had this problem? As I've searched for a solution one of the advice was to disable the warning, but how to do it?
Thanks in Advance,
The following test case runs perfectly, without showing the warning.
Please note the option verify=true on the Create Session keyword.
*** Settings ***
Library Collections
Library String
Library RequestsLibrary
Library OperatingSystem
*** Test Cases ***
Get Requests
[Tags] get
Create Session google http://www.google.com verify=true
Create Session github https://api.github.com verify=true
${resp}= Get Request google / timeout=5
Should Be Equal As Strings ${resp.status_code} 200
${resp}= Get Request github /users/bulkan timeout=5
Should Be Equal As Strings ${resp.status_code} 200
Dictionary Should Contain Value ${resp.json()} Bulkan Evcimen
Create Session along with verify=true, by default it's verify=false
Create Session test_session http://www.example.com **verify=true**
I plan on using the sample-app.py as a baseline of what I am building out and then expanding it from there. Just want to get comfortable with the instagram API and build out from there.
I am trying to use the sample-app.py provided with python-instagram. I have registered an application on instagrams website. I set it up using the default redirect uri from sample-app.py:
http://localhost:8515/oauth_callback .
I was able to authorize my instagram account to use the app, but when I click on any of the links, I get an error about the acccess-token.
When you look at the python command-line window that stays open, I get the following error:
"check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED"
It appears that when the sample app is processing the lines below, it is trying to connect to instagram, but is not able to because SSL in local host is not set up properly. How do I set up SSL so i do not get the above error?
access_token, user_info = unauthenticated_api.exchange_code_for_access_token(code)
if not access_token:
return 'Could not get access token'
api = client.InstagramAPI(access_token=access_token)
request.session['access_token'] = access_token
print ("access token="+access_token)
There are a few steps to solve this problem (it appears that it is actually several problems in aggregate causing this issue):
use openssl to create a ssl certificate and save cert to the same location as your python script. Download open ssl here: http://slproweb.com/products/Win32OpenSSL.html
You need to tweak bottle so that it will support ssl. Do so by adding the following lines in run in class WSGIRefServer(ServerAdapter):
import ssl
srv.socket = ssl.wrap_socket (
srv.socket,
certfile='server.pem', # path to certificate
server_side=True)
There is a bug in python 3 and above(https://github.com/jcgregorio/httplib2/issues/173). I am using 3.4, so the bug could be fixed in 3.5. In the instagram/oauth2.py file, change all disable_ssl_certificate_validation=False to True.