Python selenium proxy firefox - python-3.x

Does anyone have any working code for python selenium, firefox, proxy server?
There are many methods that are described on various solutions but none of them seem to work.
I'm trying to create headless firefox, then call "whatismyip.com" to test the IP. However, I always get current IP.
opts = FirefoxOptions()
opts.add_argument("--headless")
myProxy = "xxx.xxx.xxx.xxx:xxxx"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"httpProxy": myProxy,
"sslProxy": myProxy,
"proxyType": "MANUAL",
}
browser = webdriver.Firefox(options=opts)
browser.get( 'https://www.whatismyip.com/')

It would be helpful to have more information: What versions you are using and the setup. The response that was provided fix is for Chrome and not Firefox. If you are using 4.0 selenium then the DesiredCapabilities has been deprecated Legacy Selenium Desired Capabilities. The options and services should be used. The following worked with Seleinum 4.0 in and latest python library. Using binary_location with portable Firefox and Windows 10 OS. The following will change all the proxy options just remove the ones' you do not want.
from selenium import webdriver
from selenium.webdriver.common.proxy import *
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
options=Options()
s = Service('c:\\webdriver\\geckodriver.exe')
options.binary_location = r'C:\Firefox64\firefox.exe' # FF installed locaiton
myProxy = "86.111.144.194:3128"
options.proxy = Proxy({
'proxyType': ProxyType.MANUAL,
"socksVersion": 5,
'httpProxy': myProxy,
'sslProxy': myProxy,
"socksProxy": myProxy,
'noProxy':''})
driver = webdriver.Firefox(service=s, options=options)
driver.set_page_load_timeout(30)
driver.get('http://ifconfig.io')
I would use a different service to check the IP as some will want additional verification if going through TOR.

Related

Why is Selenium overriding my firefox profile

I am making python selenium script to automate some google search with firefox.
I am using python 3.7 on Windows 10 64b.
Something weird happened. When I run my python script, it’is fine.
When I compile it with Nuitka and I run the exe, Firefox is opening with some proxy added (127.0.01:53445).
So I added this line:
profile.set_preference("network.proxy.type", 0)
And again, the script run fine but when I compile it, the exe opens Firefox with a proxy.
It is a pain as this 127.0.01 proxy creates an issue to open google and my program is broken.
Does anyone already faced this weird behaviour of selenium?
Without seeing what code you are using relevant to webdriver I am mostly guessing here. I would suggest rotating the proxy as well.
import requests
from bs4 import BeautifulSoup
from random import choice
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
def proxy_generator():
# REQUIRED - pip install html5lib
response = requests.get("https://sslproxies.org/")
soup = BeautifulSoup(response.content, 'html5lib')
proxy = {'https': choice(list(map(lambda x: x[0] + ':' + x[1], list(zip(map(lambda x: x.text, soup.findAll('td')[::8]), map(lambda x: x.text, soup.findAll('td')[1::8]))))))}
return proxy
PROXY = proxy_generator()# Commented out proxy option "58.216.202.149:8118"
firefox_capabilities['proxy'] = {
"proxyType": "MANUAL",
"httpProxy": PROXY,
"ftpProxy": PROXY,
"sslProxy": PROXY
}
driver = webdriver.Firefox(capabilities=firefox_capabilities)

--headless is not an option in Chrome WebDriver for Selenium

I would like to have Selenium run a headless instance of Google Chrome to mine data from certain websites without the UI overhead. I downloaded the ChromeDriver executable from here and copied it to my current scripting directory.
The driver appears to work fine with Selenium and is able to browse automatically, however I cannot seem to find the headless option. Most online examples of using Selenium with headless Chrome go something along the lines of:
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'`
driver = webdriver.Chrome(executable_path=os.path.abspath(“chromedriver"), chrome_options=chrome_options)
driver.get("http://www.duo.com")`
However when I inspect the possible arguments for the Selenium WebDriver using the command chromedriver -h this is what I get:
D:\Jobs\scripts>chromedriver -h
Usage: chromedriver [OPTIONS]
Options
--port=PORT port to listen on
--adb-port=PORT adb server port
--log-path=FILE write server log to file instead of stderr, increases log level to INFO
--log-level=LEVEL set log level: ALL, DEBUG, INFO, WARNING, SEVERE, OFF
--verbose log verbosely (equivalent to --log-level=ALL)
--silent log nothing (equivalent to --log-level=OFF)
--append-log append log file instead of rewriting
--replayable (experimental) log verbosely and don't truncate long strings so that the log can be replayed.
--version print the version number and exit
--url-base base URL path prefix for commands, e.g. wd/url
--whitelisted-ips comma-separated whitelist of remote IP addresses which are allowed to connect to ChromeDriver
No --headless option is available.
Does the ChromeDriver obtained from the link above allow for headless browsing?
--headless is not argument for chromedriver but for Chrome. --headless Run chrome in headless mode, i.e., without a UI or display server dependencies. ChromeDriver is a separate executable that WebDriver uses to control Chrome and Webdriver is a a collection of language specific bindings to drive a browser.
I am able to run in headless mode with this set of options. I hope this will help:
from bs4 import BeautifulSoup, NavigableString
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import requests
import re
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
browser = webdriver.Chrome(chrome_options=options) # see edit for recent code change.
browser.implicitly_wait(20)
Update 12 Aug 2019:
old : browser = webdriver.Chrome(chrome_options=options)
new : browser = webdriver.Chrome(options=options)
Try
options.headless=True
The following is how I set up my headless chrome
options = webdriver.ChromeOptions()
options.headless=True
options.add_argument('window-size=1920x1080')
prefs = {
"download.default_directory": r"C:\FilePath\Download",
"download.prompt_for_download": False,
"download.directory_upgrade": True}
options.add_experimental_option('prefs', prefs)
chromedriver = (r"C:\Filepath\chromedriver.exe")
--headless is not argument for chromedriver but Chrome, you can see more arguments or Command Line Switches for chrome here

proxy server refusing connections when trying to run tor browser with selenium (using TorBrowserDriver not profile and binary) [duplicate]

I am trying to connect to a Tor browser but get an error stating "proxyConnectFailure" any ideas I have tried multiple attempts to get into the basics of Tor browser to get it connected but all in vain if any could help life could be saved big time:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary(r"C:\Users\Admin\Desktop\Tor Browser\Browser\firefox.exe")
profile = FirefoxProfile(r"C:\Users\Admin\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default")
# Configured profile settings.
proxyIP = "127.0.0.1"
proxyPort = 9150
proxy_settings = {"network.proxy.type":1,
"network.proxy.socks": proxyIP,
"network.proxy.socks_port": proxyPort,
"network.proxy.socks_remote_dns": True,
}
driver = webdriver.Firefox(firefox_binary=binary,proxy=proxy_settings)
def interactWithSite(driver):
driver.get("https://www.google.com")
driver.save_screenshot("screenshot.png")
interactWithSite(driver)
To connect to a Tor Browser through a FirefoxProfile you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile= profile, executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("http://check.torproject.org")
Browser Snapshot:
You can find a relevant discussion in How to use Tor with Chrome browser through Selenium
I would like to expand on #DebanjanB answer by adding the Linux counterpart:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen('some/path/tor-browser_en-US/Browser/start-tor-browser')
# in my case, I installed it under a folder tor-browser_en-US after
# downloading and extracting it from
# https://www.torproject.org/download/ for linux
profile = FirefoxProfile(
'some/path/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox'
# /usr/bin/firefox is default location of firefox - for me anyway
driver = webdriver.Firefox(
firefox_profile=profile, options=firefox_options,
executable_path='wherever/you/installed/geckodriver')
# I keep my geckodriver(s) in a special folder sorted by versions.
# Geckodriver downloadable here:
# https://github.com/mozilla/geckodriver/releases/
driver.get("http://check.torproject.org")
The verified answer does not work in case of opening dot onion sites(I believe that's something to do with tor network which is not allowing access to normal firefox).
As for the latest tor browser (from the tor browser bundle), starting it using selenium causes some error due to which the browser cannot start tor proxy itself causing proxy and timeout errors(doesn't matter if tor proxy is started by python or manually or not started at all). This could also be due to port 9050 or 9150 being used by tor proxy and not being available to browser's tor instance but this does not explain the error caused when no instance of tor proxy is running.
The solution i have found is to start the tor proxy as normal, manually or using os.popen("tor.exe") and configure tor browser to not start tor proxy.
here's the code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
os.popen(r'e:\\bla\\bla\\bla\\tor\\Tor\\tor.exe')
binary=FirefoxBinary(r'e:\\bla\\bla\\bla\\Tor Browser\\Browser\\firefox.exe')
fp=FirefoxProfile(r'e:\\foo\\bar\\bla\\Tor Browser\\Browser\\TorBrowser\\Data\\Browser\\profile.default')
fp.set_preference('extensions.torlauncher.start_tor',False)#note this
fp.set_preference('network.proxy.type',1)
fp.set_preference('network.proxy.socks', '127.0.0.1')
fp.set_preference('network.proxy.socks_port', 9050)
fp.set_preference("network.proxy.socks_remote_dns", True)
fp.update_preferences()
driver = webdriver.Firefox(firefox_profile=fp,firefox_binary=binary)
driver.get("http://check.torproject.org")
driver.get('https://www.bbcnewsv2vjtpsuy.onion/')
*note fp.set_preference('extensions.torlauncher.start_tor',False) on line 10 is being used to configure tor to not start its own tor instance so that it uses the proxy config and tor instance started above.
lo and behold as the tbb starts working like normal firefox bot browser

Python 3.X Selenium Headless Chrome Error 1016

I want to run headless Chrome but it keeps crashing with the following error:
[0814/155351.062:ERROR:gpu_process_transport_factory.cc(1016)] Lost UI shared context.
I have no idea what is going on here. My code is simple and works with the GUI.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless') # Enable headless
options.add_argument('--disable-gpu') # Required as I am on Windows
driver = webdriver.Chrome(chromepath,chrome_options=options) #chromepath is a raw string to the driver
driver.get('https://en.wikipedia.org/wiki/Wiki')
driver.quit()
print('Successfully opened and closed headless Chrome')
All dependencies are current:
Chrome version: 68.0.3440.106(Official Build)(64-bit)
Selenium: 3.14.0
Chromedriver: 2.41

How to suppress console error/warning/info messages when executing selenium python scripts using chrome canary

I am running python script (complete script link below) for selenium test using Chrome Canary. The test seems to be running fine, however, there are lots of error/warning/info messages displayed on the console.
Is there a way to suppress these messages? I have tried:
chrome_options.add_argument("--silent"), but does not help. I am not able to find the right solution. Appreciate any help.
Python script : Example script provided here
Python: 3.6.3
Selenium: 3.6.0
Chrome Canary: 63.0.3239.5 (64 bit)
ChromeDriver : 2.33
Try options.add_argument('log-level=3').
log-level:
Sets the minimum log level.
Valid values are from 0 to 3:
INFO = 0,
WARNING = 1,
LOG_ERROR = 2,
LOG_FATAL = 3.
default is 0.
If "--log-level" doesn't work for you (as of 75.0.3770.100 it didn't for me), this should:
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(executable_path='<path-to-chrome>', options=options)
See https://bugs.chromium.org/p/chromedriver/issues/detail?id=2907#c3
Code copied from Python selenium: DevTools listening on ws://127.0.0.1
Works for me in Python/Chrome...
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--log-level=3')
You can take help of below link.
List of Chromium Command Line Switches
"--log-level" sets the minimum log level. Valid values are from 0 to 3: INFO = 0, WARNING = 1, LOG_ERROR = 2, LOG_FATAL = 3.
I have just tested this one, it works for me (C#):
ChromeOptions options = new ChromeOptions();
options.AddArguments("--headless", "--log-level=3");
RemoteWebDriver driver = new ChromeDriver(options);
import os
os.environ['WDM_LOG_LEVEL'] = '0'
That code hides the console output for from webdriver_manager.chrome import ChromeDriverManager console outputs

Resources