I am making python selenium script to automate some google search with firefox.
I am using python 3.7 on Windows 10 64b.
Something weird happened. When I run my python script, it’is fine.
When I compile it with Nuitka and I run the exe, Firefox is opening with some proxy added (127.0.01:53445).
So I added this line:
profile.set_preference("network.proxy.type", 0)
And again, the script run fine but when I compile it, the exe opens Firefox with a proxy.
It is a pain as this 127.0.01 proxy creates an issue to open google and my program is broken.
Does anyone already faced this weird behaviour of selenium?
Without seeing what code you are using relevant to webdriver I am mostly guessing here. I would suggest rotating the proxy as well.
import requests
from bs4 import BeautifulSoup
from random import choice
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
def proxy_generator():
# REQUIRED - pip install html5lib
response = requests.get("https://sslproxies.org/")
soup = BeautifulSoup(response.content, 'html5lib')
proxy = {'https': choice(list(map(lambda x: x[0] + ':' + x[1], list(zip(map(lambda x: x.text, soup.findAll('td')[::8]), map(lambda x: x.text, soup.findAll('td')[1::8]))))))}
return proxy
PROXY = proxy_generator()# Commented out proxy option "58.216.202.149:8118"
firefox_capabilities['proxy'] = {
"proxyType": "MANUAL",
"httpProxy": PROXY,
"ftpProxy": PROXY,
"sslProxy": PROXY
}
driver = webdriver.Firefox(capabilities=firefox_capabilities)
Related
I am trying to run a selenium test but getting the above error. I think this is a compatibility issue but I am not sure as I have very basic knowledge of selenium. My geckodriver version is 0.31.0 and firefox browser version is 100 and selenium version is selenium == 4.1.0. Any idea how I can fix this?
This is my conftest.py
import pytest
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as ff_service
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver_path = ff_service(r"/usr/bin/geckodriver")
driver = webdriver.Firefox(service=driver_path,options = options) #getting error at this line
driver.maximize_window()
Does anyone have any working code for python selenium, firefox, proxy server?
There are many methods that are described on various solutions but none of them seem to work.
I'm trying to create headless firefox, then call "whatismyip.com" to test the IP. However, I always get current IP.
opts = FirefoxOptions()
opts.add_argument("--headless")
myProxy = "xxx.xxx.xxx.xxx:xxxx"
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"httpProxy": myProxy,
"sslProxy": myProxy,
"proxyType": "MANUAL",
}
browser = webdriver.Firefox(options=opts)
browser.get( 'https://www.whatismyip.com/')
It would be helpful to have more information: What versions you are using and the setup. The response that was provided fix is for Chrome and not Firefox. If you are using 4.0 selenium then the DesiredCapabilities has been deprecated Legacy Selenium Desired Capabilities. The options and services should be used. The following worked with Seleinum 4.0 in and latest python library. Using binary_location with portable Firefox and Windows 10 OS. The following will change all the proxy options just remove the ones' you do not want.
from selenium import webdriver
from selenium.webdriver.common.proxy import *
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
options=Options()
s = Service('c:\\webdriver\\geckodriver.exe')
options.binary_location = r'C:\Firefox64\firefox.exe' # FF installed locaiton
myProxy = "86.111.144.194:3128"
options.proxy = Proxy({
'proxyType': ProxyType.MANUAL,
"socksVersion": 5,
'httpProxy': myProxy,
'sslProxy': myProxy,
"socksProxy": myProxy,
'noProxy':''})
driver = webdriver.Firefox(service=s, options=options)
driver.set_page_load_timeout(30)
driver.get('http://ifconfig.io')
I would use a different service to check the IP as some will want additional verification if going through TOR.
I am trying to connect to a Tor browser but get an error stating "proxyConnectFailure" any ideas I have tried multiple attempts to get into the basics of Tor browser to get it connected but all in vain if any could help life could be saved big time:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary(r"C:\Users\Admin\Desktop\Tor Browser\Browser\firefox.exe")
profile = FirefoxProfile(r"C:\Users\Admin\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default")
# Configured profile settings.
proxyIP = "127.0.0.1"
proxyPort = 9150
proxy_settings = {"network.proxy.type":1,
"network.proxy.socks": proxyIP,
"network.proxy.socks_port": proxyPort,
"network.proxy.socks_remote_dns": True,
}
driver = webdriver.Firefox(firefox_binary=binary,proxy=proxy_settings)
def interactWithSite(driver):
driver.get("https://www.google.com")
driver.save_screenshot("screenshot.png")
interactWithSite(driver)
To connect to a Tor Browser through a FirefoxProfile you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\AtechM_03\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile= profile, executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("http://check.torproject.org")
Browser Snapshot:
You can find a relevant discussion in How to use Tor with Chrome browser through Selenium
I would like to expand on #DebanjanB answer by adding the Linux counterpart:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen('some/path/tor-browser_en-US/Browser/start-tor-browser')
# in my case, I installed it under a folder tor-browser_en-US after
# downloading and extracting it from
# https://www.torproject.org/download/ for linux
profile = FirefoxProfile(
'some/path/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox'
# /usr/bin/firefox is default location of firefox - for me anyway
driver = webdriver.Firefox(
firefox_profile=profile, options=firefox_options,
executable_path='wherever/you/installed/geckodriver')
# I keep my geckodriver(s) in a special folder sorted by versions.
# Geckodriver downloadable here:
# https://github.com/mozilla/geckodriver/releases/
driver.get("http://check.torproject.org")
The verified answer does not work in case of opening dot onion sites(I believe that's something to do with tor network which is not allowing access to normal firefox).
As for the latest tor browser (from the tor browser bundle), starting it using selenium causes some error due to which the browser cannot start tor proxy itself causing proxy and timeout errors(doesn't matter if tor proxy is started by python or manually or not started at all). This could also be due to port 9050 or 9150 being used by tor proxy and not being available to browser's tor instance but this does not explain the error caused when no instance of tor proxy is running.
The solution i have found is to start the tor proxy as normal, manually or using os.popen("tor.exe") and configure tor browser to not start tor proxy.
here's the code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
os.popen(r'e:\\bla\\bla\\bla\\tor\\Tor\\tor.exe')
binary=FirefoxBinary(r'e:\\bla\\bla\\bla\\Tor Browser\\Browser\\firefox.exe')
fp=FirefoxProfile(r'e:\\foo\\bar\\bla\\Tor Browser\\Browser\\TorBrowser\\Data\\Browser\\profile.default')
fp.set_preference('extensions.torlauncher.start_tor',False)#note this
fp.set_preference('network.proxy.type',1)
fp.set_preference('network.proxy.socks', '127.0.0.1')
fp.set_preference('network.proxy.socks_port', 9050)
fp.set_preference("network.proxy.socks_remote_dns", True)
fp.update_preferences()
driver = webdriver.Firefox(firefox_profile=fp,firefox_binary=binary)
driver.get("http://check.torproject.org")
driver.get('https://www.bbcnewsv2vjtpsuy.onion/')
*note fp.set_preference('extensions.torlauncher.start_tor',False) on line 10 is being used to configure tor to not start its own tor instance so that it uses the proxy config and tor instance started above.
lo and behold as the tbb starts working like normal firefox bot browser
I have a project and I need to create an HTTP packet interceptor in python as a proxy, but I can not use mitmproxy because it only works from the cli command and not from an internal library. I have an application and I can not tell people to run mitmproxy because that's what the app should do internally. I'm currently using a fork to launch mitmproxy from bash, but it gives me a lot of trouble creating this functionality for other operating systems.
For this reason, someone knows some way to intercept traffic as a proxy that does not depend on a command outside python ?.
mitmproxy can be load inside your python code.
from mitmproxy import proxy, options
from mitmproxy.tools.dump import DumpMaster
from mitmproxy.addons import core
class AddHeader:
def __init__(self):
self.num = 0
def response(self, flow):
self.num = self.num + 1
print(self.num)
flow.response.headers["count"] = str(self.num)
addons = [
AddHeader()
]
if __name__ == '__main__':
opts = options.Options(listen_host='127.0.0.1', listen_port=8080)
pconf = proxy.config.ProxyConfig(opts)
m = DumpMaster(None)
m.server = proxy.server.ProxyServer(pconf)
print(m.addons)
m.addons.add(addons)
# m.addons.add(core.Core())
try:
m.run()
except KeyboardInterrupt:
m.shutdown()
This code is taken from the GitHub issue.
Different result by urllib and urllib3
I can open a web page by copying the address into my chrome and urllib also returns the page source code. I just do not understand why urllib3 is returning 404 not found on this webpage when everything else works.
Below is the original code:
url = 'http://www.webmd.com/drugs/2/condition-12862/depression%20associated%20with%20bipolar%20disorder'
import urllib3
http = urllib3.PoolManager()
r = http.request('GET',url)
r.data
import urllib.request
req = urllib.request.Request(url=url)
with urllib.request.urlopen(req) as f:
print(f.read().decode('utf-8'))
My guess, you are calling behind proxy, urllib uses system proxy (if you are on linux - http_proxy enviroment variable), for urllib3 you need to specify it using urllib3 library