I am trying to run a Python + Selenium script in headless mode with Firefox using Xvfb but I am getting errors. There is not much documents or guides available for Xvfb to troubleshoot the issue so looking for assistance here.
Environment info:
OS: CentOS release 6.5 (Minimal installation)
Xvfb: xorg-x11-server-Xvfb-1.15.0
Firefox: 52.8.0
geckodriver: 0.24.0
Python: 3.6.7
Steps followed:
Once done installing the above-mentioned requirements. I started a virtual display with:
$Xvfb :1 -ac &
Also, I tried with:
$Xvfb :1 -screen 0 1024x768x24 -extension RANDR &
And then I set Display variable:
export DISPLAY=:1
When I tried to initiate Selenium WebDriver in Python console I am getting the error Connection refused:
> from selenium import webdriver
> from pyvirtualdisplay import Display
> display = Display(visible=0, size=(800, 600))
> display.start()
> driver = webdriver.Firefox()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Any help or suggestion will be much appreciated.
There is a wrapper around xvfb called PyVirtualDisplay that seems designed for this exact solution. If you simply do a pip install pyvirtualdisplay the following script should run a headless Firefox window:
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox(executable_path="/Users/username/Location/geckodriver")
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
Related
I am running selenium inside docker and getting this error on this call: driver.get(URL)
Traceback (most recent call last):
File "esdm.py", line 244, in <module>
upload_to_esdm(browser, version_url, args.im_file, args.build, args.user_name, args.password, args.apps_file)
File "esdm.py", line 78, in upload_to_esdm
browser.get(version_url)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_CONNECTION_TIMED_OUT
(Session info: headless chrome=87.0.4280.66)
This is my code:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--js-flags=--max-old-space-size=8196')
driver = webdriver.Chrome(options=chrome_options)
driver.set_window_size(1920, 1080)
driver.get(URL)
And my Dockerfile:
FROM ubuntu:18.04
WORKDIR /src
COPY . /src
RUN apt-get -y update
RUN apt-get install -y python3 python3-pip chromium-chromedriver
# set display port to avoid a crash
ENV DISPLAY=:99
RUN pip3 install selenium
ENTRYPOINT "/bin/bash"
This works properly in windows, but when running in docker it doesn't
I have a very basic Python script that runs perfectly on my local machine (Mint 19), and yet fails on a remote box (Ubuntu 16.04). Same files, both Python 3.7. I have geckodriver in /usr/local/bin and it checks out from path with gecko --version from the command line. I can't figure out what the problem is. The geckodriver.log file just says:
1541268536111 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.Mt6zAyZc7D01"
*** You are running in headless mode.
1541268546125 Marionette INFO Listening on port 33632
The error from the terminal is:
root#dev1:/home/krypterro/PycharmProjects/corbot# python3 test1.py
2018-11-03 12:28:22,442 - INFO - Application - Start
test1.py:12: DeprecationWarning: use setter for headless property instead of set_headless
opts.set_headless(headless=True)
Traceback (most recent call last):
File "test1.py", line 21, in <module>
main()
File "test1.py", line 14, in main
driver = webdriver.Firefox(options=opts)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: timed out
Here is the Python code:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def main():
logging.info('Application - Start')
# Operate in headless mode
opts = Options()
opts.set_headless(headless=True)
assert opts.headless
driver = webdriver.Firefox(options=opts)
driver.get("https://www.krypterro.com")
html_src = driver.page_source
print(html_src)
driver.close()
driver.quit()
logging.info('Application - End')
main()
I have port 4444 allowed in the firewall on the remote box, but as it's a local-to-local connection I am not sure that should even matter.
This information log...
INFO - Application - Start test1.py:12: DeprecationWarning: use setter for headless property instead of set_headless opts.set_headless(headless=True)
...implies that the set_headless opts.set_headless(headless=True) is deprecated and you have to use the setter for headless property as follows:
opts = Options()
opts.headless = True
driver = webdriver.Firefox(options=opts)
driver.get("https://www.krypterro.com")
You can find the detailed discussion in How to make firefox headless programmatically in Selenium with python?
Moving ahead as you are trying to retrive the Page Source and as the Web Application is JavaScript enabled you need to induce WebDriverWait and you can use the following solution:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://www.krypterro.com")
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//h2[contains(.,'Products and Services')]")))
html_src = driver.page_source
print(html_src)
driver.quit()
Note B: You don't need to invoke driver.close() and driver.quit() rather always invoke driver.quit() only within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
I'm trying to run chrome headless with selenium in Python.
I tried xvfb-run -a python3 message.py
And i get this error:
Traceback (most recent call last):
File "message.py", line 388, in <module>
initialize()
File "message.py", line 54, in initialize
driver = webdriver.Chrome(chrome_options = chrome_options)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.33.506178 (69ae10f91723897591ef1a3b465aa5d35011eb5e),platform=Linux 3.13.0-96-generic x86)
And initialize():
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--verbose")
chrome_options.add_argument("--log-path=chrome.log")
chrome_options.add_argument("window-size=1200x600")
chrome_options.add_argument("user-data-dir=\\")
driver = webdriver.Chrome(chrome_options = chrome_options)
driver.get('http://google.com')
I'm using Python 3.5 and chromedriver 2.33.506178 for Linux32 (Ubuntu as Server without Desktop)
I tested the same code on my other pc with linux mint, all works fine.
Hope you can help me :)
After checking my actual rights of writing/reading/executing I found my mistake. chromedriver weren't able to load while trying to execute with root rights.
After I changed chromedrivers permission all works fine now.
When trying to run a Python Selenium script via Jenkins, it crashes, with a very unhelpful exception - "selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status: 0"
The same Python script runs perfectly fine from outside Python.
I've reduced the script to a very, very basic one, simply launching Firefox -
from selenium import webdriver
if __name__ == "__main__":
print("Hello World")
driver = webdriver.Firefox()
driver.get("http://www.google.com")
driver.maximize_window()
driver.quit()
print("Goodbye World")
Output from cmd prompt:
>C:\Users\kipod>C:\python_projects\Ranger\_just_open_browser.py
>Hello World
>Goodbye World
Works fine.
Running as a Jenkins job:
>##Started by user anonymous
>Building in workspace C:\Program Files (x86)\Jenkins\workspace\Second Test
>[Second Test] $ cmd /c call >C:\Users\kipod\AppData\Local\Temp\jenkins3557827225974274191.bat
>
>C:\Program Files (x86)\Jenkins\workspace\Second Test>python >C:\python_projects\Ranger\_just_open_browser.py
>Hello World
>Traceback (most recent call last):
> File "C:\python_projects\Ranger\_just_open_browser.py", line 8, in <module>
> driver = webdriver.Firefox()
> File "C:\Python\Python36-32\lib\site->packages\selenium\webdriver\firefox\webdriver.py", line 154, in __init__
> keep_alive=True)
> File "C:\Python\Python36-32\lib\site->packages\selenium\webdriver\remote\webdriver.py", line 151, in __init__
> self.start_session(desired_capabilities, browser_profile)
> File "C:\Python\Python36-32\lib\site->packages\selenium\webdriver\remote\webdriver.py", line 240, in start_session
> response = self.execute(Command.NEW_SESSION, parameters)
> File "C:\Python\Python36-32\lib\site->packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
> self.error_handler.check_response(response)
> File "C:\Python\Python36-32\lib\site->packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
> **raise exception_class(message, screen, stacktrace)
>selenium.common.exceptions.WebDriverException: Message: Process unexpectedly >closed with status: 0**
>C:\Program Files (x86)\Jenkins\workspace\Second Test>exit 1
>Build step 'Execute Windows batch command' marked build as failure
>Finished: FAILURE
I suspect something is off with Jenkins's environment variables, but I have no idea what to check. And the exception is so unspecific ("I crashed. lols"), That I found nothing helpful on Google. And I've been searching for a while.
Versions:
Jenkins: 2.73.1
Python: 3.6.2
Geckodriver: 0.19.0
Selenium: 3.6.0
(Tried opening chrome instead of firefox - same behaviour.)
I think you need to run your python script using a jenkins slave configured to log on as a standard windows user account (for example "kipod") ? Make sure that auto login is set for the slave machine for the user account, in order to allow selenium to launch Firefox in a desktop.
Some documenation here: Step by step guide to set up master and slave machines on Windows
Here's my code for opening Chrome:
from selenium import webdriver
driver=webdriver.Chrome('C:\\Users\\Imran\\AppData\\Local\\Programs\\Python\\Python36\\selenium\\chromedriver.exe')
Error when running program:
Traceback (most recent call last):
File "C:/Users/Imran/PycharmProjects/webscraping/WF Item.py", line 6, in <module>
driver = webdriver.Chrome('C:\\Users\\Imran\\AppData\\Local\\Programs\\Python\\Python36\\selenium\\chromedriver.exe')
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 69, in __init__
desired_capabilities=desired_capabilities)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: session not created exception
from unknown error: Runtime.executionContextCreated has invalid 'context': {"auxData":{"frameId":"27028.1","isDefault":true},"id":1,"name":"","origin":"://"}
(Session info: chrome=54.0.2840.71)
(Driver info: chromedriver=2.23.409699 (49b0fa931cda1caad0ae15b7d1b68004acd05129),platform=Windows NT 6.3.9600 x86_64)
Getting the following exception:
selenium.common.exceptions.WebDriverException: Message: session not created exception
Update your chromedriver to 2.24 version from the given url:
http://chromedriver.storage.googleapis.com/index.html?path=2.24/
That worked for me.
Setting LANG=en_US.UTF-8 before executing my script works for me.
See this: https://bugs.chromium.org/p/chromedriver/issues/detail?id=1552#c43
In Python:
import os
os.environ["LANG"] = "en_US.UTF-8"
To install the chromedriver on windows, download and unzip the chromedriver_win32.zip and add the path to the folder containing the chromedriver.exe in the environment variables setting [System properties - Advanced system settings - Environment variables].