using python, selenium and phantomjs on openshift (socket binding permission denied?) - python-3.x

Question

Alright I'm at the end of my tether trying to get phantomJS to work with selenium in an openshift environment. I've downloaded the phantomjs binary using ssh and can even run it in the shell. But when it comes to starting a webdriver service using selenium I keep getting this traceback error no matter the args I put in.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/lib/openshift/576e22027628e1fb13000211/python/virtenv/venv/lib/python3.3/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 50, in __init__
service_args=service_args, log_path=service_log_path)
File "/var/lib/openshift/576e22027628e1fb13000211/python/virtenv/venv/lib/python3.3/site-packages/selenium/webdriver/phantomjs/service.py", line 50, in __init__
service.Service.__init__(self, executable_path, port=port, log_file=open(log_path, 'w'))
File "/var/lib/openshift/576e22027628e1fb13000211/python/virtenv/venv/lib/python3.3/site-packages/selenium/webdriver/common/service.py", line 33, in __init__
self.port = utils.free_port()
File "/var/lib/openshift/576e22027628e1fb13000211/python/virtenv/venv/lib/python3.3/site-packages/selenium/webdriver/common/utils.py", line 36, in free_port
free_socket.bind(('0.0.0.0', 0))
PermissionError: [Errno 13] Permission denied
Not sure what's going on, am I supposed to bind to an IP address? If so I tried using service args but that hasn't helped.

Answer 1

I came across the same issues trying to run phantomJS on my Openshift-hosted Django application, running on a Python 3 gear. Finally I managed to make it work, this is how:
The main issue to overcome is that Openshift does not allow applications to bind on localhost (nor '0.0.0.0' nor '127.0.0.1). So the point is to bind to the actual IP address of your Openshift Gear instead
You have to deal with this issue at the ghostdriver level as well as within the Python-selenium binding.
ghostdriver (phantomJS binary)
Unfortunately, as explained brilliantly by Paolo Bernardi in this post : http://www.bernardi.cloud/2015/02/25/phantomjs-with-ghostdriver-on-openshift/ you have to use a patched version of phantomjs for this, as the released version doesn't allow to bind to a specified IP. The binary linked by Paolo did not work on my Python3 cardridge, yet this one worked perfectly: https://github.com/jrestful/server/blob/master/seo/phantomjs-1.9.8-patched.tar.gz?raw=true (see question Trying to run PhantomJS on OpenShift: cannot patch GhostDriver so that it can bind on the server IP address for details)
Upload this phantomjs binary to app-root/data/phantomjs/bin (for example) and make sure it is runnable :
> chmod 711 app-root/data/phantomjs/bin/phantomjs
You can now check that you can bin to your IP like this (I chose port number 15002 for my app, I reckon you can pick any value you want above 15000):
> echo $OPENSHIFT_PYTHON_IP
127.13.XXX.XXX
> app-root/data/phantomjs/bin/phantomjs --webdriver=127.13.XXX.XXX:15002
PhantomJS is launching GhostDriver...
[INFO - 2017-03-24T13:16:36.031Z] GhostDriver - Main - running on port 127.13.XXX.XXX:15002
Ok, now kill this process and proceed to step 2 : python webdriver
Custom python-selenium webdriver for PhantomJS
The point is to add the IP address to bind to as a parameter of the PhantomJS Webdriver.
First, I defined new settings to adapt to Openshift's constraint in my settings.py
PHANTOMJS_BIN_PATH = os.path.join(os.getenv('OPENSHIFT_DATA_DIR'), 'phantomjs', 'bin', 'phantomjs')
PHANTOMJS_LOG_PATH = os.path.join(os.getenv('OPENSHIFT_LOG_DIR'), 'ghostdriver.log')
(make sure app-root/logs/ is writable, maybe you'll have to chmod it)
Then, I had to override the PhantomJS Webdriver class, to provide the ip address as an argument. Here is my own implementation:
from selenium.webdriver import phantomjs
from selenium.webdriver.common import utils
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
class MyPhantomJSService(phantomjs.service.Service):
def __init__(self, executable_path, port=0, service_args=None, log_path=None, ip=None):
if ip is None:
self.ip = '0.0.0.0'
else:
self.ip = ip
phantomjs.service.Service.__init__(self, executable_path, port, service_args, log_path)
def command_line_args(self):
return self.service_args + ["--webdriver=%s:%d" % (self.ip, self.port)]
def is_connectable(self):
return utils.is_connectable(self.port, host=self.ip)
#property
def service_url(self):
"""
Gets the url of the GhostDriver Service
"""
return "http://%s:%d/wd/hub" % (self.ip, self.port)
class MyPhantomWebDriver(RemoteWebDriver):
"""
Wrapper to communicate with PhantomJS through Ghostdriver.
You will need to follow all the directions here:
https://github.com/detro/ghostdriver
"""
def __init__(self, executable_path="phantomjs",
ip=None, port=0, desired_capabilities=DesiredCapabilities.PHANTOMJS,
service_args=None, service_log_path=None):
"""
Creates a new instance of the PhantomJS / Ghostdriver.
Starts the service and then creates new instance of the driver.
:Args:
- executable_path - path to the executable. If the default is used it assumes the executable is in the $PATH
- ip - IP sur lequel on veut se binder : c'est la spécificité de ce monkeypatch
- port - port you would like the service to run, if left as 0, a free port will be found.
- desired_capabilities: Dictionary object with non-browser specific
capabilities only, such as "proxy" or "loggingPref".
- service_args : A List of command line arguments to pass to PhantomJS
- service_log_path: Path for phantomjs service to log to.
"""
self.service = MyPhantomJSService(
executable_path,
port=port,
service_args=service_args,
log_path=service_log_path,
ip=ip)
self.service.start()
try:
RemoteWebDriver.__init__(
self,
command_executor=self.service.service_url,
desired_capabilities=desired_capabilities)
except Exception:
self.quit()
raise
self._is_remote = False
def quit(self):
"""
Closes the browser and shuts down the PhantomJS executable
that is started when starting the PhantomJS
"""
try:
RemoteWebDriver.quit(self)
except Exception:
# We don't care about the message because something probably has gone wrong
pass
finally:
self.service.stop()
Finally, invoke this custom webdriver instead of webdriver.PhantomJS(..., like this:
from .myphantomjs import MyPhantomWebDriver
browser = MyPhantomWebDriver(executable_path=settings.PHANTOMJS_BIN_PATH, service_log_path=settings.PHANTOMJS_LOG_PATH, ip=os.getenv('OPENSHIFT_PYTHON_IP'), port=15002)
From then on, you can use the browser object normally