Where to place PhantomJS exe? - python-3.x

I am trying to use PhantomJS with Selenium and Python.
My understanding is:
I will have to write Python script utilizing Selenium package which will interact with Selenium to operate on PhantomJS WebDriver to automate web application testing.
I have installed following:
Python v3.5.1.
Selenium using pip install selenium v3.7.0.
PhantomJS v2.1.1
In meantime I tested using Chrome WebDriver by placing it in PATH, and it executes without errors. Following is my script to open google.com using chrome webdriver.
from selenium import webdriver
driver = webdriver.Chrome() # or add to your PATH
driver.get('https://google.com/')
Using PhantomJS:
from selenium import webdriver
url = "http://www.google.com"
path_phantom = r'H:\phantomjs\bin\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path=path_phantom)
driver.get(url)
driver.save_screenshot(r'H:\out.png')
driver.quit()
Errors:
Traceback (most recent call last):
File "C:\Users\acer\Desktop\testing\openYoutube.py", line 5, in
driver = webdriver.PhantomJS()
File "C:\Users\acer\AppData\Local\Programs\Python\Python35-32\lib\site-package
s\selenium\webdriver\phantomjs\webdriver.py", line 51, in init
log_path=service_log_path)
File "C:\Users\acer\AppData\Local\Programs\Python\Python35-32\lib\site-package
s\selenium\webdriver\phantomjs\service.py", line 50, in init
service.Service.init(self, executable_path, port=port, log_file=open(log
_path, 'w'))
PermissionError: [Errno 13] Permission denied: 'ghostdriver.log'
Am I misplacing PhantomJS exe or missing any step ?

You can place the PhantomJS v2.1.1 binary at any location within your system and use the following code block :
from selenium import webdriver
url = "http://www.url.com.br/contact.asp"
path_phantom = r'C:\your_path\phantomjs-2.1.1-windows\bin\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path=path_phantom)
driver.set_window_size(1400,1000)
driver.get(url)
Update :
Please consider the following points and try the following code block with debug messages:
Run CCleaner tool to wipe off all the OS chores from your system.
You can opt for a System Reboot.
Try to keep the Python Application, WebBrowser binaries and the WebDriver binaries i.e. phantomjs.exe on the same drive.
from selenium import webdriver
url = "http://www.google.com"
path_phantom = r'C:\Utility\phantomjs-2.1.1-windows\bin\phantomjs.exe'
driver = webdriver.PhantomJS(executable_path=path_phantom)
print("PhantomJS browser invoked")
driver.get(url)
print("Browser Initialized")
driver.save_screenshot("C://Utility//out.png")
driver.quit()
print("Browser Closed")

Problem seems to be with the log file.
Changing path of log file solved this problem.
path_phantom = r'H:\phantomjs\bin\phantomjs.exe'
log_path=r'H:\ghostdriver.log' #changed path to a temporary file.
# service_log_path is required to change path of log file.
driver = webdriver.PhantomJS(executable_path=path_phantom,service_log_path=log_path)

From your error:
PermissionError: [Errno 13] Permission denied: 'ghostdriver.log
Seems that it try to create this file ghostdriver.log but fails because of the permissions.
As suggested in this answer, try to add the argument
service_log_path=os.path.devnull
to the function webdriver.PhantomJS().
Or make sure it is able to create the file.

Related

selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/geckodriver unexpectedly exited. Status code was: 64

I am trying to run a selenium test but getting the above error. I think this is a compatibility issue but I am not sure as I have very basic knowledge of selenium. My geckodriver version is 0.31.0 and firefox browser version is 100 and selenium version is selenium == 4.1.0. Any idea how I can fix this?
This is my conftest.py
import pytest
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as ff_service
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver_path = ff_service(r"/usr/bin/geckodriver")
driver = webdriver.Firefox(service=driver_path,options = options) #getting error at this line
driver.maximize_window()

Checking for existing webdriver when using selenium

I am trying to use selenium to control Chrome at work. The methods I have found all seem to install the webdriver every time the code is run. However, because of work from home and a rather slow VPN, this can take upto a minute. So, I am trying to check for the webdriver executable and then skip the install method if it exists.
Issue: When searching the directory, the the files list is empty.
chromedriver.exe does exist in C:\Users<user>.wdm\drivers\chromedriver\win32\103.0.5060 as installed by a previous run.
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
# set a variable for the web driver folder.
wddir = os.environ["USERPROFILE"] + "\.wdm\drivers\chromedriver\win32"
# check the existance of the folder.
# if it exists, find the driver and set it
if os.path.exists(wddir):
wdname = "chromedriver.exe"
for root, dir, files in os.walk(wddir):
if wdname in files:
driver = webdriver.Chrome(os.path.join(root, wdname))
else: #install the driver
driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
Resulting variable list:
- dir: ['103.0.5060']
- files: []
- root: C:\Users\<user>\.wdm\drivers\chromedriver\win32
- wddir: C:\Users\<user>\.wdm\drivers\chromedriver\win32
- wdname: chromedriver.exe
Since files = [], it goes on to install the driver anyway.
As an aside, is this a viable method for skipping the install? I am open to suggestions for better methods.

save document.cookie output in a file

skillshare-downloader says:
grab your cookie by typing:
document.cookie
Then it says:
Copy-paste cookie from developer console (without " if present) into example script.
Example:
from downloader import Downloader
cookie = """
ADD YOUR COOKIE HERE
"""
It adds an extra step.
Is there any way we can save document.cookie output to a file so that we can just read the cookie from the file instead of going to the console and type document.cookie and copy-paste the output?
I checked How to write console.log to a file instead. I also checked Python open browser and run javascript function. It suggests using Selenium or webbroser module. However, I am not sure how to approach this problem.
What can I do?
Assuming that you are using chrome:
Install selenium by running in a terminal pip install selenium
Install a chromedriver via a manager (It allows you to control chrome) by running in a terminal pip install webdriver-manager
Create a file example.py and paste this inside
#import the selenium webdriver and the chromedriver
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from time import sleep
#trying to stop skillshare from detecting we a are a bot
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
#create the instance of chrome
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
#get command used to open an url
driver.get('https://www.skillshare.com/')
#login part
#press on sign in
driver.find_element_by_css_selector("a.button.alt-white-ghost.transparent.initialized").click()
#change email here
driver.find_element_by_name("email").send_keys("testmail#mail.com")
#change password here
driver.find_element_by_name("password").send_keys("fakepassword")
#wait a second
sleep(1)
#click on sign in
driver.find_element_by_xpath("//span[text()='Sign In']/parent::button").click()
#wait 3 seconds for the login
sleep(3)
#execute_script is used to execute the command in the browser console, using return here to store it in a variable
cookie = driver.execute_script('return document.cookie')
#python way of creating a file on the given path and write the cookie inside it
f = open("D:\cookie.txt", "w")
f.write(cookie)
f.close()
#closing the chrome instance
driver.close()
open a terminal and type python example.py and it will run the script

Running Headless Chrome using Python 3.6 on AWS Lambda - Permissions Error

I have struggled to get Headless Chrome running on AWS Lambda for days. It works fine on EC2 but when I try it on Lambda, I just get "Message: 'chromedriver' executable may have wrong permissions.
The modules are zipped with the chromedriver and headless-chromium executables in the root directory of the zip file. The total zipped file I upload to S3 is 52mb but extracted it is below the 250mb limit so I don't think that is the issue.
Python Zip Folder Structure Image
from selenium import webdriver
def lambda_handler(event, context):
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--disable-application-cache")
options.add_argument("--disable-infobars")
options.add_argument("--no-sandbox")
options.add_argument("--hide-scrollbars")
options.add_argument("--enable-logging")
options.add_argument("--log-level=0")
options.add_argument("--v=99")
options.add_argument("--single-process")
options.add_argument("--ignore-certificate-errors")
options.add_argument("--homedir=/tmp")
options.binary_location = "/var/task/headless-chromium"
driver = webdriver.Chrome("/var/task/chromedriver", chrome_options=options)
driver.get("https://www.google.co.uk")
title = driver.title
driver.close()
return title
if __name__ == "__main__":
title = lambda_handler(None, None)
print("title:", title)
A few posts on the web have reported compatibility issues that may have caused problems so I have the specific executable versions for Chrome and ChromeDriver from the web, where others seem to on previous success EC2 and other means.
DOWNLOAD SOURCES FOR HEADLESS CHROME AND CHROMEDRIVER
(stable) https://github.com/adieuadieu/serverless-chrome/releases/tag/v1.0.0-37
(https://sites.google.com/a/chromium.org/chromedriver/downloads) Download unavailable so retrieved from the source below
https://chromedriver.storage.googleapis.com/index.html?path=2.37/
Can anyone help me crack this?
I found a solution for this problem few minutes ago.
When use chromedriver in Lambda Function (I think) it need permission can write. but when chrome driver file is in 'task' folder or 'opt' folder, user can only have read permission.
Only folder can change permission in Lambda Function is 'tmp' folder.
So I move the chrome driver file to 'tmp' folder. and it works.
like this.
os.system("cp ./chromedriver /tmp/chromedriver")
os.system("cp ./headless-chromium /tmp/headless-chromium")
os.chmod("/tmp/chromedriver", 0o777)
os.chmod("/tmp/headless-chromium", 0o777)
chrome_options.binary_location = "/tmp/headless-chromium"
driver = webdriver.Chrome(executable_path=r"/tmp/chromedriver",chrome_options=chrome_options)

Deploying Selenium on Heroku(Status code -6)

I'm trying to run a script via scheduler which uses selenium but it shows the following error - Message: Service /app/.apt/opt/google/chrome/chrome unexpectedly exited. Status code was: -6
I've used both the buildpacks -
https://github.com/heroku/heroku-buildpack-chromedriver.git
https://github.com/heroku/heroku-buildpack-xvfb-google-chrome
The script is:
chrome_exec_shim = "/app/.apt/opt/google/chrome/chrome"
opts = webdriver.ChromeOptions()
opts.binary_location = chrome_exec_shim
driver = webdriver.Chrome(executable_path=chrome_exec_shim, chrome_options=opts)
What you should do is download the Chrome driver here.
You can either put it in the chrome package that way you don't need to set the path at all. ( In my experience better to put in the path) or you can just give the path to the downloaded driver it can be in the project folder (recommend).
Just change the variable chrome_exec_shim to the path of the driver.
chrome_exec_shim = "/app/.apt/opt/google/chrome/chrome"
opts = webdriver.ChromeOptions()
opts.binary_location = chrome_exec_shim
opts.addArguments("--no-sandbox");
opts.addArguments("--disable-gpu");
driver = webdriver.Chrome(executable_path=chrome_exec_shim, chrome_options=opts)
Try with this code.
You must add the arguments of the chromeoption and it will work.
I tried this and its working for me.
After downloading the chromedriver, it was giving an error that binary was not found.
Gave the address of chrome in the executable path and the path of chrome driver in the chrome options.
That too resulted in the error, and after adding --disable-gpu and --no-sandbox arguments in the chrome options, it got resolved.
Thanks for the help... :)
The code that ran, at last, is below -
from selenium import webdriver
import os
chrome_exec_shim = os.environ.get("GOOGLE_CHROME_BIN", "chromedriver")
opts = webdriver.ChromeOptions()
opts.binary_location = chrome_exec_shim
opts.add_argument('--disable-gpu')
driver = webdriver.Chrome(executable_path='/app/development/chromedriver', chrome_options=opts)

Resources