I need to automate tasks to extract information from websites using the uBlock plugin with Chrome driver using the selenium module in Python 3.
I am running my code remotely without a GUI on the remote machine - for that I am using xvfb-run to simulate a desktop environment where Chrome launches with a specific window size.
The remote machine has the following Debian operating system:
uname -a
Linux mem 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux
These were the steps I took to configure my environment and code on the remote machine:
1 - To configure my environment, I installed this version of Google Chrome:
google-chrome --version
Google Chrome 86.0.4240.111
2 - Check the versions of Python 3 and selenium I installed:
python --version
Python 3.7.3
pip freeze
selenium==3.141.0
3 - Check the xvfb-run version:
apt-cache policy xvfb
2:1.20.4-1+deb10u1
4 - With these packages configured, I obtained the chromedriver_linux64.zip from this list (the version 86.0.4240.22 below is the most recent one that is the same major version of the installed google-chrome):
https://chromedriver.storage.googleapis.com/index.html
https://chromedriver.storage.googleapis.com/index.html?path=86.0.4240.22/
5 - To be able to use the uBlock extension of Chrome, I needed to install an extension that is able to produce a .crx archive file of other installed extensions.
For this, I used CRX Extractor/Downloader:
https://chrome.google.com/webstore/detail/crx-extractordownloader/ajkhmmldknmfjnmeedkbkkojgobmljda
6 - After using that extension, I got my ublock.crx file to test.
I managed to use the binary in chromedriver_linux64.zip without the extension to launch a Chrome instance and do some basic crawling.
But when I tried to use ublock.crx in my code, I got an exception.
The code was this:
This is the exception produced:
ublock.crx error
selenium.common.exceptions.SessionNotCreatedException: Message: session not
created: cannot process extension #1
from unknown error: cannot unzip
I am launching it from my program like this:
from selenium import webdriver
option = webdriver.ChromeOptions()
option.add_extension(ublock_crx_file_path)
driver = webdriver.Chrome(executable_path=driver_path, options=option)
I have made sure the path of ublock_crx_file_path is valid and points to the file I obtained from Chrome.
Hopefully someone can shed light on this?
Related
I am looking to install Chrome Driver (For my selenium project) on an Amazon Linux 2 (Arm 64).
I have done this on Ubuntu and seems pretty straight, it also seems straight on AL2 x86, like this script from Intoli but it 404 when looking for Chrome on ARM :(. So far I see Chrome is not available for ARM so I see the alternative is to use Chromium.
I tried installing it manually but it seems there is no compiled on Chromium downloads page for ARM.
Finally i found the Electron's unofficial webdrivers, which seem to be the answer, and after downloading, installing dependencies (libXcursor, libXfixes, libXdamage) and running it:
[ec2-user#scraping1 current]$ chromedriver
Starting ChromeDriver 100.0.4896.143 (6bf1afe83487405ea0aff37182f05e3db45559c2-refs/branch-heads/4951#{#831}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
And run my project I get:
2.7.3 :004 > Selenium::WebDriver::Service.driver_path = "/usr/bin/chromedriver"
=> "/usr/bin/chromedriver"
session = Capybara::Session.new
Webdrivers::BrowserNotFound: Failed to find Chrome binary.
Which means i don't have Chrome installed, but then how to install it? I have seen this question which seems to be my same problem but it is unclear to me how to install Chromium; I see it is needed to download and use the chromedriver compiled for ARM64 which I have done already.
I am executing selenium test via Jenkins server in AWS Ubuntu.
I was getting chrome binary not found error so I set chrome binary in my code.
System.setProperty("webdriver.chrome.driver","/var/lib/jenkins/.m2/repository/webdriver/chromedriver/linux64/83.0.4103.39/chromedriver");
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("applicationCacheEnabled", true);
ChromeOptions options = new ChromeOptions();
options.merge(capabilities);
options.setBinary("/usr/bin/google-chrome-stable");
options.addArguments("--headless");
options.addArguments("--no-sandbox");
options.addArguments("--remote-debugging-port=9222");
options.addArguments("--disable-infobars");
options.addArguments("--disable-dev-shm-usage"); //Linux
options.addArguments("--disable-browser-side-navigation");
options.addArguments("--disable-gpu"); //Windows
options.addArguments("--disable-web-security");
driver = new ChromeDriver(options);
Then the error changed to: no chrome binary at /usr/bin/google-chrome-stable
Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103#{#416}) on port 14665
Only local connections are allowed.
ChromeDriver was started successfully.
INFO [main] (ControlCenter.java:108)- START
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 3, Time elapsed: 2.116 s <<< FAILURE! - in TestSuite
[ERROR] com.info.end2end.ExcelAccountToFusion.onTestSetup Time elapsed: 2.032 s <<< FAILURE!
org.openqa.selenium.WebDriverException:
**unknown error: no chrome binary at /usr/bin/google-chrome-stable**
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'jenkins-it02', ip: '10.113.0.187', os.name: 'Linux', os.arch: 'amd64', os.version: '5.3.0-1019-aws', java.version: '11.0.7'
Driver info: driver.version: ChromeDriver
This is how my usr/bin is:
And this is message when I try to launch browser in command line:
I tried the solution given by #DebanjanB at Cannot find Chrome binary with Selenium in Python for older versions of Google Chrome but no luck. Any help is much appreciated as I have spent 2 days on this already.
Not sure if setBinary() should be pointing to /usr/bin/google-chrome-stable.
As per the documentation in How To Install Google Chrome 78 On a RHEL/CentOS 7 and Fedora Linux to install and use the latest google-chrome using Yum you need to follow the sequence below:
Open the Terminal application. Grab 64bit Google Chrome installer.
Type the following command to download 64 bit version of Google Chrome:
wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
Install Google Chrome and its dependencies on a CentOS/RHEL, type:
sudo yum install ./google-chrome-stable_current_*.rpm
Start Google Chrome from the CLI:
google-chrome &
Outputs from yum command:
Finally, you need to use the following line to set the chrome binary:
options.setBinary("/usr/bin/google-chrome");
Additional Considerations
Ensure that:
Execute your #Test as non-root user.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
Reference
You can find a couple of relevant discussions in:
Cannot find Chrome binary with Selenium in Python for older versions of Google Chrome
I did not have the actual fix for this issue but I could get around my actual blocker by installing Chromium instead of Chrome. Difference I found is the installation directory of both.
I installed Chrome and whereis Google-Chrome-Stable gives:
qa_user#jenkins:~$ whereis google-chrome-stable google-chrome-stable: /usr/bin/google-chrome-stable /usr/share/man/man1/google-chrome-stable.1.gz
And qa_user#jenkins:~$ google-chrome-stable gives: google-chrome-stable: command not found
Now for Chromium:
qa_user#jenkins:~$ whereis chromium-browser gives: chromium-browser: /usr/bin/chromium-browser /usr/lib/chromium-browser /etc/chromium-browser /usr/share/chromium-browser /usr/share/man/man1/chromium-browser.1.gz
and qa_user#jenkins~$ chromium-browser [21304:21304:0607/135202.629230:ERROR:browser_main_loop.cc(1473)] Unable to open X display. Identifies and opens Chromium (X display issue can be solved by adding --headless option in actual Selenium code).
So I am gonna use Chromium instead of Chrome until I/Someone here find the fix for the actual Chrome issue here.
UPDATE:
After my below solution, I uninstalled Chromium and kept only Chrome for further testing. Surprisingly, chrome issue got fixed. I am able to run tests in Chrome now. Not sure how this is happening but I think it has something to do with dependency package that comes with Chrome.
I know this is strange but installing Chrome and Chromium and then uninstalling Chromium worked for me.
I have installed selenium and the chrome web driver and made it accessible via the path variable of my operating system (using Linux virtual machine on windows and python3). I run this simple code and get the below error. Any ideas as to what may cause this error?
Thanks in advance!
Selenium version installed:
selenium-3.141.0 urllib3-1.25.8
Chrome driver installed (latest version for chrome browser 80) :
ChromeDriver 80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987#{#882})
Chrome browser installed :
Google Chrome 80.0.3987.132
Python version :
Python 3.6.9
import selenium
from selenium import webdriver
options = webdriver.chrome.options.Options()
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
chromedriver = '/usr/bin/chromedriver'
print('test0') #is being printed
driver = webdriver.Chrome('/usr/bin/chromedriver',options=options)
print('test') #not being printed
driver.get('http:google.com')
Error message when running the code : The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.
When I try to run google-chrome with (sudo google-chrome) I get this message:
[56:56:0316/235226.650275:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
I tried modifying the google-chrome file in the /usr/bin but it is read-only so the workaround I found online was to add the options '--no-sandbox' in my code above. But hasn't resolved the issue.
Resolved : had a python file named queue in working directory and had to rename it
Solution:
This error "AttributeError: 'LifoQueue' object has no attribute 'put' selenium webdriver" is caused by having in your working directory a file with the name Queue. Rename that.
For this error "The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed."
If you tried all possible solutions and still get issues then if you are using Linux virtual machine that seems to be causing the problem and I suggest to install docker desktop on windows and download a docker image with all the libraries you will be needing to run selenium apps .
I am trying to create a webcrawler using Selenium, but I get this error when I try to create the webdriver object.
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between 70 and 73
(Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 6.1.7601 SP1 x86_64)
I downloaded the latest version of chromedriver (2.45) which requires Chrome 70-73. My current Chrome version is 68.0.3440.106 (Official Build) (64-bit), which is the latest. I tried downloading an "older" chrome version (71) and when I tried installing it, the installer indicated that I had a newer version already installed.
There doesn't seem to be any previous Chromedriver releases available for download, even though the website says there is. I couldn't find them.
I don't quite understand how version 71 is older than 68?
Is there a Chrome version newer than 68 actually available, or an older version of chromedriver i can use with Chrome 68?
Does anyone have any other suggestions?
This is the code that i'm trying to execute:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
user = 'XXXXXXX'
pwd = 'XXXXXXX'
chromedriver = "...\...\...\chromedriver.exe"
driver = webdriver.Chrome(executable_path=chromedriver) # Error occurs at this line
driver.get("http://www.facebook.com")
assert "Facebook" in driver.title
time.sleep(5) # So i can see something!
elem = driver.find_element_by_id("email")
elem.send_keys(user)
time.sleep(5) # So i can see something!
elem = driver.find_element_by_id("pass")
elem.send_keys(pwd)
time.sleep(5) # So i can see something!
elem.send_keys(Keys.RETURN)
driver.close()
For me, upgrading the driver did the trick. Just run:
brew cask upgrade chromedriver
and then try running your test again. Hope it helps!
You can find the older versions of chrome driver here.
I dont think it is a good idea to install chrome from sources other than the official channel and installation of the same can cause issues. See if the google update service is running in your PC. This will automatically update the chrome version to latest. Mine is running Version 71.0.3578.98 (Official Build) (64-bit).
This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between 70 and 73
(Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 6.1.7601 SP1 x86_64)
...implies that Chrome version must be between 70 and 73
Your main issue is the version compatibility between the binaries you are using as follows :
You are using chromedriver=2.45
Release Notes of chromedriver=2.45 clearly mentions the following :
Supports Chrome v70-72
You are using chrome=68.0
Release Notes of ChromeDriver v2.41 clearly mentions the following :
Supports Chrome v67-69
So there is a clear mismatch between ChromeDriver v2.45 and the Chrome Browser v68.0
Solution
Upgrade ChromeDriver to current ChromeDriver v2.45 level.
Keep Chrome version between Chrome v70-72 levels. (as per ChromeDriver v2.45 release notes)
Take a System Reboot.
Execute your #Test.
Alternative
Somehow I feel there are 2 versions of Chrome browser installed in your system. If that is the case you need to mention the absolute location of the Chrome binary within your program and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location = "C:/Program Files (x86)/Google/Chrome/Application/chrome.exe"
driver = webdriver.Chrome(chrome_options=options, executable_path="C:/Utility/BrowserDrivers/chromedriver.exe", )
driver.get('http://google.com/')
You can find a detailed discussion in Set chrome browser binary through chromedriver in Python
Note: You can find a relevant discussion in Session not created exception: Chrome version must be >= x.y.z when using Selenium Webdriver with Chrome
Reference
You can find a relevant detailed discussion in:
How to work with a specific version of ChromeDriver while Chrome Browser gets updated automatically through Python selenium
There are two options to resolve this issue:
1. If your Chrome version is not updated -> Update it
Steps: 1. Go to Help -> About Google Chrome -> Chrome will automatically look for updates(update Chrome to the latest version)
2. If your chrome version is already up to date -> Then you need to upgrade you chrome driver version
Here is the link: http://chromedriver.chromium.org/downloads
I encountered the same problem. I tried installing a downgraded version of Chrome (current stable was 74 and the driver required chrome version must be between 70-73) but I wasn't able to do so.
I found another way. This link will show you which version is compatible with your current google-chrome (to know your version the command is google-chrome --version)
This link will guide you as to how to install chrome driver with zip file. The commands are:
cd
wget <URL to zip file>
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
sudo chown root:root /usr/bin/chromedriver
sudo chmod +x /usr/bin/chromedriver
In case the error comes No such file or directory: '/usr/lib/chromium-browser/chromedriver' OR the same error persists
Repeat the above procedure with the path /usr/lib/chromium-browser/chromedriver instead of /usr/bin/chromedriver
For me, google-chrome version 74 worked with ChromeDriver version 73
If you end up here and you're using Laravel / Dusk, try:
php artisan dusk:chrome-driver
Your chrome version is very old. Version 68 is not the latest version. For all operating systems, Chrome is currently in version 71+. Please see the list here.
Alternatively, this is the link for downloading older versions.
In Summary:
Find your chrome version (Help-> About Google Chrome)
Find your chromedriver version, if you already have one. (For me "chromedriver.exe -v" gave me the version on windows environment.)
Visit the official chrome webdriver page (http://chromedriver.chromium.org/downloads)
Download the chromedriver matching your chrome browser version, from the above download location.
Have a glance at the release notes on this download page towards lower section of the page, which gives an clear idea regarding which driver to use for which version of browser.
You can put the downloaded chromedriver binary in the path environment.
There you go. All the best!
Just download chromedriver extension 2.8 or 2.9 from here.
Extract it and add this extension to /usr/local/bin
I faced this issue due to mismatch of chromedriver and chrome version. I followed the below steps to resolve my issue:
1) First, find out which version of Chrome you are using. My Chrome version was 77.0.3865.90.
2) Take the Chrome version number, remove the last part, and append the result to URL "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_". For example, with Chrome version 77.0.3865.90, you'd get a URL "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_77.0.3865".
3) Use the URL created in the last step to retrieve the version of ChromeDriver to use. For example, the above URL will get your a file containing "77.0.3865.40". (The actual number may change in the future, of course.)
4) I then did :
webdriverdownloader chrome:77.0.3865.40 on my mac terminal.
After this your Chromedriver and Chrome version must be in sync and the issue should be fixed.
You have to match the chromedriver version number and your web browser chrome version ,
If both of the version match then it is best.
>
For Example -
1- your web browser chrome version is 77 ( For checking your web browser chrome version
got to your web browser help and then About Google Chrome there you will find
your web browser chrome version ) .
2- Now download same version here 77 chromedriver from https://chromedriver.chromium.org/downloads .
3- Make google chrome as your default web browser .
All done .
Your code will run fine.
from selenium import webdriver
driver=webdriver.Chrome("chromedriver.exe")
# Below line of code provide
driver.get('http://python.org')
html_doc=driver.page_source
print("Html code of http://python.org web page :",html_doc)
Environment:
Centos 7
GOOGLE CHROME V61
SELENIUM WEBDRIVER 3.5.3
ChromeDriver 2.30/2.32
Tried running manually running google chrome inside jenkins-slave
google-chrome --no-sandbox --disable-setuid-sandbox &
Was giving me this error:
[31339:31350:1003/144118.591084:ERROR:bus.cc(395)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1003/144118.702053:ERROR:nacl_helper_linux.cc(310)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
Webdriver Error Info:
Chrome failed to start: crashed
(Driver info: chromedriver=2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.4.8-20.46.amzn1.x86_64 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.06 seconds
Build info: version: '3.5.3', revision: 'a88d25fe6b', time: '2017-08-29T12:42:44.417Z'
System info: host: 'jenkins-slavev2', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.8-20.46.amzn1.x86_64', java.version: '1.8.0_102'
Driver info: driver.version: ChromeDriver
This error message...
NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
...implies that you have no setuid sandbox in your system, hence the program was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session.
Solution
As you are seeing the error even with configuring --no-sandbox and --disable-setuid-sandbox, as per the documentation in Linux SUID Sandbox Development google-chrome needs a SUID helper binary to turn on the sandbox on Linux. In majority of the cases you can install the proper sandbox for you using the command:
build/update-linux-sandbox.sh
This program will install the proper sandbox for you in /usr/local/sbin and tell you to update your .bashrc if required.
However, there can be some exceptions as an example, if your setuid binary is out of date, you will get messages such as:
Running without the SUID sandbox!
Or
The setuid sandbox provides API version X, but you need Y
You are using a wrong version of the setuid binary!
In these cases, you need to:
Build chrome_sandbox whenever you build chrome (ninja -C xxx chrome chrome_sandbox instead of ninja -C xxx chrome)
After building, execute update-linux-sandbox.sh.
# needed if you build on NFS!
sudo cp out/Debug/chrome_sandbox /usr/local/sbin/chrome-devel-sandbox
sudo chown root:root /usr/local/sbin/chrome-devel-sandbox
sudo chmod 4755 /usr/local/sbin/chrome-devel-sandbox
Finally, you have to include the following line in your ~/.bashrc (or .zshenv):
export CHROME_DEVEL_SANDBOX=/usr/local/sbin/chrome-devel-sandbox
tl; dr
Security Considerations - ChromeDriver - Webdriver for Chrome
There were some missing dependencies that caused this issue (https://www.centos.org/forums/viewtopic.php?t=60908&start=10)
yum install liberation-mono-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts
Installing above packages solved the problem!
Hope it helps.