Selenium on Raspberry Pi headless 2022 - python-3.x

I have a weather station and take a screenshot of a webpage (windy.com) once an hour to integrate into my display which runs on a Pi 3
It was working perfectly until I had an SD card crash and had to rebuild it - I stupidly didn't have a backup of the SD card
I now have Pi OS Bullseye, Python 3.9.2 and Selenium 4.1.3
This is the old program that was working on PI OS Buster with Python 3.7 and Selenium 1.9ish
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from PIL import Image
import pygame
options = Options()
options.headless = True
options.add_argument('--user-agent="Mozilla/5.0 (Windows NT x.y; rv:10.0) Gecko/20100101 Firefox/10.0"')
driver = webdriver.Chrome('/usr/lib/chromium-browser/chromedriver', options=options)
url = "https://embed.windy.com/embed2.html?lat=53.514&lon=-2.944&detailLat=53.393&detailLon=-2.134&width=530&height=490&zoom=4&level=surface&overlay=wind&product=ecmwf&menu=&message=true&marker=&calendar=now&pressure=true&type=map&location=coordinates&detail=&metricWind=mph&metricTemp=%C2%B0C&radarRange=-1"
driver.get(url)
driver.fullscreen_window()
driver.implicitly_wait(1)
driver.set_window_size(530,445)
driver.save_screenshot('weather.png')
basewidth = 530
img = Image.open('weather.png')
driver.quit()
wpercent = (basewidth / float(img.size[0]))
hsize = int((float(img.size[1]) * float(wpercent)))
img = img.resize((basewidth, hsize), Image.ANTIALIAS)
img.save('/home/pi/images/weather_icons/kia.bmp')}
It will not work with the new Pi Bullseye, Python3.9 and Selenium 2.4
I had so many errors and gradually got rid of them all but now I'm left with this error and I cannot figure out how to fix it
This is the error
Traceback (most recent call last):
File "/home/pi/map.py", line 6, in <module>
driver = webdriver.Chrome(service=s)
File "/home/pi/.local/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "/home/pi/.local/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 90, in __init__
self.service.start()
File "/home/pi/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 98, in start
self.assert_process_still_running()
File "/home/pi/.local/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 110, in assert_process_still_running
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: Service /usr/lib/chromium-browser/chromium-browser-v7 unexpectedly exited. Status code was: -5
This is the modified code I had to use to get this far
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from PIL import Image
path = ('/usr/lib/chromium-browser')
s = Service('/usr/lib/chromium-browser/chromium-browser-v7')
driver = webdriver.Chrome(service=s)
options = webdriver.ChromeOptions()
options.set_capability("goog:loggingPrefs", {'performance': 'ALL'})
url = "https://embed.windy.com/embed2.html?lat=53.514&lon=-2.944&detailLat=53.393&detailLon=-2.134&width=530&height=490&zoom=4&level=surface&overlay=wind&product=ecmwf&menu=&message=true&marker=&calendar=now&pressure=true&type=map&location=coordinates&detail=&metricWind=mph&metricTemp=%C2%B0C&radarRange=-1"
driver.get(url)
driver.fullscreen_window()
driver.implicitly_wait(1)
driver.set_window_size(530,445)
driver.save_screenshot('weather.png')
basewidth = 530
img = Image.open('weather.png')
driver.quit()
wpercent = (basewidth / float(img.size[0]))
hsize = int((float(img.size[1]) * float(wpercent)))
img = img.resize((basewidth, hsize), Image.ANTIALIAS)
img.save('/home/pi/images/weather_icons/kia.png')
My chromium-chromedriver is the latest version (98.0.4758) and my chromium browser is Version 98.0.4758.106
Can anyone tell me how to get this working without having to go back to the old versions of everything?

Related

AttributeError: 'Options' object has no attribute 'set_headless'

I am tried to use a headless google chrome to scrapy some website content in macOS Big Sur, this is my python 3 code:
from webbrowser import Chrome
from dolphin.common.commonlogger import CommonLogger
from selenium.webdriver import chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
class FetchMusic:
def __init__(self):
print("fetch...")
if __name__ == '__main__':
opts = Options()
opts.set_headless()
assert opts.headless # Operating in headless mode
browser = Chrome(executable_path=r"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
options=opts)
browser.implicitly_wait(3)
browser.get('https://ca.finance.yahoo.com/quote/AMZN/profile?p=AMZN')
results = browser.find_elements_by_xpath('//*[#id="quote-header-info"]/div[3]/div/div/span[1]')
for result in results:
print(result.text)
when I run this code, shows error:
Traceback (most recent call last):
File "/Users/dolphin/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/212.5457.59/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Users/dolphin/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/212.5457.59/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/dolphin/source/reddwarf/backend/pydolphin/dolphin/biz/music/fetch.py", line 18, in <module>
opts.set_headless()
AttributeError: 'Options' object has no attribute 'set_headless'
python-BaseException
Process finished with exit code 1
what should I do to make it work?
Instead of
opts = Options()
opts.set_headless()
please use
options = ChromeOptions()
options.add_argument("--headless")
or
options = Options()
options.add_argument("--headless")
Imports :
from selenium.webdriver.chrome.options import Options as ChromeOptions

How do I downgrade my Chrome Version from version 94.0.4606.71 to version 94.0.4606.61

I am working on a Voice assistant using Python. I keep getting this error while running my code...
Traceback (most recent call last):
File "C:\Users\Admin\PycharmProjects\pythonProject\Python_Bot.py", line 40, in <module>
browser_driver.get('https://www.youtube.com/')
File "C:\Users\Admin\Python3.9\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\Admin\Python3.9\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Admin\Python3.9\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
(Session info: chrome=94.0.4606.71)
From the Session info: chrome=94.0.4606.71 message, I found out that I'm using Chrome version 94.0.4606.71 when my driver version is 94.0.4606.61. (I used Selenium's Chrome driver manager to find out my Chrome Driver version) My selenium version is up to date btw. So I'm wondering if downgrading the Chrome version from my current version to the version of my driver would get rid of this error. If that's the solution how do I downgrade Chrome safely without facing any issues? If downgrading Chrome isn't the solution, then what is?
Btw I asked a similar question over here.
(The code for my Voice assistant if it's required...)
import datetime
import webbrowser
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from webdriver_manager.chrome import ChromeDriverManager
import speech_recognition as sr
import pyttsx3
import pyaudio
import os
import random
import gtts
browser_driver = webdriver.Chrome(ChromeDriverManager().install())
r1 = sr.Recognizer()
r2 = sr.Recognizer()
r3 = sr.Recognizer()
engine = pyttsx3.init()
voices = engine.getProperty('voices')
engine.setProperty('voice', voices[0].id)
with sr.Microphone() as source:
print('Listening...')
engine.say("Hey I'm your bot, Trevor! What can I do for you today?")
engine.runAndWait()
audio = r3.listen(source)
# From here
if 'YouTube' in r2.recognize_google(audio):
r2 = sr.Recognizer()
with sr.Microphone() as source:
print("What do you want to see?", end='')
audio = r2.listen(source)
keyword = audio
browser_driver.get('https://www.youtube.com/')
elem = browser_driver.find_element_by_id('search')
elem.send_keys(keyword , Keys.RETURN)
browser_driver.quit()
try:
get = r2.recognize_google(audio)
print(get)
except sr.UnknownValueError:
print('Error on your side')
except sr.RequestError:
print('Error on my side')
# Till here is the code to run a YouTube vid
PS: Both the code and the error message are indented as per Pycharm's indentation
I've been facing this issue for days now so I would really appreciate some help...
It seems you're calling the web driver in the wrong way.
Try:
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
service = Service('theWebDriverPATH\\chromedriver.exe')
chrome_options = Options()
chrome_options.add_argument \
(r"--user-data-dir=C:\\Users\\yourWindowsUser\\AppData\\Local\\Google\\Chrome\\User Data")
chrome_options.add_argument(r'--profile-directory=ThePofileYouWantToUse')
driver = webdriver.Chrome(service=service, options=chrome_options)
If you have the correct version of the web driver as you mentioned, it should work
Take note that theWebDriverPATH should be something in the form of C:\\Users\\nicoc\\PycharmProjects
EDIT I would suggest updating both Chrome and the driver at the latest version available (currently 94.0.4606.71 and 95.0.4638.17) to avoid issues

My selenium program isn't able to click on an input box

I have made a program to grab a youtube link and type it on a website but its not able to find the input box...plz tell me why
import time
from selenium import webdriver
import pyautogui as auto
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('chromedriver.exe')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
auto.hotkey('ctrl', 't')
auto.typewrite('https://www.youtubemp3.pro/en2')
auto.press('enter')
time.sleep(3)
search_bar = web.find_element_by_xpath('//*[#id="videoURL"]')
search_bar.click()
search_bar.send_keys(link)
The error I'm getting is:
Traceback (most recent call last):
File "e:\pranil\python\doing shit with python\songs downloader.py", line 17, in <module>
search_bar = web.find_element_by_xpath('//*[#id="videoURL"]')
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Pranil.DESKTOP-TLQKP4G.000\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="videoURL"]"}
(Session info: chrome=83.0.4103.116)
The xpath you provided to find an element doesn't seem to work ""//[#id="videoURL"]"" I also opened up a YouTube video to see what you were looking for but I couldn't find videoURL if you are trying to get the URL I recommend you have selenium click on the share button and grab the link from there.
import time
from selenium import webdriver
import pyautogui as auto
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('./chromedriver')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
web.execute_script("window.open('https://www.youtubemp3.pro/en2', 'new_window')")
web.switch_to_window(web.window_handles[-1])
ylink = web.find_element_by_id("videoURL")
ylink.click()
ylink.send_keys(link)
submit = web.find_element_by_name("submitForm")
submit.click()
you do not have to use pyautogui it will work better if you use seleniu
try to find the element with it's ID.
import time
from selenium import webdriver
url = 'https://www.youtube.com/results?search_query=closer+8d'
web = webdriver.Chrome('../assets/chromedriver')
web.get(url)
time.sleep(3)
link = web.find_element_by_id('video-title')
link = link.get_attribute('href')
web.get('https://www.youtubemp3.pro/en2/')
time.sleep(3)
search_bar = web.find_element_by_id('txt-url')
search_bar.click()
search_bar.send_keys(link)

Why the error occured python3 with selenium

I learning python recent, but I have some error.
environment python3 , chrome , webdriver(chrome)
from selenium import webdriver
import time
import random
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
driver = webdriver.Chrome("./chromedriver.exe")
mobile_emulation = { "deviceName": 'Nexus 5' }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
driver.get("https:/xxx.com")
num = random.randint(11111111 , 99999999)
red = driver.find_element_by_class_name("***")
red.click()
numBox = driver.find_element_by_name("***")
numBox.send_keys(int(num))
reader = driver.find_element_by_id("***")
reader.send_keys("***")
comment = driver.find_element_by_css_selector(" ***")
comment.click()
and result error is here
Traceback (most recent call last):
File "C:\python\pad\pad.py", line 16, in <module>
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 156, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\***\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 254, in start_session
self.session_id = response['sessionId']
TypeError: string indices must be integers
I think the error because number of this code includes Decimal. but I cant find such number .
please give me advise
This error message...
Traceback (most recent call last):
File "C:\python\pad\pad.py", line 16, in <module>
driver = webdriver.Remote(command_executor='https:xxx.com',desired_capabilities = chrome_options.to_capabilities())
.
TypeError: string indices must be integers
...implies that there was a TypeError while invoking webdriver.Remote() method.
As per your code trials as you are using webdriver.Remote() with the argument command_executor perhaps you were trying to execute your tests in Selenium Grid Configuration.
As per the documentation documentation:
command_executor : remote_connection.RemoteConnection object used to execute commands.
Example:
command_executor='http://127.0.0.1:4444/wd/hub'
The complete implementation:
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities = chrome_options.to_capabilities())
Note: Here we have considered that the Selenium Grid Hub and Selenium Grid Node are configured, up and running successfully with default configuration on the localhost.
Solution (Python 3.6)
Your effective code block will be:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
#driver = webdriver.Remote(command_executor='https:xxx.com', desired_capabilities = chrome_options.to_capabilities())
driver = webdriver.Remote(command_executor='http://127.0.0.1:4444/wd/hub', desired_capabilities = chrome_options.to_capabilities())
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
driver.quit()

How to run chrome headless browser

I have been trying to set up chromes headless browser on my mac but I am getting errors.
I tried following these tutorials for reference:
https://intoli.com/blog/running-selenium-with-headless-chrome/
https://duo.com/decipher/driving-headless-chrome-with-python
and using these stackoflow pages
How to get Chromedriver in headless mode to run headless? and selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH error with chrome
The headless browser worked on phantomjs but I know selenium no longer wants us to use this. This is what I am running right now: (almost exactly one the responses on stack overflow)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("http://www.duo.com")
print("Chrome Browser Initialized in Headless Mode")
this is my response:
Traceback (most recent call last):
File "headless_browser.py", line 47, in <module>
driver = webdriver.Chrome(chrome_options=chrome_options)
File "/Users/BCohen/anaconda3/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/Users/BCohen/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/Users/BCohen/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/Users/BCohen/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/Users/BCohen/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.36.540469 (1881fd7f8641508feb5166b7cae561d87723cfa8),platform=Mac OS X 10.13.2 x86_64)
The error does gives us the hint as follows :
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.36.540469 (1881fd7f8641508feb5166b7cae561d87723cfa8),platform=Mac OS X 10.13.2 x86_64)
The error says that the ChromeDriver didn't find the Chrome binary in the expected location.
There can be two solutions as follows :
As per the Requirements the Chrome installation needs to be in a definite location.
As an alternative you can pass the absolute path of the Chrome binary through an instance of Options class as follows :
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location = "/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
Finally while initiating the WebDriver and WebClient you need to send the argument Key executable_path along with the Value chrome_path.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.binary_location = "/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
options = Options()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
chrome_path = "/home/ec2-user/chrome/chromedriver"
driver = webdriver.Chrome(executable_path=chrome_path, chrome_options=options)
driver.get("http://www.duo.com")
print("Chrome Browser Initialized in Headless Mode")
Alternative
Invoking Google Chrome Browser in Headless Mode programatically have become much easier with the availability of the method set_headless(headless=True) as follows :
Documentation :
set_headless(headless=True)
Sets the headless argument
Args:
headless: boolean value indicating to set the headless option
Sample Code :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.set_headless(headless=True)
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'C:\path\to\chromedriver.exe')
driver.get("http://google.com/")
print ("Headless Chrome Initialized")
driver.quit()
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
option = Options()
option.headless = True
driver = webdriver.Chrome(executable_path=r'C:\path\to\chromedriver.exe', options = option)

Resources