Preface: I have very limited experience with linux and selenium in general. Also this is a multipart question.
Hi there,
I've written a script that I want to run on my Raspberry Pi model 3B which is running Raspbian OS.
I originally built it on a Windows machine (pretty stupid mistake to build it for the wrong platform but anyway) and now need to adapt it so it runs on the Pi.
The main thing I am concerned about is the usage of selenium and chromedriver. I have downloaded chromium-chromedriver (chromium-chromedriver 34.0.1847.116-0ubuntu2 in armhf (Release)) from this repository.
First Question
The file downloads as a .deb which I have never seen before. It opts to install, however I have no idea where it installs as it does not give the option to install to a different directory. I've tried search for it with find, but nothing comes up. Where does it go by default in Raspbian OS?
Second Question
I know I will have to change some expressions because I am using chromium-chromedriver instead of chrome driver, but I am unsure which ones as I don't know selenium that well. This is the main instance here. Do I just change all instances of "Chrome" including "chrome_options" to "Chromium/chromium_options"?
from selenium.webdriver import Chrome
from contextlib import closing
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
#Scrape out the table information
with closing(Chrome(chrome_options=chrome_options)) as driver:
Lastly, for the purpose of finding the process with psutil and ending it with process.kill(), the text ID of the driver in task manager should just be chromium-chromedriver.exe right?
Any and all help and advice appreciated.
Thank you all for your time.
L
Related
I'm new on Selenium and I'm trying to do a task using Internet Explorer (I'm using Python 3.8)
In order to uderstand the commands of Selenium, I tried to run the simple code below
from selenium.webdriver.common.keys import Keys
import time
driver=webdriver.Ie(executable_path="C:\\Users\\usuario\\Downloads\\IEDriverServer.exe")
driver.get("https://www.google.com.br/")
driver.set_page_load_timeout(10)
driver.find_element_by_name("q").send_keys("Ronaldinho Gaucho")
driver.find_element_by_name("btnK").send_keys(Keys.ENTER)
driver.maximize_window()
driver.refresh()
The page opens, however, nothing is typed on the search bar on Google website, I have seen this code in a Youtube video and it has worked well, but when I try on my computer, it does not works (it does not raise any error on my terminal)
Anyone can help?, what I should been looking for?
Thanks in advance
I hope you are all well =)
this is my first post/question on stackoverflow =)
I'm writing this after trying all the answers in this thread. (the last answer I tried was from Youssof H.)
I'm a python newbie and trying to write a script that can help me upload products to a website.
Since I need to be logged in to be able to add products, I figured why not use a browser profile where I'm already logged in, instead of writing code to get this done (I figured using a browser profile would be easier)
I've been trying to get this working for many hours now and I don't seem to be able to solve this by myself.
When I run the code, instead of opening Chromium it keeps opening google-chrome. Prior to trying to use chromium I tried it with chrome, if I open chrome it opens google-chrome-stable but when I run the python file it runs google-chrome.
My operating system is Mint 20.1 (Cinnamon) and I use Visual Studio Code
If anyone can help me with this it would be highly appreciated =)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Do not use this path that is extracted from "chrome://version/"
exec_path_chrome = "/usr/bin/chromium"
exec_path_driver = "/home/equinix/drivers/chromedriver"
ch_options = Options() # Chrome Options
# Extract this path from "chrome://version/"
ch_options.add_argument(
"user-data-dir = /home/equinix/.config/chromium/Default")
# Chrome_Options is deprecated. So we use options instead.
driver = webdriver.Chrome(executable_path=exec_path_driver, options=ch_options)
driver.get("https://duckduckgo.com/")
I dont have the same paths but, you can pinpoint the exact binary that you want to use, either chromium or chrome
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Do not use this path that is extracted from "chrome://version/"
exec_path_chrome = "/usr/lib/bin/chromium"
exec_path_driver = "/home/art/drivers/chromedriver"
ch_options = Options() # Chrome Options
# Extract this path from "chrome://version/"
ch_options.add_argument(
"user-data-dir = /home/art/.config/chromium/Default")
# in your case its "/usr/lib/bin/chromium/chromium"
ch_options.binary_location = '/usr/lib/chromium/chromium'
#to point to google-chrome-stable
#ch_options.binary_location = '/opt/google/chrome/google-chrome'
# Chrome_Options is deprecated. So we use options instead.
driver = webdriver.Chrome(executable_path=exec_path_driver, options=ch_options)
driver.get("https://duckduckgo.com/")
I use such a simple construction, but it only works when you have one google chrome profile.
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
chrome_options = Options()
chrome_options.add_argument("user-data-dir=C:/Users/yourusername/AppData/Local/Google/Chrome/User Data/")
browser = webdriver.Chrome(chrome_options=chrome_options)
browser.get("https://google.com/")
The target of this project is to automate checking sites with Microsoft edge browser using selenium-python i downloaded the webdriver for the edge legacy from this link and i went for the latest release 17134 extracted it with out any problems now lets say i want to visit facebook in an automated way with firefox using the geckodriver
firefox code sample with selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.options import Options
# setting up headless option for faster execution
options = Options()
options.headless = True
browser = (webdriver.Firefox(options=options))
browser.get('https://www.facebook.com/')
but when I try to use Microsoft edge that is built in windows 10 I get an attribute error
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.edge.options import Options
options = Options()
options.headless = True
#browser = webdriver.edge(options=options)
browser = webdriver.edge()
ps : when I uncomment this part (browser = webdriver.edge(options=options)) I get module not found error
what is the right way to call Microsoft edge browser , or what I am doing wrong
when I use Edge and try to make Edge headless. I also find it hard to do that with slight changes as Chrome. And I refer to the official documentation and get a official solution. Besides selenium, you need to install msedge-selenium-tools, just pip install itpip install msedge-selenium-tools. And use Edge Class in msedge tools. Just like:
from msedge.selenium_tools import Edge
driver = Edge(executable_path='where')
And if we want to make Edge headless, we need to use EdgeOptions Class which selenium.webdriver doesn't offer. selenium.webdriver only provides us with ChromeOptions, FirefoxOptions and Ie's. EdgeOptions is in a separated package msedge.selenium_tools.Then we add argument as what we do on Firefox or Chrome. Before that, we need to set the attribute use_chromium as True. The whole codes:
from msedge.selenium_tools import EdgeOptions
from msedge.selenium_tools import Edge
# make Edge headless
edge_options = EdgeOptions()
edge_options.use_chromium = True # if we miss this line, we can't make Edge headless
# A little different from Chrome cause we don't need two lines before 'headless' and 'disable-gpu'
edge_options.add_argument('headless')
edge_options.add_argument('disable-gpu')
driver = Edge(executable_path='where', options=edge_options)
Hope it helps. Sorry for my awkward explaination.
I am using this WebDriver Package. Works perfectly. This package auto-download and runs your system-compatible browser smoothly. If you want to install and run a specific version, that is also possible. To know the instructions click here.
This code is for Selenium 4 [Python 3.10.*]
class MyEdge:
def get_browser(self):
options = webdriver.EdgeOptions()
# If you want to avoid popup browser use '--headless'
options.add_argument('--headless')
# Ref: https://learn.microsoft.com/en-us/microsoft-edge/webdriver-chromium/?tabs=python#using-chromium-specific-options
self.driver = webdriver.Edge(options= options,service=Service(EdgeChromiumDriverManager().install()))
return self.driver
More optional arguments:
--headless, --no-sandbox, '--disable-gpu', '--window-size=1280x1696', '--user-data-dir=/tmp/user-data', '--hide-scrollbars', '--enable-logging', '--log-level=0', , '--single-process', '--data-path=/tmp/data-path', '--ignore-certificate-errors', '--homedir=/tmp', '--disk-cache-dir=/tmp/cache-dir'
Make sure to impore:
# Import
from selenium import webdriver
from selenium.webdriver.edge.service import Service
from webdriver_manager.microsoft import EdgeChromiumDriverManager
I try to refer to the official documentation for WebDriver for Microsoft Edge (EdgeHTML). But I did not get any information about the Headless mode in it.
WebDriver (EdgeHTML)
I also try to refer to some old threads to find any information on this topic. It looks like we cannot use the Headless mode with the MS Edge legacy browser.
Headless Edge driven through Selenium by C#
I found one article that also said that 'User cannot use IE10, IE11, Edge, Opera & Safari for headless testing.'
Headless Browsers Testing using Selenium Webdriver
From the above references, it looks like you cannot use Headless mode with the MS Edge legacy browser.
As a workaround, I suggest you try to make a test with the MS Edge Chromium browser. I found that it supports Headless mode.
Using Chromium-Specific Options
I am writing an program for a web automation in python. Is here a ways to hide the geckodriver? So that the console (see picture) won't show up when I start the program.
console of geckodriver
here is a fraction of my code:
from selenium import webdriver
from selenium import *
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC`
driver=webdriver.Firefox()
wait=WebDriverWait(driver,120)
url = r"http://google.com"
driver.get(url) #This line starts the console (see picture)
To prevent geckodriver from displaying any windows, you need to pass -headless argument:
from selenium import webdriver
options = webdriver.FirefoxOptions()
options.add_argument('-headless')
driver = webdriver.Firefox(options=options)
driver.get('https://www.example.com')
...
driver.quit()
This worked for me in C#. It blocks both geckodriver and firefox window
FirefoxOptions f = new FirefoxOptions();
f.AddArgument("-headless");
var ffds = FirefoxDriverService.CreateDefaultService();
ffds.HideCommandPromptWindow = true;
driver = new FirefoxDriver(ffds,f);
I was able to do that after implementing PyVirtualDisplay
sudo pip install pyvirtualdisplay # Install it into your Virtual Environment
Then just import Display as follows:
from pyvirtualdisplay import Display
Then, before fetching, start the virtual display as follows:
# initiate virtual display with 'visible=0' activated
# this way you will hide the browser
display = Display(visible=0, size=(800, 600))
# Start Display
display.start()
...
# Do your fetching/scrapping
...
# Stop Display
display.stop()
I hope it helps
Here the approach which has solved it in my case. I used #thekingofravens suggestion but realized that it is just enough to create the Run.bat file which he mentions in his post. Previously I ran my programms in IDLE with F5. Because of this the Geckodriver popped up every few seconds.
My Solution: I just made the Run.bat file with the same code:
cd C:\PathToYourFileWhichIsCausingTheGeckodriverToPopUp
python FileWhichIsCausingTheGeckodriverToPopUp.py
And thats it. Just start this file when you want to run your code and the Geckodriver won't pop up. (That it works without the whole path to you program, Python has to be in PATH.)
Also, of course you can run your program from the command line with the same commands like above and without creating an extra bat file.
You have to take care of a couple of stuffs here:
Keep the useful imports. Unusual imports like from selenium import * (in your code) must be avoided.
Keep your code minimal. wait=WebDriverWait(driver,120) have no usage in your code block.
When you use the raw r switch remember to use single quotes '...'
If you initialize the webdriver instance remember to call the quit() method.
Be careful about indentation while using Python.
Here is the minimal code which uses geckodriver to open the url http://www.google.com, print the Page Title and quits the driver instance.
from selenium import webdriver
driver=webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
url = r'http://www.google.com'
driver.get(url)
print("Page Title is : %s" %driver.title)
driver.quit()
Console Output:
Page Title is : Google
So I found a very generic workaround to solve this on Windows (in Linux it doesn't seem to be an issue in the first place, can't speak for OSX).
So you need to make three files and its very awkward.
First, you make a file. call it start.bat (or anything) and put the following code in it:
wscript.exe "C:\Wherever\invisible.vbs" "C:\Some Other Place\Run.bat"
This will be the top level batch script. It will be visible for a split second while it launches a visual basic script and passes a batch script as an argument. The purpose of this is to make the next console invisible. Next we make the VBscript, invisible.vbs:
CreateObject("Wscript.Shell").Run """" & WScript.Arguments(0) & """", 0, False
Finally, we make the script that invisible.vbs is supposed to hide, we could call it Run.bat
cd C:\Wherever
python script_using_geckodriver.py
What happens is, as follows:
The first .bat file launches invisible.vbs
invisible.vbs launches the second .bat file without showing it on screen.
The second .bat file then launches the python program. Python (and geckodriver) output to the invisible cmd therefore hiding the geckodriver console window.
P.S. all of this works with PyInstaller to produce a single redistributable package the user can just click on.
Credit harrymc # superuser.com for this solution, which I found when trying to solve an unrelated problem. I tested and realized it was cross applicable to this.
https://superuser.com/questions/62525/run-a-batch-file-in-a-completely-hidden-way
I have been trying to import tinytag into python to be able to read mp3 tags but I keep receiving the same error. This is the code I am running
from tinytag import TinyTag
tag = TinyTag.get('/some/music.mp3)
print(tag.album)
and the error I recieve from this is
ImportError: No module named 'tinytag'
If anyone could give me any information on how to fix this would be greatly appreciated or can suggest another reader to use that is compatible with python 3.
Like you, I'm new to Python and I struggled with this but I worked it out eventually. I'm sure there is better way, but this worked (on windows, with my examples)
I installed a python module called easy_install (bundled with
setuptools). you can Google this. In the directory \Python26\Scripts you should see an exe file called easy_install if this has worked
Then I downloaded TinyTag to my pc eg
\downloads\tinytag-0.6.1.tar.gz
Then in note pad I wrote a small text file called myinstall.bat with
the contents
easy_install C:/downloads/tinytag-0.6.1.tar.gz
pause
then saved it into \Python26\Scripts and ran it (the pause keeps the
window open so you can see it worked)
Subsequently I started using some software called JetBrains to code with (it's commercial but there is a free edition) and that has an install tool built in which is even easier) I hope this helps