I am attempting to run a DAG that will use selenium to scrape a web page every week on Cloud Composer.
I have already tried to give the path when creating the WebDriver.Chrome() instance to a driver that I uploaded to GCS, though I imagine this is not the best way to do this.
Airflow is giving this error
Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
If you have any tips as to updating Cloud Composer's PATH variable, would be greatly appreciated. If I need to put in more info, drop a comment and I'll add on.
So there was no official answer and the Slack channel for neither Composer nor GKE were able to help. The real problem was that the binaries were not on Composer. Best answer for right now is to manually ssh into all of your GKE airflow-workers and install Google Chrome yourself: https://linuxize.com/post/how-to-install-google-chrome-web-browser-on-ubuntu-18-04/
Then place the chromedriver for the correct version of Chrome you installed in your dags/dependencies folder and reference it on instantiation of your Webdriver object. Hope this helps!
You can create a Docker File and give mention the command to install chrome in the Docker File. Or else as mentioned by Alex you can manually install chrome on all worker nodes.
Follow this tutorial to connect to your worker nodes using Cloud shell- https://towardsdatascience.com/connect-airflow-worker-gcp-e79690f3ecea
Once inside the worker Run the following to install Chrome-
sudo apt-get update
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb
If you get some dependency error then run the below command and again run the install command
sudo apt --fix-broken install
To check the chrome installation run -
google-chrome --version
And now check where the chrome binary is installed
which google-chrome-stable
Copy this path and put it in the Selenium options in binary_location
options = webdriver.ChromeOptions()
options.binary_location= '/usr/bin/google-chrome-stable'
browser = webdriver.Chrome(ChromeDriverManager().install(),chrome_options=options);
If you are looking for Chrome Driver you can install it on the go while creating the webdriver object shown above
Related
and hope you are all doing well.
I am trying to install Docker on one of the front-end nodes my university's compute cluster, which uses CentOS 7.9. Unfortunately, I am not in the list of sudoers, and therefore cannot use any sudo commands or even the yum installer.
I also do not have write access to /etc, and therefore I am not able to modify any of the files there (like sysctl.conf). I also cannot run sysctl or yum commands.
So far, I have managed to download the binaries for Docker 20 in .tgz format, but after extracting, I cannot run the docker daemon, as I do not have root privileges.
I have tried following this guide: https://docs.docker.com/engine/security/rootless/ but it requires executing commands in sudo mode and modifying files in /etc, and I am unable to follow through completely.
Is there another way to install and run docker, or should I just give up completely? I would really appreciate it if someone could please let me know. Thank you very much!
Best Regards
I am trying to install python 3.11 without the Mac developer tools, which will give me python3.9. But when I used the vscode to open a python file, the command line installer poped up and showed the following message 'the python3 command requires the command line developer tools ....'. But I can run the python3 command in the terminal successfully.
The annoying part is the pop up window. I found some suggestions about relink or rebuild the Xcode / command line tools. But in my case, there is no Xcode path. Please let me know if there is a way to configure the path correctly.
Thank you,
I checked the link of python3 in /usr/bin
I install the python directly from the http://python.org
Do you use Homebrew in your terminal?
You can try to run brew link --overwrite --dry-run python#3.11 in your terminal to see if you need to do an update for your env path. If you do need it, the terminal will give you suggestions to relink, ex. brew link --overwrite python#3.11 or rm '/usr/local/bin/2to3-3.11'
I also had an issue when installing python3.11 through Homebrew in my terminal, it said I must use xcode-select --install to install python3.11.
So later on, I followed this video step by step, and now it works. Whenever I check python3 --version, it returns Python 3.11.1. It also allows me to install another library that depends on python3.11.
Maybe it is worth checking the current version in your terminal and seeing if need to install again the package.
Puppeteer preinstalled chromium wasn't working at all so I installed ubuntu one by apt install chromium-browser and changed executable path to usr/bin/chromium-browser
Now, puppeteer doesn't throw any errors but I'm getting this one:
[935270:935270:1222/220755.094434:ERROR:browser_main_loop.cc(1434)] Unable to open X display.
I tried everything I red on forums but didn't work.
I am just assuming that you are executing this on a terminal only Ubuntu? In my case it was the Ubuntu Windows subsystem for Linux.
As usual when you have a solution the problem is mostly what the error says. There is no display.
Unable to open X display.
There is a solution for this called X virtual framebuffer or xvfb. https://en.wikipedia.org/wiki/Xvfb
The way I understand it it is aiming at acting as a display for all the intents of the client but is not visible to the user.
So one solution for the problem I had was to install xvfb and then run the command that requires a display with this tool. In my case it was protractor but the pupeteer command should look similar:
sudo apt-get install xvfb
xvfb-run --auto-servernum npm run protractor
PS. I got inspired by this blog entry https://remarkablemark.org/blog/2020/12/12/headless-test-in-github-actions-workflow/
Puppet Version: 3.8.7
I have been working on building some system monitoring boxes and have ran into an issue when it comes to installing group yum packages. The normal course of installing packages of course isn't working but I figured that I would at least be able to work around this by including an exec to run the install as a command (like below):
exec { "GNOME Desktop":
command => "/usr/bin/yum -y groups install 'GNOME Desktop'",
timeout => 600,
}
There is an available module on the puppet forge that seems to do what I want but it's not compatible with our version of puppet and we are not in a place to upgrade at this time.
I also tried the setup that was listed in the below server fault question but it also did not work for me:
https://serverfault.com/questions/127460/how-do-i-install-a-yum-package-group-with-puppet
I have also manually been able to run the following command but when I exec it as a puppet command, it fails:
/usr/bin/yum -y groups install "GNOME Desktop"
Why is this? I assumed that puppet is just issuing the command in the exact same way the terminal would?
Changing the time out (or removing it) had zero effect, the issue is with the version of puppet and the ability to install group packages. I ended up installing the desktop environment in my kickstart file and ran puppet for everything else.
I am trying to use selenium on RHEL(Red Hat).
In order to use Chrome browser, I need to install the chromedriver.
Does anyone know how to install this, or what command I need to use? I did not find any resources on this that worked.
Really appreciate any help.
Thank you.
I followed a process which worked for me.
I am using Redhat 7.4, and I've been through a lot to uninstall and install chrome and firefox, got partially successful with chrome and this is what I tried.
In the Linux terminal
Go to the root by typing
su -
Then type
rpm -qa
then (ctrl+f) and search for the keywords "google" or " chrome" or "mozilla" or "firefox" to check whether they are present or not and if they are available, then uninstall( remove ) them by the following command ( if you want to remove them )
yum remove (theWholeName)
this removes the application
Then install chrome by the following process
type this in terminal
wget
https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm
then a package gets downloaded in your home folder
Then type the following command - this installs the packages
yum -y install (name of the downloaded folder with extension)
in this case it will be
yum -y install google-chrome-stable_current_x86_64.rpm
This installs google chrome and creates an icon under Application->Internet->googleChrome
IMPportant:
technically, by clicking this googlechrome icon- this should open the browser, but it doesn't open the browser for me, I dont know whats the problem and I appreciate any suggestion regarding this issue.
I open google chrome through terminal using this command
google-chrome --no-sandbox
this opens the chrome browser.