AWS Lambda Python 3.7 Web Scraping - "Could not get version for Chrome with this command: google-chrome --version" - python-3.x

Question

Via an S3 bucket, I've uploaded a lambda function along with its dependencies as a ZIP file. The lambda function is a web scraper with the following initial code to get the scraper started:
import json
import os
import pymysql
import boto3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--window-size=1280x1696')
chrome_options.add_argument('--user-data-dir=/tmp/user-data')
chrome_options.add_argument('--hide-scrollbars')
chrome_options.add_argument('--enable-logging')
chrome_options.add_argument('--log-level=0')
chrome_options.add_argument('--v=99')
chrome_options.add_argument('--single-process')
chrome_options.add_argument('--data-path=/tmp/data-path')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--homedir=/tmp')
chrome_options.add_argument('--disk-cache-dir=/tmp/cache-dir')
chrome_options.binary_location = os.getcwd() + "/bin/headless-chromium"
browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=chrome_options)
When I try to test the lambda function, I get the following error in the console:
{
"errorMessage": "Could not get version for Chrome with this command: google-chrome --version",
"errorType": "ValueError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 67, in lambda_handler\n browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=chrome_options)\n",
" File \"/var/task/webdriver_manager/chrome.py\", line 24, in install\n driver_path = self.download_driver(self.driver)\n",
" File \"/var/task/webdriver_manager/manager.py\", line 32, in download_driver\n driver_version, is_latest = self.__get_version_to_download(driver)\n",
" File \"/var/task/webdriver_manager/manager.py\", line 23, in __get_version_to_download\n return self.__get_latest_driver_version(driver), True\n",
" File \"/var/task/webdriver_manager/manager.py\", line 17, in __get_latest_driver_version\n return driver.get_latest_release_version()\n",
" File \"/var/task/webdriver_manager/driver.py\", line 54, in get_latest_release_version\n self._latest_release_url + '_' + chrome_version())\n",
" File \"/var/task/webdriver_manager/utils.py\", line 98, in chrome_version\n .format(cmd)\n"
]
}
In response, I tried editing the utils.py file in the webdriver_manager dependency folder, by using other commands like 'chrome --version' and 'chromium-browser --version' instead of 'google-chrome --version' under the function definition of 'chrome_version()', but got the similar error of not being able to the get the chrome version from the new command:
def chrome_version():
pattern = r'\d+\.\d+\.\d+'
cmd_mapping = {
OSType.LINUX: 'google-chrome --version',
OSType.MAC: r'/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version',
OSType.WIN: r'reg query "HKEY_CURRENT_USER\Software\Google\Chrome\BLBeacon" /v version'
}
cmd = cmd_mapping[os_name()]
stdout = os.popen(cmd).read()
version = re.search(pattern, stdout)
if not version:
raise ValueError(
'Could not get version for Chrome with this command: {}'
.format(cmd)
)
return version.group(0)
Can anyone tell me what command I should be using instead of 'google-chrome --version'?

Answer 1

By default, Google Chrome does not exists on the container that runs our lambda functions.
I'm implementing similar solutions but with JavaScript and the way I solve is by using a micro-browser (Chromium) using the following packages:
"chrome-aws-lambda": "^1.19.0",
"puppeteer-core": "^1.19.0"
For Python, here is a tutorial that might help in your situation.
https://robertorocha.info/setting-up-a-selenium-web-scraper-on-aws-lambda-with-python/