Selenium XVFB - Unable to receive message from renderer - linux

Question

Overview:
Selenium scraper works perfectly in headless mode. Spawning a virtual display shows no errors via XVFB:
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
vdisplay.stop()
But when I try to run them together, it errors out with:
[ERROR] SessionNotCreatedException: Message: session not created
from disconnected: Unable to receive message from renderer (Session info: chrome=96.0.4664.0)
Traceback:
Traceback (most recent call last):
File "/var/task/slack_main.py", line 34, in handler
scrape_price(asin_list)
File "/var/task/slack_main.py", line 58, in scrape_price
driver = webdriver.Chrome("/opt/chromedriver",options=options)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog",
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 92, in __init__
RemoteWebDriver.__init__(
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 275, in __init__
self.start_session(capabilities, browser_profile)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 365, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
self.error_handler.check_response(response)
File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
Configuration:
Below is my complete Selenium and XVFB configuration:
from selenium import webdriver
from selenium_stealth import stealth
from xvfbwrapper import Xvfb
vdisplay = Xvfb()
vdisplay.start()
options = webdriver.ChromeOptions()
prefs = {"browser.downloads.dir": "//tmp//", "download.default_directory": "//tmp//", "directory_upgrade": True}
options.add_experimental_option("prefs", prefs)
options.binary_location = '/opt/chrome/chrome'
#options.add_argument('--headless') #toggled on and off when running with or without XVFB
options.add_argument('--no-sandbox')
options.add_argument("--disable-gpu")
options.add_argument("--window-size=1280x1696")
options.add_argument("--single-process")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-dev-tools")
options.add_argument("--no-zygote")
options.set_capability('unhandledPromptBehavior', 'ignore')
options.add_argument("download.default_directory=/tmp")
driver = webdriver.Chrome("/opt/chromedriver",options=options)
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
fix_hairline=True,
)
vdisplay.stop()
driver.close()
Why is it not connecting to the display? My guess is it has something to do with the '--headless' toggle?
Versions and Tools:
Selenium version 3.141.0
xvfbwrapper version 0.2.9
Docker is used to compile and push to AWS Lambda, base image used (no changes are made in docker file with or without XVFB)
Edit:
Found a pull request for XVFB configuration in the Github Repo of my base image. Even used the exact same code from the pull request and I still recieve the exact same error. Maybe this has something to do with AWS offboard?