I am running selenium inside docker and getting this error on this call: driver.get(URL)
Traceback (most recent call last):
File "esdm.py", line 244, in <module>
upload_to_esdm(browser, version_url, args.im_file, args.build, args.user_name, args.password, args.apps_file)
File "esdm.py", line 78, in upload_to_esdm
browser.get(version_url)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_CONNECTION_TIMED_OUT
(Session info: headless chrome=87.0.4280.66)
This is my code:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--js-flags=--max-old-space-size=8196')
driver = webdriver.Chrome(options=chrome_options)
driver.set_window_size(1920, 1080)
driver.get(URL)
And my Dockerfile:
FROM ubuntu:18.04
WORKDIR /src
COPY . /src
RUN apt-get -y update
RUN apt-get install -y python3 python3-pip chromium-chromedriver
# set display port to avoid a crash
ENV DISPLAY=:99
RUN pip3 install selenium
ENTRYPOINT "/bin/bash"
This works properly in windows, but when running in docker it doesn't
Related
I want to deploy a container image with some Python libs and Python code which should be used as AWS lambda function. I followed the following blog post:
https://aws.amazon.com/de/blogs/machine-learning/using-container-images-to-run-pytorch-models-in-aws-lambda/
Locally on my machine I can build and run the Docker image successfully. From Docker perspective I thought this is ok. But when I use AWSSAMCLI, I get the following issue:
F:\AWS>sam build && sam deploy --guided
Your template contains a resource with logical ID "ServerlessRestApi", which is a reserved logical ID in AWS SAM. It could result in unexpected behaviors and is not recommended.
Building codeuri: F:\AWS runtime: None metadata: {'Dockerfile': 'Dockerfile', 'DockerContext': 'F:\\\AWS\\inference-ai'} architecture: x86_64 functions: InferenceFunction
Building image for InferenceFunction function
Setting DockerBuildArgs: {} for InferenceFunction function
Step 1/13 : FROM public.ecr.aws/lambda/python:3.9
---> 7afff3a8d417
Step 2/13 : COPY requirements.txt ./
---> Using cache
---> 3e5e3aa3f81d
Step 3/13 : RUN yum install -y gcc git unzip
---> Using cache
---> 9ce3cda3d21a
Step 4/13 : RUN yum install -y python3-devel.x86_64
---> Using cache
---> ffcd1e85aa7d
Step 5/13 : RUN python3.9 -m pip install cython
---> Using cache
---> 178bccdc8433
Step 6/13 : RUN python3.9 -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
---> Using cache
---> 389d5f5d0a59
Step 7/13 : RUN python3.9 -m pip install git+https://github.com/gautamchitnis/cocoapi.git#cocodataset-master#subdirectory=PythonAPI
---> Using cache
---> 6a1b0cc8c2c7
Step 8/13 : RUN python3.9 -m pip install -r requirements.txt
---> Using cache
---> 4111e9474c42
Step 9/13 : RUN curl https://gitlab.com/username/inference-ai/-/archive/main/inference-ai-main.zip --output project.zip
---> Using cache
---> 580214e52eb2
Step 10/13 : RUN unzip project.zip
---> Using cache
---> 7d49342aeca1
Step 11/13 : RUN mv inference-ai-main inference-ai
---> Using cache
---> c5ed0c584cbd
Step 12/13 : WORKDIR inference-ai
---> Using cache
---> 768f6092aa72
Step 13/13 : CMD ["test-infer-with-coco.lambda_handler"]
---> Using cache
---> d09be07424dd
Successfully built d09be07424dd
Successfully tagged inferencefunction:latest
Traceback (most recent call last):
File "runpy.py", line 194, in _run_module_as_main
File "runpy.py", line 87, in _run_code
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\__main__.py", line 12, in <module>
cli(prog_name="sam")
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metric.py", line 166, in wrapped
raise exception # pylint: disable=raising-bad-type
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metric.py", line 124, in wrapped
return_value = func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\utils\version_checker.py", line 41, in wrapped
actual_result = func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\cli\main.py", line 87, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 182, in cli
do_cli(
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 262, in do_cli
ctx.run()
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\build_context.py", line 255, in run
modified_template = builder.update_template(
File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\build\app_builder.py", line 321, in update_template
absolute_output_path = pathlib.Path(
File "pathlib.py", line 1180, in resolve
File "pathlib.py", line 205, in resolve
OSError: [WinError 123] Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch: 'inferencefunction:latest'
F:\AWS>
I have no idea why an invalid file will be used (colon character). How can I solve this issue?
Dockerfile:
FROM public.ecr.aws/lambda/python:3.9
COPY requirements.txt ./
RUN yum install -y gcc git unzip
RUN yum install -y python3-devel.x86_64
RUN python3.9 -m pip install cython
RUN python3.9 -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
RUN python3.9 -m pip install git+https://github.com/gautamchitnis/cocoapi.git#cocodataset-master#subdirectory=PythonAPI
RUN python3.9 -m pip install -r requirements.txt
RUN curl https://gitlab.com/username/inference-ai/-/archive/main/inference-ai-main.zip --output project.zip
RUN unzip project.zip
RUN mv inference-ai-main inference-ai
WORKDIR inference-ai
CMD ["test-infer-with-coco.lambda_handler"]
template.yaml:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Globals:
Function:
Timeout: 50
MemorySize: 5000
Api:
BinaryMediaTypes:
- image/png
- image/jpg
- image/jpeg
Resources:
InferenceFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Architectures:
- x86_64
Events:
Inference:
Type: Api
Properties:
Path: /inference-ai
Method: post
Metadata:
Dockerfile: Dockerfile
DockerContext: ./inference-ai
# DockerTag: python3.9-v1
Outputs:
InferenceApi:
Description: "API Gateway endpoint URL for Prod stage for Inference function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/inference-ai/"
InferenceFunction:
Description: "Inference Lambda Function ARN"
Value: !GetAtt InferenceFunction.Arn
InferenceFunctionIamRole:
Description: "Implicit IAM Role created for Inference function"
Value: !GetAtt InferenceFunction.Arn
I am getting an error with this playbook and am not sure where to look. Perhaps something isn't defined right in my host file? (I'm told the playbook is good)
YML Playbook
- hosts: fortigates
collections:
- fortinet.fortios
connection: httpapi
vars:
vdom: "root"
ansible_httpapi_use_ssl: yes
ansible_httpapi_validate_certs: no
ansible_httpapi_port: 443
tasks:
- name: Configure global attributes.
fortios_system_global:
vdom: "{{ vdom }}"
system_global:
admintimeout: "23"
hostname: "FortiGate02"
Host file
[fortigates]
fortigate01 ansible_host=192.168.0.103 ansible_user="admin" ansible_password="password"
[fortigates:vars]
ansible_network_os=fortinet.fortios.fortios
#ansible_python_interpreter=/usr/bin/python3
Error Output
TASK [Configure global attributes.] ****************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.module_utils.connection.ConnectionError: addinfourl instance has no attribute 'getheaders'
fatal: [fortigate01]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File "/root/.ansible/tmp/ansible-local-454799bt3QT/ansible-tmp-1593138436.55-45584-34169098305172/AnsiballZ_fortios_system_global.py", line 102, in \n _ansiballz_main()\n File "/root/.ansible/tmp/ansible-local-454799bt3QT/ansible-tmp-1593138436.55-45584-34169098305172/AnsiballZ_fortios_system_global.py", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File "/root/.ansible/tmp/ansible-local-454799bt3QT/ansible-tmp-1593138436.55-45584-34169098305172/AnsiballZ_fortios_system_global.py", line 40, in invoke_module\n runpy.run_module(mod_name='ansible_collections.fortinet.fortios.plugins.modules.fortios_system_global', init_globals=None, run_name='main', alter_sys=True)\n File "/usr/lib/python2.7/runpy.py", line 188, in run_module\n fname, loader, pkg_name)\n File "/usr/lib/python2.7/runpy.py", line 82, in _run_module_code\n mod_name, mod_fname, mod_loader, pkg_name)\n File "/usr/lib/python2.7/runpy.py", line 72, in _run_code\n exec code in run_globals\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_global.py", line 2075, in \n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_global.py", line 2043, in main\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_global.py", line 1544, in fortios_system\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/modules/fortios_system_global.py", line 1533, in system_global\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 173, in set\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 146, in get_mkey\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 137, in get_mkeyname\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible_collections/fortinet/fortios/plugins/module_utils/fortios/fortios.py", line 126, in schema\n File "/tmp/ansible_fortios_system_global_payload_CQaHFo/ansible_fortios_system_global_payload.zip/ansible/module_utils/connection.py", line 185, in rpc\nansible.module_utils.connection.ConnectionError: addinfourl instance has no attribute 'getheaders'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
I have installed the Ansible Galaxy package, as per the documentation
# ansible-galaxy collection install fortinet.fortios
Process install dependency map
Starting collection install process
Skipping 'fortinet.fortios' as it is already installed
Same issue on Ubuntu 18.04 (WSL).
I fixed it by installing ansible with pip3.
# remove ansible
sudo apt remove ansible
# install python3 & pip3
sudo apt install python3 python3-pip
# install ansible with pip3
pip3 install ansible --user
# update the environment PATH variable for ansible commands
echo "export PATH=$PATH:$HOME/.local/bin" >> ~/.bashrc
source ~/.bashrc
# install fortios module
ansible-galaxy collection install fortinet.fortios
I am trying to run a Python + Selenium script in headless mode with Firefox using Xvfb but I am getting errors. There is not much documents or guides available for Xvfb to troubleshoot the issue so looking for assistance here.
Environment info:
OS: CentOS release 6.5 (Minimal installation)
Xvfb: xorg-x11-server-Xvfb-1.15.0
Firefox: 52.8.0
geckodriver: 0.24.0
Python: 3.6.7
Steps followed:
Once done installing the above-mentioned requirements. I started a virtual display with:
$Xvfb :1 -ac &
Also, I tried with:
$Xvfb :1 -screen 0 1024x768x24 -extension RANDR &
And then I set Display variable:
export DISPLAY=:1
When I tried to initiate Selenium WebDriver in Python console I am getting the error Connection refused:
> from selenium import webdriver
> from pyvirtualdisplay import Display
> display = Display(visible=0, size=(800, 600))
> display.start()
> driver = webdriver.Firefox()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Any help or suggestion will be much appreciated.
There is a wrapper around xvfb called PyVirtualDisplay that seems designed for this exact solution. If you simply do a pip install pyvirtualdisplay the following script should run a headless Firefox window:
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
# now Firefox will run in a virtual display.
# you will not see the browser.
browser = webdriver.Firefox(executable_path="/Users/username/Location/geckodriver")
browser.get('http://www.google.com')
print browser.title
browser.quit()
display.stop()
I'm trying to run chrome headless with selenium in Python.
I tried xvfb-run -a python3 message.py
And i get this error:
Traceback (most recent call last):
File "message.py", line 388, in <module>
initialize()
File "message.py", line 54, in initialize
driver = webdriver.Chrome(chrome_options = chrome_options)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/chrome/webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.33.506178 (69ae10f91723897591ef1a3b465aa5d35011eb5e),platform=Linux 3.13.0-96-generic x86)
And initialize():
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--verbose")
chrome_options.add_argument("--log-path=chrome.log")
chrome_options.add_argument("window-size=1200x600")
chrome_options.add_argument("user-data-dir=\\")
driver = webdriver.Chrome(chrome_options = chrome_options)
driver.get('http://google.com')
I'm using Python 3.5 and chromedriver 2.33.506178 for Linux32 (Ubuntu as Server without Desktop)
I tested the same code on my other pc with linux mint, all works fine.
Hope you can help me :)
After checking my actual rights of writing/reading/executing I found my mistake. chromedriver weren't able to load while trying to execute with root rights.
After I changed chromedrivers permission all works fine now.
Here's my code for opening Chrome:
from selenium import webdriver
driver=webdriver.Chrome('C:\\Users\\Imran\\AppData\\Local\\Programs\\Python\\Python36\\selenium\\chromedriver.exe')
Error when running program:
Traceback (most recent call last):
File "C:/Users/Imran/PycharmProjects/webscraping/WF Item.py", line 6, in <module>
driver = webdriver.Chrome('C:\\Users\\Imran\\AppData\\Local\\Programs\\Python\\Python36\\selenium\\chromedriver.exe')
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 69, in __init__
desired_capabilities=desired_capabilities)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Users\Imran\AppData\Local\Programs\Python\Python36\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: session not created exception
from unknown error: Runtime.executionContextCreated has invalid 'context': {"auxData":{"frameId":"27028.1","isDefault":true},"id":1,"name":"","origin":"://"}
(Session info: chrome=54.0.2840.71)
(Driver info: chromedriver=2.23.409699 (49b0fa931cda1caad0ae15b7d1b68004acd05129),platform=Windows NT 6.3.9600 x86_64)
Getting the following exception:
selenium.common.exceptions.WebDriverException: Message: session not created exception
Update your chromedriver to 2.24 version from the given url:
http://chromedriver.storage.googleapis.com/index.html?path=2.24/
That worked for me.
Setting LANG=en_US.UTF-8 before executing my script works for me.
See this: https://bugs.chromium.org/p/chromedriver/issues/detail?id=1552#c43
In Python:
import os
os.environ["LANG"] = "en_US.UTF-8"
To install the chromedriver on windows, download and unzip the chromedriver_win32.zip and add the path to the folder containing the chromedriver.exe in the environment variables setting [System properties - Advanced system settings - Environment variables].