UndetectedChromedriver Extension Issue - python-3.x

I can't use extensions on UndetectedChromedriver PYPI Package (Python). If I use it with normal selenium its works, but not with this package. I tried to install extensions directly from webstore, but Chrome Webstore Alert is not an Alert to handle with selenium is a Window Event, so we need to use AutoIT, Pyautogui, etc... To handle that.
The only thing is working is loading profiles, but... I'm working for multiprocess windows, is working, but I need to create houndred of windows and then delete them. And I can't clone profiles, because UndetectedChromedriver doesn't work, i need to create manually.
Finally i tried with Google Chrome Enterprise Bundle, then I used Extensions policy to install forced the extension for all profiles. And yes, is working, but if I enabled that, selenium, doesn't work properly.
The error traceback log is:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\threading.py", line 950, in _bootstrap_inner
self.run()
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\andre\OneDrive\Documentos\(A1)_Inicio\(A2)_CyberEspacio\LAB\(A1)_Programador123\(A1)_Programming_(Section)\VSCode Snippets\python\selenium\app.py", line 72, in test
seleniumCaptchaSolver.reCaptchaServiceLogin(apiKey='MYAPIKEY', solverType = SeleniumCaptchaSolverType().Capmonster)
File "C:\Users\andre\OneDrive\Documentos\(A1)_Inicio\(A2)_CyberEspacio\LAB\(A1)_Programador123\(A1)_Programming_(Section)\VSCode Snippets\python\selenium\modules\seleniumCaptchaSolver.py", line 103, in reCaptchaServiceLogin
self.__driver.get('chrome-extension://pabjfbciaedomjjfelfafejkppknjleh/popup.html')
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\site-packages\undetected_chromedriver\__init__.py", line 535, in get
return super().get(url)
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 447, in get
self.execute(Command.GET, {'url': url})
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 435, in execute
self.error_handler.check_response(response)
File "C:\Users\andre\anaconda3\envs\selenium-env\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot determine loading status
from disconnected: received Inspector.detached event
(Session info: chrome=103.0.5060.134)
This happen only when chrome-extension://pabjfbciaedomjjfelfafejkppknjleh/popup.html is opened to login (Send APi Key). I can login etc... But when Policy is Activated I can't because of that issue.
Anyone here know how to fix that or properly use extensions in UndetctedChromedriver?
Note: This error only happens if i load chrome-extension://pabjfbciaedomjjfelfafejkppknjleh/popup.html link, others links works.

I found this solution:
import undetected_chromedriver as uc
import os
working_dir = os.getcwd()
# Im using proxy extension
proxy_plugin = f'{working_dir}/proxy_plugin'
options = uc.ChromeOptions()
options.add_argument(f'--load-extension={proxy_plugin}')
# {proxy_plugin} path to extension folder, I tried to import .zip file
# and this doesnt working, maybe you can try import .crx file
# Also, I use extensions.ui.developer_mode
options.add_experimental_option('prefs', { 'extensions.ui.developer_mode': True })
driver = uc.Chrome(options = options)
extensions.ui.developer_mode
Yet, I seen this pages:
Load unpacked Chrome extension programmatically
Installing extension into V2

Related

Python xlwings: EventError: Command failed: Parameter error. (-50)

I wish I could use python to execute the Excel macro, so I tried to use the package xlwings to implement it.
The OS of my laptop is macOS Catalina (ver.: 10.15.7), my compiler is PyCharm (ver.: 2021.2.3), my Python version is 3.8.8, I used Anaconda (Ver.: 22.11.1) as my interpreter, my excel version is 16.66.1 (Microsoft Excel for Mac).
I faced the error "Command Error -1743: The User has declined permission" when I tried to use this package originally, and I solved this issue by installing an old compiler & using the old version of the compiler to run my code. My privacy setting for automation in the app Setting was shown below: (This is NOT the question I want to ask, but I'm not sure if it is also related to the issue I faced, so I still attached it here. I had uninstalled the old version of my compiler already.)
I wish I could implement an existing macro (called Hi) in an existing Excel file (called [StakeOverflow]HelloWorld.xlsm) through Python (represented as MY_PYTHON_FILE.py below), like the snapshot below:
My Excel file and my python code were stored on OneDrive. My macro code was shown below:
Sub Hi()
MsgBox "Good morning!"
End Sub
My python code was shown below:
import xlwings as xw
import time
wb = xw.Book('/Users/<MY NAME>/OneDrive/MY PATH DETAILS/[StakeOverflow]HelloWorld.xlsm')
time.sleep(10)
app = wb.app
macro_vba = app.macro("Hi")
macro_vba()
The code looks really simple, but I still faced the error. My Excel was not opened automatically, and I even could not open the Excel file manually thereafter. The error was shown below:
/Users/.../.conda/envs/Program/bin/python "/Users/.../OneDrive/.../MY_PYTHON_FILE.py"
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4914, in open
impl= self.impl(name)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/_xlmac.py", line 366, in __call__
raise KeyError(name_or_index)
KeyError: '[stakeoverflow] helloworld.xlsm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 482, in __call___
return self.AS_appdata.target().event (self._code, params, atts, codecs=self.AS_appdata).send(timeout, sendflags)
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/aem/aemsend.py", line 92, in send
raise EventError(errornum, errormsg, eventresult)
aem.aemsend.EventError: Command failed: Parameter error. (-50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.../OneDrive/.../MY_PYTHON_FILE.py", line 18, in <module>
wb = xw.Book('/Users/.../OneDrive/.../[stakeoverflow] helloworld.xlsm')
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 876, in __init__
impl= app.books.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/main.py", line 4921, in open
impl = self.impl.open(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/xlwings/ xlmac.py", line 420, in open
self.app.xl.open_workbook(
File "/Users/.../.conda/envs/Program/lib/python3.9/site-packages/aeosa/appscript/reference.py", line 518, in __call__
raise CommandError(self, (args, kargs), e, self.AS_appdata) from e
appscript.reference.CommandError: Command failed:
OSERROR: -50
MESSAGE: Parameter error.
COMMAND: app(pid=1647).open_workbook (workbook_file_name='/users/.../onedrive/.../[stakeoverflow] helloworld.xlsm', update_links=k.do_not_update_links, read_only=None, format=None, password=None, write_reserved_password=None, ignore_read_only_recommended=None, origin=None, delimiter=None, editable=None, notify=None, converter=None, add_to_mru=None, timeout=-1)
Process finished with exit code 1
I tried to use terminal to run my python code, but I could not solve the problem, either.
I tried to open the Excel file manually thereafter, the error was shown below:
Excel cannot open the file ’[StakeOverflow]HelloWorld.xlsm’
because the file format or file extension is not valid. Verify
that the file has not been corrupted and that the file
extension matches the format of the file.
I tried to Google this error, but few solutions was found. It seems that it is related to the issue of external storage location. I tried to create another Excel file with the same name & macro code on my desktop and try again, and the error would disappear. (However, our company stored the files on OneDrive, so I wish I could utilise the file online.)
Just wondering if anyone here faced this situation before?
I found the answer by myself today. The issue is related to the naming issue rather than the permission issue.
As we could see that the error is KeyError: '[stakeoverflow] helloworld.xlsm', which implies that the problem is here. (Maybe because the system could not find the Excel file with this name.)
I tried to change the name from [stakeoverflow] helloworld.xlsm to helloworld.xlsm, and the error was gone. It seems that when xlwings want to open an Excel file, it would check the validity of the file name and change the name into smaller cases. If the file name contains special characters which are not allowed (e.g., "[]"), then the error would occur.
Notice that I could store the Excel file with these special characters in our laptop & OneDrive, but xlwings did not accept them.
Hope it is helpful to those who face this issue when using xlwings!

How can I open a file with its correct program (e.g. ".blend" with Blender and ".webloc" with Chrome) inside a Python program?

I'm working with IDLE on Mac and I'm trying to make a small program that opens a random file from a folder (it is actually a test for a bigger project). In the folder, I have many types of files like ".blend", ".m4a", ".py" and ".webloc" but I expect to have even more in the future. I would like my code to open a random one with their respective program (Blender, QuickTime Player, IDLE, Chrome...) but so far I have not found any way to do it. Is it possible? The most I've been able to do is open Google Chrome from my Windows computer. It doesn't work on my Mac (maybe because it is .app instead of .exe?) and I can only open programs, but not files. Here's the code I used for that:
import subprocess
subprocess.Popen(['C:\Program Files (x86)\Google\Chrome\Application\\chrome.exe', '-new-tab'])
When I enter that on Mac (but with the correct file path for Mac):
import subprocess
subprocess.Popen(['/Applications/Google Chrome.app', '-new-tab'])
It gives me this error (could it be because the file path is written incorrectly? I copied it with right-click on the Chrome file and clicking "copy as path"):
>>>
=============== RESTART: /Users/jaimewalter/Desktop/Test/Test3.py ==============
Traceback (most recent call last):
File "/Users/jaimewalter/Desktop/Test/Test3.py", line 3, in <module>
subprocess.Popen(['/Applications/Google Chrome.app', '-new-tab'])
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/Applications/Google Chrome.app'
>>>
And here's my code for the random file selector:
import random
files = ["Test1.blend", "Test2.m4a", "Test3.py", "Test4.webloc"]
open_this = random.choice (files)
print(open_this)
if open_this == "Test1.blend":
print("Opening Test1.blend")
#now it should open Test1.blend on a new Blender window (/Users/jaimewalter/Desktop/Test/Test1.blend)
elif open_this == "Test2.m4a":
print("Opening Test2.m4a")
#now it should open Test2.m4a on a new QuickTime Player window (/Users/jaimewalter/Desktop/Test/Test2.m4a)
elif open_this == "Test3.py":
print("Opening Test3.py")
#now it should open Test3.py on a new IDLE window or preferably runs the code inside directly it if that's possible (/Users/jaimewalter/Desktop/Test/Test3.py)
elif open_this == "Test4.webloc":
print("Opening Test4.webloc")
#now it should open Test4.webloc on a new Chrome or Safari window (/Users/jaimewalter/Desktop/Test/Test4.webloc)
What should I use to open the files inside the code? Thanks in advance
I already solved it. I used
import subprocess
subprocess.call(["open", "Test1.blend"])
I think with Linux it's xdg-open

Selenium code running locally but not on server

Both the server and my computer have geckodriver 0.26.0, Firefox 71, and Selenium 3.141.0.
My computer has MacOS Mojave with python 3.8 and the server is CentOS 7 with python 3.7. The code runs perfectly on my computer, but it returns errors on the server.
I don't remember how, but I have been getting different errors depending on if I add breakpoints to it, or if I run it in terminal or submit the job in SLURM.
On terminal:
File "Main.py", line 230, in <module>
main()
File "Main.py", line 179, in main
dfs=get_data(stations, inidate, findate)
File "Main.py", line 113, in get_data
list_files=return_list_day(date)
File "Main.py", line 66, in return_list_day
driver.get(webdir)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
or
Submitted on SLURM (and most often error):
Traceback (most recent call last):
File "Main.py", line 230, in <module>
main()
File "Main.py", line 179, in main
dfs=get_data(stations, inidate, findate)
File "Main.py", line 113, in get_data
list_files=return_list_day(date)
File "Main.py", line 66, in return_list_day
driver.get(webdir)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
self.execute(Command.GET, {'url': url})
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/user/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Timeout loading page after 300000ms
I can use Selenium without issues in another unrelated code that finishes perfectly fine. So I don't understand why Selenium is having a hard time here or where it comes from.
The code fails here:
## Commented parts are cuz I tried running with the MOZ_HEADLESS tag on terminal
## Didnt make a difference as far as I could tell.
#options = Options()
#options.headless = True
driver = webdriver.Firefox()#options=options)
driver.get(webdir) ##<-- HERE
## Added these because of other replies to this issue I found
driver.implicitly_wait(7)
time.sleep(3)
.
.
.
driver.close()
That bit of codes retrieves a list from the webpage and is ran inside a for-loop. The error is not in the retrieving part, is in driver.get(webdir). I can't share the website since it's literally like looking into the server of a partner institution. webdir is a directory and I am basically waiting till its contents are loaded so I can retrieve its file names.
I know you can't help much without a website, but do the errors I've shown give any indication as to what the problem might be?. Can the retrieving of a specific website behave differently based on OS? I have googled the errors, found questions here, read through them and applied them to see how it changed, and nothing did or I got a different error (either one of the two above).
I found this that states incompatibility between gecko and Mozilla, but since I can successfully run another (unrelated) code with the exact calling and usage of Selenium (only different URL given) then I don't think that's my issue.
Thanks for any help! Let me know what other information I could give that might help.
Edit:
It is not the same as the question that was linked, since I have given it sleep time and didn't change anything. It has 40 GB of ram allocated so its not dying out of too little memory. Which are the solutions shown in this question.
For anyone with this issue, the server updated Firefox to version 79.0 and now it works without any issues. Nothing else was changed as far as I was let known.
I assume the version change fixed it but I don't know exactly how. It's worth a try if anyone else was experiencing the same as I was with different errors depending on how it was run.

Key error while updating all modules in oddo10

using command - python3 odoo-bin --addons=addons,/opt/git_addons/project_abcd -u all &
when i tryied to update modules on server, I am geeting Internal server Error and Error log says:
Traceback (most recent call last):
File "/opt/odoo/odoo/modules/registry.py", line 83, in new
odoo.modules.load_modules(registry._db, force_demo, status, update_module)
File "/opt/odoo/odoo/modules/loading.py", line 373, in load_modules
force, status, report, loaded_modules, update_module, models_to_check)
File "/opt/odoo/odoo/modules/loading.py", line 270, in load_marked_modules
perform_checks=perform_checks, models_to_check=models_to_check
File "/opt/odoo/odoo/modules/loading.py", line 153, in load_module_graph
registry.setup_models(cr, partial=True)
File "/opt/odoo/odoo/modules/registry.py", line 300, in setup_models
model._setup_fields(partial)
File "/opt/odoo/odoo/models.py", line 2853, in _setup_fields
field.setup_full(self)
File "/opt/odoo/odoo/fields.py", line 505, in setup_full
self._setup_regular_full(model)
File "/opt/odoo/odoo/fields.py", line 2178, in _setup_regular_full
invf = comodel._fields[self.inverse_name]
KeyError: 'standard_id'
Please help to resolve this error.
Please find the standard_id field in all modules.
Upgrade module which have standard_id field.
If you update -u all with this command line interface then it'll update all your base module first and then you custom modules.
So it might be the reason where your module consist this field and odoo registry can't find it.
With this information it's impossible to say reason for this. One of your modules is trying to refer to a field named stadard_id which doesn't exist.
Try to update your modules one by one and see which one gives this error. Then it's easier to troubleshoot it further.
There may be some dependencies missing from __manifest__.py file.

How to use paralledots api in app engine?

I want to check text similarity using paralleldots api in app engine, but when setting the api key in app engine using.
paralleldots.set_api_key("XXXXXXXXXXXXXXXXXXXXXXXXXXX")
App engine giving Error:
with open('settings.cfg', 'w') as configfile:
File "/home/ulti72/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/stubs.py", line 278, in __init__
raise IOError(errno.EROFS, 'Read-only file system', filename)
IOError: [Errno 30] Read-only file system: 'settings.cfg'
INFO 2019-03-17 10:43:59,852 module.py:835] default: "GET / HTTP/1.1" 500 -
INFO 2019-03-17 10:46:47,548 client.py:777] Refreshing access_token
ERROR 2019-03-17 10:46:50,931 wsgi.py:263]
Traceback (most recent call last):
File "/home/ulti72/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/home/ulti72/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/home/ulti72/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/home/ulti72/Desktop/koda/main.py", line 26, in <module>
paralleldots.set_api_key("7PR8iwo42DGFB8qpLjpUGJPqEQHU322lqTDkgaMrX7I")
File "/home/ulti72/Desktop/koda/lib/paralleldots/config.py", line 13, in set_api_key
with open('settings.cfg', 'w') as configfile:
File "/home/ulti72/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/stubs.py", line 278, in __init__
raise IOError(errno.EROFS, 'Read-only file system', filename)
IOError: [Errno 30] Read-only file system: 'settings.cfg'
The paralleldots api, seems to want to save a settings.cfg file to the local filesystem in response to that call. Which is not allowed in the 1st generation standard environment and only allowed for files in the /tmp filesystem in the 2nd generation.
The local development server was designed for the 1st generation standard env and enforces the restriction with that error. It has limited support for the 2nd generation env, see Python 3.7 Local Development Server Options for new app engine apps.
Things to try:
check if specifying the location of the settings.cfg is supported and if so make it reside under /tmp. Maybe the local development server allows that or you switch to some other local development method than the development server.
check if saving the settings using an already open file handler is supported and, if so, use one obtained from Cloud Storage client library, something along these lines: How to zip or tar a static folder without writing anything to the filesystem in python?
check if set_api_key() supports some other method of persisting the API key than saving the settings to a file
check if it's possible to specify the API key for every subsequent call so you don't have to persist it using set_api_key() (maybe using a common wrapper function for convenience)

Resources