Unable to add a cookie using Selenium & Splinter - python-3.x

Per below, I'm not sure how to troubleshoot this pretty simple usage scenario.
I have script (that I run about once a month) that functionally does the identical thing and which used to work as of a month ago.
I'd appreciate any pointers on places to start looking into why this does not work.
$ python3
Python 3.6.1 (default, Mar 23 2017, 16:49:06)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from selenium import webdriver
>>> from splinter import Browser
>>> chrome_options = webdriver.ChromeOptions()
>>> browser = Browser('chrome')
>>> browser.cookies.add({'aaa':'bbb'})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/dummyuser/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/splinter/driver/webdriver/cookie_manager.py", line 28, in add
self.driver.add_cookie({'name': key, 'value': value})
File "/Users/dummyuser/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 708, in add_cookie
self.execute(Command.ADD_COOKIE, {'cookie': cookie_dict})
File "/Users/dummyuser/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/Users/dummyuser/.virtualenvs/my-virtualenv/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unable to set cookie
(Session info: chrome=62.0.3202.94)
(Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.13.1 x86_64)

you should open the url first, and load the cookies, then you can open the next url with cookies.you can also open like that if you want open the same url:
driver = webdriver.Chrome(executable_path=r'X:\home\xxx\chromedriver.exe')
cookies = pickle.load(open("cookies.pkl", "rb"))
driver.get("https://www.douban.com/")
for cookie in cookies:
driver.add_cookie(cookie)
driver.get("https://www.douban.com/")
hope this helps

The answer of Florent B. works for me as well, just want to put it into the right place.
browser.cookies.add needs to be called after some browser.visit(...)
see Florent's comment:
The method browser.cookies.add is bound to the current domain which is undefined in your example. You need to set the domain first with driver.get('http://...')

Related

python webbrowser module not working on mac 16.2.1

I'm trying to use the webbrowser module to open a web page.
The first attempt leads to nowhere. code hangs, nothing happens.
After a keyboard interrupt the same command - and everything works just fine.
>>> import webbrowser
>>> webbrowser._browsers
{}
>>> webbrowser.register_standard_browsers()
^CTraceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/webbrowser.py", line 555, in register_standard_browsers
raw_result = subprocess.check_output(cmd, stderr=subprocess.DEVNULL)
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 505, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1141, in communicate
stdout = self.stdout.read()
KeyboardInterrupt
>>> webbrowser._browsers
{'macosx': [None, <webbrowser.MacOSXOSAScript object at 0x10f08c5e0>], 'chrome': [None, <webbrowser.MacOSXOSAScript object at 0x10f08c670>], 'firefox': [None, <webbrowser.MacOSXOSAScript object at 0x10f08c760>], 'safari': [None, <webbrowser.MacOSXOSAScript object at 0x10efb4370>]}
Traceback tells me it stucks somewhere in the register process.
So i tried to register all standard browsers -> same behavior. But after keyboard interrupt webbrowser._browsers returned registered browsers.
Tried to register chrome browser - also the same.
webbrowser._browsers['chrome'][1].open("https://www.google.com/")
also needs an interrupt but that command opens the site directly after keyboard interrupt
Problem is the same from py version 3.9 - 3.11. Also on pycharm within or without a virtual environment.
Can anybody reproduce this or tell me what's going on?

Got error from message.py - paramiko "object of type 'bool' has no len()" when executing fabric.Connection.run()

I use fabric 2.6.0, paramiko 2.9.2 and invoke 1.4.0
Is this bug or something incompatible I got an error like this.
File "/usr/local/lib/python3.7/dist-packages/paramiko/message.py",
line 274, in add_string
self.add_int(len(s)) TypeError: object of type 'bool' has no len()
when I set dry=True I got like this.
>>> conn.run('touch hello.txt', dry=True)
touch hello.txt
<Result cmd='touch hello.txt' exited=0>
Here's the complete Error that I got.
Python 3.7.5 (default, Dec 9 2021, 17:04:37)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fabric import Connection
>>> conn = Connection('192.168.1.16')
>>> conn.open()
>>> conn.is_connected
True
>>> conn.run('touch hello.txt')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<decorator-gen-3>", line 2, in run
File "/usr/local/lib/python3.7/dist-packages/fabric/connection.py", line 30, in opens
return method(self, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/fabric/connection.py", line 723, in run
return self._run(self._remote_runner(), command, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/invoke/context.py", line 102, in _run
return runner.run(command, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/invoke/runners.py", line 380, in run
return self._run_body(command, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/invoke/runners.py", line 431, in _run_body
self.start(command, self.opts["shell"], self.env)
File "/usr/local/lib/python3.7/dist-packages/fabric/runners.py", line 57, in start
self.channel.update_environment(env)
File "/usr/local/lib/python3.7/dist-packages/paramiko/channel.py", line 72, in _check
return func(self, *args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/paramiko/channel.py", line 332, in update_environment
self.set_environment_variable(name, value)
File "/usr/local/lib/python3.7/dist-packages/paramiko/channel.py", line 72, in _check
return func(self, *args, **kwds)
File "/usr/local/lib/python3.7/dist-packages/paramiko/channel.py", line 361, in set_environment_variable
m.add_string(value)
File "/usr/local/lib/python3.7/dist-packages/paramiko/message.py", line 274, in add_string
self.add_int(len(s))
TypeError: object of type 'bool' has no len()
and here my ssh config
Host *
Port 22
User ubuntu
IdentityFile /home/ubuntu/.ssh/id_rsa
when I run ssh 192.168.1.16 from shell can successfully connect with remote machine.
After debugging for a while, I realize that this error happen because of miss configuration.
The thing that actually happen is that I put some environment variable on fabric.yaml so that will loaded automatically when fabric run and the problem happen when I want to open connection to the remote server.
What happen in the back when I run command with fabric is, fabric will open ssh connection using paramiko and then try to send the local environment variable that I set in fabric.yaml to the remote machine (this thing that cause the error)
You can't send environment variable form local machine to remote using ssh without put the setting into your sshd_config for the SendEnv and AcceptEnv for your local and remote machine or the alternatives is enable PermitUserEnvironment yes this only for remote machine and don't forget to create environment file in your remote home directory (home when you do ssh).
Hope my mistake can help another who also has similiar problem.

Error configuring a USB device using pyvisa

So, as context, I'm writing some controller software in Python for a RTM3004 oscilloscope for laboratory use. I had it working just fine on my laptop (Win10) and the previous laboratory computer (Linux), but upon changing to a new measurement computer, I can no longer connect to the device using Pyvisa. Here is a console log of the error, which happens whenever I attempt to connect to the device via Pyvisa:
Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvisa as visa
>>> rm= visa.ResourceManager()
>>> rm.list_resources()
('ASRL1::INSTR', 'ASRL3::INSTR', 'USB0::2733::470::103028::0::INSTR')
>>> ID='USB0::2733::470::103028::0::INSTR'
>>> instr=rm.open_resource(ID)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa\highlevel.py", line 3304, in open_resource
res.open(access_mode, open_timeout)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa\resources\resource.py", line 297, in open
self.session, status = self._resource_manager.open_bare_resource(
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa\highlevel.py", line 3232, in open_bare_resource
return self.visalib.open(self.session, resource_name, access_mode, open_timeout)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa_py\highlevel.py", line 167, in open
sess = cls(session, resource_name, parsed, open_timeout)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa_py\sessions.py", line 323, in __init__
self.after_parsing()
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa_py\usb.py", line 81, in after_parsing
self.interface = self._intf_cls(
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\pyvisa_py\protocols\usbtmc.py", line 293, in __init__
self.usb_dev.set_configuration()
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\usb\core.py", line 905, in set_configuration
self._ctx.managed_set_configuration(self, configuration)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\usb\core.py", line 113, in wrapper
return f(self, *args, **kwargs)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\usb\core.py", line 159, in managed_set_configuration
self.backend.set_configuration(self.handle, cfg.bConfigurationValue)
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\usb\backend\libusb0.py", line 509, in set_configuration
_check(_lib.usb_set_configuration(dev_handle, config_value))
File "C:\Users\Hbeam\AppData\Local\Programs\Python\Python39\lib\site-packages\usb\backend\libusb0.py", line 447, in _check
raise USBError(errmsg, ret)
usb.core.USBError: [Errno None] b'libusb0-dll:err [set_configuration] could not set config 1: win error: The parameter is incorrect.\r\n'
As well as the information I get out of pyvisa-info:
Machine Details:
Platform ID: Windows-10-10.0.18362-SP0
Processor: Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
Python:
Implementation: CPython
Executable: c:\users\hbeam\appdata\local\programs\python\python39\python.exe
Version: 3.9.4
Compiler: MSC v.1928 64 bit (AMD64)
Bits: 64bit
Build: Apr 6 2021 13:40:21 (#tags/v3.9.4:1f2e308)
Unicode: UCS4
PyVISA Version: 1.11.3
Backends:
ivi:
Version: 1.11.3 (bundled with PyVISA)
Binary library: Not found
py:
Version: 0.5.2
ASRL INSTR: Available via PySerial (3.5)
USB INSTR: Available via PyUSB (1.1.1). Backend: libusb0
USB RAW: Available via PyUSB (1.1.1). Backend: libusb0
TCPIP INSTR: Available
TCPIP SOCKET: Available
GPIB INSTR:
Please install linux-gpib (Linux) or gpib-ctypes (Windows, Linux) to use this resource type. Note that installing gpib-ctypes will give you access to a broader range of funcionality.
No module named 'gpib'
I've been racking my brain over this for two workdays now with no solution in sight, so I thought it best to ask for help, since I'm obviously no programming genius. Thank you in advance.

Python Selenium: How to use debugger_address option in chrome driver for Remote Debugging?

I want to launch my selenium chrome web driver to current existing session.
So I am trying to connect via remote debugging instead of starting a new instance.
https://sites.google.com/a/chromium.org/chromedriver/help/operation-not-supported-when-using-remote-debugging
So what I did is launched chrome at 127.0.0.1:6813 by cmd.
C:\Program Files (x86)\Google\Chrome\Application>chrome.exe '127.0.0.1:6813'
A new instance of chrome started.
Now in my code I added debugger_address="127.0.0.1:6813" using chrome options.
chrome_options.debugger_address="127.0.0.1:6813"
But my Spyder IDE gives following error:
runfile('C:/Users/Gupta Niwas/Downloads/Programming/Projects/Mi/temp7.py', wdir='C:/Users/Gupta Niwas/Downloads/Programming/Projects/Mi')
Traceback (most recent call last):
File "<ipython-input-9-77e28441ef9f>", line 1, in <module>
runfile('C:/Users/Gupta Niwas/Downloads/Programming/Projects/Mi/temp7.py', wdir='C:/Users/Gupta Niwas/Downloads/Programming/Projects/Mi')
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Gupta Niwas/Downloads/Programming/Projects/Mi/temp7.py", line 22, in <module>
browser = webdriver.Chrome(chrome_options=chrome_options,executable_path=cpath)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 75, in __init__
desired_capabilities=desired_capabilities)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 156, in __init__
self.start_session(capabilities, browser_profile)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 245, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 314, in execute
self.error_handler.check_response(response)
File "C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: unknown error: cannot connect to chrome at 127.0.0.1:6813
from chrome not reachable
(Driver info: chromedriver=2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT 10.0.17134 x86_64)
Full code:
import pyautogui
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException,TimeoutException
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
fpath="C:/Users/Gupta Niwas/Downloads/Softwares/Browsers/Drivers/geckodriver-v0.21.0-win64/geckodriver.exe"
cpath="C:/Users/Gupta Niwas/Downloads/Softwares/Browsers/Drivers/chromedriver_win32/chromedriver.exe"
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("start-maximized")
chrome_options.debugger_address="127.0.0.1:6813"
#browser = webdriver.Firefox(executable_path=fpath)
browser = webdriver.Chrome(chrome_options=chrome_options,executable_path=cpath)
#browser.execute_async_script("function(){(function(a){document.body.appendChild(a);a.setAttribute('href', 'http://google.com');a.dispatchEvent((function(e){e.initMouseEvent('click', true, true, window, 0, 0, 0, 0, 0, true, false, false, false, 0, null);return e;}(document.createEvent('MouseEvents'))))}(document.createElement('a')));}")
browser.get('https://youngeinsteins.com/')
pyautogui.hotkey('ctrl','t')
WebDriverWait(browser,100).until(EC.visibility_of_element_located((By.CSS_SELECTOR,'html.gtie9.js.flexbox.canvas.canvastext.webgl.no-touch.geolocation.postmessage.no-websqldatabase.indexeddb.hashchange.history.draganddrop.websockets.rgba.hsla.multiplebgs.backgroundsize.borderimage.borderradius.boxshadow.textshadow.opacity.cssanimations.csscolumns.cssgradients.no-cssreflections.csstransforms.csstransforms3d.csstransitions.fontface.generatedcontent.video.audio.localstorage.sessionstorage.webworkers.applicationcache.svg.inlinesvg.smil.svgclippaths.dk_fouc body.home.page-template-default.page.page-id-15 div.subscribe-popup-wrap div.subscribe-popup div.close svg.icon-cross polygon')))
cross=browser.find_element_by_css_selector('html.gtie9.js.flexbox.canvas.canvastext.webgl.no-touch.geolocation.postmessage.no-websqldatabase.indexeddb.hashchange.history.draganddrop.websockets.rgba.hsla.multiplebgs.backgroundsize.borderimage.borderradius.boxshadow.textshadow.opacity.cssanimations.csscolumns.cssgradients.no-cssreflections.csstransforms.csstransforms3d.csstransitions.fontface.generatedcontent.video.audio.localstorage.sessionstorage.webworkers.applicationcache.svg.inlinesvg.smil.svgclippaths.dk_fouc body.home.page-template-default.page.page-id-15 div.subscribe-popup-wrap div.subscribe-popup div.close svg.icon-cross polygon')
cross.click()
I know this answer may not be useful to you anymore but for those who are still suffering from this problem, I have found a solution.
First of all, when you try to create a chrome instance you need to create a new profile as well, for that instance.
here is a command for that:
In order to run this command, you will have to add chrome to system path or go to the directory where chrome is installed, for windows it is usually this path: C:\Program Files (x86)\Google\Chrome\Application
chrome.exe -remote-debugging-port=9014 --user-data-dir="<AnyDirectoryOfYourChoice>"
Now you should have an instance of chrome running in this instance you can open anything you want and once you are done, you can use this python code below to connect with this instance of chrome.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9014")
driver = webdriver.Chrome(options=chrome_options)
Once you have connected successfully driver will have the reference to your chrome instance and you can control as we normally do in selenium.
This worked for me:
chrome_options.add_argument('--remote-debugging-port=9222')
There seems to be a way to also start a remote debugging server with firefox.
firefox.exe --start-debugger-server 9224 -profile C:\<profilepath>
but since theadd_experimental_option()-method is sole available to the "Chromeoptions" class it might not be an easy,direct conversion to firefox.

Strange invalid pickle protocol errors when using Dill

Recently, Dill has completely stopped working for me. It does this:
>>> import dill
>>> dill.dumps([1,2,3])
b'\x80\x03]q\x00(K\x01K\x02K\x03e.'
>>> dill.loads(_)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python34\lib\site-packages\dill\dill.py", line 260, in loads
return load(file)
File "C:\Python34\lib\site-packages\dill\dill.py", line 250, in load
obj = pik.load()
File "C:\Python34\lib\pickle.py", line 1039, in load
dispatch[key[0]](self)
File "C:\Python34\lib\pickle.py", line 1066, in load_proto
raise ValueError("unsupported pickle protocol: %d" % proto)
ValueError: unsupported pickle protocol: 93
The number is different every time. This started happening maybe a month ago; reinstalling Dill with pip didn't help.
From stepping through it with a debugger it looks like it's correctly reading in the protocol version from the beginning of the data, then reading one of the first instructions in the pickle data and interpreting it as the protocol version for some reason. I don't really know though, since I don't know much about how pickle works.

Resources