I have one question:I want to test "select" and "input".can I write it like the code below:
original code:
12 class Sinaselecttest(unittest.TestCase):
13
14 def setUp(self):
15 binary = FirefoxBinary('/usr/local/firefox/firefox')
16 self.driver = webdriver.Firefox(firefox_binary=binary)
17
18 def test_select_in_sina(self):
19 driver = self.driver
20 driver.get("https://www.sina.com.cn/")
21 try:
22 WebDriverWait(driver,30).until(
23 ec.visibility_of_element_located((By.XPATH,"/html/body/div[9]/div/div[1]/form/div[3]/input"))
24 )
25 finally:
26 driver.quit()
# #测试select功能
27 select=Select(driver.find_element_by_xpath("//*[#id='slt_01']")).select_by_value("微博")
28 element=driver.find_element_by_xpath("/html/body/div[9]/div/div[1]/form/div[3]/input")
29 element.send_keys("杨幂")
30 driver.find_element_by_xpath("/html/body/div[9]/div/div[1]/form/input").click()
31 driver.implicitly_wait(5)
32 def tearDown(self):
33 self.driver.close()
I want to test Selenium "select" function.so I choose sina website to select one option and input text in textarea.then search it .but when I run this test,it has error:
Traceback (most recent call last):
File "test_sina_select.py", line 32, in tearDown
self.driver.close()
File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 688, in close
self.execute(Command.CLOSE)
File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 376, in execute
return self._request(command_info[0], url, body=data)
File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 399, in _request
resp = self._conn.request(method, url, body=body, headers=headers)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 68, in request
**urlopen_kw)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 81, in request_encode_url
return self.urlopen(method, url, **urlopen_kw)
File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 247, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 617, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 617, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 617, in urlopen
release_conn=release_conn, **response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 271, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=51379): Max retries exceeded with url: /session/2e64d2a1-3c7f-4221-96fe-9d0b1c102195/window (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
----------------------------------------------------------------------
Ran 1 test in 72.106s
who can tell me why?thanks
This error message...
MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=51379): Max retries exceeded with url: /session/2e64d2a1-3c7f-4221-96fe-9d0b1c102195/window (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused')))
...implies that the call to self.driver.close() method failed raising MaxRetryError.
A couple of things:
First and foremost as per the discussion max-retries-exceeded exceptions are confusing the traceback is somewhat misleading. Requests wraps the exception for the users convenience. The original exception is part of the message displayed.
Requests never retries (it sets the retries=0 for urllib3's HTTPConnectionPool), so the error would have been much more canonical without the MaxRetryError and HTTPConnectionPool keywords. So an ideal Traceback would have been:
ConnectionError(<class 'socket.error'>: [Errno 1111] Connection refused)
But again #sigmavirus24 in his comment mentioned ...wrapping these exceptions make for a great API but a poor debugging experience...
Moving forward the plan was to traverse as far downwards as possible to the lowest level exception and use that instead.
Finally this issue was fixed by rewording some exceptions which has nothing to do with the actual connection refused error.
Solution
Even before self.driver.close() within tearDown(self) is invoked, the try{} block within test_select_in_sina(self) includes finally{} where you have invoked driver.quit()which is used to call the /shutdown endpoint and subsequently the web driver & the client instances are destroyed completely closing all the pages/tabs/windows. Hence no more connection exists.
You can find a couple of relevant detailed discussion in:
PhantomJS web driver stays in memory
Selenium : How to stop geckodriver process impacting PC memory, without calling
driver.quit()?
In such a situation when you invoke self.driver.close() the python client is unable to locate any active connection to initiate a clousure. Hence you see the error.
So a simple solution would be to remove the line driver.quit() i.e. remove the finally block.
tl; dr
As per the Release Notes of Selenium 3.14.1:
* Fix ability to set timeout for urllib3 (#6286)
The Merge is: repair urllib3 can't set timeout!
Conclusion
Once you upgrade to Selenium 3.14.1 you will be able to set the timeout and see canonical Tracebacks and would be able to take required action.
References
A couple of relevent references:
Adding max_retries as an argument
Removed the bundled charade and urllib3.
Third party libraries committed verbatim
Just had the same problem. The solution was to change the owner of the folder with a script recursively. In my case the folder had root:root owner:group and I needed to change it to ubuntu:ubuntu.
Solution: sudo chown -R ubuntu:ubuntu /path-to-your-folder
Use Try and catch block to find exceptions
try:
r = requests.get(url)
except requests.exceptions.Timeout:
#Message
except requests.exceptions.TooManyRedirects:
#Message
except requests.exceptions.RequestException as e:
#Message
raise SystemExit(e)
Related
I have a problem during the execution of my python script from crontab, which consists of an insert operation in the firestore database.
db.collection(u'ab').document(str(row["Name"])).collection(str(row["id"])).document(str(row2["id"])).set(self.packStructure(row2))
When I execute normally with python3 script.py command it works, but when I execute it from crontab it return the following error:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/axatel/angel_bridge/esportazione_firebase/main.py", line 23, in <module>
dato.getDati(dato, db, cursor, cursor2, fdb, select, anagrafica)
File "/home/axatel/angel_bridge/esportazione_firebase/dati.py", line 19, in getDati
db.collection(u'ab').document(str(row["Name"])).collection(str(row["id"])).document(str(row2["id"])).set(self.packStructure(row2))
File "/home/axatel/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/document.py", line 234, in set
write_results = batch.commit()
File "/home/axatel/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/batch.py", line 147, in commit
metadata=self._client._rpc_metadata,
File "/home/axatel/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/gapic/firestore_client.py", line 1121, in commit
request, retry=retry, timeout=timeout, metadata=metadata
File "/home/axatel/.local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/home/axatel/.local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/home/axatel/.local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/home/axatel/.local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/home/axatel/.local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 DNS resolution failed for service: firestore.googleapis.com:443
I really don't understand what's the problem, because the connection at the database works every time the script is started in both ways.
Is there a fix for this kind of issue?
I found something that might be helpful. There is nice troubleshooting guide and there is a part there, which seems to be related:
If your command works by invoking a runtime like python some-command.py perform a few checks to determine that the runtime
version and environment is correct. Each language runtime has quirks
that can cause unexpected behavior under crontab.
For python you might find that your web app is using a virtual
environment you need to invoke in your crontab.
I haven't seen such error running Firestore API, but this seems to match to your issue.
I found the solution.
The problem occured because the timeout sleep() value was lower than expected, so the database connection function starts too early during boot phase of machine. Increasing this value to 45 or 60 seconds fixed the problem.
#time.sleep(10) # old version
time.sleep(60) # working version
fdb = firebaseConnection()
def firebaseConnection():
# firebase connection
cred = credentials.Certificate('/database/axatel.json')
firebase_admin.initialize_app(cred)
fdb = firestore.client()
if fdb:
return fdb
else:
print("Error")
sys.exit()
I am running a script on AWS (Ubunut) EC2 instance. It's a web scraper that uses selenium/chromedriver and headless chrome to scrape some webpages. I've had this script running previously with no problems, but today I'm getting an error. Here's the script:
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--window-size=1420,1080')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
options.add_argument("--disable-notifications")
options.binary_location='/usr/bin/chromium-browser'
driver = webdriver.Chrome(chrome_options=options)
#Set base url (SAN FRANCISCO)
base_url = 'https://www.bandsintown.com/en/c/san-francisco-ca?page='
events = []
for i in range(1,90):
#cycle through pages in range
driver.get(base_url + str(i))
pageURL = base_url + str(i)
print(pageURL)
When I run this script from ubuntu, I get this error:
Traceback (most recent call last):
File "BandsInTown_Scraper_SF.py", line 91, in <module>
driver = webdriver.Chrome(chrome_options=options)
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/home/ubuntu/.local/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 76, in start
stdin=PIPE)
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
I confirmed that I'm running the same version of Chromedriver/Chromium Browser:
ChromeDriver 79.0.3945.130 (e22de67c28798d98833a7137c0e22876237fc40a-refs/branch-heads/3945#{#1047})
Chromium 79.0.3945.130 Built on Ubuntu , running on Ubuntu 18.04
For what it's worth, I have this running on a mac, and I do have multiple web scraping scripts like this one running on the same EC2 instance (only 2 scripts so far, so not that much).
Update
I'm now getting these errors as well when trying to run this script on ubuntu:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 284, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f90945757f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
^[[B File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
^[[B^[[A^[[A _stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.bandsintown.com', port=443): Max retries exceeded with url: /en/c/san-francisco-ca?page=6 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f90945757f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "BandsInTown_Scraper_SF.py", line 39, in <module>
res = requests.get(url)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.bandsintown.com', port=443): Max retries exceeded with url: /en/c/san-francisco-ca?page=6 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f90945757f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
Finally, here's my currently monthly AWS usage, which doesn't show any memory quota being exceed.
This error message...
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
...implies that the operating system was unable to allocate memory to initiate/spawn a new session.
Additionally, this error message...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.bandsintown.com', port=443): Max retries exceeded with url: /en/c/san-francisco-ca?page=6 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f90945757f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
...implies that your program have successfully iterated till Page 5 and while on Page 6 you see this error.
I don't see any issues in your code block as such. I have taken your code, made some minor adjustments and here is the execution result:
Code Block:
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
base_url = 'https://www.bandsintown.com/en/c/san-francisco-ca?page='
for i in range(1,10):
#cycle through pages in range
driver.get(base_url + str(i))
pageURL = base_url + str(i)
print(pageURL)
Console Output:
https://www.bandsintown.com/en/c/san-francisco-ca?page=1
https://www.bandsintown.com/en/c/san-francisco-ca?page=2
https://www.bandsintown.com/en/c/san-francisco-ca?page=3
https://www.bandsintown.com/en/c/san-francisco-ca?page=4
https://www.bandsintown.com/en/c/san-francisco-ca?page=5
https://www.bandsintown.com/en/c/san-francisco-ca?page=6
https://www.bandsintown.com/en/c/san-francisco-ca?page=7
https://www.bandsintown.com/en/c/san-francisco-ca?page=8
https://www.bandsintown.com/en/c/san-francisco-ca?page=9
Deep dive
This error is coming from subprocess.py:
self.pid = _posixsubprocess.fork_exec(
args, executable_list,
close_fds, tuple(sorted(map(int, fds_to_keep))),
cwd, env_list,
p2cread, p2cwrite, c2pread, c2pwrite,
errread, errwrite,
errpipe_read, errpipe_write,
restore_signals, start_new_session, preexec_fn)
However, as per the discussion in OSError: [Errno 12] Cannot allocate memory this error OSError: [Errno 12] Cannot allocate memory is related to RAM / SWAP.
Swap Space
Swap Space is the memory space in the system hard drive that has been designated as a place for the os to temporarily store data which it can no longer hold with in the RAM. This gives you the ability to increase the amount of data your program can keep in its working memory. The swap space on the hard drive will be used primarily when there is no longer sufficient space in RAM to hold in-use application data. However, the information written to I/O will be significantly slower than information kept in RAM, but the operating system will prefer to keep running application data in memory and use swap space for the older data. Deploying swap space as a fall back for when your system’s RAM is depleted is a safety measure against out-of-memory issues on systems with non-SSD storage available.
System Check
To check if the system already has some swap space available, you need to execute the following command:
$ sudo swapon --show
If you don’t get any output, that means your system does not have swap space available currently. You can also verify that there is no active swap using the free utility as follows:
$ free -h
If there is no active swap in the system you will see an output as:
Output
total used free shared buff/cache available
Mem: 488M 36M 104M 652K 348M 426M
Swap: 0B 0B 0B
Creating Swap File
In these cases you need to allocate space for swap to use as a separate partition devoted to the task and you can create a swap file that resides on an existing partition. To create a 1 Gigabyte file you need to execute the following command:
$ sudo fallocate -l 1G /swapfile
You can verify that the correct amount of space was reserved by executing the following command:
$ ls -lh /swapfile
#Output
$ -rw-r--r-- 1 root root 1.0G Mar 08 10:30 /swapfile
This confirms the swap file has been created with the correct amount of space set aside.
Enabling the Swap Space
Once the correct size file is available we need to actually turn this into swap space. Now you need to lock down the permissions of the file so that only the users with specific privileges can read the contents. This prevents unintended users from being able to access the file, which would have significant security implications. So you need to follow the steps below:
Make the file only accessible to specific user e.g. root by executing the following command:
$ sudo chmod 600 /swapfile
Verify the permissions change by executing the following command:
$ ls -lh /swapfile
#Output
-rw------- 1 root root 1.0G Apr 25 11:14 /swapfile
This confirms only the root user has the read and write flags enabled.
Now you need to mark the file as swap space by executing the following command:
$ sudo mkswap /swapfile
#Sample Output
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=6e965805-2ab9-450f-aed6-577e74089dbf
Next you need to enable the swap file, allowing the system to start utilizing it executing the following command:
$ sudo swapon /swapfile
You can verify that the swap is available by executing the following command:
$ sudo swapon --show
#Sample Output
NAME TYPE SIZE USED PRIO
/swapfile file 1024M 0B -1
Finally check the output of the free utility again to validate the settings by executing the following command:
$ free -h
#Sample Output
total used free shared buff/cache available
Mem: 488M 37M 96M 652K 354M 425M
Swap: 1.0G 0B 1.0G
Conclusion
Once the Swap Space has been set up successfully the underlying operating system will begin to use it as necessary.
Probably what has happened is that Chromium browser is updated, now takes more memory (or perhaps leaks memory worse..you don't say how many urls it gets before dying)
As a work around, launch a larger instance size. Do don't say what instance size you are using but if you have a t3.micro try a t3.medium instead.
There is an easy to understand chart here https://www.ec2instances.info/?region=eu-west-1
If you have launched an instance and want to resize it without rebuilding from scratch then use the console to take it to state stopped, alter the size and start again
I am trying to read data from an ESA server in a PyDev Eclipse 3.8.1 Environment using Python 3.5.3. An example of a product link is given here:
"https://scihub.copernicus.eu/dhus/odata/v1/Products('c7208694-dedb-4f47-96c0-c8fb03512ff5')
You will need authentication credentials to access the website.
In my Eclipse environment I have manually added in proxy settings by going to Windows-> General -> Network Connections. Authentication is not required for the proxy.
To get the content of the webpage, I use a Python get request to send my query as below:
url = r"https://scihub.copernicus.eu/dhus/odata/v1/Products('c5dc59f0-b041-4f76-a685-49be63491270')"
r = requests.get(url, verify=False, proxies=proxies, auth=auth)
I need the verify=False to disable the certificate check and proxies is a dictionary storing the proxy addresses with the relevant port. Server credentials are given with auth which is a tuple of a username and password.
When I send the request to the server, I get the following error:
Traceback (most recent call last):
File "/lib/python3.5/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/lib/python3.5/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/lib/python3.5/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/lib/python3.5/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/lib/python3.5/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
conn.connect()
File "/lib/python3.5/site-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/lib/python3.5/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fecf00acbe0>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/lib/python3.5/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/lib/python3.5/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='scihub.copernicus.eu', port=443): Max retries exceeded with url: /dhus/odata/v1/Products('c5dc59f0-b041-4f76-a685-49be63491270') (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fecf00acbe0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "download_lib2/download_lib/scihub/download.py", line 325, in <module>
download_url(dl_link, auth, temp_dir)
File "download_lib2/download_lib/scihub/download.py", line 153, in download_url
online = check_url_online(url_to_download, auth)
File "download_lib2/download_lib/scihub/download.py", line 51, in check_url_online
r = requests.get(url, verify=False, proxies=proxies, auth=auth)
File "/lib/python3.5/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/lib/python3.5/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/lib/python3.5/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/lib/python3.5/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='scihub.copernicus.eu', port=443): Max retries exceeded with url: /dhus/odata/v1/Products('c5dc59f0-b041-4f76-a685-49be63491270') (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fecf00acbe0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
I have tried the same request outside of Eclipse, in IDLE, and I am able to successfully connect and read the content. Is there something wrong with my current Eclipse/PyDev environment that will not allow me to send a get request?
Why is Python not able to connect? I am able to send get requests to this and other websites using other projects in Eclipse (e.g. https://podaac-opendap.jpl.nasa.gov/opendap/allData/oscar/preview/L4/oscar_third_deg/) with no problem whatsoever.
Edit: Added in versions
The socks timeout indicates that your problem is actually the connection to your proxy as opposed to the ESA server. Additionally, I believe the proxy settings in Eclipse are only used for plugins/software updates and not by any code you write.
As an initial step, check your proxy dictionary keys are correct and verify your proxy URLS if you can.
You mention you have other Eclipse projects that can access the internet; it's worth checking what proxy settings they implement and seeing if there are any discrepancies with your current setup.
I am getting a requests module exception error when I try to access the http://www.acastipharma.com/ website. I am not having problems with any other website so I believe this is a website specific issue. Here is some example code
import requests
initialURL = 'http://www.acastipharma.com/'
r = requests.get(initialURL)
When I run this code I get an error message that terminates with
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.acastipharma.com', port=443): Max retries exceeded with url: /investors/ (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)'),))
An internet search indicates that the problem might be with acastipharma's SSL certificate. I tried installing pyopenssl to make sure I had the latest version of the the module thats checks SSL certificates but that did not solve the problem. I also tried running the requests.get statement with the verify=False option but that was also unsuccessful.
r = requests.get(initialURL, verify=False)
If anybody has any ideas on how to resolve this issue I would appreciate the assistance. I also tried using the older urllib.request package but ran into the same error.
This is an update to my original question: The error message I posted was from trying to run the requests command on one of the acastipharma's website subpages, here is the complete error message I get when I run the code exactly as shown in this question:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
_context=self)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
self._sslobj.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.acastipharma.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 640, in send
history = [resp for resp in gen] if allow_redirects else []
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 640, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 218, in resolve_redirects
**adapter_kwargs
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.acastipharma.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:645)'),))
I am using Python 3.5.1. I am on a Mac using High Sierra version 10.13.2. I am using requests 2.18.4. Since posting the question I believe the problem lies with my IDE PyCharm's environment. If I use my Python 3.5 environment I have the problem as shown in this question, if I switch the project interpreter to a Python 3.6 Anaconda environment requests will work but unfortunately mysql won't import. Thanks
The OpenSSL version is OpenSSL 0.9.8zh 14 Jan 2016
This (very old and long unsupported) version of OpenSSL does not support the newer ciphers required by this specific web server. This server only supports ECDHE key exchange, which is not supported yet by OpenSSL 0.9.8. This means that the client only offers ciphers to the server which the server will not accept and due to no common ciphers the server will close the connection with an SSL handshake alert.
We can ignore this SSL Error using the following:
import warnings
from urllib3.exceptions import InsecureRequestWarning
warnings.simplefilter('ignore',InsecureRequestWarning)
Hello i have a simple problem that has suddenly developed on my code, until yesterday it was working correctly, this means it opened a web page and entered login details (is a test for a bigger app), however now whenever it tries to connect to the login page i get a timeout error. I have updated Selenium and Chrome Web Driver to the latest and it is still failing to connect, this is my code:
#importing libraries, selenium is the base of the script, time to add pauses when appropiate or necessary and select and request will stay until i determine that they are not necessary for the final script.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
import requests
#defining browser to use the webdriver with chrome, and navigating to Makor.
browser = webdriver.Chrome()
browser.get('http://mrmprod/Login.aspx')
And this is the complete error code:
Traceback (most recent call last):
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\Scripts\Input
test.py", line 7, in
browser = webdriver.Chrome()
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\chrome\webdriver.py",
line 69, in init
desired_capabilities=desired_capabilities)
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 92, in init
self.start_session(desired_capabilities, browser_profile)
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 234, in execute
response = self.command_executor.execute(driver_command, params)
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 407, in execute
return self._request(command_info[0], url, body=data)
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 439, in _request
resp = self._conn.getresponse()
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\http\client.py",
line 1197, in getresponse
response.begin()
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\http\client.py",
line 297, in begin
version, status, reason = self._read_status()
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\http\client.py",
line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\AMSUser\AppData\Local\Programs\Python\Python35-32\lib\socket.py",
line 575, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
I asked the network administrator and he has told me that there hasn't being any change on the network settings since i originally got the code working.
If you have reviewed the aforementioned then do the following steps:
You have to include inside the parenthesis the chromedriver path.
Check the code below:
CODE SAMPLE FOR WINDOWS OS
browser = webdriver.Chrome('C:\\User\\YOUR_USERS_NAME\\YOUR_PATH\\chromedriver.exe')
CODE SAMPLE FOR LINUX OS
browser = webdriver.Chrome('usr\\bin\\YOUR_PATH\\chromedriver')