Parsing/Writing SysML v2? - python-3.x

I'm trying to find an existing parser (preferably in Python) to write and read SysML v2.
The official Github repo by SysML v2 Submission Team (SST) has placed some examples online already (For instance https://github.com/Systems-Modeling/SysML-v2-Release/blob/master/sysml/src/examples/Vehicle%20Example/VehicleUsages.sysml ), but I'm unable to get the Eclipse Plugin running properly. (Nothing is shown in the preview windows, no error message either).
There's a repo for the SysML-v2-API-Python-Client which I installed and executed the given sample code (https://github.com/Systems-Modeling/SysML-v2-API-Python-Client), but all I'm getting is this message:
$ python3.8 Test1.py
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f9492d1b9a0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "Test1.py", line 38, in <module>
api_response = api_instance.delete_branch_by_project_and_id(
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/api/branch_api.py", line 62, in delete_branch_by_project_and_id
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/api/branch_api.py", line 144, in delete_branch_by_project_and_id_with_http_info
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/api_client.py", line 364, in call_api
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/api_client.py", line 181, in __call_api
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/api_client.py", line 431, in request
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/rest.py", line 256, in DELETE
File "/home/qohelet/.local/lib/python3.8/site-packages/sysml_v2_api_client-2021.post9-py3.8.egg/sysml_v2_api_client/rest.py", line 163, in request
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 75, in request
return self.request_encode_url(
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 97, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
return self.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
return self.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 747, in urlopen
return self.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /projects/project_id_example/branches/branch_id_example (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f9492d1b9a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Am I supposed to have anything running? Is there anywhere a tutorial or manual on how to use it?
Or are there any other parsers out there?

There are no parsers currently available for Python. The API Python Client is an example implementation to interact with the Pilot Server API Implementation. You would need to run the server to interact with the API using the python client.

Related

MinIO | Failed to establish a new connection: [Errno -2] Name or service not known

Goal: download all files within a folder, in a MinIO bucket.
Since I want to dynamically download all files in a folder, I use:
objs = list(client.list_objects(bucket, recursive=True)).sort()
That way, I can iterate over each file name for download using:
for o in objs:
obj = client.get_object(bucket, o)
Downloading a specified file also fails:
obj = client.get_object(bucket, 'industry/gri/esg/ESG_COMP.csv')
Code:
from sdg.datasource.MinioConn import MinioConn
client = MinioConn().client()
bucket = 'synthetic-data-gen'
obj = client.get_object(bucket, 'industry/gri/esg/ESG_COMP.csv')
print('###')
print(obj)
objs = list(client.list_objects(bucket, recursive=True)).sort()
dfs = []
for o in objs:
obj = client.get_object(bucket, o)
dfs.append(pd.read_csv(obj).iloc[:, 0].tolist()) # 1 column csvs
MinioConn:
from minio import Minio
class MinioConn:
def __init__(self,
host='foo.bar.foo.bar.com:9000',
access_key='CENSORED', secret_key='CENSORED',
secure=False):
self.host = host
self.access_key = access_key
self.secret_key = secret_key
self.secure = secure
def client(self):
return Minio(self.host, self.access_key, self.secret_key,
secure=self.secure)
Traceback:
Traceback (most recent call last):
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/home/me/miniconda3/envs/sdg/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/http/client.py", line 1010, in _send_output
self.send(msg)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/http/client.py", line 950, in send
self.connect()
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc6f4ee17f0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/me/miniconda3/envs/sdg/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/runpy.py", line 144, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/mnt/c/Users/me/Documents/GitHub/repo/project/foo/__init__.py", line 5, in <module>
from sdg.sdg import generate_data
File "/mnt/c/Users/me/Documents/GitHub/repo/project/foo/sdg.py", line 10, in <module>
from sdg.industry.gri.generator import Generator as GRIGenerator
File "/mnt/c/Users/me/Documents/GitHub/repo/project/foo/industry/gri/generator.py", line 40, in <module>
class Generator:
File "/mnt/c/Users/me/Documents/GitHub/repo/project/foo/industry/gri/generator.py", line 346, in Generator
dfs = get_esgs()
File "/mnt/c/Users/me/Documents/GitHub/repo/project/foo/industry/gri/generator.py", line 61, in get_esgs
obj = client.get_object(bucket, 'industry/gri/esg/ESG_COMP.csv')
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/minio/api.py", line 1151, in get_object
return self._execute(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/minio/api.py", line 394, in _execute
region = self._get_region(bucket_name, None)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/minio/api.py", line 461, in _get_region
response = self._url_open(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/minio/api.py", line 266, in _url_open
response = self._http.urlopen(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/poolmanager.py", line 375, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
return self.urlopen(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
return self.urlopen(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/home/me/miniconda3/envs/sdg/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='foo.bar.foo.bar.com', port=9000): Max retries exceeded with url: /synthetic-data-gen?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6f4ee17f0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
The error message:
socket.gaierror: [Errno -2] Name or service not known
implies that DNS resolution on the host name (foo.bar.foo.bar.com) failed. You likely need to check your DNS configuration or fix the host name if it is incorrect.

npm run serve works but docker-compose fails

I'm trying to test a containerized app on my local machine. The app uses a combination of ruby on rails and vue code. If I run "npm run serve" it will fire up a server and I can connect to it locally in a browser (ex. "http://localhost:xxxx/")
If I try to run "docker-compose up -d" I get the error messages below.
Does anyone see what the issue might be? The code was developed for linux but I'm testing on mac, if that makes a difference.
working version:
npm run serve
failed version with error:
$ docker-compose up -d
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 670, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1255, in request
File "http/client.py", line 1301, in _send_request
File "http/client.py", line 1250, in endheaders
File "http/client.py", line 1010, in _send_output
File "http/client.py", line 950, in send
File "docker/transport/unixconn.py", line 43, in connect
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 439, in send
File "urllib3/connectionpool.py", line 726, in urlopen
File "urllib3/util/retry.py", line 410, in increment
File "urllib3/packages/six.py", line 734, in reraise
File "urllib3/connectionpool.py", line 670, in urlopen
File "urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1255, in request
File "http/client.py", line 1301, in _send_request
File "http/client.py", line 1250, in endheaders
File "http/client.py", line 1010, in _send_output
File "http/client.py", line 950, in send
File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker/api/client.py", line 214, in _retrieve_server_version
File "docker/api/daemon.py", line 181, in version
File "docker/utils/decorators.py", line 46, in inner
File "docker/api/client.py", line 237, in _get
File "requests/sessions.py", line 543, in get
File "requests/sessions.py", line 530, in request
File "requests/sessions.py", line 643, in send
File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 200, in perform_command
File "compose/cli/command.py", line 60, in project_from_options
File "compose/cli/command.py", line 152, in get_project
File "compose/cli/docker_client.py", line 41, in get_client
File "compose/cli/docker_client.py", line 170, in docker_client
File "docker/api/client.py", line 197, in __init__
File "docker/api/client.py", line 221, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(61, 'Connection refused'))
[55686] Failed to execute script docker-compose
figured it out, had to have docker engine running before I tried to run docker-compose.

How to Make Python AWS S3 Boto3 Code Robust to Intermittent WiFi

I am trying to create a program on a Raspberry Pi that will intermittently take pictures and send them to AWS S3.
About half the time, everything works fine. Problem is that the WiFi receiver on the Raspberry Pi that I have is pretty awful, and it loses connection constantly. I know it isn't my WiFi because all other devices on the network are fine. After the connection is lost, it searches until it finds it again, usually 15-30 seconds or so. Then the cycle repeats after a minute or two.
I am trying to create code that will attempt to connect to S3, and on failure, will continue trying until the connection is restored. So far, however, connection errors still cause a hard stop.
import boto3
s3 = boto3.resource('s3')
allbuckets = 0
while allbuckets == 0:
try:
allbuckets = s3.buckets.all()
except:
allbuckets = 0
for bucket in allbuckets:
print(bucket.name)
If the connection is solid, this works fine:
>>> %Run boto_test.py
derek-raspberrypi-pictures
stpaul-academy-middleschool-showcase-2020-part2
If the connection has been temporarily lost, however:
>>> %Run boto_test.py
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 343, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0xb52a6950>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pi/Documents/boto_test.py", line 15, in <module>
for bucket in allbuckets:
File "/home/pi/.local/lib/python3.7/site-packages/boto3/resources/collection.py", line 83, in __iter__
for page in self.pages():
File "/home/pi/.local/lib/python3.7/site-packages/boto3/resources/collection.py", line 161, in pages
pages = [getattr(client, self._py_operation_name)(**params)]
File "/home/pi/.local/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/client.py", line 622, in _make_api_call
operation_model, request_dict, request_context)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/client.py", line 641, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 137, in _send_request
success_response, exception):
File "/home/pi/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 256, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 183, in __call__
if self._checker(attempts, response, caught_exception):
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 251, in __call__
caught_exception)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
File "/home/pi/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/endpoint.py", line 269, in _send
return self.http_session.send(request)
File "/home/pi/.local/lib/python3.7/site-packages/botocore/httpsession.py", line 283, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.amazonaws.com/"
Any suggestions on how to make this robust to the intermittent WiFi connection would be greatly appreciated.
Remove the power management from the wifi interface on the RPi which should keep it active.
$ sudo iwconfig wlan0 power off

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')

Anyone has an idea how I can solve this error?
I am trying to download data and I have connected a posgresql server with the database navigator as localhost. Files are successfully downloaded for like 3 minutes and than suddenly the download stops and returns this error message:
.
..
...
run_filing 201500419349300305
run_filing 201540439349301594
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/django/core/management/base.py", line 369, in execute
output = self.handle(*args, **options)
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/return/management/commands/load_filings.py", line 123, in handle
self.run_filing(filing)
File "/Users/Georg/PycharmProjects/990-xml-database/irsdb/return/management/commands/load_filings.py", line 65, in run_filing
parsed_filing = self.xml_runner.run_filing(object_id)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/xmlrunner.py", line 111, in run_filing
this_filing.process(verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/filing.py", line 232, in process
self._download(verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/filing.py", line 70, in _download
stream_download(self.URL, self.filepath, verbose=verbose)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/irsx/file_utils.py", line 22, in stream_download
response = requests.get(url, stream=True)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/Users/Georg/PycharmProjects/990-xml-database/venv/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
Attached is a screenshot of the connectionpool.py code.

Requests library raises urllib connection error

Requests library is giving me an error I've never encountered before, while trying to download an image from my website.
Here is the download function
def dl_jpg(url, filePath, fileName):
fullPath = filePath + fileName + '.jpg'
r = requests.get(url,allow_redirects=True)
open(fullPath,'wb').write(r.content)
The URL I'm entering is:
http://www.deepfrybot.ga/uploads/c4c7936ef4218dbe7014cb543049168b.jpg
Here's the error message
Last login: Fri Nov 1 03:36:32 2019 from 116.193.136.13
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
[ec2-user#ip ~]$ cd dfb-master
[ec2-user#ip dfb-master]$ sudo python3 mainr.py
enter url (n) if auto: http://www.deepfrybot.ga/uploads/c4c7936ef4218dbe7014cb543049168b.jpg
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 57, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib64/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib64/python3.7/http/client.py", line 1244, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1290, in _send_request
self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.7/http/client.py", line 1239, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib64/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib64/python3.7/http/client.py", line 966, in send
self.connect()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fdce843ed50>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.deepfrybot.ga', port=80): Max retries exceeded with url: /uploads/c4c7936ef4218dbe7014cb543049168b.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdce843ed50>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mainr.py", line 32, in wrapper
return job_func(*args, **kwargs)
File "mainr.py", line 138, in main_p
dl_jpg(url, get_abs_file('images/'), file_name)
File "mainr.py", line 67, in dl_jpg
r = requests.get(url,allow_redirects=True)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-
packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='www.deepfrybot.ga', port=80): Max retries exceeded with url: /uploads/c4c7936ef4218dbe7014cb543049168b.jpg (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdce843ed50>: Failed to establish a new connection: [Errno -2] Name or service not known'))
I'm clueless as to why this is happening, a little help would be much appreciated
lxop said "
The script is failing to resolve the name 'www.deepfrybot.ga'. Does
your EC2 instance have DNS access?
"
And after checking and researching a bit on it I came to know that some specific EC2 instances/urllib3 are sometimes not able to access free top level domains (like .ga,.ml,.tk) as they have a general tendency to be malicious and also since they generally lack an SSL certificate.
These domains with SSL certificates work fine though!
And if you are hosting on say heliohost.org (which i am), simply change the domain in the script from yourwebsite.ga to yourwebsite.yourhost.domain
That should solve it!

Resources