Trying to connect a Colab Notebook to a MongoDB on Atlas.
from pymongo import MongoClient
uri = "mongodb+srv://MYUSERNAME:mypassword#mydatabase.mongodb.net/test"
client = MongoClient(uri)
I am getting a CongfigurationError:
"dnspython" module must be installed to use mongodb+srv:// URIs.
I installed the module.
pip install dnspython
Got the message back
Requirement already satisfied: dnspython in /usr/local/lib/python3.6/dist-packages (1.16.0)
Do not know what is wrong.
This worked a few days ago with another colab notebook (and another database).
Here is the entire error message:
ConfigurationError Traceback (most recent call last)
<ipython-input-30-a6c89e14e64f> in <module>()
----> 1 client = MongoClient(uri)
1 frames
/usr/local/lib/python3.6/dist-packages/pymongo/mongo_client.py in __init__(self, host, port, document_class, tz_aware, connect, type_registry, **kwargs)
522 for entity in host:
523 if "://" in entity:
--> 524 res = uri_parser.parse_uri(entity, port, warn=True)
525 seeds.update(res["nodelist"])
526 username = res["username"] or username
/usr/local/lib/python3.6/dist-packages/pymongo/uri_parser.py in parse_uri(uri, default_port, validate, warn)
316 elif uri.startswith(SRV_SCHEME):
317 if not _HAVE_DNSPYTHON:
--> 318 raise ConfigurationError('The "dnspython" module must be '
319 'installed to use mongodb+srv:// URIs')
320 is_srv = True
ConfigurationError: The "dnspython" module must be installed to use mongodb+srv:// URIs
You have to restart runtime to have the changes take effect:
!pip install dnspython
Restart runtime Runtime -> Restart runtime...
Run your code
Try installing pymongo[srv] and [tls]
!pip3 install pymongo[srv]
!pip3 install pymongo[tls]
change mongodb+srv:// to mongodb:// and it will work
Related
I am using Poetry version 1.1.7.
Running poetry install yields this error.
Can this be resolved through bash or is this explicitly to do with my network? Note: Internet connectivity it perfectly fine in browsers.
me#LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/CompositeKey/CompositeKey$ poetry install
Updating dependencies
Resolving dependencies... (225.5s)
ConnectionError
HTTPSConnectionPool(host='pkgs.dev.azure.com', port=443): Max retries exceeded with url: <CENSORED> (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f53950c3ee0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
at ~/.poetry/lib/poetry/_vendor/py3.8/requests/adapters.py:516 in send
512│ if isinstance(e.reason, _SSLError):
513│ # This branch is for urllib3 v1.22 and later.
514│ raise SSLError(e, request=request)
515│
→ 516│ raise ConnectionError(e, request=request)
517│
518│ except ClosedPoolError as e:
519│ raise ConnectionError(e, request=request)
520│
A 401 Unauthorized Error is thrown when pasting the URL into my browser.
I did:
poetry self update 1.0.10
poetry install
Terminal is giving me installs now, as it should.
me#LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/CompositeKey/CompositeKey$ poetry self update 1.0.10
Updating to 1.0.10
- Downloading poetry-1.0.10-linux.tar.gz 100%
Poetry (1.0.10) is installed now. Great!
me#LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/CompositeKey/CompositeKey$ poetry install
Updating dependencies
Resolving dependencies... (101.5s)
Writing lock file
Package operations: 167 installs, 0 updates, 0 removals
- Installing six (1.16.0)
...
I have the same issue and it turns out to be connection refused by server due to exceeding the max connections limit of the mirror repo.
By default poetry's max installer.max-workers is set to number_of_cores + 4. This would be a problem when you run poetry on a server with like 24 cores and it is reasonable for the server side to refuse that many of connections.
The work around is to config this value manually before executing poetry install, you can use the following command to do limit the max connections:
poetry config installer.max-workers 4
With the current version 0.22.2 there is a interactive tool to enter interactive data and see the results. Its called Libsvm GUI.
I never managed to have it running in Jupyter notebook.
Having seen that there is a binder option. When trying this (which should not depend on my computer environment) errors come up.
https://scikit-learn.org/stable/auto_examples/applications/svm_gui.html
Automatically created module for IPython interactive environment
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-e5e1b6a6b155> in <module>
6
7 import matplotlib
----> 8 matplotlib.use('TkAgg')
9
10 from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
/srv/conda/envs/notebook/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs)
305 f"for the old name will be dropped %(removal)s.")
306 kwargs[new] = kwargs.pop(old)
--> 307 return func(*args, **kwargs)
308
309 # wrapper() must keep the same documented signature as func(): if we
/srv/conda/envs/notebook/lib/python3.7/site-packages/matplotlib/__init__.py in use(backend, warn, force)
1305 if force:
1306 from matplotlib.pyplot import switch_backend
-> 1307 switch_backend(name)
1308 else:
1309 # Finally if pyplot is not imported update both rcParams and
/srv/conda/envs/notebook/lib/python3.7/site-packages/matplotlib/pyplot.py in switch_backend(newbackend)
234 "Cannot load backend {!r} which requires the {!r} interactive "
235 "framework, as {!r} is currently running".format(
--> 236 newbackend, required_framework, current_framework))
237
238 rcParams['backend'] = rcParamsDefault['backend'] = newbackend
ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, as 'headless' is currently running
Seeing the first error, it seems even with the untouched binder environment there is something wrong. But I am not sure if it is with binder or with the code itself.
What can I try to make it working?
I have very strange error :
If I would like to download one model I get
python3.6/site-packages/urllib3/contrib/pyopenssl.py in recv_into(self, *args, **kwargs)
303 try:
--> 304 return self.connection.recv_into(*args, **kwargs)
305 except OpenSSL.SSL.SysCallError as e:
SSLError: ("read error: Error([('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')],)",)
But, if I download another model in same workspace download normally.
model = Model(ws, 'model1')
model.download(target_dir=os.getcwd() + '/outputs/1/', exist_ok=True)
# this download normaly
model = Model(ws, 'model2')
model.download(target_dir=os.getcwd() + '/outputs/2/', exist_ok=True)
# This give me an SSL error
Some points:
This model already worked, but suddenly doesn't wont to download
My network is probably not a problem, because else the first model wouldn't download,...
This is indeed odd. I assume that it this is consistently reproducing between model1 and 2. Which version of openssl are you using?
python -c "import sys; print(sys.OPENSSL_VERSION)"
I run an automated python job on an EMR cluster that updates Amazon Athena Tables.
It was running well until few days ago (on python 2.7 and 3.7). Here is the script:
from pyathenajdbc import connect
import yaml
config = yaml.load(open('athena-config.yaml', 'r'))
statements = config['statements']
staging_dir = config['staging_dir']
conn = connect(s3_staging_dir=staging_dir, region_name='eu-west-1')
try:
with conn.cursor() as cursor:
for statement in statements:
cursor.execute(statement)
finally:
conn.close()
The athena-config.yaml has a staging directory and few Athena Statements.
Here is the Error:
You are using pip version 9.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Unrecognized option: -server
create_tables.py:5: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(open('athena-config.yaml', 'r'))
/mnt/conda/lib/python3.7/site-packages/jpype/_core.py:210: UserWarning:
-------------------------------------------------------------------------------
Deprecated: convertStrings was not specified when starting the JVM. The default
behavior in JPype will be False starting in JPype 0.8. The recommended setting
for new code is convertStrings=False. The legacy value of True was assumed for
this session. If you are a user of an application that reported this warning,
please file a ticket with the developer.
-------------------------------------------------------------------------------
""")
Traceback (most recent call last):
File "create_tables.py", line 10, in <module>
region_name='eu-west-1')
File "/mnt/conda/lib/python3.7/site-packages/pyathenajdbc/__init__.py", line 69, in connect
driver_path, log4j_conf, **kwargs)
File "/mnt/conda/lib/python3.7/site-packages/pyathenajdbc/connection.py", line 68, in __init__
self._start_jvm(jvm_path, jvm_options, driver_path, log4j_conf)
File "/mnt/conda/lib/python3.7/site-packages/pyathenajdbc/util.py", line 25, in _wrapper
return wrapped(*args, **kwargs)
File "/mnt/conda/lib/python3.7/site-packages/pyathenajdbc/connection.py", line 97, in _start_jvm
jpype.startJVM(jvm_path, *args)
File "/mnt/conda/lib/python3.7/site-packages/jpype/_core.py", line 219, in startJVM
_jpype.startup(jvmpath, tuple(args), ignoreUnrecognized, convertStrings)
RuntimeError: Unable to start JVM
at loadJVM(native/common/jp_env.cpp:169)
at loadJVM(native/common/jp_env.cpp:179)
at startup(native/python/pyjp_module.cpp:159)
As far as I understand the issue in convertStrings being deprecated. Can anyone help me resolve that? I cannot understand why this """) comes before the traceback, and what changed in past days to break the code. Thanks!
Got the same issue today. Try to downgrade JPype1 to 0.6.3. JPype1 released 0.7.0 today, which is not compatible with old interfaces.
The issue appears to be that the package is calling the JVM with an unrecognized argument -server. The previous version was ignoring those sort of errors allowing things to proceed. To get the same behavior with 0.7.0, the flag ignoreUnrecognized would need to be set to True. Likely this needs to be send to pyathenajdbc to correct the defect which placed the bogus argument into the startJVM in the first place.
Looking at the source the -server is hardcoded into the module.
if not jpype.isJVMStarted():
_logger.debug('JVM path: %s', jvm_path)
args = [
'-server',
'-Djava.class.path={0}'.format(driver_path),
'-Dlog4j.configuration=file:{0}'.format(log4j_conf)
]
if jvm_options:
args.extend(jvm_options)
_logger.debug('JVM args: %s', args)
jpype.startJVM(jvm_path, *args)
cls.class_loader = jpype.java.lang.Thread.currentThread().getContextClassLoader()
It is assuming a particular JVM which accepts -server as an argument.
I have a project code in Python Notebook and it ran all good when Spark was hosted in Bluemix.
We are running the following code to connect to Netezza (on premises) which worked fine in Bluemix.
VT = sqlContext.read.format('jdbc').options(url='jdbc:netezza://169.54.xxx.x:xxxx/BACC_PRD_ISCNZ_GAPNZ',user='XXXXXX', password='XXXXXXX', dbtable='GRACE.CDVT_LIVE_SPARK', driver='org.netezza.Driver').load()'
However, after migration to DatascienceExperience, we are getting the following error. I have established the secure gateway and its all working fine, but this code is not running. I think the issue is with the Netezza driver. If it is the case, is there a way we can explicitly import the class/driver so the above code can be executed. Please help how we can address the issue.
Error Message:
/usr/local/src/spark20master/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
65 s = e.java_exception.toString()
/usr/local/src/spark20master/spark/python/lib/py4j-0.10.3-src.zip /py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
317 raise Py4JJavaError(
318 "An error occurred while calling {0}{1} {2}.\n".
--> 319 format(target_id, ".", name), value)
320 else:
321 raise Py4JError(
Py4JJavaError: An error occurred while calling o212.load.
: java.lang.ClassNotFoundException: org.netezza.driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:607)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:844)
at java.lang.ClassLoader.loadClass(ClassLoader.java:823)
at java.lang.ClassLoader.loadClass(ClassLoader.java:803)
at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:38)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createC onnectionFactory$1.apply(JdbcUtils.scala:49)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createC onnectionFactory$1.apply(JdbcUtils.scala:49)
at scala.Option.foreach(Option.scala:257)
You can install a jar file by adding a cell with an exclamation mark that runs a unix tool to download the file, in this example wget:
!wget https://some.public.host/yourfile.jar -P ${HOME}/data/libs
After downloading the file you will need to restart your kernel.
Note this approach assumes your jar file is publicly available on the Internet.
Notebooks in Bluemix and notebooks in DSX (Data Science Experience) currently use the same backend, so they have access to the same pre-installed drivers. Netezza isn't among them. As Chris Snow pointed out, users can install additional JARs and Python packages into their service instances.
You probably created a new service instance for DSX, and did not yet install the user JARs and packages that the old one had. It's a one-time setup, therefore easy to forget when you've been using the same instance for a while. Execute these commands in a Python notebook of the old instance on Bluemix to check for user-installed things:
!ls -lF ~/data/libs
!pip freeze
Then install the missing things into your new instance on DSX.
There is another way to connect to Netezza using ingest connector which
is by default enabled in DSX.
http://datascience.ibm.com/docs/content/analyze-data/python_load.html
from ingest import Connectors
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
NetezzaloadOptions = {
Connectors.Netezza.HOST : 'hostorip',
Connectors.Netezza.PORT : 'port',
Connectors.Netezza.DATABASE : 'databasename',
Connectors.Netezza.USERNAME : 'xxxxx',
Connectors.Netezza.PASSWORD : 'xxxx',
Connectors.Netezza.SOURCE_TABLE_NAME : 'tablename'}
NetezzaDF = sqlContext.read.format("com.ibm.spark.discover").options(**NetezzaloadOptions).load()
NetezzaDF.printSchema()
NetezzaDF.show()
Thanks,
Charles.