I am trying to load my saved model from s3 using joblib
import pandas as pd
import numpy as np
import json
import subprocess
import sqlalchemy
from sklearn.externals import joblib
ENV = 'dev'
model_d2v = load_d2v('model_d2v_version_002', ENV)
def load_d2v(fname, env):
model_name = fname
if env == 'dev':
try:
model=joblib.load(model_name)
except:
s3_base_path='s3://sd-flikku/datalake/doc2vec_model'
path = s3_base_path+'/'+model_name
command = "aws s3 cp {} {}".format(path,model_name).split()
print('loading...'+model_name)
subprocess.call(command)
model=joblib.load(model_name)
else:
s3_base_path='s3://sd-flikku/datalake/doc2vec_model'
path = s3_base_path+'/'+model_name
command = "aws s3 cp {} {}".format(path,model_name).split()
print('loading...'+model_name)
subprocess.call(command)
model=joblib.load(model_name)
return model
But I get this error:
from sklearn.externals import joblib
ImportError: cannot import name 'joblib' from 'sklearn.externals' (C:\Users\prane\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\externals\__init__.py)
Then I tried installing joblib directly by doing
import joblib
but it gave me this error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in load_d2v_from_s3
File "/home/ec2-user/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/home/ec2-user/.local/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()
File "/usr/lib64/python3.7/pickle.py", line 1088, in load
dispatch[key[0]](self)
File "/usr/lib64/python3.7/pickle.py", line 1376, in load_global
klass = self.find_class(module, name)
File "/usr/lib64/python3.7/pickle.py", line 1426, in find_class
__import__(module, level=0)
ModuleNotFoundError: No module named 'sklearn.externals.joblib'
Can you tell me how to solve this?
You should directly use
import joblib
instead of
from sklearn.externals import joblib
It looks like your existing pickle save file (model_d2v_version_002) encodes a reference module in a non-standard location – a joblib that's in sklearn.externals.joblib rather than at top-level.
The current scikit-learn documentation only talks about a top-level joblib – eg in 3.4.1 Persistence example – but I do see a reference in someone else's old issue to a DeprecationWarning in scikit-learn version 0.21 about an older scikit.external.joblib variant going away:
Python37\lib\site-packages\sklearn\externals\joblib_init_.py:15:
DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and
will be removed in 0.23. Please import this functionality directly
from joblib, which can be installed with: pip install joblib. If this
warning is raised when loading pickled models, you may need to
re-serialize those models with scikit-learn 0.21+.
'Deprecation' means marking something as inadvisable to rely-upon, as it is likely to be discontinued in a future release (often, but not always, with a recommended newer way to do the same thing).
I suspect your model_d2v_version_002 file was saved from an older version of scikit-learn, and you're now using scikit-learn (aka sklearn) version 0.23+ which has totally removed the sklearn.external.joblib variation. Thus your file can't be directly or easily loaded to your current environment.
But, per the DeprecationWarning, you can probably temporarily use an older scikit-learn version to load the file the old way once, then re-save it with the now-preferred way. Given the warning info, this would probably require scikit-learn version 0.21.x or 0.22.x, but if you know exactly which version your model_d2v_version_002 file was saved from, I'd try to use that. The steps would roughly be:
create a temporary working environment (or roll back your current working environment) with the older sklearn
do imports something like:
import sklearn.external.joblib as extjoblib
import joblib
extjoblib.load() your old file as you'd planned, but then immediately re-joblib.dump() the file using the top-level joblib. (You likely want to use a distinct name, to keep the older file around, just in case.)
move/update to your real, modern environment, and only import joblib (top level) to use joblib.load() - no longer having any references to `sklearn.external.joblib' in either your code, or your stored pickle files.
You can import joblib directly by installing it as a dependency and using import joblib,
Documentation.
Maybe your code is outdated. For anyone who aims to use fetch_mldata in digit handwritten project, you should fetch_openml instead. (link)
In old version of sklearn:
from sklearn.externals import joblib
mnist = fetch_mldata('MNIST original')
In sklearn 0.23 (stable release):
import sklearn.externals
import joblib
dataset = datasets.fetch_openml("mnist_784")
features = np.array(dataset.data, 'int16')
labels = np.array(dataset.target, 'int')
For more info about deprecating fetch_mldata see scikit-learn doc
none of the answers below works for me, with a little changes this modification was ok for me
import sklearn.externals as extjoblib
import joblib
for this error, I had to directly use the following and it worked like a charm:
import joblib
Simple
In case the execution / call to joblib is within another .py program instead of your own (in such case even you have installed joblib, it still causes error from within the calling python programme unless you change the code, i thought would be messy), I tried to create a hardlink:
(windows version)
Python> import joblib
then inside your sklearn path >......\Lib\site-packages\sklearn\externals
mklink /J ./joblib .....\Lib\site-packages\joblib
(you can work out the above using a ! or %, !mklink....... or %mklink...... inside your Python juptyter notebook , or use python OS command...)
This effectively create a virtual folder of joblib within the "externals" folder
Remarks:
Of course to be more version resilient, your code has to check for the version of sklearn is >= 0.23 again before hand.
This would be alternative to changing sklearn vesrion.
When getting error:
from sklearn.externals import joblib it deprecated older version.
For new version follow:
conda install -c anaconda scikit-learn (install using "Anaconda Promt")
import joblib (Jupyter Notebook)
I had the same problem
What I did not realize was that joblib was already installed!
so what you have to do is replace
from sklearn.externals import joblib
with
import joblib
and that is it
After a long investigation, given my computer setup, I've found that was because an SSL certificate was required to download the dataset.
I'm a bit new to the Python world. I'm using Python3 and having hard times with imports.
I was using PyCharm on Windows to write an application. Everything was working through the project until I switched to Linux and VS Code.
Now I can't use absolute imports for importing modules from other packages in the same project.
For example, I'd like to import from module cards all available card types.
I tested the classes and everything is ok. I'm only getting this issue with the import stuff.
The project structure:
/
|-cards
|-__init__.py
|-card.py
|-monster_card.py
|-spell_card.py
|-trap_card.py
|-ritual_card.py
|-deck
|-__init__py
|-deck.py
|-system
# This is the code in __init__.py in cads package
from . trap_card import TrapCard
from . spell_card import SpellCard
from . ritual_card import RitualCard
from . monster_card import MonsterCard
__all__ = [TrapCard, SpellCard, RitualCard, MonsterCard]
# The following line, for example, does not work from inside another package
# I'm trying to import the modules in cards from deck
from cards import TrapCard, MonsterCard, SpellCard, RitualCard
When I try to import packages from another folders I get this error message:
Traceback (most recent call last):
File "/root/git-repos/Yu-Gi-Oh/decks/deck.py", line 3, in
from cards import TrapCard, MonsterCard, SpellCard, RitualCard
ModuleNotFoundError: No module named 'cards'
When you call import *, python search the module from sys.path. You need to add your root dir into sys.path before call import stmt.
For your case, your root dir is /.
like:
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
from cards import *
other way
Add file __init__.py into your root dir for make it become a module. Then change from cards import * to from .cards import *.
I am trying to run my Flask application with an Apache server using mod_wsgi, and it has been a bumpy road, to say the least.
It has been suggested that I should try to run my app's .wsgi file using Python to make sure it is working.
This is the contents of the file:
#!/usr/bin/python
activate_this = '/var/www/Giveaway/Giveaway/venv/bin/activate_this.py'
with open(activate_this) as f:
code = compile(f.read(), "somefile.py", 'exec')
exec(code)
import sys
import logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0,"/var/www/Giveaways/")
from Giveaways import application
application.secret_key = 'Add your secret key'
However, when I run it, I get this error:
ImportError: No module named 'zlib'
And no, I am not using some homebrewed version of Python - I installed Python 3 via apt-get.
Thanks for any help.
does the contents of somefile.py include the gzip package? in which case you may have to install gzip package via pip or similar
I'm trying to import gcm as following:
from . import gcm
And i get:
SystemError: Parent module '' not loaded, cannot perform relative import
I tried also called:
from gcm import GCM
by pip install python-gcm but i get the follwoing error:
module' object has no attribute 'GCM
Don't use the relative import syntax, from . import gcm. Instead, put the library where the Python interpreter can find it (via pip install python-gcm) and call from gcm import GCM in your program.
However, if you still get an error while attempting to import, you should add a comment to the project's open Python3 import issue thread on GitHub and let them know, since there still appears to be some ambiguity if this bug has been fixed yet or not.
Make it run temporarily
python-gcm depends on the requests package. Make sure it is installed. Invoke pip install requests to install it.
Download the ZIP archive of the master branch.
Extract the archive into a folder python-gcm-master.
Patch the file python-gcm-master/gcm/__init__.py. Use
import gcm
#GCM = gcm.GCM
instead of
import gcm
GCM = gcm.GCM
In this folder create a python file test.py. This file should contain example code from the python-gcm package:
from gcm.gcm import GCM
gcm = GCM(API_KEY)
data = {'param1': 'value1', 'param2': 'value2'}
# Plaintext request
reg_id = '12'
gcm.plaintext_request(registration_id=reg_id, data=data)
# JSON request
reg_ids = ['12', '34', '69']
response = gcm.json_request(registration_ids=reg_ids, data=data)
# Extra arguments
res = gcm.json_request(
registration_ids=reg_ids, data=data,
collapse_key='uptoyou', delay_while_idle=True, time_to_live=3600
)
Run this file using python3 test.py. Be aware that the example uses from gcm.gcm import GCM instead of the default from gcm import GCM
If an gcm.gcm.GCMAuthenticationException is thrown, you can see that it is imported correctly and you need to adjust your API_KEY. Adjust the test.py file according to your needs.
Make it run permanently
As far as you tried a relative import, I think you already came across issue 65. Apparently they broke the import mechanism for python3. However, it works per default on python2, but it is buggy with python3. The issue in the issue tracker is still open.
How do I solve an ImportError: No module named 'cStringIO' under Python 3.x?
From Python 3.0 changelog:
The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
From the Python 3 email documentation it can be seen that io.StringIO should be used instead:
from io import StringIO
from email.generator import Generator
fp = StringIO()
g = Generator(fp, mangle_from_=True, maxheaderlen=60)
g.flatten(msg)
text = fp.getvalue()
I had the same issue because my file was called email.py. I renamed the file and the issue disappeared.
I had the issue because my directory was called email. I renamed the directory to emails and the issue was gone.