Good morning!
I'm using a code (with python 3.8) that is running both in a local PC and in a ssh server. In one point, I'm loading data from a pickle using the next piece of code:
from os.path import exists
import _pickle as pickle
def load_pickle(pickle_file):
if exists(pickle_file):
with open(pickle_file, 'rb') as f:
loaded_dic = pickle.load(f)
return loaded_dic
else:
return 'Pickle not found'
pickle_file is a string with the path of the pickle. If the pickle exists, the function returns a dictionary, while if it doesn't exist, it returns the string 'Pickle not found'.
In my local PC, the code works perfectly, loading the dict without problems. However, in the ssh server, theoretically, the dict is loaded, but, if I try to access to it, jus typing loaded_dic, it throws the following error:
AttributeError: 'NoneType' object has no attribute 'axes'
Due to it, the rest of my code fails when it try to use the variable loaded_dic.
Thank you very much in advance!
I have a similar problem. For me it happens as I store pandas DataFrames in a dictionary and save this dict as a pickle with pandas version '1.1.1'.
When I read the dictionary pickle with pandas version '0.25.3' on another server, I get the same error.
Both have pickle version 4.0 and I do not have a solution yet, other than upgrading to similar pandas versions.
I made a small example, it also happens when I store just a DataFrame, Saving it on one machine:
import pandas as pd
print("Pandas version", pd.__version__)
df = pd.DataFrame([1, 2, 3])
df.to_pickle('df.pkl')
Pandas version 1.1.1
Then loading it on another machine:
import pandas as pd
print("Pandas version", pd.__version__)
df = pd.read_pickle('df.pkl')
print(type(df))
Pandas version 0.25.3
<class 'pandas.core.frame.DataFrame'>
print(len(df))
results in this error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-8-9f6b6d8c3cd3> in <module>
----> 1 print(len(df))
/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in __len__(self)
994 Returns length of info axis, but here we use the index.
995 """
--> 996 return len(self.index)
997
998 def dot(self, other):
/opt/conda/lib/python3.7/site-packages/pandas/core/generic.py in __getattr__(self, name)
5173 or name in self._accessors
5174 ):
-> 5175 return object.__getattribute__(self, name)
5176 else:
5177 if self._info_axis._can_hold_identifiers_and_holds_name(name):
pandas/_libs/properties.pyx in pandas._libs.properties.AxisProperty.__get__()
AttributeError: 'NoneType' object has no attribute 'axes'
Avi's answer helped me. I had pickled with a later version of Pandas and was trying to read the pickle file with an earlier version.
This code clearly doesn't work anywhere; it doesn't return loaded_dict, a local variable, so nothing can use it. Change it to:
return pickle.load(f)
and the caller will receive the loaded dict instead of the default return value, None.
Update for edited question: With the return, your code works as written. Your pickle file on said machine must have the result of pickling None stored in it, rather than whatever you expected. Or your code is broken in some other place we haven't seen. The loading code is fine, and behaving exactly as its supposed to.
Related
I am working on topic inference on a new corpus given a previously derived lda model. I am able to load the model perfectly, while I am not able to load the id2word file to create the corpora.Dictionary object needed to map the new corpus into numbers: the load method returns a dict attribute error that I don't know why. Below is the minimal code that replicates the situation, and I have attached the code (and packages used) here.
Thank you in advance for your response...
import numpy as np
import os
import pandas as pd
import gensim
from gensim import corpora
import datetime
import nltk
model_name = "lda_sub_full_35"
dictionary_name = "lda_sub_full_35.id2word"
model_for_inference = gensim.models.LdaModel.load(model_name, mmap='r')
print('Successfully load the model')
lda_dictionary = corpora.Dictionary.load(dictionary_name, mmap='r')
I expect to have both the dictionary and the model loaded, but it turns out that when I load the dictionary, I got the below error:
File "topic_inference.py", line 31, in <module>
lda_dictionary = corpora.Dictionary.load(dictionary_name, mmap='r')
File "/topic_modeling/env/lib/python3.8/site-packages/gensim/utils.py", line 487, in load
obj._load_specials(fname, mmap, compress, subname)
AttributeError: 'dict' object has no attribute '_load_specials'```
How were the contents of the lda_sub_full_35.id2word file originally saved?
Only if it was saved by a Gensim corpora.Dictionary object's .save() method should it be loaded as you've tried, with corpora.Dictionary.load().
If, by any chance, it was just a plain Python dict saved via some other method of writing a pickle()-created object, then you would need to load it in a symmetrically-matched way. That might be as simple as:
import pickle
with open(path, 'rb') as f:
lda_dictionary = pickle.load(f)
I'm converting a code from Python 2.7 to Python 3.8.
In its Python 2.7 version, I had to use downgraded versions of scipy and numpy in order to avoid a TypeError (see below). With Python 3.8, these downgraded versions of scipy and numpy are not available anymore and I get this error, which I'm unable to fix.
1. Setup
Previous : MacOS Catalina 10.15.7, Python 2.7.16, numpy 1.9.0, scipy 1.0.1
New : MacOS Catalina 10.15.7, Python 3.8.6, numpy 1.20.3, scipy 1.4.0
2. Code
It happens when calling scipy.integrate.odeint :
y_trajectory = scipy.integrate.odeint(growth_a_derivs, y_start, t_array, atol=eps, args=myargs)
With growth_a_derivs a function, y_start, t_array numpy arrays with dtype float64, eps a float and myargs a tuple of length 1 containing a dictionary.
3. Error
The following traceback pops :
Traceback (most recent call last):
<long blah blah blah>
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyxcosmo/icosmo_cosmo.py", line 149, in growth_a
result=growth_a_ode(cosmo, a, unnorm=unnorm)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyxcosmo/icosmo_cosmo.py", line 671, in growth_a_ode
y_trajectory = scipy.integrate.odeint(growth_a_derivs, y_start, t_array, atol=eps, args=myargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/scipy/integrate/odepack.py", line 242, in odeint
output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, ml, mu,
TypeError: Cannot cast array data from dtype('O') to dtype('float64') according to the rule 'safe'
So the last line is in scipy's integrate pckg, the full call to odeint is :
output = _odepack.odeint(func, y0, t, args, Dfun, col_deriv, ml, mu,
full_output, rtol, atol, tcrit, h0, hmax, hmin,
ixpr, mxstep, mxhnil, mxordn, mxords,
int(bool(tfirst)))
Inside this, the error is in a compiled scipy .so file, thus I can't analyze this error further. I just know that the call to growth_a_derivs isn't problematic (its print() calls all work). But I don't know whether its a numpy or scipy problem, and how to overcome it.
4. Resolution !
I was able to understand the problem thanks to #RFoxtea answer below.
I believe this issue was caused by the change in numpy version.
The output of growth_a_derivs is :
np.array([f1,f2])
with :
type(f1) = numpy.float64
type(f2) = numpy.ndarray
f2.shape = (1,)
With numpy 1.9, this gives :
np.array([f1,f2]) = [1.234, np.array([1.234])]
np.array([f1,f2]).dtype = numpy.float64
With numpy 1.20, this gives :
np.array([f1,f2]) = [1.234, np.array([1.234])]
np.array([f1,f2]).dtype = numpy.object
And that is not accepted by scipy.integrate.odeint.
However, notice that scipy.integrate.solve_ivp is ok with that.
Thanks for your help !
At some point in running scipy.integrate.odeint, Numpy is told to convert an array of Python objects into an array of floats, and it's answering you that it can't do that. Your description suggests the problem has to be with y_start, t_start or (maybe, I'm not sure) the return value of growth_a_derivs. Please make sure that these all have an appropriate dtype that can be converted into a float. Could be very easy to fix.
Thanks to #RFoxtea who provided the right solution.
However there is an interesting alternative. You can use scipy.integrate.solve_ivp instead of scipy.integrate.odeint.
Solve_ivp can handle an output of funthat is with dtype object.
The other day I was working on a project for an Image captioning model on Keras. But when I am running it, I am facing a host of error. Note that I am using Atom Editor and a virtual environment in Python, Running everything from a Command-line.
train_features = load_photo_features(os.path('C:/Users/neelg/Documents/Atom_projects/Main/features.pkl'), train)
In this line, I am receiving this error==>
File "C:\Users\neelg\Documents\Atom_projects\Main\Img_cap.py", line 143
train_features = load_photo_features(os.path('C:/Users/neelg/Documents/Atom_projects/Main/features.pkl'), train)
^
SyntaxError: invalid syntax
I think that the syntax is correct regarding the function, yet the error persists. So, in a seperate file I copied the function and tried to isolate problem.
Code for the standalone function:-
from pickle import load
import os
def load_photo_features(filename, dataset):
all_features = load(open(filename, 'rb'))
features = {k: all_features[k] for k in dataset}
return features
filename = 'C:/Users/neelg/Documents/Atom_projects/Main/Flickr8k_text/Flickr8k.trainImages.txt'
train_features = load_photo_features(os.path('C:/Users/neelg/Documents/Atom_projects/Main/features.pkl'), train)
Now, A different type of problem crops up:
Traceback (most recent call last):
File "C:\Users\neelg\Documents\Atom_projects\Main\testing.py", line 10, in <module>
train_features = load_photo_features(os.path('C:/Users/neelg/Documents/Atom_projects/Main/features.pkl'), train)
TypeError: 'module' object is not callable
Any help? I am trying to import the Flickr_8k dataset, which contains random pictures and another small dataset which are the labels of those photographs...
P.S=>Pls send suggestions after testing the code on tour own editors before submitting because I suspect there is some core problem arising due to the System encoding(As suggested by some others). Also, it is not possible to load the whole code due to it's length and requirement of multiple files.
This error comes from the fact that you're calling os.path which is a module not a function. Just remove it, you don't need it in this use-case, a string is enough for filename in open
I was about to ask you the same question with #ted why do you use os.path when you are trying to load the file.
Normally, I am using the following code for loading from pickle:
def load_obj(filename):
with open(filename, "rb") as fp:
return pickle.load(fp, enconding = 'bytes')
Furthermore, if I try something like that it works:
from pickle import load
import os
import pdb
def load_photo_features(filename):
all_features = load(open(filename, 'rb'))
pdb.set_trace()
#features = {k: all_features[k] for k in dataset}
#return features
train_features = load_photo_features('train.pkl')
I do not know what is the dataset input to proceed, but loading of the pickle file works fine.
I am following this tutorial: https://github.com/fastai/fastai/tree/master/courses/dl2/imdb_scripts
I downloaded the pre-trained model in part 3b.
I want to open the .h5 files and look/use the weights. I tried to use python to do this, but it is not opening.
Here’s the code I used:
import tables
import pandas as pd
filename = “…bwd_wt103.h5”
file = tables.open_file(filename)
Here’s the error:
OSError: HDF5 error back trace
File “C:\ci\hdf5_1525883595717\work\src\H5F.c”, line 511, in H5Fopen
unable to open file
File “C:\ci\hdf5_1525883595717\work\src\H5Fint.c”, line 1604, in H5F_open
unable to read superblock
File “C:\ci\hdf5_1525883595717\work\src\H5Fsuper.c”, line 413, in H5F__super_read
file signature not found
End of HDF5 error back trace
Unable to open/create file 'C:/Users/Rishabh/Documents/School and Work/Classes/8
Fall2019/Senior Design/ULMFiT/Wiki Data/wt103/models/bwd_wt103.h5'
I also used The HDF Group HDF Viewer: https://support.hdfgroup.org/products/java/release/download.html
But that didn’t work either. It gave an error saying “Failed to open the file… Unsupported format”
Is there a way to load the weights in Python? I ultimately want to access the last layer of the stacked LSTMS to create word embeddings.
Thanks in advance.
That's because it's a torch model. You can load it on your local machine using torch like so:
>>> import torch
>>> filename = "bwd_wt103.h5"
>>> f = torch.load(filename, map_location=torch.device('cpu'))
Now, let's explore it:
>>> type(f)
OrderedDict
>>> len(f.keys())
15
>>> list(f.keys())
['0.encoder.weight',
'0.encoder_with_dropout.embed.weight',
'0.rnns.0.module.weight_ih_l0',
'0.rnns.0.module.bias_ih_l0',
'0.rnns.0.module.bias_hh_l0',
'0.rnns.0.module.weight_hh_l0_raw',
'0.rnns.1.module.weight_ih_l0',
'0.rnns.1.module.bias_ih_l0',
'0.rnns.1.module.bias_hh_l0',
'0.rnns.1.module.weight_hh_l0_raw',
'0.rnns.2.module.weight_ih_l0',
'0.rnns.2.module.bias_ih_l0',
'0.rnns.2.module.bias_hh_l0',
'0.rnns.2.module.weight_hh_l0_raw',
'1.decoder.weight']
You can access the weights of 0.rnns.2.module.weight_hh_l0_raw like so:
>>> wts = f['0.rnns.2.module.weight_hh_l0_raw']
>>> wts.shape
torch.Size([1600, 400])
I wrote a class where some data are fitted. Since the fitting takes very long when lots of data have to be fitted, I want to save the fit-object of this class so I do not have to repeat the fitting when I want to use the fitted data later. Using pickle, I get the following error calling the save method on an object:
AttributeError: Can't pickle local object 'ConstantModel.__init__.<locals>.constant'
I only have this problem when pickle the fitted data, pickle works if I save the object before fitting.
Is there a way to pickle fitted data or is there a nice workaround?
class pattern:
def fitting(self):
mod_total = lmfit.models.ConstantModel()
pars_total = mod_total.guess(self.y, x=self.x)
self.fit = mod_total.fit(self.y, pars_total, x=self.x)
def save(self, path):
with open(path, 'wb') as filehandler:
pickle.dump(self, filehandler)
I found a solution for this problem: Using dill instead of pickle works (as I want it to do).