I am a newbie and then trying to learn fastai data block api
Here is the mistake, the code is exactly the same as the tutorial：
coco = untar_data(URLs.COCO_TINY)
path=coco/'train.json'
images, lbl_bbox = get_annotations(coco/'train.json')
img2bbox = dict(zip(images, lbl_bbox))
get_y_func = lambda o:img2bbox[o.name]
data = (ObjectItemList.from_folder(coco)
.split_by_rand_pct()
.label_from_func(get_y_func)
.transform(get_transforms(), tfm_y=True)
.databunch(bs=1, num_workers=0,collate_fn=bb_pad_collate))
data.show_batch(rows=2, ds_type=DatasetType.Valid, figsize=(6,6))
Then the error is：
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\IPython\core\interactiveshell.py", line
3326, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-7-25e60680c0ba>", line 15, in <module>
data.show_batch(rows=2, ds_type=DatasetType.Valid, figsize=(6,6))
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 185, in show_batch
x,y = self.one_batch(ds_type, True, True)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 168, in one_batch
try: x,y = next(iter(dl))
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 75, in __iter__
for b in self.dl: yield self.proc_batch(b)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 348,__next__
data = _utils.pin_memory.pin_memory(data)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
55, in pin_memory
return [pin_memory(sample) for sample in data]
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
55, in <listcomp>
return [pin_memory(sample) for sample in data]
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
47, in pin_memory
return data.pin_memory()
RuntimeError: CUDA error: unknown error
Regarding this error, the forums on the Internet are all setting the parameters of the DataLoader, but it does not seem to be used here
How would I go about this?
I also got this error. me it fixed by edit the following two lines in the head of notebook:
import os
os.environ['CUDA_VISIBLE_DEVICES']='2'
Related
I am new to Deep Learning and my first Project is FACIAL EMOTION RECOGINISTION
I am trying to use this DeepFace library but seems kind of stuck at the moment can anyone help ?
import cv2
from cv2 import cvtColor
from deepface import DeepFace
import matplotlib.pyplot as plt
img = cv2.imread('Images\happy\happy_001.jpg')
# plt.imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB))
# plt.show()
predictions = DeepFace.analyze(img, actions = ['age', 'gender', 'race', 'emotion'])
and the error i am getting is
Traceback (most recent call last):
File "C:\Users\asus\OneDrive - Graphic Era University\Desktop\ML AND AI\FACE RECOG\test.py", line 11, in <module>
predictions=DeepFace.analyze(img, actions = ['age', 'gender', 'race', 'emotion'])
File "C:\Python 3.9\lib\site-packages\deepface\DeepFace.py", line 355, in analyze
models['gender'] = build_model('Gender')
File "C:\Python 3.9\lib\site-packages\deepface\DeepFace.py", line 61, in build_model
model = model()
File "C:\Python 3.9\lib\site-packages\deepface\extendedmodels\Gender.py", line 49, in loadModel
gender_model.load_weights(home+'/.deepface/weights/gender_model_weights.h5')
File "C:\Python 3.9\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Python 3.9\lib\site-packages\h5py\_hl\files.py", line 507, in __init__
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "C:\Python 3.9\lib\site-packages\h5py\_hl\files.py", line 220, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 106, in h5py.h5f.open
OSError: Unable to open file (truncated file: eof = 232972459, sblock->base_addr = 0, stored_eof = 537149760)
I certainly don't know how to solve this .. can anyone help?
I am using VS CODE with python 3.9.6
Check that file: "h5py\h5f.pyx", line 106, in h5py.h5f.open. I guess your issue rises from there. Try to re-install the libs. Also, here your problem was discussed and, probably, solved: https://github.com/keras-team/keras/issues/6221
I am getting this error on the line 273 even though I'm not using the placeholder in my code.
Using TensorFlow backend.
Traceback (most recent call last):
File "GA5.py", line 273, in <module>
model = load_trained_model_from_checkpoint(config_path,checkpoint_path,training=True,trainable=True,seq_len=SEQ_LEN)
File "/home/kosimadukwe/.local/lib/python3.8/site-packages/keras_bert/loader.py", line 163, in load_trained_model_from_checkpoint
model, config = build_model_from_config(
File "/home/kosimadukwe/.local/lib/python3.8/site-packages/keras_bert/loader.py", line 46, in build_model_from_config
model = get_model(
File "/home/kosimadukwe/.local/lib/python3.8/site-packages/keras_bert/bert.py", line 85, in get_model
inputs = get_inputs(seq_len=seq_len)
File "/home/kosimadukwe/.local/lib/python3.8/site-packages/keras_bert/layers/inputs.py", line 12, in get_inputs
return [keras.layers.Input(
File "/home/kosimadukwe/.local/lib/python3.8/site-packages/keras_bert/layers/inputs.py", line 12, in <listcomp>
return [keras.layers.Input(
File "/usr/pkg/lib/python3.8/site-packages/keras/engine/input_layer.py", line 175, in Input
input_layer = InputLayer(batch_input_shape=batch_shape,
File "/usr/pkg/lib/python3.8/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/pkg/lib/python3.8/site-packages/keras/engine/input_layer.py", line 84, in __init__
input_tensor = K.placeholder(shape=batch_input_shape,
File "/usr/pkg/lib/python3.8/site-packages/keras/backend/tensorflow_backend.py", line 517, in placeholder
x = tf.placeholder(dtype, shape=shape, name=name)
AttributeError: module 'tensorflow' has no attribute 'placeholder'
I have searched similar errors and added these import lines as suggested
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
I am using Python3.8 and Tensorflow2.0 and Keras-bert. and from the tests ive done its most likely due to Keras-Bert.
Can anyone give me suggestions? I'm all out of options.
I'm trying to fit a linear model to some data using the code below. I'm getting the error below. I think the error has an issue with the '%' in the field name. I have many fields in my data with this naming convention. Does anyone know how to solve this issue with statsmodel?
code:
mod = ols('fieldA%'+'~'+'fieldB',data=smp_df).fit()
error:
Traceback (most recent call last):
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\IPython\core\interactiveshell.py", line 3267, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-24-9e4f478cefb9>", line 3, in <module>
mod = ols('fieldA%'+' ~'+'fieldB',data=smp_df).fit()
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\statsmodels\base\model.py", line 155, in from_formula
missing=missing)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\statsmodels\formula\formulatools.py", line 65, in handle_formula_data
NA_action=na_action)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\highlevel.py", line 310, in dmatrices
NA_action, return_type)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\highlevel.py", line 165, in _do_highlevel_design
NA_action)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\highlevel.py", line 70, in _try_incr_builders
NA_action)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\build.py", line 689, in design_matrix_builders
factor_states = _factors_memorize(all_factors, data_iter_maker, eval_env)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\build.py", line 354, in _factors_memorize
which_pass = factor.memorize_passes_needed(state, eval_env)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\eval.py", line 474, in memorize_passes_needed
subset_names = [name for name in ast_names(self.code)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\eval.py", line 474, in <listcomp>
subset_names = [name for name in ast_names(self.code)
File "C:\Users\username\AppDataPython\envs\py36\lib\site-packages\patsy\eval.py", line 105, in ast_names
for node in ast.walk(ast.parse(code)):
File "C:\Users\username\AppDataPython\envs\py36\lib\ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
CIM_ID_SALES %
^
SyntaxError: invalid syntax
I'm trying to multiprocess the fit_generator.
These are the problems that I face.
trainable_model.fit_generator(load_random_cached_bottlenecks(BATCH_SIZE, label_map, training_addr_label_map, train_npy_dir, 'h5py', h5py_file_train),epochs = EPOCHS, steps_per_epoch=iterations_per_epoch_t, validation_data = load_random_cached_bottlenecks(BATCH_SIZE, label_map, validation_addr_label_map, val_npy_dir, 'h5py', h5py_file_val), validation_steps=iterations_per_epoch_v, workers = 1, callbacks = callback_list, use_multiprocessing = True, max_queue_size = 32)
The main arguments that are causing problem: workers and use_multiprocessing.
When worker=1, use_multiprocessing=True/False runs with no problem.
If workers=5, use_multiprocessing=True its throwing errors. The weird thing is its running, but at some random iteration I'm getting errors like
KeyError: 'Unable to open object (bad local heap signature)'
or
KeyError: 'Unable to open object (wrong B-tree signature)'
Im using h5py to read the files. I have written custom generator for this purpose.
def load_random_cached_bottlenecks(batch_size, label_map,
addr_label_map, dirs, comp_type = 'h5py', hdf5_file = None):
'''
Parameters
----------
batch_size: Number of bottlenecks to be loaded along with the labels
label_map: The dictionary that maps the class_names and the index
addr_label_map: The dictionary that maps addrs of bottlenecks and the labels
hdf5_file: This is the hdf5 file object with reading enabled.
Returns
-------
batch: (bottlenecks_train, bottlenecks_labels) a batch of them which is equal to batch_size
'''
while True:
chosen_h5py = np.random.choice(dirs, size = batch_size)
# chosen_h5py = [dirs[i] for i in batch_index]
labels_for_chosen_h5py = [label_map[addr_label_map[i]] for i in chosen_h5py]
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
h5py_onehot = to_categorical(labels_for_chosen_h5py, num_classes = LABEL_LENGTH)
# print (h5py_data.shape)
yield (h5py_data, h5py_onehot)
I have referred here, but couldn't solve my problem.
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 677, in _data_generator_task
generator_output = next(self._generator)
File "general_model.py", line 263, in load_random_cached_bottlenecks
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 677, in _data_generator_task
generator_output = next(self._generator)
File "general_model.py", line 263, in load_random_cached_bottlenecks
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "general_model.py", line 263, in <listcomp>
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "general_model.py", line 263, in <listcomp>
h5py_data = np.array([hdf5_file[i] for i in chosen_h5py])
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/opt/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 177, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
File "/opt/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py", line 177, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
KeyError: 'Unable to open object (wrong B-tree signature)'
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: 'Unable to open object (bad symbol table node signature)'
Traceback (most recent call last):
File "general_model.py", line 437, in <module>
train_with_bottlenecks(args, label_map, trainable_model, non_trainable_model, iterations_per_epoch_t, iterations_per_epoch_v)
File "general_model.py", line 326, in train_with_bottlenecks
validation_steps=iterations_per_epoch_v, workers = 4, callbacks = callback_list, use_multiprocessing = True, max_queue_size = 32)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 2194, in fit_generator
generator_output = next(output_generator)
File "/opt/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get
six.reraise(value.__class__, value, value.__traceback__)
File "/opt/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
KeyError: 'Unable to open object (wrong B-tree signature)'
Any help is appreciated! Thanks in Advance!
This is not a solution per-se but this solved this problem for me.
I got the a similar error: OSError: Can't read data (wrong B-tree signature)
when trying to use fit_generator when this one reads data from a hdf5_file, also inside an anaconda3 virtual env.
In my case I created a new virtual environment and re-installed the needed dependencies of the specific versions in which it was supposed to work, with this my code ran smoothly.
I'm trying to carry out the tutorial named "Training a classifier" with PyTorch.
WHen trying to debug this part of the code :
import matplotlib.pyplot as plt
import numpy as np
# functions to show an image
def imshow(img):
img = img / 2 + 0.5 # unnormalize
npimg = img.numpy()
plt.imshow(np.transpose(npimg, (1, 2, 0)))
# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()
# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
I get this error message :
Files already downloaded and verified Files already downloaded and verified
Files already downloaded and verified Files already downloaded and verified Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 277, in
_fixup_main_from_path
run_name="__mp_main__")
File "D:\Anaconda\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "D:\Anaconda\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
dataiter = iter(trainloader)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 136, in
_check_not_importing_main
is not going to be frozen to produce an executable.)
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Traceback (most recent call last):
File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
dataiter = iter(trainloader)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
w.start()
File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj) File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 65, in
__init__
reduction.dump(process_obj, to_child)
File "D:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe
All the previous lines in the tutorial are working perfectly.
Does someone know how to solve this, please ?
Thanks a lot in advance
The question happened because Windows cannot run this DataLoader in 'num_workers' more than 0.
You can see where the trainloader come from.we can see
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
shuffle=True, num_workers=2)
We need to change the 'num_workers' to 0.like this:
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
shuffle=True, num_workers=0)
Every trainloaders need to change like this.
Got the same error. The following workaround works for me:
def run():
# code goes here
if __name__ == '__main__':
run()
This doesn't look to be a PyTorch problem. Try executing the code in Jupyter notebooks and other environment troubleshooting.
you need to add a if-clause protection as stated in the pytorch docs:
https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing