Cannot open lmdb db file in Caffe2 - pytorch

I installed Caffe2 from source for Ubuntu.
I don't use Anaconda and my python is python2.
which python gives me
/usr/bin/python
I can import lmdb in python.
I can run tutorials from this page
My problem in testing one of the examples is "Can't open lmdb file".
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-19-c9a97f268658> in <module>()
1 # The parameter initialization network only needs to be run once.
2 # Now all the parameter blobs are initialized in the workspace.
----> 3 workspace.RunNetOnce(train_model.param_init_net)
4
5 # Creating an actual network as a C++ object in memory.
/home/pytorch/build/caffe2/python/workspace.py in RunNetOnce(net)
199 C.Workspace.current._last_failed_op_net_position,
200 GetNetName(net),
--> 201 StringifyProto(net),
202 )
203
/home/pytorch/build/caffe2/python/workspace.py in CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs)
178 def CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs):
179 try:
--> 180 return func(*args, **kwargs)
181 except Exception:
182 op_id = op_id_fetcher()
RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: /home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb of type lmdbError from operator:
output: "dbreader_/home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "/home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" }
I have all files and have read access.
I compiled pytorch with USE_LMDB=ON.
I see the same issues but can't find how they solved the issues.
My laptop system is small with Ubuntu 16.4, with Nvidia Cuda GeForce GT 750M, so quite old model.
Then I found in one page. They solved the issue with changing LMDB_MAP_SIZE in lbdb.cc. So I changed to 4G, but still have error.
constexpr size_t LMDB_MAP_SIZE = 4294967296;//4gb 1099511627776; // 1 TB
The problem is at this line
workspace.RunNetOnce(train_model.param_init_net)
How to solve the issue?

Related

Torchtext import error on macbook air m1. Used pip3 to install all of it

While the installation went smooth, I keep facing this error when I try to import torchtext on my macbook air m1
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
Input In [10], in <cell line: 1>()
----> 1 import torchtext
File ~/Library/Python/3.8/lib/python/site-packages/torchtext/__init__.py:6, in <module>
3 from torch.hub import _get_torch_home
5 # the following import has to happen first in order to load the torchtext C++ library
----> 6 from torchtext import _extension # noqa: F401
8 _TEXT_BUCKET = "https://download.pytorch.org/models/text/"
10 _CACHE_DIR = os.path.expanduser(os.path.join(_get_torch_home(), "text"))
File ~/Library/Python/3.8/lib/python/site-packages/torchtext/_extension.py:64, in <module>
59 # This import is for initializing the methods registered via PyBind11
60 # This has to happen after the base library is loaded
61 from torchtext import _torchtext # noqa
---> 64 _init_extension()
File ~/Library/Python/3.8/lib/python/site-packages/torchtext/_extension.py:58, in _init_extension()
55 if not _mod_utils.is_module_available("torchtext._torchtext"):
56 raise ImportError("torchtext C++ Extension is not found.")
---> 58 _load_lib("libtorchtext")
59 # This import is for initializing the methods registered via PyBind11
60 # This has to happen after the base library is loaded
61 from torchtext import _torchtext
File ~/Library/Python/3.8/lib/python/site-packages/torchtext/_extension.py:50, in _load_lib(lib)
48 if not path.exists():
49 return False
---> 50 torch.ops.load_library(path)
51 return True
File ~/Library/Python/3.8/lib/python/site-packages/torch/_ops.py:220, in _Ops.load_library(self, path)
215 path = torch._utils_internal.resolve_library_path(path)
216 with dl_open_guard():
217 # Import the shared library into the process, thus running its
218 # static (global) initialization code in order to register custom
219 # operators with the JIT.
--> 220 ctypes.CDLL(path)
221 self.loaded_libraries.add(path)
File /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py:365, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
362 self._FuncPtr = _FuncPtr
364 if handle is None:
--> 365 self._handle = _dlopen(self._name, mode)
366 else:
367 self._handle = handle
OSError: dlopen(/Users/pratik/Library/Python/3.8/lib/python/site-packages/torchtext/lib/libtorchtext.so, 0x0006): Symbol not found: __ZN3c1012OptionalType3getENS_4Type24SingletonOrSharedTypePtrIS1_EE
Referenced from: /Users/pratik/Library/Python/3.8/lib/python/site-packages/torchtext/lib/libtorchtext.so
Expected in: /Users/pratik/Library/Python/3.8/lib/python/site-packages/torch/lib/libtorch_cpu.dylib
I don't quite understand what I can do about this. I uninstalled and reinstalled it all to avoid an error I was having previously. But now, this comes up.

Azure ML Tabular Dataset : missing 1 required positional argument: 'stream_column'

For the Python API for tabular dataset of AzureML (azureml.data.TabularDataset), there are two experimental methods which have been introduced:
download(stream_column, target_path=None, overwrite=False, ignore_not_found=True)
mount(stream_column, mount_point=None)
Parameter stream_column has been defined as The stream column to mount or download.
What is the actual meaning of stream_column? I don't see any example any where?
Any pointer will be helpful.
The stack trace:
Method download: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/tmp/ipykernel_11561/3904436543.py in <module>
----> 1 tab_dataset.download(target_path="../data/tabular")
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_base_sdk_common/_docstring_wrapper.py in wrapped(*args, **kwargs)
50 def wrapped(*args, **kwargs):
51 module_logger.warning("Method {0}: {1} {2}".format(func.__name__, _method_msg, _experimental_link_msg))
---> 52 return func(*args, **kwargs)
53 return wrapped
54
/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/data/_loggerfactory.py in wrapper(*args, **kwargs)
130 with _LoggerFactory.track_activity(logger, func.__name__, activity_type, custom_dimensions) as al:
131 try:
--> 132 return func(*args, **kwargs)
133 except Exception as e:
134 if hasattr(al, 'activity_info') and hasattr(e, 'error_code'):
TypeError: download() missing 1 required positional argument: 'stream_column'
Update on 5th March, 2022
I posted this as a support ticket with Azure. Following is the answer I have received:
As you can see from our documentation of TabularDataset Class,
the “stream_column” parameter is required. So, that error is occurring
because you are not passing any parameters when you are calling the
download method. The “stream_column” parameter should have the
stream column to download/mount. So, you need to pass the column name
that contains the paths from which the data will be streamed.
Please find an example here.

Google Colab - tensowflow object detection api - 'function' object has no attribute 'called'

I encountered the following error when I try to test the object detection api model_builder_test.py.
!apt-get install -y -qq protobuf-compiler python-pil python-lxml
!git clone --quiet https://github.com/tensorflow/models.git
import os
os.chdir('models/research')
!protoc object_detection/protos/*.proto --python_out=.
import sys
sys.path.append('/content/models/research/slim')
%run object_detection/builders/model_builder_test.py
The following error appears after running the model_builder_test.py
.W0220 03:22:35.097244 140099951081344 deprecation.py:323] From
/content/models/research/object_detection/anchor_generators/grid_anchor_generator.py:59:
to_float (from tensorflow.python.ops.math_ops) is deprecated and will
be removed in a future version. Instructions for updating: Use tf.cast
instead. .. WARNING: The TensorFlow contrib module will not be
included in TensorFlow 2.0. For more information, please see: *
https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue.
..................s
---------------------------------------------------------------------- Ran 22 tests in 0.203s
OK (skipped=1)
--------------------------------------------------------------------------- AttributeError Traceback (most recent call
last) in ()
----> 1 get_ipython().magic('run object_detection/builders/model_builder_test.py')
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py
in magic(self, arg_s) 2158 magic_name, _, magic_arg_s =
arg_s.partition(' ') 2159 magic_name =
magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2160 return self.run_line_magic(magic_name, magic_arg_s) 2161 2162
-------------------------------------------------------------------------
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py
in run_line_magic(self, magic_name, line) 2079
kwargs['local_ns'] = sys._getframe(stack_depth).f_locals 2080
with self.builtin_trap:
-> 2081 result = fn(*args,**kwargs) 2082 return result 2083
in run(self, parameter_s, runner, file_finder)
/usr/local/lib/python3.6/dist-packages/IPython/core/magic.py in
(f, *a, **k)
186 # but it's overkill for just that one bit of state.
187 def magic_deco(arg):
--> 188 call = lambda f, *a, **k: f(*a, **k)
189
190 if callable(arg):
/usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py
in run(self, parameter_s, runner, file_finder)
740 else:
741 # regular execution
--> 742 run()
743
744 if 'i' in opts:
/usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py
in run()
726 def run():
727 runner(filename, prog_ns, prog_ns,
--> 728 exit_ignore=exit_ignore)
729
730 if 't' in opts:
/usr/local/lib/python3.6/dist-packages/IPython/core/pylabtools.py in
mpl_execfile(fname, *where, **kw)
175 matplotlib.interactive(is_interactive)
176 # make rendering call now, if the user tried to do it
--> 177 if plt.draw_if_interactive.called:
178 plt.draw()
179 plt.draw_if_interactive.called = False
AttributeError: 'function' object has no attribute 'called'
This is how I overcame the issue:
install prompt-toolkit to the version 1.0.15, as explained in the link below
https://github.com/jupyter/jupyter_console/issues/158
restart the runtime to activate the package
use '!python' instead of '%run'

Tensorflow cifar10.maybe_download_and_extract() gives error

I try to run a CNN model on tensorflow using cifar10 library. However, when I import cifar10 and run the maybe_download_and_extract() method, it gives the following error. I am running it on tensorflow (1.5.0) and Python 3.6.4 :: Anaconda, Inc.
---------------------------------------------------------------------------
UnrecognizedFlagError Traceback (most recent call last)
<ipython-input-37-02a754d7036a> in <module>()
----> 1 cifar10.maybe_download_and_extract()
D:\Google Drive\Akademik\Ozyegin\CS566\Project 2\cifar10.py in maybe_download_and_extract()
~\anaconda3\lib\site-packages\tensorflow\python\platform\flags.py in __getattr__(self, name)
82 # a flag.
83 if not wrapped.is_parsed():
---> 84 wrapped(_sys.argv)
85 return wrapped.__getattr__(name)
86
~\anaconda3\lib\site-packages\absl\flags\_flagvalues.py in __call__(self, argv, known_only)
628 suggestions = _helpers.get_flag_suggestions(name, list(self))
629 raise _exceptions.UnrecognizedFlagError(
--> 630 name, value, suggestions=suggestions)
631
632 self.mark_as_parsed()
UnrecognizedFlagError: Unknown command line flag 'f'
Add following line to your jupyter notebook:
tf.app.flags.DEFINE_string('f', '', 'kernel')

Python 3.4 plistlib doesn't work (str vs bytes errors)

So, I started on a new toy project and decided I'd use Python 3 for the first time...
In [1]: import plistlib
In [2]: with open("/Volumes/Thunderbay/CURRENT/Music/iTunes/iTunes Library.xml") as itl:
library = plistlib.load(itl)
...:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-6459a022cb71> in <module>()
1 with open("/Volumes/Thunderbay/CURRENT/Music/iTunes/iTunes Library.xml") as itl:
----> 2 library = plistlib.load(itl)
3
/usr/local/Cellar/python3/3.4.3_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plistlib.py in load(fp, fmt, use_builtin_types, dict_type)
984 fp.seek(0)
985 for info in _FORMATS.values():
--> 986 if info['detect'](header):
987 P = info['parser']
988 break
/usr/local/Cellar/python3/3.4.3_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plistlib.py in _is_fmt_xml(header)
556
557 for pfx in prefixes:
--> 558 if header.startswith(pfx):
559 return True
560
TypeError: startswith first arg must be str or a tuple of str, not bytes
hmm ok, let's give it a hint:
In [3]: with open("/Volumes/Thunderbay/CURRENT/Music/iTunes/iTunes Library.xml") as itl:
library = plistlib.load(itl, fmt=plistlib.FMT_XML)
...:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-ef5f06b44ec2> in <module>()
1 with open("/Volumes/Thunderbay/CURRENT/Music/iTunes/iTunes Library.xml") as itl:
----> 2 library = plistlib.load(itl, fmt=plistlib.FMT_XML)
3
/usr/local/Cellar/python3/3.4.3_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plistlib.py in load(fp, fmt, use_builtin_types, dict_type)
995
996 p = P(use_builtin_types=use_builtin_types, dict_type=dict_type)
--> 997 return p.parse(fp)
998
999
/usr/local/Cellar/python3/3.4.3_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/plistlib.py in parse(self, fileobj)
323 self.parser.EndElementHandler = self.handle_end_element
324 self.parser.CharacterDataHandler = self.handle_data
--> 325 self.parser.ParseFile(fileobj)
326 return self.root
327
TypeError: read() did not return a bytes object (type=str)
plistlib is in the standard library, but from the problems above I have the feeling it has not actually been converted to Python 3?
Anyway, my actual question: is it possible to open an XML plist file with plistlib in Python 3.4.3?
surely I'm missing something obvious here perhaps... just noticed the Py2 version of plistlib (which works!) has a different interface, so someone has actually modified the code of the library for inclusion with Py3...
Thanks to #J Presper Eckert for giving me a clue about what to look for...
I then found this article:
http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html#the-binary-option
which suggests the answer is simply to open the file in binary mode, tried it and it works!
with open("/Volumes/Thunderbay/CURRENT/Music/iTunes/iTunes Library.xml", 'rb') as itl:
library = plistlib.load(itl)
Got same error message using python 3.4.3 against .plist file ( mac property list, configuration file in xml ).
Try this (worked for me):
copy plist / xml file to new file.
change new file extension to '.txt'
open newfile.txt in textEdit (mac), notepad++(win) or similar
save as .txt file with UTF-8 encoding & plaintext only.
Now, when you read newfile.txt, you won't see "startswith first arg must be str or a tuple of str, not bytes".
Cheers

Resources