I am trying to install scikit-learn so that it doesn't have access to ATLAS (other BLAS and LAPACK libraries okay). Is there is a way to set the install so that ATLAS isn't used? Sorry for the basic question but I couldn't see a good way of doing it.
The reason I want to do this: I'm working on a cluster where I can't build atlas, but the pre-installed version of it is broken (compiled on 16 cores, only 8 cores on node, pthread problems), so when I install scikit-learn my scikit-learn tries to fire up these libraries and I get errors.
Thanks
EDIT: It seems I had some things wrong, and actually scipy is still broken after I compile it against openblas. In one of the unit tests I'm getting:
assertion !pthread_create(&thr->thrH, &attr, rout, arg) failed, line 111 of file /var/tmp/portage/sci-libs/atlas-3.10.1-r2/work/ATLAS//src/threads/ATL_thread_start.c
Bit of a longshot now, but any idea why my scipy build is even looking at ATLAS?
Numpy, scipy and scikit-learn will all try to link to ATLAS if they find it. While scikit-learn can build against its own embedded subset of BLAS if no ATLAS or other BLAS/LAPACK is found on the system, scipy will require a BLAS/LAPACK implementation anyway.
What you could do is build your own versions of numpy, scipy and scikit-learn against a locally built of either ATLAS or OpenBLAS (take the master branch on github for the latter).
To do so you need to customize the site.cfg file in the numpy and scipy source folders prior to building.
Also you should build OpenBLAS with something like:
make TARGET=NEHALEM NO_AFFINITY=1 NUM_THREADS=8 USE_OPENMP=0
Related
My team and I are using python 3.7.5 and a virtual environments. We are developing and using a fairly informal package. Its an engineering tool and due to the informal and fast pace we arent using a 0.0.0 versioning scheme yet. Hopwever, I would like an easy way to know exactly which version of the tool people are using if they run into issues. And I need a solution that doesn't require manually updating a version number in setuptools because I don't trust my coworkers to do that.
We are using python setuptools and pip to generate wheel files and install the packages into our virtual environments.
My question is: How can I get the commit sha of a particular package that is installed in the virtual environment at run time?
If the solution involves adding some code to embedd that data via setuptools during installation, thats good too.
I've been trying to get CUDA to work with TensorFlow for a while now because the neural nets I've been building are now taking hours to train on my CPU, and it'd be great to get that big speed boost. However, whenever I try to use it with TensorFlow (it works with PyTorch, but I want to learn multiple APIs), it tells me that one of the .dll files needed to run CUDA doesn't exist, when it actually does.
I've downloaded and replaced that .dll with other versions from dll-files.com. I've tried uninstalling and reinstalling TensorFlow, CUDA, and cuDNN. I've tried different versions of CUDA, but that only caused all the .dll files to not be found (and yes, I did change the CUDA_PATH value). I've tried switching the PATH between C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0 and C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin to see if that changed anything.
If anyone could help with this, that would be much appreciated.
The errors I get when I run tf.test.is_gpu_available()
The file existing
Try installing a different older version of the CUDA toolkit on top of the version you have installed already. It fixed it for me however I also had to import all the previous dlls from the latest cuDNN toolkit into the new legacy CUDA toolkit installs as well.
Have you checked if your TF version was compatible with your CUDA version?
Check the compatibility matrix here: https://www.tensorflow.org/install/source#tested_build_configurations
Unless you compile TF from source, CUDA 11 is not supported yet.
In any case, I would avoid downloading dll from the website you mentioned.
I have Python 3.7 installed (system-wide) along with packages like Numpy, Pandas, pptx, xlsxwriter, and some more. Recently, I learned about virtual environments (yes I am very late to the party), and clearly see the benefits. But I am confused about a few things. Hopefully I can get answers here.
In most of my projects, I use common packages like Numpy, Pandas, Matplotlib, mysql.connector, etc. Should I install these system-wide? Or should I install them in each virtual environment that I create for each project, which, for example, amounts to installing Pandas 10 times in 10 different virtual environments?
If I install packages (system wide) that are not part of the Python Standard Library, for example, pptx and mysql.connector, AND I create a virtual environment, will I have access to these packages from within a virtual environment, or should I install them in the virtual environment as well?
What about a module like jupyter notebook, where it is not part of any project specifically, but I love using it for simple code development and testing (simple plots, etc.). Should this be installed system wide?
I am considering un-installing Python 3.7 and all the packages from my computer, and doing a fresh install of Python 3.8. I want to approach this the "right" way. I would think packages like Numpy and Pandas (I do quite a bit of data manipulation/plotting) should be installed system wide, and each virtual environment should have access to it automatically, and a more specialized package (for me at least) like pptx should be installed in a particular virtual environment for a particular project. Is my assumption correct about this?
Please advise on how to best approach package installation in the context of having virtual environments.
EDIT: based on Ni's reply, I would ask one more question: Are there modules (for example, python-dateutil, that might be used in many projects) and/or circumstances where it makes sense to install a module system-wide?
In general, I never install packages system wide.
You might install packages which require specific versions of Numpy, in your environments. In those cases, if you update the system wide version of Numpy, the package in the environment might break and you won't know that happened.
Yes, you can access them from virtual environment. But in general, don't install packages system wide
Again, I wouldn't install that system wide. For e.g., you might have environments running different python versions, which might not be compatible with the same version of Jupyter
Seems like you're doing a lot of data science work - you might want to use Anaconda to help you manage your virtual environments and package installations
I've previously inferenced TensorFlow graphs from C++. Now I'm embarking on working out how to inference PyTorch graphs via C++.
My first question is, how can I know the recommended version of cuDNN to use with LibTorch, or if I'm doing my own PyTorch compile?
Determining the recommended CUDA version is easy. Upon going to https://pytorch.org/ and choosing the options under Quick Start Locally (PyTorch Build, Your OS, etc.) the site makes it pretty clear that CUDA 10.1 is recommended, but there is no mention of cuDNN version and upon Googling I'm unable to find a definitive answer for this.
From what I understand about PyTorch on ubuntu, if you use the Python version you have to install the CUDA driver (ex. so nvidia-smi works, version 440 currently), but the CUDA and cuDNN install are not actually required beyond the driver because they are included in the pip3 package, is this correct? If so, then is there a command I can run in a Python script that shows the version of CUDA (expected to be 10.1) and cuDNN that the pip pre-compiled .whl uses? I suspect there is such a command but I'm not familiar enough with PyTorch yet to know what that may be or how to look it up.
I've ran into compile and inferencing errors using C++ with TensorFlow when I was not using the specific recommended version of cuDNN for a certain version of TensorFlow and CUDA so I'm aware these version can be sensitive and I have to make the right choices from the get-go. If anybody can assist in determining the recommended version of cuDNN for a certain version of PyTorch that would be great.
CUDA is supported via the graphics card driver, AFAIK there's no separate "CUDA driver". The system graphics card driver pretty much just needs to be new enough to support the CUDA/cudNN versions for the selected PyTorch version. To the best of my knowledge backwards compatibility is included in most drivers. For example a driver that supports CUDA 10.1 (reported via nvidia-smi) will also likely support CUDA 8, 9, 10.0
If you installed with pip or conda then a version of CUDA and cudNN are included with the install. You can query the actual versions being used in python with torch.version.cuda and torch.backends.cudnn.version().
I work on a Python project that in one place callse Julia's code, and in other uses OpenCV.
Unfortunately, pyJulia prefers Python interpreter to be dynamically linked to the libpython. (I know I can build a custom Julia system image, but I fear the build delays when I want to test a development version of my Julia code from Python.)
What has worked so far, is using Spack instead of Conda. Python built by Spack has a shared libpython and Spack's repository does include a recent opencv.
Unfortunately, contrary to Conda, Spack is designed around a paradigm of compiling everything, rather than downloading binaries. The installation time of opencv is well over 1 hour, which barely is acceptable for a one-off install in the development environment, but is dismayingly long to build a Docker image.
So I have a thought: maybe it is possible to integrate my own Python with the rest of the Conda ecosystem?
This isn't a full solution, but Spack does support binary packages, as well as GitLab build pipelines to build them in parallel and keep them updated. What it does not have (yet) is a public binary mirror, so that you could install these things very quickly from pre-existing builds. That's in the works.
So, if you like the Spack approach, you can set up your own binary caches and automated builds for your dev environment.
I am not sure what the solution would be with Conda. You could make your own conda-forge packages, but I think if you deviate from the standard ones, you may end up reimplementing a lot of packages to support your use case. On the other hand, they may accept patches to make your particular configuration work.