I am trying to install a specific version of pytorch that is compatible with a specific cuda driver version with pipenv. The pytorch website shows how to to this with pip:
pip3 install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html
I tried to convert this into an entry in my Pipfile like this:
[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/torch_stable.html"
verify_ssl = false
pytorch = {version="==1.3.1+cu92", index="pytorch"}
torchvision = {version="==0.4.2+cu92", index="pytorch"}
However, this does not work. The dependency with this version can not be resolved. I am not sure if the url that is listed with the -f parameter in the pip3 command is even a valid source for pipenv.
I could install both libraries by just passing the command through to pip like this:
pipenv run pip install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html
but I am not really satisfied with that solution since the dependencies are not in the Pipfile and I have to manually document the usage of this command.
The problem with the approach above lies in the structure of https://download.pytorch.org/whl/torch_stable.html. Pipenv can only find torch versions 0.1 to 0.4.1 because all others have the cuda (or cpu) version as a prefix e.g. cu92/torch-0.4.1-cp27-cp27m-linux_x86_64.whl.
But the cuda version is a subdirectory. So if you change the url of the source to the cuda version and only specify the torch version in the dependencies it works.
[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu92"
verify_ssl = false
[packages]
torch = {index = "pytorch",version = "==1.4.0"}
The only problem I encountered is that numpy is not recognized as a dependency of pytoch 1.4.0. But this seems to be a problem of the specific pytorch wheel. With version 1.3.1 or 1.5.1 and a recent pipenv version it works.
So if after the installation with pipenv install, the command pipenv run python -c "import torch" throws an error, numpy must be added manually.
Solved: This is how to install Torch CORRECTLY in Pipenv. Most people do this incorrectly even in various "solutions" I've seen from other people... I've researched to the bottom of this problem and found out the real, intended solution that everyone should use.
This technique is the correct one and it's documented here in the pipenv repository:
https://github.com/pypa/pipenv/issues/4961#issuecomment-1045679643
I've just run into this issue right now. Take whichever URL the Torch website tells you to use and do this:
pipenv install torch --index https://download.pytorch.org/whl/cu113
Of course, replace https://download.pytorch.org/whl/cu113 with the URL indicated for you by Torch. This updates the Pipfile correctly and worked in my system. Installed pytorch and enabled correct CUDA support.
Related
I installed the transformers in the Macbook Pro M1 Max
Following this, I installed the tokenizers with
pip install tokenizers
It showed
Collecting tokenizers
Using cached tokenizers-0.12.1-cp39-cp39-macosx_12_0_arm64.whl
Successfully installed tokenizers-0.12.1
It seems to use the correct architecture for the whl file
When I import it I get
'/Users/myname/miniforge3/envs/tf/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
I see that this problem used to happen to others before. Any thoughts on how to fix this?
James Briggs method works but produces the following error
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
The Issue
After installing Rust and Cargo, we must source the environment file. This is the missing step in the previous answer.
The Solution
The workaround to solving this is to type the following in the terminal, right after installing Rust:
source "$HOME/.cargo/env"
Then, you can install transformers with the following code snippet:
pip install transformers
If using Anaconda we switch to a terminal window and create a new ARM environment like so:
CONDA_SUBDIR=osx-arm64 conda create -n ml python=3.9 -c conda-forge
now get in to ml envoriment
conda activate ml
run inside the env
conda env config vars set CONDA_SUBDIR=osx-arm64
needs to restart env
conda deactivate
get into to env
conda activate ml
PyTorch Installation
To get started we need to install PyTorch v1.12. For now, this is only available as a nightly release.
pip3 install -U --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Side note: The transformers library uses tokenizers built in Rust (it makes them faster). Because we are using this new ARM64 environment we may get ERROR: Failed building wheel for tokenizers. If so, we install Rust (in the same environment) with:
curl — proto ‘=https’ — tlsv1.2 -sSf https://sh.rustup.rs | sh
restart your env
conda deactivate
conda activate ml
than you can install transformer comes with tokenizers or only install tokenizers
pip install tokenizers or pip install transformer
thanks to James Briggs
You can try
conda install -c huggingface transformers
I got this error too. Solved it after a lot of trial & error.
The Problem: my brew was still running on Rosetta. Fixed that by uninstalling, cleaning and reinstalling. So everything seemed to run fine. Except this problem still kept cropping up
Until I discovered that pip is quite agressive in caching. So it caches the build even if the architecture changed. Solution: pip cache purge. Or remove the whole cache directory which you find with pip cache info
After testing most of the solutions provided I finally got it working by doing
brew install ffmpeg
sudo pip install tokenizers
🛠️🚀
I have a problem with importing TensorFlow. I have tried multiple versions of Numpy, Python, and TensorFlow and I still get the following error:
struct_pb2.TypeSpecProto.NDARRAY_SPEC
AttributeError: NDARRAY_SPEC
I have tried using conda and pip for installation and neither one works. I have no idea what might be the cause of this problem and it started happening about a week ago before that TensorFlow was working fine!
I believe you are using windows, and you have an incompatible version of tensorflow installed or you are missing a dependency. First make sure you have the following installed correct version of Visual C++ installed for your version of windows.
https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0
https://aka.ms/vs/16/release/vc_redist.x64.exe here is the direct link.
If it still doesn't work, enable longpaths,
https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing
If you are having a clash with other packages, create a new conda environment first if you haven't already, and install tensorflow like this.
conda create -n tfenv
conda activate tfenv
conda install tensorflow
Then try to import tensorflow as tf again.
I am running anaconda on windows; When i try to install pytorch in the anaconda prompt I get the following error
ModuleNotFoundError: No module named 'torchvision'
i checked the openssl version via conda list and it is 1.1.g3
I also moved the lib crypto and libssl from the anaconda/library/bin to anaconda3/DLLs (as per one the prior solutions.
What could be the issue? Please help
Use the commands given in the official pytorch website. Or you can use pip to install too. Commands for both conda and pip are given in pytorch website.
When installing Theano anaconda automatically tries to install pygpu despite this being an optional dependency. I have deleted the .theanorc file from my windows user directory.
Also when running my application Theano tries to load from the GPU. It's like it remembers somehow?
conda install theano
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment
C:\Users\zebco\Miniconda3\envs\py35:
The following NEW packages will be INSTALLED:
libgpuarray: 0.6.9-vc14_0
pygpu: 0.6.9-py36_0
theano: 0.9.0-py36_0
Proceed ([y]/n)?
As you can see I've only specified to install theano yet conda wants to install everything including optional dependancies.
Your assumption that pygpu is optional is dependent on the package manager you are using.
Regular Python (pip)
If you are using a direct Python install (obtained using brew or Python site) then you would be using pip to install theano. This basically comes from
https://pypi.python.org/pypi/Theano/1.0.0
If you download the file and unzip it. Open setup.py, you will see below lines
install_requires=['numpy>=1.9.1', 'scipy>=0.14', 'six>=1.9.0'],
So they are set as the dependencies for this package. Which means when you install theano you will also get numpy, scipy and six.
Anaconda Python (conda)
Now coming to Anaconda python. Anaconda doesn't use a package format that PyPI or pip uses. It uses its own format. In case of Anaconda you should be using conda to install the packages you need and not pip.
Conda has channels which is nothing but a repository which has some packages available. You can install a package from any channel using below
conda install -c <channel-name> <package-name>
The default channel is conda-forge. If you look at the theano package over there
https://anaconda.org/conda-forge/theano/files
And download and extract it. There will be a info/recipe/meta.yml file. You will notice below content in the same
requirements:
build:
- ca-certificates 2017.7.27.1 0
- certifi 2017.7.27.1 py36_0
- ncurses 5.9 10
- openssl 1.0.2l 0
- python 3.6.2 0
- readline 6.2 0
- setuptools 36.3.0 py36_0
- sqlite 3.13.0 1
- tk 8.5.19 2
- xz 5.2.3 0
- zlib 1.2.11 0
run:
- python
- setuptools
- six >=1.9.0
- numpy >=1.9.1
- scipy >=0.14
- pygpu >=0.6.5,<0.7
Which specifies that if you want to run this package then pygpu is also on of its dependencies. So conda downloads pygpu as a dependency which you though was optional (which is probably true if you were using regular python and pip)
Update:
Usually, 'Optional Dependency' is an oxymoron. Something optional is not a dependency, a dependency is a software package another piece of software depends on to function for features.
One may get by without a dependency if the dependency does not interact with the package except for one atomized feature which is not being used. As a beginner I would suggest you not take this path.
I am not super familiar with Theano, but Theano can use the system's GPU to speed up its computations, and it seems to me pygpu and gpulibarray are what enable this functionality. Which means it is not optional.
I believe pygpu is 'optional' if you do not wish to use the GPU for speeding up computation (only done if the GPU is powerful enough to be useful for this).
The --no-deps command above allows you to install a package without its dependencies but that is rarely wise, unless one really knows what they are doing. As a beginner I would not recommend you go down this path yet. Conda was designed specifically to ensure scientific packages are easily managed with all necessary stuff installed without any fuss or muss. pip is a general python package manager, but is not built specifically for scientific packages.
If you wish to install theano without installing its dependencies, then you have one of three options:
use conda install theano --no-deps.
Install it using pip instead of conda, using pip install theano. This will install theano, numpy, scipy and six but not pygpu and libgpuarray.
Create a custom conda build file for Theano. Documentation is at:
https://conda.io/docs/user-guide/tasks/build-packages/index.html
Original Answer:
You probably know this already but, use this command instead:
conda install theano --no-deps
This does not install dependencies of the package. If you already have the essential dependencies installed, as it would seem, this should work out for you.
libgpuarray is a dependency of pygpu. With this command switch neither will be installed.
Can you share the .yaml file that you edited?
I'm trying to install Blocks on my Linux machine and I have some troubles. How can I fix this? I'm doing the following steps:
Installing Anaconda for python2.
Theano depends on numpy 1.10.1 version. If at this step I will run import theano in python shell then it will work perfectly.
Okay, it's time for Blocks. I'm installing stable version using pip install --user git+git://github.com/mila-udem/blocks.git \
-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt
After this step if I import theano it gives the following: RuntimeError: module compiled against API version a but this version of numpy is 9.
I looked at requirements.txt and see that Blocks depends on numpy 1.9.3.
I uninstalled Theano, Blocks, downgraded numpy using conda install numpy=1.9.3, then just run again pip install --user git+git://github.com/mila-udem/blocks.git \
-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt and still while importing theano it gives me RuntimeError: module compiled against API version a but this version of numpy is 9.
How can I overcome this problem?
This could be a problem with the Anaconda distribution. You could try updating all of Anaconda via conda update conda and conda update --all. You could also try changing Blocks requirements.txt to refer to numpy 1.10.1 since it's likely that Blocks won't care about the changes from 1.9 to 1.10.
– Daniel Renshaw