I installed the transformers in the Macbook Pro M1 Max
Following this, I installed the tokenizers with
pip install tokenizers
It showed
Collecting tokenizers
Using cached tokenizers-0.12.1-cp39-cp39-macosx_12_0_arm64.whl
Successfully installed tokenizers-0.12.1
It seems to use the correct architecture for the whl file
When I import it I get
'/Users/myname/miniforge3/envs/tf/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
I see that this problem used to happen to others before. Any thoughts on how to fix this?
James Briggs method works but produces the following error
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
The Issue
After installing Rust and Cargo, we must source the environment file. This is the missing step in the previous answer.
The Solution
The workaround to solving this is to type the following in the terminal, right after installing Rust:
source "$HOME/.cargo/env"
Then, you can install transformers with the following code snippet:
pip install transformers
If using Anaconda we switch to a terminal window and create a new ARM environment like so:
CONDA_SUBDIR=osx-arm64 conda create -n ml python=3.9 -c conda-forge
now get in to ml envoriment
conda activate ml
run inside the env
conda env config vars set CONDA_SUBDIR=osx-arm64
needs to restart env
conda deactivate
get into to env
conda activate ml
PyTorch Installation
To get started we need to install PyTorch v1.12. For now, this is only available as a nightly release.
pip3 install -U --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Side note: The transformers library uses tokenizers built in Rust (it makes them faster). Because we are using this new ARM64 environment we may get ERROR: Failed building wheel for tokenizers. If so, we install Rust (in the same environment) with:
curl — proto ‘=https’ — tlsv1.2 -sSf https://sh.rustup.rs | sh
restart your env
conda deactivate
conda activate ml
than you can install transformer comes with tokenizers or only install tokenizers
pip install tokenizers or pip install transformer
thanks to James Briggs
You can try
conda install -c huggingface transformers
I got this error too. Solved it after a lot of trial & error.
The Problem: my brew was still running on Rosetta. Fixed that by uninstalling, cleaning and reinstalling. So everything seemed to run fine. Except this problem still kept cropping up
Until I discovered that pip is quite agressive in caching. So it caches the build even if the architecture changed. Solution: pip cache purge. Or remove the whole cache directory which you find with pip cache info
After testing most of the solutions provided I finally got it working by doing
brew install ffmpeg
sudo pip install tokenizers
🛠️🚀
Related
I need sklearn that's why I tried installing scikit-learn in my conda env using conda install scikit-learn but it results in:
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- scikit-learn -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0|>=3.5,<3.6.0a0']
Your python: python=3.9
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The following specifications were found to be incompatible with your system:
- feature:/linux-64::__cuda==9.1=0
- feature:|#/linux-64::__cuda==9.1=0
Your installed version is: 9.1
The same appears if I try conda install -c anaconda scikit-learn or conda install -c conda-forge scikit-learn. How to resolve this?
Ok, so the problem was that my env was running Python 3.9.0 but scikit-learn isn't compatible with 3.9 so I created a new env while specifically mentioning the Python version by
conda create -n newEnv python=3.7.3
python 3.7.3 was running in my base environment so I chose that.
I have just encountered this problem, and my OS is Windows10. At first I tried to uninstall scikit-learn package and reinstall it, but it failed. Then I searched this website which focuses on python package using wheel. Luckily scikit-learn compatible with python3.9.0 now is available. After downloading the proper wheel file, I cd into the download folder and use 'pip install <wheel file name>' to install it. It worked for my situation. Wishes for you.
I am running anaconda on windows; When i try to install pytorch in the anaconda prompt I get the following error
ModuleNotFoundError: No module named 'torchvision'
i checked the openssl version via conda list and it is 1.1.g3
I also moved the lib crypto and libssl from the anaconda/library/bin to anaconda3/DLLs (as per one the prior solutions.
What could be the issue? Please help
Use the commands given in the official pytorch website. Or you can use pip to install too. Commands for both conda and pip are given in pytorch website.
I'm currently trying to create a TensorFlow estimator to run Ludwig's training model on Azure ML Compute with various pip and conda packages like so:
estimator= TensorFlow(source_directory= project_folder,
compute_target=compute_target, script_params=script_params,
entry_script='./train.py', pip_packages=dependencies, conda_packages =
["tensorflow"], use_gpu =True)
One of the pip packages is gmpy, but it will not install and throws an error: fatal error: gmp.h: No such file or directory compilation terminated. error: command 'gcc' failed with exit status 1.
This prevents Ludwig from installing and causes the imagine to fail to build
When I run Ludwig locally in a python virtual environment on Ubuntu, I'm able to work around this issue by running “sudo apt-get install libgmp3-dev” instead of pip install gmpy. When I try adding Gmpy2 as a library to the estimator, it throws the same error, and it seems that libgmp3-dev doesn't have a pip or conda equivalent. I tried adding the gmpy and gmpy2 .whl files directly to the environment but the wheel files were not recognized as compatible.
Is there some way to add RUN sudo apt-get install libgmp3-dev to the dockerfile so that the docker container made by the estimator has this already installed without needing to create a custom dockerfile? I noticed that the TensorFlow estimator class has an "environment_definition" flag that can take a DockerSection but I can't find any examples of how they work.
Looks like they have gmpy2 on conda-forge channel
https://anaconda.org/conda-forge/gmpy2
you can also should be able to reference dependency from # git
I want to use Assimulo and Sundials for the solution of differential algebraic equations in Python and therefore I am trying to install it on Ubuntu.
For the installation of Sundials, I followed the installation instructions and as I understand it worked well.
% cmake -DCMAKE_INSTALL_PREFIX=/usr/local/lib/sundials-3.1.1/ ~/opt/sundials/sundials-3.1.1
% make
% make install
Then I tried to install Assimulo with the command pip3 install Assimulo, but I get an error message. I also tried to follow the instructions on Installation - Assimulo 3.0 documentation by downloading the installation files and install it with the following command. It results in the same error message.
sudo python3 setup.py install --sundials-home=/usr/local/lib/sundials-3.1.1
This is the error message I get:
target build/src.linux-x86_64-3.6/assimulo/thirdparty/hairer/dopri5module.c does not exist:
Assuming dopri5module.c was generated with "build_src --inplace" command.
error: 'assimulo/thirdparty/hairer/dopri5module.c' missing
What is wrong and how can I fix it? Any help would be appreciated!
I got the same error when installing on macos via pip install assimulo, after pip-installing numpy and cython.
For me, using a conda env did the trick:
Creating the conda env: conda create -n your_name_goes_here
conda activate your_name_goes_here
conda install python=3.6 (I noticed you can also use 3.7)
conda install -c conda-forge assimulo
I also had the same error message. As suggested in the other answer, you can get a compiled package from Conda. But if you want to compile from source yourself, it looks to me that PyPI source tarball doesn't contain all needed files. At least some *.pyf files are missing. So, I used SVN repo instead:
svn checkout https://svn.jmodelica.org/assimulo/tags/Assimulo-3.0/ assimulo
By compiling this source tree, I managed to get pass the original error you had, but I'm now having another build error that I don't know yet how to solve:
ssimulo/solvers/sundials.c: In function '__pyx_f_8assimulo_7solvers_8sundials_5CVode_initialize_cvode':
assimulo/solvers/sundials.c:33274:31: error: too many arguments to function 'CVodeCreate'
__pyx_v_self->cvode_mem = CVodeCreate(__pyx_t_3, __pyx_t_4);
I am a little bit confused....
I installed anaconda on my computer (I have windows 10).
Normally, when I want to install a package I simply do "pip install package_name" or "conda install package_name" and it is done.
First question: what is the difference between pip and conda?
Now I tried to install xgboost and it was really complicated I tried lot of things nothings worked until I install something called miniconda.
There it works but now, when I do "conda install package_name" it install it in miniconda3/lib/site _package and I have to copy/paste it in Anaconda3/lib/site_package if I want it to work.
Second question: how can I ask to the computer that "conda install
package_name" install it directly in anaconda3 and not miniconda3?
Finally I tried to install the package "surprise" for recommended systems. Both "pip install" or "conda install" failed.
I went in github and got the file "surprise" from https://github.com/NicolasHug/Surprise
I tried to copy it in Anaconda3/lib/site_package but it doesn't work.
When I do from surprise import Reader I did not get the error "no module name surprise" anymore but I get "cannot import name 'Reader'"
Last question: how can I make it work? I think I have to build it but
I do not now how...
Thank you in advance for anyone that can explain all this for me :-)
Similarly to you, I had issues installing the surprise package.
I tried both pip install surprise and conda install surprise unsuccessfully.
conda install -c conda-forge scikit-surprise
conda install -c conda-forge/label/gcc7 scikit-surprise
conda install -c conda-forge/label/cf201901 scikit-surprise
I found those on the anconda website and the first one worked for me.
Hopefully this would help you as well
pip vs conda
pip is a package manager that facilitates installation, upgrade,
and uninstallation of python packages. It also works with virtual python environments.
conda is a package manager for any software (installation, upgrade and uninstallation).
It also works with virtual system environments.
Conda is a packaging tool and installr that aims to do more than what pip does;
handle library dependencies outside of the Python packages as well as the Python packages themselves.
Conda also creates a virtual environment, like virtualenv does.
For more see here
Anaconda vs miniconda
The open source version of Anaconda is an easy-to-install
high performance Python and R distribution with a package manager,
environment manager and collection of 720+ open source packages.
It also comes with the options to install RStudio.
The "lite" version of Anaconda without the collection of 720 packages.
The downside is that you need to type in command line commands,
"conda install PACKAGENAME"
And Last
To install this package with conda run:
conda install -c anaconda py-xgboost=0.60
Update for surprise
The easiest way is to use pip (you'll need numpy):
$ pip install numpy
$ pip install scikit-surprise
Or you can clone the repo and build the source (you'll need Cython and numpy):
$ git clone https://github.com/NicolasHug/surprise.git
$ python setup.py install