How to install conda packages in the aarch64 from x86-64 architecture - linux

Question

May I know how can I get the installed packages from a particular environment from x86-64 architecture linux and
how can I create a new conda environment in aarch64 architecture using the same package?
First in the x86-64 architecture linux machine called L2, I export the package
conda list --export > envconda.txt
When I open the envconda.txt, it is
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
_r-mutex=1.0.0=anacondar_1
I changed the platform : linux-64 to linux-aarch64 because I am going to install the packages in the aarch64 architecture.
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-aarch64
_libgcc_mutex=0.1=main
_r-mutex=1.0.0=anacondar_1
In aarch64 linux machine called L1, I create a conda environment
conda create -n envtest --file envconda.txt
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- setuptools==36.4.0=py36_1
- kiwisolver==1.1.0=pypi_0
- pyyaml==3.13=pypi_0
- jedi==0.10.2=py36_2
- libgcc==5.2.0=0
- jsonschema==2.6.0=py36_0
- ptyprocess==0.5.2=py36_0
- prompt_toolkit==1.0.15=py36_0
- libstdcxx-ng==9.1.0=hdf63c60_0
- tqdm==4.36.1=pypi_0
- tomli==1.2.3=pypi_0
- astor==0.7.1=pypi_0
- argparse==1.4.0=pypi_0
- pycparser==2.19=pypi_0
- testpath==0.3.1=py36_0
- cudnn==7.6.5=cuda10.2_0
- asn1crypto==0.22.0=py36_0
- dataclasses==0.8=pypi_0
- platformdirs==2.4.0=pypi_0
- krbcontext==0.10=pypi_07
- decorator==4.1.2=py36_0
- lazy-object-proxy==1.7.1=pypi_0
- gsl==2.2.1=0
- pexpect==4.2.1=py36_0
- icu==54.1=0
- freetype==2.5.5=2
- bleach==1.5.0=py36_0
- matplotlib==3.1.1=pypi_0
- wheel==0.29.0=py36_0
- cudatoolkit==10.2.89=hfd86e86_1
- glib==2.50.2=1
- kneed==0.7.0=pypi_0
- sqlite==3.13.0=0
- importlib-metadata==1.7.0=pypi_0
- python==3.6.2=0
- jpeg==9b=0
- pango==1.40.3=1
- fontconfig==2.12.1=3
- resampy==0.2.2=pypi_0
- nbformat==4.4.0=py36_0
- pixman==0.34.0=0
- scikit-learn==0.21.3=pypi_0
- termcolor==1.1.0=pypi_0
- typed-ast==1.5.4=pypi_0
- keras-applications==1.0.8=pypi_0
- harfbuzz==0.9.39=2
- libffi==3.2.1=1
- jupyter_client==5.1.0=py36_0
- gssapi==1.6.9=pypi_0
- curl==7.54.1=0
- keras==2.2.4=pypi_0
- isort==5.10.1=pypi_0
- simplegeneric==0.8.1=py36_1
- joblib==0.14.0=pypi_0
- pypandoc==1.6.3=pypi_0
- python-dateutil==2.8.2=pypi_0
- ipython_genutils==0.2.0=py36_0
- pyparsing==2.4.2=pypi_0
- ca-certificates==2022.6.15=ha878542_0
- krb5==1.13.2=0
- path.py==10.3.1=py36_0
- markdown==3.0.1=pypi_0
- requests-kerberos==0.12.0=pypi_0
- hdfs==2.5.8=pypi_0
- traitlets==4.3.2=py36_0
- tornado==4.5.2=py36_0
- librosa==0.7.0=pypi_0
- pyasn1==0.4.8=pypi_0
- blas==1.0=mkl
- zlib==1.2.11=0
- libogg==1.3.2=h14c3975_1001
- mkl==2017.0.3=0
- terminado==0.6=py36_0
- libflac==1.3.1=hf484d3e_1002
- python-levenshtein==0.12.2=pypi_0
- werkzeug==0.14.1=pypi_0
- pyspark==2.3.2=pypi_0
- urllib3==1.26.9=pypi_0
- bzip2==1.0.6=3
- html5lib==0.9999999=py36_0
- pywavelets==1.1.1=pypi_0
- zeromq==4.1.5=0
- pykerberos==1.2.1=pypi_0
Current channels:
- https://repo.anaconda.com/pkgs/main/linux-aarch64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-aarch64
- https://repo.anaconda.com/pkgs/r/noarch
- https://conda.anaconda.org/conda-forge/linux-aarch64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
May I know how can I install the packages successfully in the aarch64 architecture?
Last but not least, when I install package using pip install numpy, I got this error Illegal instruction (core dumped)
For this issue, may I know how can I solve this also in linux aarch64 architecture?

Answer 1

Very unlikely this will work for multiple reasons:
Package support off the major platforms (osx-64, linux-64, win-64) is sparse, especially further back in time. A concrete example is cudatoolkit, which only has linux-aarch64 builds starting with version 11.
Overgeneralized environment. The more packages included in an environment, the more difficult it becomes to solve it, and solving across platforms aggravates this problem. I would, for example, remove any Jupyter-related packages completely. In the future, try to plan ahead to have dedicated environments to specific projects, and only install the packages that are absolutely required.
Some packages are completely incompatible. For example mkl is architecture specific.
Nevertheless, if you want to attempt recreating an approximation of the environment, there are some options. First, one cannot achieve this with conda list --export - that simply does not handle environments that have packages from PyPI installed.
PyPI-centric Approach
Because so much of the environment is from PyPI, my first inclination is to recommend abandoning the Conda components and going a pip route. That is, use
pip list --format=freeze > requirements.txt
to capture the Python packages, then create a new environment with something like:
environment.yaml
name: foo
channels:
- conda-forge
- nodefaults
dependencies:
- python=3.6
- pip
- pip:
- -r requirements.txt
With both requirements.txt and environment.yaml in the same folder, the environment is created with
## "foo" is arbitrary - pick something descriptive
conda env create -n foo -f environment.yaml
Retaining some Conda packages
You could also try keeping some parts from Conda by mixing together a conda env export and the previous pip list. Specifically, export a minimal environment definition, with
conda env export --from-history > environment.yaml
Edit this file to include a specific version of Python, remove any packages that are not available for linux-aarch64 (like mkl), and add the pip: section, as above:
environment.yaml
#...
dependencies:
- python=3.6
# ...
- pip
- pip:
- -r requirements.txt
This is then used with:
conda env create -n foo -f environment.yaml
Expect to iterate several times to discover what cannot be found for the platform. I would strongly recommend using mamba instead of Conda in order to minimize this solving time.