Can't install Azure packages with pip: ruamel.yaml error - azure

Question

I'm having trouble installing the following packages in a new python 3.9.7 virtual environment on Arch Linux.
My requirements.txt file:
joblib
python-dotenv
azure-cli==2.29.2
azureml-core
Steps to reproduce:
python3 -m venv venv
source venv/bin/activate
pip install -U pip
pip install -r requirements.txt
Here's the relevant part of the output of the last command, it errors when trying to install ruamel.yaml:
Using legacy 'setup.py install' for ruamel.yaml, since package 'wheel' is not installed.
Installing collected packages: pycparser, urllib3, idna, chardet, cffi, certifi, six, requests, PyJWT, oauthlib, cryptography, requests-oauthlib, python-dateutil, isodate, azure-nspkg, wrapt, tabulate, pyyaml, PySocks, pyopenssl, PyNaCl, pygments, portalocker, msrest, msal, MarkupSafe, jmespath, colorama, bcrypt, azure-mgmt-nspkg, azure-core, argcomplete, applicationinsights, adal, websocket-client, vsts, pyparsing, pyasn1, psutil, pkginfo, pathlib2, paramiko, msrestazure, msal-extensions, knack, jinja2, jeepney, invoke, humanfriendly, deprecated, backports.weakref, azure-mgmt-datalake-nspkg, azure-mgmt-core, azure-common, azure-cli-telemetry, xmltodict, sshtunnel, semver, SecretStorage, scp, ruamel.yaml, pytz, PyGithub, pathspec, packaging, ndg-httpsclient, jsonpickle, jsondiff, javaproperties, fabric, docker, distro, contextlib2, backports.tempfile, azure-synapse-spark, azure-synapse-managedprivateendpoints, azure-synapse-artifacts, azure-synapse-accesscontrol, azure-storage-common, azure-multiapi-storage, azure-mgmt-web, azure-mgmt-trafficmanager, azure-mgmt-synapse, azure-mgmt-storage, azure-mgmt-sqlvirtualmachine, azure-mgmt-sql, azure-mgmt-signalr, azure-mgmt-servicefabricmanagedclusters, azure-mgmt-servicefabric, azure-mgmt-servicebus, azure-mgmt-security, azure-mgmt-search, azure-mgmt-resource, azure-mgmt-reservations, azure-mgmt-relay, azure-mgmt-redis, azure-mgmt-redhatopenshift, azure-mgmt-recoveryservicesbackup, azure-mgmt-recoveryservices, azure-mgmt-rdbms, azure-mgmt-privatedns, azure-mgmt-policyinsights, azure-mgmt-network, azure-mgmt-netapp, azure-mgmt-msi, azure-mgmt-monitor, azure-mgmt-media, azure-mgmt-marketplaceordering, azure-mgmt-maps, azure-mgmt-managementgroups, azure-mgmt-managedservices, azure-mgmt-loganalytics, azure-mgmt-kusto, azure-mgmt-keyvault, azure-mgmt-iothubprovisioningservices, azure-mgmt-iothub, azure-mgmt-iotcentral, azure-mgmt-imagebuilder, azure-mgmt-hdinsight, azure-mgmt-extendedlocation, azure-mgmt-eventhub, azure-mgmt-eventgrid, azure-mgmt-dns, azure-mgmt-devtestlabs, azure-mgmt-deploymentmanager, azure-mgmt-datamigration, azure-mgmt-datalake-store, azure-mgmt-datalake-analytics, azure-mgmt-databoxedge, azure-mgmt-cosmosdb, azure-mgmt-containerservice, azure-mgmt-containerregistry, azure-mgmt-containerinstance, azure-mgmt-consumption, azure-mgmt-compute, azure-mgmt-cognitiveservices, azure-mgmt-cdn, azure-mgmt-botservice, azure-mgmt-billing, azure-mgmt-batchai, azure-mgmt-batch, azure-mgmt-authorization, azure-mgmt-applicationinsights, azure-mgmt-appconfiguration, azure-mgmt-apimanagement, azure-mgmt-advisor, azure-loganalytics, azure-keyvault-keys, azure-keyvault-administration, azure-keyvault, azure-identity, azure-graphrbac, azure-functions-devops-build, azure-datalake-store, azure-cosmos, azure-cli-core, azure-batch, azure-appconfiguration, antlr4-python3-runtime, python-dotenv, joblib, azureml-core, azure-cli
Running setup.py install for ruamel.yaml: started
Running setup.py install for ruamel.yaml: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /home/zjeffer/Documents/school/howest-mct/semester5/mlops/labo/project/mlops-project/venv/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ouy4o2zq/ruamel-yaml_763ae0866f7e4425a0965a00130752b0/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ouy4o2zq/ruamel-yaml_763ae0866f7e4425a0965a00130752b0/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8y_97owt/install-record.txt --single-version-externally-managed --compile --install-headers /home/zjeffer/Documents/school/howest-mct/semester5/mlops/labo/project/mlops-project/venv/include/site/python3.9/ruamel.yaml
cwd: /tmp/pip-install-ouy4o2zq/ruamel-yaml_763ae0866f7e4425a0965a00130752b0/
Complete output (649 lines):
sys.argv ['/tmp/pip-install-ouy4o2zq/ruamel-yaml_763ae0866f7e4425a0965a00130752b0/setup.py', 'install', '--record', '/tmp/pip-record-8y_97owt/install-record.txt', '--single-version-externally-managed', '--compile', '--install-headers', '/home/zjeffer/Documents/school/howest-mct/semester5/mlops/labo/project/mlops-project/venv/include/site/python3.9/ruamel.yaml']
test compiling test_ruamel_yaml
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/ruamel
copying .ruamel/__init__.py -> build/lib.linux-x86_64-3.9/ruamel
creating build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./main.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./util.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./tokens.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./scalarfloat.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./scanner.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./comments.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./loader.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./serializer.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./resolver.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./nodes.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./dumper.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./configobjwalker.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./timestamp.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./error.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./reader.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./cyaml.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./parser.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./__init__.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./representer.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./events.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./scalarbool.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./composer.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./scalarint.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./emitter.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./anchor.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./compat.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./constructor.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./scalarstring.py -> build/lib.linux-x86_64-3.9/ruamel/yaml
copying ./LICENSE -> build/lib.linux-x86_64-3.9/ruamel/yaml
running build_ext
building '_ruamel_yaml' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/ext
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/home/zjeffer/Documents/school/howest-mct/semester5/mlops/labo/project/mlops-project/venv/include -I/usr/include/python3.9 -c ext/_ruamel_yaml.c -o build/temp.linux-x86_64-3.9/ext/_ruamel_yaml.o
In file included from ext/_ruamel_yaml.c:523:
ext/_ruamel_yaml.h:10: warning: "PyString_CheckExact" redefined
<a whole bunch of compiling errors & warnings>
The full log file can be found here
Note the line Using legacy 'setup.py install' for ruamel.yaml, since package 'wheel' is not installed.. If I first install the wheel package, the same error appears when installing the packages.
Installing ruamel.yaml manually works fine, but the azure packages still can't be installed.
EDIT: Installing every package one by one installs everything without errors. Why?
Changing step 3 to pip install -U pip wheel ruamel.yaml doesn't change anything.

Answer 1

The ruamel.yaml documentation states that it should be installed using:
pip install ruamel.yaml
so using the legacy [python] setup.py install should not be used.
Your full log indicates that it is tried to install two versions of ruamel.yaml:
Collecting ruamel.yaml<=0.17.16,>=0.17.10
Using cached ruamel.yaml-0.17.16-py3-none-any.whl (109 kB)
and
Collecting ruamel.yaml<=0.15.89,>=0.15.35
Using cached ruamel.yaml-0.15.89.tar.gz (306 kB)
Apart from a potential problem with two versions being installed, the second one will not succeed as the 0.15 ruamel.yaml series (the last one uploaded to PyPI in July 2019) doesn't support Python 3.9 first released in Oct 2020.
So when you install each package step by step, you get the latest version or ruamel.yaml. Why that step by step installation doesn't at with some package tries to install ruamel.yaml==0.15.89 as a dependency is unclear.
When I install the four packages listed in your requirements.txt, then ruamel.yaml doesn't get installed but pipdeptree indicates that azureml-core==1.36.0.post2 has possible conflicting dependencies.
When I use your requirements.txt I get the same error.
When I use your requirements.txt in a Python 3.7 venv (so that the old ruamel.yaml wheel can be installed) installation works. From that you can see that the package azureml-core==1.14.0 is installed instead of the azureml-core==1.36.0.post2