Python: testing & loading different git commits of same package - python-3.x

Question

I have a python package on github, and I can install different commit versions of it using e.g. pip3 install git+https://github.com/my/package#commithash. I would like to benchmark various different commits against each other, ideally comparing two versions within the same python script, so that I can plot metrics from different versions against each other. To me, the most obvious way to do this would be to install multiple different versions of the same package simultaneously, and access them using a syntax something like
import mypackage_commithash1 as p1
import mypackage_commithash2 as p2
results1 = p1.do_something()
results2 = p2.do_something()
plot_comparison(results1, results2)
But as far as I can see, python doesn't support multiple packages of the same name like this, although https://pypi.org/project/pip3-multiple-versions goes some of the way. Does anyone have any suggestions for ways to go about doing these sorts of comparison within a python script?

Answer 1

That's too broad of a question to give a clear answer...
Having two versions of the same project running in the same environment, same interpreter session is difficult, near impossible.
First, maybe have a look at this potentially related question:
Versions management of python packages
1. From reading your question, another solution that comes to mind would be to install the 2 versions of the project in 2 different virtual environments. Then in a 3rd virtual environment I would run code that looks like this (kind of untested pseudo-code, some tweaking will be required):
environments = [
'path/to/env1',
'path/to/env2',
]
results = []
for environment in environments:
output = subprocess.check_output(
[
environment + 'bin/python',
'-c',
'import package; print(package.do_something())',
],
)
results.append(parse_output(output))
plot_comparison(results)
2. Another approach, would be to eventually use tox to run the test program in different environments containing each a different version of the project. Then have an extra environment to run the code that would interpret and compare the results (maybe written on the filesystem?).
3. Maybe one could try to hack something together with importlib. Install the 2 versions under 2 different paths (pip install --target ...). Then in the test code, something like that:
modify sys.path to include the path containing version 1
import (maybe importlib can help)
run test 1
modify sys.path to remove the path containing version 1 and include the path containing version 2
import again (maybe importlib.reload is necessary)
run test 2
compare results