If you develop a ton of python applications and you need to test under a lot of different versions, and by a lot I mean overlapping major/minor versions (like 3.5.3 and 3.5.4), then a good option is to use pyenv. Along with tox, you can easily test your application against various major/minor versions.
Here is how to do it.
Start by installing pyenv. The instructions provided here are pretty clear to follow.
Once it is installed, you can install (in pyenv) all the python versions you need by running pyenv install <major.minor.patch>. So, for example
%> pyenv install 3.6.2
If you are in a hurry, it seems like you can run the command in parallel in various terminals without issues. (I’ve done it and did not have any problem.)
Then, go to your project. To define the specific versions you want to test with, run pyenv local <major.minor.patch>. So, for example
%> pyenv local 2.7.13 3.5.4 3.6.2
This will set the available python versions to those you specified whenever you are in that directory.
Disclaimer: Although pyenv will let you do it, as far as I know, setting the same major.minor with different patch is useless because you won’t be able to tell them apart. Eg.: If you run pyenv local 3.5.3 3.5.4 there doesn’t seem to be a way to run 3.5.4 with that configuration.
Then you can install tox, using pip or apt install. Here we install version 2.7.0 with pip.
%> pip install tox==2.7.0
Then, create your tox.ini file. Here is a sample file.
[tox]
envlist = py27, py35, py36
[testenv]
commands =
python -V
pip install -e requirements.txt
python test_runner.py
In the [tox] block, we have the 3 python versions to test. In the [testenv] block, we have the actual sequence of test commands. The first one, python -V, simply outputs the python version being run. Then, we install the dependencies with pip and we finally execute the tests.
To run tox, assuming you installed it with pip, the command is
%> python -m tox
That’s it! It should now test your app with the 3 configured versions of python.
Beware of bad configurations
A huge trap I have found with tox is that it is very very easy to not test the python versions properly. That’s why I run python -V before my tests. The first time you run tox over a project, you should seek this output out to confirm it is really testing the version it should be.
An easy way to test the wrong thing is to set the patch version in the envlist. So, if instead of having
[tox] envlist = py27, py35, py36
I have
[tox] envlist = py2713, py354, py362
Although it seems legit, it actually tests 3 times with python 2.7.13 and there is no warning, no error, no bell, sweet nothing. And it somewhat refers to the disclaimer above about configuring pyenv with the same minor.major but different patch, there is no way to tell them apart.
So again, make sure it really tests what you want it to.
Bonus
Since we’re at it, if you want to follow conventions, you likely want to run flake8 too. Here is an example file that does it.
[tox]
envlist = flake8, py2713, py354, py362
[testenv]
commands =
python -V
pip install -r requirements.txt
coverage run --parallel test_runner.py
[testenv:flake8]
deps=flake8
commands=flake8
Another trap to avoid, is to run flake8 within the [testenv] block. It will still work, but it will
run flake8 for each python version you define in your envlist, so 3 times in that example, rather than just once.
Comments? Improvements? Mistakes? Use the comments zone below!