Status Of Python Development

Salutations, people from the Interweb!

Pop question. How many package management tools are available for Python development?

Ignoring system package managers like APT, DNF/YUM and similar I can name, without a particular order: egg, easy_install, setuptools, wheels, pip, venv, Virtualenv, pyenv, poetry, pipenv, pants, Anaconda, conda and miniconda. Did I forgot something?

To be honest, not all are for managing libraries and some overlap in their functionalities, also I'm exaggerating for comedic purposes. Nonetheless what they don't achieve is keeping things simple, at least in my opinion.

I would love to work only with Python 3.x and call it a day, unfortunately after more than ten years from its introduction there are still tools not yet ported from version "2.7" - pwntools, I'm looking at you - and the dynamic nature of the language tend to encourage rapid iterations of libraries that could became problematic when packaging and deploying a project. So we have two interpreters different enough to be incompatible and libraries that are difficult to distribute by third parties.

What Workflow?

Right now, other than pip, I'm using a combination of pyenv and Virtualenvwrapper, in this way I can potentially manage more than a couple of Python interpreters¹ and I can use an heterogeneous collection of Python modules that could be installed independently by different projects. About the last part, yes, if two different virtual environments use the same library, it's duplicated on the disk and no, I haven't verified if there's a way to share the common code.

I start by installing pyenv in my homedir via curl plus its pyenv-register and pyenv-virtualenvwrapper plugins, the first one is for registering the system-wide Python interpreters and the second for configuring virtual environments:

$ curl https://pyenv.run | sh
  [...]
  a few line of output
  [...]
$ git clone https://github.com/doloopwhile/pyenv-register.git \
  ~/.pyenv/plugins/pyenv-register
$ git clone https://github.com/pyenv/pyenv-virtualenvwrapper.git \
  ~/.pyenv/plugins/pyenv-virtualenvwrapper

followed by adding the relevant initialization in my .zshenv:

# add pyenv to PATH
if [[ -d "$HOME/.pyenv" ]]; then
    export PATH="$HOME/.pyenv/bin:$PATH"
    eval "$(pyenv init -)"
    export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"
    #eval "$(pyenv virtualenv-init -)"
    pyenv virtualenvwrapper_lazy
fi

and .zshrc:

# virtualenvwrapper
export WORKON_HOME=~/.virtualenvs
#export PROJECT_HOME=~/Devel

# pyenv + virtualenvwrapper
export VIRTUAL_ENV_DISABLE_PROMPT=yes # virtualenv indicator disabled cause moved to RPROMPT above

The observant reader should have noted above that the pyenv-virtualenv plugin was explicitly disabled, that's because I've found out it interferes with pyenv-virtualenvwrapper. Also, the default Virtualenv prompt indicator is disabled because I like to have it on the Zsh's "right prompt". And yes, installing from a git repository appears indeed fragile, but looking at the number of commits of the last few years it seems quite stable. To be honest it looks a big hack, behind the scenes it mangles the command $PATH variable in an ugly way and relies to a collection of shell stubs, so the above should be interpreted more as "not changing a lot stable"

Oh the horrors!

Anyway, the next step is to register the available system Python interpreters. In my case, at the time of writing, I have both the 2.7.13 and 3.5.3 versions installed²:

$ pyenv versions
* system (set by ~/.pyenv/version)
$ pyenv register /usr/bin/python2.7
  [...]
  bunch of output
  [...]
$ pyenv register /usr/bin/python3.5
  [...]
  similar to the output above
  [...]
$ pyenv versions
* system (set by /home/dnc/.pyenv/version)
  system-2.7.13
  system-3.5.3

Now it's possible to configure the Python interpreter system-wide with the pyenv global command or temporarily on a single terminal instance with the pyenv shell command.

Splendid.

Real life example

Below follows an example of creating a virtual environmet for Pelican, a Python static site generator that the loyal reader has seen mentioned on this pages before. What follows was verified on a Debian Stretch install, the Debian stable version at the time of writing, I haven't tested it on anything else.

After installing pyenv, if the shell isn't reopened, it's required to initialize the Virtualenvwapper plugin³, on the first run it will download and unpack the required Python packages like setuptools, pip and wheel:

$ pyenv shell system-3.5.3
$ pyenv versions
  system
  system-2.7.13
* system-3.5.3 (set by PYENV_VERSION environment variable)
$ pyenv virtualenvwrapper
  [...]
  output for virtualenv initialisation
  [...]

I should mention that forgotten remnants of previously installed Python scripts or libraries in your commands path like ~/.local/bin or ~/.local/lib can cause weird interactions or strange errors, as I've lost an afternoon chasing what I was doing wrong replicating the above steps on a different system.

Anywho the last step consist in creating/naming the virtual environment and optionally associating a project directory to it:

$ mkdir ~/scsm
$ mkvirtualenv -a ~/scms pelican
$ pip install 'pelican==3.6.3' livereload Markdown \
Pillow Pygments
  [...]
  more output
  [...]
$ pip list
Package         Version
--------------- -------
blinker         1.4
docutils        0.14
feedgenerator   1.9
Jinja2          2.10
livereload      2.6.0
Markdown        3.0.1
MarkupSafe      1.1.0
pelican         3.6.3
Pillow          5.4.1
pip             19.0.1
Pygments        2.3.1
python-dateutil 2.7.5
pytz            2018.9
setuptools      40.6.3
six             1.12.0
tornado         5.1.1
Unidecode       1.0.23
wheel           0.32.3

Now I have Pelican ready to run on an isolated Python environment.

Awesome!

Colophon

The inquisitive reader would wonder why it was specified an exact version of the Pelican package. The reason is about a regression in the footnotes generator⁴ that I think was introduced in version 3.7 and I'm guilty to not have reported it yet. Last time I checked it was still present. In my defense I've tried to isolate the bug, first by git bisecting, but that was unfruitful because the blamed commit was a big refactoring of the template logic, and then debugging in the classic way, i.e. reasoning about the code, but when I started to dig Markdown and Jinja2 I lost patience and got distracted with other things...