Status Of Python Development

Salutations, people from the Interweb!

Pop question. How many package management tools are available for Python development?

Ignoring system package managers like APT, DNF/YUM and similar I can name, without a particular order: egg, easy_install, setuptools, wheels, pip, venv, Virtualenv, pyenv, poetry, pipenv, pants, Anaconda, conda and miniconda. Did I forgot something?

To be honest, not all are for managing libraries and some overlap in their functionalities, also I'm exaggerating for comedic purposes. Nonetheless what they don't achieve is keeping things simple, at least in my opinion.

I would love to work only with Python 3.x and call it a day, unfortunately after more than ten years from its introduction there are still tools not yet ported from version "2.7" - pwntools, I'm looking at you - and the dynamic nature of the language tend to encourage rapid iterations of libraries that could became problematic when packaging and deploying a project. So we have two interpreters different enough to be incompatible and libraries that are difficult to distribute by third parties.

What Workflow?

Right now, other than pip, I'm using a combination of pyenv and Virtualenvwrapper, in this way I can potentially manage more than a couple of Python interpreters¹ and I can use an heterogeneous collection of Python modules that could be installed independently by different projects. About the last part, yes, if two different virtual environments use the same library, it's duplicated on the disk and no, I haven't verified if there's a way to share the common code.

I start by installing pyenv in my homedir via curl plus its pyenv-register and pyenv-virtualenvwrapper plugins, the first one is for registering the system-wide Python interpreters and the second for configuring virtual environments:

$ curl https://pyenv.run | sh
  [...]
  a few line of output
  [...]
$ git clone https://github.com/doloopwhile/pyenv-register.git \
  ~/.pyenv/plugins/pyenv-register
$ git clone https://github.com/pyenv/pyenv-virtualenvwrapper.git \
  ~/.pyenv/plugins/pyenv-virtualenvwrapper

followed by adding the relevant initialization in my .zshenv:

# add pyenv to PATH
if [[ -d "$HOME/.pyenv" ]]; then
    export PATH="$HOME/.pyenv/bin:$PATH"
    eval "$(pyenv init -)"
    export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"
    #eval "$(pyenv virtualenv-init -)"
    pyenv virtualenvwrapper_lazy
fi

and .zshrc:

# virtualenvwrapper
export WORKON_HOME=~/.virtualenvs
#export PROJECT_HOME=~/Devel

# pyenv + virtualenvwrapper
export VIRTUAL_ENV_DISABLE_PROMPT=yes # virtualenv indicator disabled cause moved to RPROMPT above

The observant reader should have noted above that the pyenv-virtualenv plugin was explicitly disabled, that's because I've found out it interferes with pyenv-virtualenvwrapper. Also, the default Virtualenv prompt indicator is disabled because I like to have it on the Zsh's "right prompt". And yes, installing from a git repository appears indeed fragile, but looking at the number of commits of the last few years it seems quite stable. To be honest it looks a big hack, behind the scenes it mangles the command $PATH variable in an ugly way and relies to a collection of shell stubs, so the above should be interpreted more as "not changing a lot stable"

Oh the horrors!

Anyway, the next step is to register the available system Python interpreters. In my case, at the time of writing, I have both the 2.7.13 and 3.5.3 versions installed²:

$ pyenv versions
* system (set by ~/.pyenv/version)
$ pyenv register /usr/bin/python2.7
  [...]
  bunch of output
  [...]
$ pyenv register /usr/bin/python3.5
  [...]
  similar to the output above
  [...]
$ pyenv versions
* system (set by /home/dnc/.pyenv/version)
  system-2.7.13
  system-3.5.3

Now it's possible to configure the Python interpreter system-wide with the pyenv global command or temporarily on a single terminal instance with the pyenv shell command.

Splendid.

Real life example

Below follows an example of creating a virtual environmet for Pelican, a Python static site generator that the loyal reader has seen mentioned on this pages before. What follows was verified on a Debian Stretch install, the Debian stable version at the time of writing, I haven't tested it on anything else.

After installing pyenv, if the shell isn't reopened, it's required to initialize the Virtualenvwapper plugin³, on the first run it will download and unpack the required Python packages like setuptools, pip and wheel:

$ pyenv shell system-3.5.3
$ pyenv versions
  system
  system-2.7.13
* system-3.5.3 (set by PYENV_VERSION environment variable)
$ pyenv virtualenvwrapper
  [...]
  output for virtualenv initialisation
  [...]

I should mention that forgotten remnants of previously installed Python scripts or libraries in your commands path like ~/.local/bin or ~/.local/lib can cause weird interactions or strange errors, as I've lost an afternoon chasing what I was doing wrong replicating the above steps on a different system.

Anywho the last step consist in creating/naming the virtual environment and optionally associating a project directory to it:

$ mkdir ~/scsm
$ mkvirtualenv -a ~/scms pelican
$ pip install 'pelican==3.6.3' livereload Markdown \
Pillow Pygments
  [...]
  more output
  [...]
$ pip list
Package         Version
--------------- -------
blinker         1.4
docutils        0.14
feedgenerator   1.9
Jinja2          2.10
livereload      2.6.0
Markdown        3.0.1
MarkupSafe      1.1.0
pelican         3.6.3
Pillow          5.4.1
pip             19.0.1
Pygments        2.3.1
python-dateutil 2.7.5
pytz            2018.9
setuptools      40.6.3
six             1.12.0
tornado         5.1.1
Unidecode       1.0.23
wheel           0.32.3

Now I have Pelican ready to run on an isolated Python environment.

Awesome!

Colophon

The inquisitive reader would wonder why it was specified an exact version of the Pelican package. The reason is about a regression in the footnotes generator⁴ that I think was introduced in version 3.7 and I'm guilty to not have reported it yet. Last time I checked it was still present. In my defense I've tried to isolate the bug, first by git bisecting, but that was unfruitful because the blamed commit was a big refactoring of the template logic, and then debugging in the classic way, i.e. reasoning about the code, but when I started to dig Markdown and Jinja2 I lost patience and got distracted with other things...

Nautilus And FUSE

Greetings folks!

Later today I decided to tackle my other remaining big gripe I had with Nautilus¹: an user can mount a FUSE based file-system but the file manager refuses to unmount it when clicking on the relative button, so the user is forced to open a shell and to execute the fusermount command with the appropriate option.

It's a long term bug, I've lived with it for far too much time, that's why I finally decided to fix it once and for all. After downloading the sources of the version currently available on Debian stable, during the documentation phase, I found the RedHat bug report n. 1460203. Skimming to it, I stumbled to this line:

The solution is to add "allow_root" to your /etc/fstab and enable "user_allow_other" in your /etc/fuse.conf.

Wait, Wait. Is it true? Why no one told me about it before?

٩(ˊᗜˋ*)و

Yatta!

Indeed, I can confirm that the two parameters permit to unmount a FUSE based file-system from Nautilus. Even if it isn't the proper solution², I'm really happy that I can finally put to rest this ancient and annoying bug without spending much time on it.

Killing buffers in Emacs

Good day Interweb surfers!

I've used Emacs for several years but until today I've never realized that the kill-this-buffer function should be invoked only by menu:

This command can be reliably invoked only from the menu bar, otherwise it could decide to silently do nothing.

According to the Interwebs, around 10 years ago the function was made menu-bar specific, but originally could be invoked freely. I haven't investigated the rationale behind the change, but I'm pretty sure I've picked it up more than 10 years ago.

The observant reader would correctly guess that only this morning, amusingly enough, it decided to silently do nothing and I have no idea what I did differently¹.

To not be surprised again in the future, at first I modified the binding like this:

(global-set-key (kbd "<f8>") 
                '(lambda ()
                   (interactive)
                   (kill-buffer (current-buffer))))

but immediately after I found out that the kill-current-buffer function, defined in simple.el, seems to do a slightly better job at killing buffers than my lambda above and it's probably the spiritual successor of the kill-this-buffer function.

A few days ago I also fixed the cursor position when correcting typos with Flyspell² and when reopening a buffer as root, that always annoyed me but I continued forgetting about it:

;; flyspell
(eval-after-load 'flyspell
  '(define-key flyspell-mode-map (kbd "C-.")
     '(lambda ()
        (interactive)
        (let ((origin (point)))
          (flyspell-correct-at-point)
          (goto-char origin)))))

and

;; open the current buffer as a different user or root as default
(defun reopen-this-buffer-as-another-user (user)
  "Re-open the current buffer with tramp and sudo,
  using root as default"
  (interactive "sRe-open as user (default: root): ")
  (when (string= "" user)
    (setq user "root"))
  (let ((origin (point)))
    (find-alternate-file 
      (concat (format "/sudo:%s@localhost:" user)
              buffer-file-name))
    (goto-char origin)))

The only remarkable thing about the two definitions above is I couldn't use the save-excursion function because in both I hit two different corner cases: the first is related to the insertion of text adjacent to the saved position and the second is about changing buffers³.