ProgramFan's Writing Space Site Logo

Deploy python working environments with pip under restricted connections

Python, PIP, Working Env, and Featured
02 Oct 2015 ProgramFan

Background

One can often use pre-built python distributions to setup python working environments, for example Anaconda, Enthought Canopy. But these distributions has the following common problems, which reduces their usefulness:

  1. It’s hard to upgrade packages without internet connections. It’s also hard to install packages not contained in the distribution.

  2. It does not play well with existing packages on the target systems. For example, Anaconda bundles mpich, which conflict with system MPIs.

  3. These distributions has poor performances. Since they are built on relatively old platforms and possibly with no optimizations, they are often slower than native python. For example, I found anaconda python on a RHEL 6.3 system very slow to start.

For me, it’s essential to deploy my python working environments with the following considerations:

  1. Easy to install/upgrade packages on demand, under restricted connections. Some systems I work with does not have internet connections.

  2. It shall integrate well with existing python packages. Some packages such as PyQt5 requires deep system integration so I would better install them with system package manager.

  3. Good performance. The distributed packages shall run fast and be responsive on the target platform.


  Read the rest of this post

User-local python package install with pip

Python, PIP, and Featured
20 Sep 2015 ProgramFan

Motivation

PyENV has done a fairly good job in managing user-local python versions, as well as per-version pip packages. However, some packages such as PyQt5 need deep system integration and can not be installed using pip. This makes packages such as jupyter-qtconsole upset. The use of shims in PyENV also forces any provided command to be started with a shell script (so a shell process). This makes packages such as python-powerline much slower.

To solve the above problems, I choose to go back to the standard python user-local install process and choose pip user install. pip user install relys on system python to work correctly, and integrates tightly with system packages. That is to say, if one install PyQt5 using system package manager, he/she can use it in a user-local toyplot package, avoiding install PyQt5 manually (since not possible with pip). Moreover, binaries provided by packages are directly avaliable to the system, which greatly improves the speed of packages such as python-powerline [1]

1. python-powerline uses a binary powerline-client to accelerate shell prompt rendering, which is much more responsive than shell scripts. The latter causes visiable lag even on recent ivebridge CPUs.

  Read the rest of this post