Python ports: should we use wheel files?

[Rainer Müller]

On 2018-03-31 12:11, Enrico Maria Crisostomo wrote:
> The definition of a wheel file is the following:
> 
>> Wheel (in this context) is a project that adds the bdist_wheel command to distutils/setuptools. This produces a cross platform binary packaging format (called “wheels” or “wheel files” and defined in PEP 427) that allows Python libraries, even those including binary extensions, to be installed on a system without needing to be built locally.  In the case in point: one TensorFlow dependency has a native component that needs to be built locally.

I see two reasons why python modules need natively compiled binary files:

a) for performance
b) they link against another library

In case of b), using pre-compiled binary files is out of the question,
because we want to link to our own libraries and not against libraries
in /usr/lib.

Can we rely on Portfile authors to check what is linked?
Are they aware which libraries this python module links to before
submitting the Portfile?

It would probably always be safe to use wheels for platform "any", but
there is also no advantage over setup.py for them.

If we had a working pypi2port, this would probably be easy to test on a
wide range of python ports...

> Now, the problem.  macOS is a platform where TensorFlow is _built_ and _tested_ by upstream.  Some of its dependencies (e.g.: grpcio) are packaged and distributed as wheel files.  Hence, what upstream supports is the state of a system after the dependencies are installed from wheel files.
> 
> Given the above assumption, my impression is that we have an opportunity to leverage wheel files instead of rebuild each dependency from source.  The advantages in my opiion are:
> 
>   * The _state_ of the system after the installation of the wheel files shall be _exactly_ se seen, tested and validated by upstream.

Might as well be a downside, as it takes away all options to patch or
adapt...

We would also lose +universal variants for python ports, but I did not
check if we actually have such ports.

>   * We're not duplicating effort already put in by upstream.
> 
>   * We may not be introducing subtle problems.

However, nothing would change for users. We already provide
(pre-compiled) archives for most python ports, so there is no immediate
benefit of switching a python module port to use a wheel.

> I'm well aware that the philosophy of MacPorts (and BSD ports in general) is to rebuild every package from source.  But I'm wondering, for the sake of argument, whether a parallel can be drawn between (Python, wheel file, PyPi) and (Java, *AR files, Maven Central).  After all we already ship Java byte-code without recompiling it, and sometimes Java projects ship precompiled native libraries.

The situation for Java is not fully comparable, as we do not provide the
runtime environment, but rely on the JRE of the system. Whether we can
safely ship pre-compiled Java code is still not clear to me.

Have any other package managers picked up wheels to build packages?

Rainer