Python ports: should we use wheel files?

[Enrico Maria Crisostomo]

> On 2 Apr 2018, at 08:35, Joshua Root <jmr at macports.org> wrote:
> 
> Good summary from Rainer, which covers most of what I was going to say.
> 
> On 2018-4-1 12:53 , Rainer Müller wrote:
>> I see two reasons why python modules need natively compiled binary files:
>> 
>> a) for performance
>> b) they link against another library
>> 
>> In case of b), using pre-compiled binary files is out of the question,
>> because we want to link to our own libraries and not against libraries
>> in /usr/lib.
> 
> I'll just add that some extensions use libraries that do not ship with
> the OS at all, which makes binary wheels impossible unless you either
> statically link or just assume the user has the libs installed in some
> preselected place (which will probably be /usr/local/lib).
> 
> The direction among packagers in general seems to be towards encouraging
> upstream authors to make it easy (or at least possible) for others to
> reproduce their build process. See <https://reproducible-builds.org/>.
> 
> - Josh

Thanks everybody for sharing your point of view.  I was not aware of all the subtleties and your insights have been very helpful.

Rainer: 

> In case of b), using pre-compiled binary files is out of the question,
> because we want to link to our own libraries and not against libraries
> in /usr/lib.

Thanks for you detailed answer.  Linked libraries are indeed an issue.

> Can we rely on Portfile authors to check what is linked?
> Are they aware which libraries this python module links to before
> submitting the Portfile?

I wondered whether a Portfile author should check and I recognise it's a brittle process at best.

> It would probably always be safe to use wheels for platform "any", but
> there is also no advantage over setup.py for them.

I agree.

> We would also lose +universal variants for python ports, but I did not
> check if we actually have such ports.

Thanks for pointing this out, I wasn't fully aware.

> Whether we can
> safely ship pre-compiled Java code is still not clear to me.

I had quite a few surprises in the past while I was maintaining Logstash/ElasticSearch for FreeBSD or a running Jenkins on Solaris.  These project (used to?) include native libraries which sometimes broke on certain OS versions.  As far as I can tell, though, if it's pure Java byte code meant to run on a compliant runtime, then I would say it's safe.

Ryan:

> For what macOS versions and architectures are they compiled? I assume the list is smaller than what MacPorts currently supports.

The list is actually rather small:

  * tensorflow-1.6.0-...-macosx_10_11_x86_64.whl

And for py-grpcio pretty much the same:

  * grpcio-1.10.0-...-macosx_10_11_x86_64.whl

tensorflow-1.6.0-cp27-cp27m-macosx_10_11_x86_64.whl

> An additional question is: can you fully trust the binary code of all
> the hundreds of python packages? (It's also true that you should not
> always trust the sources until you checked them, but binaries are
> easier to infect without noticing.)

Thanks Mojca, that's a very good point.

Joshua:

> The direction among packagers in general seems to be towards encouraging
> upstream authors to make it easy (or at least possible) for others to
> reproduce their build process. See <https://reproducible-builds.org/>.
 
Thanks Joshua.  Yes, that a great initiative.



I think you've given me enough reasons to try and build everything from source.

Thanks,
-- 
Enrico