PortGroup directory hierarchy/priority

[Clemens Lang]

Hi,

On Mon, Apr 04, 2016 at 10:42:16AM +0200, René J.V. Bertin wrote:
> Typically paths defined through variables indeed, or configure
> settings,

Paths and flags should be part of configuration files installed by a
port, e.g. CMake config files or pkg-config files.

> plus the occasional pre- or post- block.

I agree that this is a valid use case.

> >That would potentially cause ports to build different depending on
> >which packages you have installed in your environment, which is not
> >reproducible.
> 
> Yes, but that is already the case when a user replaces a mainstream
> port with one from an alternative source.

Yes, but then the transitive dependency tree of the changed port was
also changed. A overwritten PortGroup could change the behavior of ports
that do not have the provider of the PortGroup in their dependencies.

> There's only one way to enforce the reproducible build principle all
> the way, and that's to remove the whole possibility of building from
> source so that users can only install official, binary packages.

No, that's (a) not true and (b) actually prevents one of the goals of
reproducibility, namely that the average user can verify that the
binaries served by us are exactly what they should be and we haven't
backdoored them.

> > The behavior of a port might even be affected by ports that are not
> > in its transitive dependency tree. 
> 
> Can you think of an example other than the one below?

Sure, let's assume you change the github 1.0 PortGroup, because you need
an additional feature… that could change how other ports fetch, and thus
the source that is being built be other ports.

> > The more I think about this, the more I get the impression a
> > PortGroup should only be affecting its port tree, and nothing else.
> 
> To draw an analogy, you prefer a Prohibition-like approach to a more
> permissive one that provides the tools to achieve do certain things in
> a controlled way?

I prefer an approach that minimizes the probability to break things.
Yes, this limits the developers of Portfiles somewhat, but it doesn't
make things impossible -- the PortGroup and affected ports can always be
copied.


On Mon, Apr 04, 2016 at 05:13:25PM +0200, René J.V. Bertin wrote:
> On Monday April 04 2016 16:22:27 Rainer Müller wrote:
> 
> >A user already installed port A with the official port group foo-1.0.
> >Then they add a new source, overriding the port group foo-1.0, which
> >redefines some paths. This will not cause A to be outdated and no
> >rebuild of A will be triggered. Now installing B, which depends on A,
> >will use the new port group. The result is unspecified and the build
> >would also most likely fail.
> 
> Yes, I already pointed that out in a previous message.
> 
> This should *not* be the case if the current lookup rules for
> _resources continue to be applied, but a higher priority central
> location (var/macports/sources/PortGroups) is added where copies of
> (symlinks to) PortGroups are installed.
> In that scenario, foo-1.0 will only be replaced when the user installs
> the custom port A. Nothing will change if the custom source contains a
> rogue foo-1.0 PortGroup that does not belong to a variant of port A,
> or if that custom port A is never installed.

The whole installation idea does not work with the current codebase. If
a PortGroup changes, the information in the PortIndex needs to be
updated, but the PortIndex uses the modification date of Portfiles to
determine whether the information is current.

That's also why enabling a source cannot silently change the behavior of
a port: it's index has already been generated, and there is currently no
way of knowing that it should be re-generated with the new PortGroup.
Adding code to track this would add significant complexity to achieve
something that is already possible by copying PortGroups and ports.

> > needed, or the version of the port group could be changed.
> 
> Changing a PortGroup version is a no-go as far as I'm concerned, as it
> makes it impossible to build existing ports against an alternative
> PortGroup without modification.

You can not change how a port behaves without touching the Portfile. See
the above rationale on the PortIndex.

> > As soon as a port group was overridden for a port we would have to
> > disable binary archives completely for this port. 
> 
> Not necessarily, and in fact that depends more on the associated
> *port* than on the PortGroup. My custom cmake PortGroup for instance
> doesn't introduce any incompatible changes (except possibly for ports
> that happen not to be built as intended with the current PortGroup ...
> but those ports are buggy anyway).

Well, and exactly the exception you mentioned there means that for the
general case, this must be disabled. We should not provide a solution
that "works mostly" and breaks in weird ways in corner cases. We want a
solution that works always.

> > However, if you copy the port into the customized ports tree with
> > the new port group, custom binary archives can always be distributed
> > from the corresponding archive_sites for this ports tree.
> 
> Sure, I could just fork of all MacPorts while I'm at it ...

There are people who provide custom port trees with associated binary
archives. Yes, it's complicated, but it can and is being done.


I think most of this discussion could be solved by merging your changes
into the main port tree, couldn't it? I think that's what we should be
working on, rather than discussing elaborate strategies to selectively
override parts of port trees.

-- 
Clemens