Re: privoxy-pki-bundle not Behaving as Desired – Request for Assistance

[Daniel J. Luke]

On May 24, 2022, at 12:07 PM, Ryan Schmidt <ryandesign at macports.org> wrote:
> I'm not convinced that I would be happy to see such a feature. Just a moment's consideration shows how many different parts of MacPorts would need modifications to support it, not only in base but also in the buildbot and in the package hosting infrastructure. And in the end, what we end up with is a rather fundamental violation of a principle we currently rely on, which is that a given version and revision of a port is (unless there is a bug) reproducible, immutable, deterministic.

The assumption you're making is that any possible implementation will be bad? ;-)

I don't think it's worth bikeshedding this when no one has done any work - but if we step back and consider that all we'd be trying to do is automate the process you already go through when reading a comment and rev-bumping a port, it doesn't seem like it's a noncomputable problem.

> To get there, we'd have to modify the registry to store what version of curl-ca-bundle was installed at the time that privoxy-pki-bundle was installed, and we'd have to extend the portindex to hold information about which ports should trigger a reinstall for another port, and we'd have to enhance base's understanding of outdatedness to make use of this new information, and enhance "port outdated" to display that information.

There are a lot of possible implementation paths.

Yes, there would be additional information we'd need to store, and yes base would have to understand this information.

> And suppose we have modified the buildbot to know that it must rebuild privoxy-pki-bundle and upload a new package. (Currently it decides whether to build a new package in part based on whether a package with the correct filename already exists on the server, which now won't be sufficient criteria.)

One possible implementation would be to use one of the existing discriminators (epoch, version, revision) or add an additional one to indicate that a new package is necessary - yes this transition would involve a lot of moving pieces - but the buildbot could use the same (or very similar) criteria.

> It takes awhile for newly uploaded packages to sync from the private server to the public one, so let's say by 2pm the new curl-ca-bundle and privoxy-pki-bundle packages have synced to the public server (although there's no guarantee that they'd both be there at the same time -- the new curl-ca-bundle package, having been built first, might arrive 30-60 minutes ahead of privoxy-pki-bundle). Our mirrors sync from the public server at varying intervals, so let's say that by 10pm the new packages are on all the mirrors. Prior to that time, old packages of the same filename might still be on some mirrors. What happens if a user upgrades to the new curl-ca-bundle at 5pm? If the new curl-ca-bundle package is available on any mirror they access, that'll be used, otherwise the new curl-ca-bundle will be built from source. Fine. Then MacPorts will want to reinstall privoxy-pki-bundle, even though its version and revision are the same, due to the new hypothetical feature. It will check for a package on a mirror and it will find one! But will it be an old package or a newly rebuilt one?

This is assuming that any possible implementation would result in non-reproducible builds, which I also wouldn't support.

> No one knows.

... because it doesn't exist yet. I think it would be more useful to try to specify what we'd want from a hypothetical feature like this rather than just claim that it's not possible for it to work, maybe something like:

- builds must still be reproducible
- buildbot must be able to determine it needs to build a new package
- a new package must not replace a previously built package
- end users must still be able to use pre-built packaages

(probably more)

On the other hand, if no one is actually doing the work /and/ prominent project leaders are saying they could never support such a feature, it's unlikely that anyone will ever try to make improvements in this area, so it's probably not even worth time defining what it would take for it to be useful and added to MacPorts.

> Or perhaps the proposal is that package filenames should grow even longer with an additional identifier -- which would hopefully be empty for ports not opting in to this new thing so that all our existing packages are not invalidated. If so, what would the new identifier be?

There isn't really a proposal yet (I don't think anyone is working on code) - but there are of course possibilities here. We could (ab)use epoch, just have portindex increment the revision for the ports that request to be rebuilt when a new version of something is added, add a new int that we auto-increment, hash over the (portname, epoch, version, revision) of deps that a port says it cares about [and countless others that would maybe take me more than 5 minutes to come up with ;-)]

-- 
Daniel J. Luke