Re: privoxy-pki-bundle not Behaving as Desired – Request for Assistance

[Ryan Schmidt]

On May 24, 2022, at 14:08, Daniel J. Luke wrote:

> On May 24, 2022, at 12:07 PM, Ryan Schmidt wrote:
>> I'm not convinced that I would be happy to see such a feature. Just a moment's consideration shows how many different parts of MacPorts would need modifications to support it, not only in base but also in the buildbot and in the package hosting infrastructure. And in the end, what we end up with is a rather fundamental violation of a principle we currently rely on, which is that a given version and revision of a port is (unless there is a bug) reproducible, immutable, deterministic.
> 
> The assumption you're making is that any possible implementation will be bad? ;-)

No, just that with a few moments of thinking about the proposed feature, I could foresee several problems that would arise, and I wanted to convey some of those to at least communicate that it's a complex problem to solve. If somebody thinks this feature is so valuable that they want to devote time to solving the problems and coming up with a workable solution that's fine. Perhaps with more than a few moments' thought that would be possible.


> I don't think it's worth bikeshedding this when no one has done any work - but if we step back and consider that all we'd be trying to do is automate the process you already go through when reading a comment and rev-bumping a port, it doesn't seem like it's a noncomputable problem.

I don't think the term bikeshedding is applicable here at this time. Bikeshedding refers to devolving into discussions about unimportant details. I don't believe I was doing that.

One possibility is to just introduce a new portfile keyword (the rebuild-if-these-ports-were-updated keyword) that MacPorts base doesn't actually use (just like base recognizes but does not use known_fail) but that the buildbot or another server process could use to increase the revision of the appropriate port and commit it. Then all other existing processes would go into effect to build and deploy the new package, update the portindexes, etc. Of course this opens up the possibility of difficulties in recognizing where the revision keyword is in any arbitrary portfile, especially one that has subports or one that does not specify the revision keyword. It's the same problem we encounter whenever we want to increase the revision of ports that depend on a library whose major version has increased. It's the reason why we now recommend all ports and subports specify their revision even when it is 0 but not all portfiles do that yet and even if they did it can still be difficult to identify programmatically which revision keyword to increase when there's more than one.


> This is assuming that any possible implementation would result in non-reproducible builds, which I also wouldn't support.

I didn't mean to suggest that builds would not be reproducible, just to point out that if we reuse archive filenames then depending on the freshness of the file on whichever server the user happens to reach they may receive the new correct file or an old incorrect one. Therefore it seems desirable not to reuse archive filenames, and the question then becomes how to differentiate the filenames.


> On the other hand, if no one is actually doing the work /and/ prominent project leaders are saying they could never support such a feature, it's unlikely that anyone will ever try to make improvements in this area, so it's probably not even worth time defining what it would take for it to be useful and added to MacPorts.

I don't want my comments to dissuade anyone from working on anything in MacPorts that they wish to work on. Again I just wanted to share the initial problems that I perceived with this proposed feature.


>> Or perhaps the proposal is that package filenames should grow even longer with an additional identifier -- which would hopefully be empty for ports not opting in to this new thing so that all our existing packages are not invalidated. If so, what would the new identifier be?
> 
> There isn't really a proposal yet

It is proposed in this thread that MacPorts could have a certain feature.


> (I don't think anyone is working on code)

I wouldn't recommend anyone begin writing any code until it is discussed how the feature should work. That should avoid spending time writing code that won't work.


> - but there are of course possibilities here. We could (ab)use epoch,

epoch can't be used because epoch is not part of the archive filename, and because it is within the portfile author's control when to change the epoch.


> just have portindex increment the revision for the ports that request to be rebuilt when a new version of something is added,

How? The PortIndex file is generated by extracting certain information from each port. Right now, an up-to-date PortIndex file contains the fact that privoxy-pki-bundle has epoch 0, version 3.0.33, revision 3, because that's what it said in the privoxy-pki-bundle Portfile when that PortIndex entry was written. Suppose tomorrow I update curl-ca-bundle to a new version. By what means is the portindex program supposed to know that now, it should record revision 4 in the PortIndex file entry for privoxy-pki-bundle? If you are suggesting that it should read the existing entry from the PortIndex file, increment revision, and write it back, that won't work because it is valid to delete the PortIndex file and regenerate it from scratch.


> add a new int that we auto-increment,

Via a method like what I suggested above, where a server-side process modifies this new int in the Portfile and commits it? If not that, then how?


> hash over the (portname, epoch, version, revision) of deps that a port says it cares about

Perhaps (excluding epoch, as explained above). You wouldn't be able to tell if one hash is before or after another hash, so whereas with epoch and revision we can compare ints numerically and for version we can compare with vercmp, the only comparison we could do with hashes is equality. If MacPorts hypothetically had the ability for a user to keep multiple version_revision_hash of each port installed (as today it can keep multiple version_revision installed), then it may be difficult for a user to understand the order of those hashes. Yes, they could use "port -v installed" and look at the installation date, but MacPorts doesn't sort the list by date.