Mirror size & completeness of binaries

[Ryan Schmidt]

On Mar 29, 2018, at 12:55, Mojca Miklavec wrote:

> On 29 March 2018 at 18:21, Rainer Müller wrote:
>> On 2018-03-29 16:19, Ryan Schmidt wrote:
>>> At the same time, we need to consider how MacPorts base will respond to the situation. Currently, it assumes all packages are available on all mirrors. Therefore, it tries to find a package on three mirrors before giving up. If many mirror operators choose to exclude an OS version, this could leave a user building from source, even when a binary exists on a more-distant mirror; I want to avoid that. One workaround is to recommend that mirror operators who exclude some packages should also add http redirects for excluded packages to their web server configuration, redirecting those requests back to the main (full) mirror. We can update our sample configurations on the Mirroring wiki page to reflect this.
>> 
>> We can easily split our definition of mirror_sites and only list the
>> URLs that will provide packages for a particular version.
>> 
>> I started looking at #56053 [1] yesterday and I think it would actually
>> be helpful if we would use a configuration file with the same syntax as
>> archive_sites.conf instead of the messy mirror_sites.tcl we have right
>> now, as it makes assumptions about the internals of base.
>> 
>> I am not sure either how we could do such a reorganisation without
>> wasting a lot of bandwidth, but we could at least start with the new
>> scheme as of 10.14.
> 
> If we start with 10.14 and the (hopefully) three new builders, we
> won't be wasting any *additional* bandwidth as we would need to put
> those binaries somewhere anyway.
> 
> As far as other builders are concerned: some packages might never be
> downloaded, but others might be downloaded 100 or 1000 times or ever
> more by various users. So as long as we don't transfer one full
> terrabyte of data at once, I don't think this would pose a serious
> issue. The biggest problem is in *additional disk space*. By:
> - cleaning up the old packages
> - transferring just one platform (or even less) at a time
> this should hopefully not be such a big deal.

I have not yet heard any reason why we need to change the file structure of the packages server.


> PS: if we gradually move packages.macports.org to a different
> subdomain, we could, say, one year after the move, reuse that
> subdomain for package index :)

If by "package index" you mean the one-web-page-per-port system that we've long desired but not yet implemented, then I had always assumed it would be located under www.macports.org/ports; we should stop creating separate hostnames for every new thing and instead try to create a single cohesive web experience.