Mirror size & completeness of binaries

[Mojca Miklavec]

On 29 March 2018 at 18:21, Rainer Müller wrote:
> On 2018-03-29 16:19, Ryan Schmidt wrote:
>> At the same time, we need to consider how MacPorts base will respond to the situation. Currently, it assumes all packages are available on all mirrors. Therefore, it tries to find a package on three mirrors before giving up. If many mirror operators choose to exclude an OS version, this could leave a user building from source, even when a binary exists on a more-distant mirror; I want to avoid that. One workaround is to recommend that mirror operators who exclude some packages should also add http redirects for excluded packages to their web server configuration, redirecting those requests back to the main (full) mirror. We can update our sample configurations on the Mirroring wiki page to reflect this.
>
> We can easily split our definition of mirror_sites and only list the
> URLs that will provide packages for a particular version.
>
> I started looking at #56053 [1] yesterday and I think it would actually
> be helpful if we would use a configuration file with the same syntax as
> archive_sites.conf instead of the messy mirror_sites.tcl we have right
> now, as it makes assumptions about the internals of base.
>
> I am not sure either how we could do such a reorganisation without
> wasting a lot of bandwidth, but we could at least start with the new
> scheme as of 10.14.

If we start with 10.14 and the (hopefully) three new builders, we
won't be wasting any *additional* bandwidth as we would need to put
those binaries somewhere anyway.

As far as other builders are concerned: some packages might never be
downloaded, but others might be downloaded 100 or 1000 times or ever
more by various users. So as long as we don't transfer one full
terrabyte of data at once, I don't think this would pose a serious
issue. The biggest problem is in *additional disk space*. By:
- cleaning up the old packages
- transferring just one platform (or even less) at a time
this should hopefully not be such a big deal.

Mojca

PS: if we gradually move packages.macports.org to a different
subdomain, we could, say, one year after the move, reuse that
subdomain for package index :)