Mirror size & completeness of binaries

[Mojca Miklavec]

On 29 March 2018 at 16:19, Ryan Schmidt wrote:
> On Mar 27, 2018, at 06:56, Mojca Miklavec wrote:
>
>> I don't know about the best way to do it, but I would like to suggest to provide macports mirorring in two different sizes: a small one and a complete one.
>>
>> While I'm a heavy supporter of providing support for legacy systems, I see no reason to mirror files for them on all of our mirrors and cause troubles to them. I would suggest to mirror by default just the latest version of any given source and binary and only support the latest three OSes there. Then we could have additional files to support older systems on a smaller set of mirrors, just on those where it would not cause any additional troubles to them. Since the number of users of legacy systems is much smaller, this should not have a heavy impact on bandwidth to that smaller number of mirrors either.
>
> I do not wish to create separate small and complete mirrors. I wish to keep a single complete mirror at our primary rsync.macports.org server, and leave it up to each individual mirror operator to decide for themselves whether they want to mirror all of it or whether they want to exclude some older systems, and if so which systems they wish to exclude. It is my assumption that it should be easy to specify exclusions in the rsync command. We would update the Mirroring wiki page with those instructions, along with our recommendations for which older OS versions to consider excluding.

Yes, exclusions are possible, but would much easier to do if we had
the OS on the top dir.

> At the same time, we need to consider how MacPorts base will respond to the situation. Currently, it assumes all packages are available on all mirrors. Therefore, it tries to find a package on three mirrors before giving up. If many mirror operators choose to exclude an OS version, this could leave a user building from source, even when a binary exists on a more-distant mirror; I want to avoid that. One workaround is to recommend that mirror operators who exclude some packages should also add http redirects for excluded packages to their web server configuration, redirecting those requests back to the main (full) mirror. We can update our sample configurations on the Mirroring wiki page to reflect this.

Ideally we would need to store the information about which mirror
offers what packages (and potentially also do some daily(?) health
checks) and only add the supported mirrors to the list of older OSes.
I don't know how mirroring currently works though.

>> I'm not saying this should be implemented immediately, but I would certainly start thinking about that before we add additional four mirrors (three legacy ones and 10.14).
>
> As I understood our libstdc++ to libc++ transition plan for 10.6-10.8 systems, we will not create new builders. We will instead repurpose existing builders for libc++, and delete the old libstdc++ archives for 10.6-10.8 system when we do so.

We may do that, but I'm super sceptic about the totally untested
transition to go right. (That said, I still prefer a transition with
problems than postponing that transition for another five years.)

I would still be in favour to set up *one* libc++ builder first and
publish the packages somewhere. It could be
http://binaries.macports.org/10.X/<name>/ or anything along those
lines, on a single server, no mirrors. We certainly need a fresh
builder with delete_la_files setting, different default compiler etc.
(Again, I would seize the opportunity and try to switch to xz at the
same time, but I admit that's a different/unrelated story; I would
also be grateful for feedback to
https://trac.macports.org/ticket/52000#comment:12)
Once the major problems are fixed (I expect there will be quite some)
and at least a few people can confirm a smooth transition was
possible, set up all three builders, switch the default, shut down the
old builders and potentially remove the old packages (either
immediately or, say, after two weeks).

Optional: We could certainly modify macports to check for both
locations. And thus first put 10.6-10.8/libc++ to that new location,
delete the old archives after two weeks, then potentially slowly move
the others. It should be straightforward to write a script to relocate
the binaries if some mirroring sites would want to run that locally
first to avoid additional traffic. If we explain the situation well in
advance, this should be doable.

I know it sounds far-stretched, but I have a feeling this could help
in the long run if done correctly.

Mojca