Mirror size & completeness of binaries

[Mojca Miklavec]

On 2 April 2018 at 17:15, Ryan Schmidt wrote:
> On Apr 2, 2018, at 08:11, Arno Hautala wrote:
>> On Fri, Mar 30, 2018 at 5:20 PM, Rainer Müller wrote:
>>>
>>> For example, in a future archive_sites.conf:
>>>
>>> name       macports
>>> platform   darwin 18
>>> urls       https://packages.macports.org/10.14/ \
>>>           https://foo.bar.packages.macports.org/10.14/
>>>
>>> name       macports
>>> platform   darwin 17
>>> urls       https://packages.macports.org/10.13/ \
>>>           https://bar.baz.packages.macports.org/10.13/
>>>
>>> ... and so on.
>>>
>>> With this layout, it would be trivial to spot if one of the mirrors does
>>> not actually provide packages for this version. When just excluding
>>> *darwin_X*, not so easy.
>
> I understand... I agree, that does seem like a good reason.
>
> I'm not completely sold on suddenly using macOS version numbers here, instead of Darwin version numbers like we do in the archive filenames, but I guess it doesn't matter.

I don't really care which name is being used. I though of 10.13
because that gives us consistency with buildbot names and might be
easier to understand by people not so intimate with Darwin versions,
but the main point would be to agree whether this kind of
configuration is desirable.

I see lots of benefits in the long run, the main ones being the
ability to put libc++-based binaries for legacy systems online right
away, but also to allow much easier OS selection filtering by mirrors
(and giving immediate feedback about which OSes a particular mirror
supports).

> We could also have separate top-level directories to distinguish between libc++ and libstdc++, to ease our 10.6-10.8 transition. We could decide that libc++ is our default and that we will only mark the libstdc++ directories, so our top-level directories at present would be "10.5_libstdcxx", "10.6_libstdcxx", "10.7_libstdcxx", "10.8_libstdcxx", "10.9", "10.10", etc.

Precisely.

> After this MacPorts 2.Y is released and the relevant mpbb and buildbot changes are deployed, we can set up buildbot workers for 10.6-10.8 libc++, which will then upload their archives to new 10.6/10.7/10.8 directories. To initially set up those new directories, we would run a one-time script that duplicates the hierarchies from the 10.6_libstdcxx/10.7_libstdcxx/10.8_libstdcxx directories using hardlinks. Then we run Josh's script in the 10.6/10.7/10.8 directories to delete any archives that use libstdc++. Then we can trigger builds of those ports on the 10.6-10.8 libc++ workers.

I would prefer to start completely from scratch for 10.6-10.8 libc++
even if the resulting archives are then compatible. There are those
delete_la_files and other settings that might influence which files
are being installed, if we switch to a different default compiler, all
binary files will be slightly different etc.

> Any change to our packages hierarchy will also have an effect on our CDN, namely that the CDN will have to pull new copies of the files from the origin server. I'm sure the CDN can handle that, but it will mean slightly slower file delivery the first time any file is requested from any of the CDN's edge locations (there are 12). It will also mean an increase in traffic to our origin server at FAU; we should notify them beforehand and make sure it's not a problem for them.

Yes, we need to communicate this. But in any case I would not try to
do the transition of more than one platform at a time.

Mojca