Mirror size & completeness of binaries

[Ryan Schmidt]

On Apr 2, 2018, at 08:11, Arno Hautala wrote:

> On Fri, Mar 30, 2018 at 5:20 PM, Rainer Müller wrote:
>> 
>> For example, in a future archive_sites.conf:
>> 
>> name       macports
>> platform   darwin 18
>> urls       https://packages.macports.org/10.14/ \
>>           https://foo.bar.packages.macports.org/10.14/
>> 
>> name       macports
>> platform   darwin 17
>> urls       https://packages.macports.org/10.13/ \
>>           https://bar.baz.packages.macports.org/10.13/
>> 
>> ... and so on.
>> 
>> With this layout, it would be trivial to spot if one of the mirrors does
>> not actually provide packages for this version. When just excluding
>> *darwin_X*, not so easy.

I understand... I agree, that does seem like a good reason.

I'm not completely sold on suddenly using macOS version numbers here, instead of Darwin version numbers like we do in the archive filenames, but I guess it doesn't matter.


> Couldn't the same be achieved by creating a second hierarchy
> consisting only of symlinks to the locations under the original
> hierarchy?

It doesn't matter if they're symlinks or hardlinks; either way, something has to initially create the second hierarchy of links from our existing files, and we have to keep that second hierarchy updated whenever we deploy new archives. This shouldn't be too difficult; I'm happy to do that.

Using hardlinks has the advantage that once we have transitioned to a new version of MacPorts that knows about downloading from these new locations (let's call it MacPorts 2.Y), we can simply remove the code that updates the original locations and "rm -rf" the old files on the mirror.

Before we begin deploying this plan, we should inform the mirror administrators, verify that they're using rsync's -H flag that detects hardlinks, as recommended in our mirroring instructions, and ask them if they have any concerns about this change.

We could also have separate top-level directories to distinguish between libc++ and libstdc++, to ease our 10.6-10.8 transition. We could decide that libc++ is our default and that we will only mark the libstdc++ directories, so our top-level directories at present would be "10.5_libstdcxx", "10.6_libstdcxx", "10.7_libstdcxx", "10.8_libstdcxx", "10.9", "10.10", etc.

After this MacPorts 2.Y is released and the relevant mpbb and buildbot changes are deployed, we can set up buildbot workers for 10.6-10.8 libc++, which will then upload their archives to new 10.6/10.7/10.8 directories. To initially set up those new directories, we would run a one-time script that duplicates the hierarchies from the 10.6_libstdcxx/10.7_libstdcxx/10.8_libstdcxx directories using hardlinks. Then we run Josh's script in the 10.6/10.7/10.8 directories to delete any archives that use libstdc++. Then we can trigger builds of those ports on the 10.6-10.8 libc++ workers.

Any change to our packages hierarchy will also have an effect on our CDN, namely that the CDN will have to pull new copies of the files from the origin server. I'm sure the CDN can handle that, but it will mean slightly slower file delivery the first time any file is requested from any of the CDN's edge locations (there are 12). It will also mean an increase in traffic to our origin server at FAU; we should notify them beforehand and make sure it's not a problem for them.