[MacPorts] #54800: Set up a private server for archives that aren't distributable to speed up Travis CI builds

[MacPorts]

#54800: Set up a private server for archives that aren't distributable to speed up
Travis CI builds
-----------------------------+---------------------
  Reporter:  yan12125        |      Owner:  admin@…
      Type:  enhancement     |     Status:  new
  Priority:  Normal          |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:                  |
-----------------------------+---------------------

Comment (by ryandesign):

 Replying to [comment:8 ryandesign]:
 > * On the buildbot server, set up a new directory outside of the rsync
 directory to store nondistributable archives.

 Done. I've used a separate script to manually populate this directory with
 all current archives from all builders. After buildbot is modified to
 upload nondistributable archives there, I can run the separate script once
 more to sync everything up, then run another script to remove
 distributable archives from this directory, leaving only the
 nondistributable ones.

 > * Change mpbb so that it uploads nondistributable archives to there. (We
 don't need to worry about cleaning up old nondistributable archives; the
 existing [ticket:56181 cleanup script] can probably be used for that.)

 https://github.com/macports/mpbb/pull/9

 Buildbot configuration changes are also needed to supply the new arguments
 to mpbb.

 https://github.com/macports/macports-infrastructure/pull/8

 > * At NameCheap, make 2 new hostnames on macports.org for this: one for
 the origin (on the buildbot server) and one for the CDN.

 Done.

 Still to do: automating the renewal of the Let's Encrypt SSL certificates.

 > * On the buildbot server, configure nginx to serve the new directory via
 that new origin hostname (and also a local-network-only hostname for use
 by the buildbot), and to allow access only from MaxCDN IP addresses (and
 localhost).

 Done.

 > * At MaxCDN, add a new zone for this with the new CDN hostname and
 configure the zone to allow access only from Travis IPs.

 Done.

 > * Modify Travis config to also find packages at the new CDN hostname by
 adding an entry to archive_sites.conf.

 Still to do. Another possibility is setting `archive_site_local` in
 macports.conf.

 > * Modify buildbot config to also find packages at the new local-network-
 only hostname by adding an entry to archive_sites.conf.

 The problem is that mpbb has its own separate copy of macports.conf, which
 references its own separate copy of sources.conf. However, I don't see any
 way to reference a separate copy of archive_sites.conf. Using
 `archive_site_local` in macports.conf is an alternative; this is included
 in https://github.com/macports/mpbb/pull/9. However, it means that the
 standard set of archive sites is still used; I would like to find a way to
 disable that.

 > * Maybe the preceding 2 happen in mpbb; if so, do we have a way in mpbb
 to differentiate whether we're running on Travis or buildbot?

 mpbb sets up macports.conf in mpbb-checkout. Travis doesn't use mpbb-
 checkout, so at present these changes will only affect buildbot. I've
 added arguments so that the archive site URLs can be specified when
 calling mpbb-checkout, so that Travis and Buildbot could request different
 URLs.

-- 
Ticket URL: <https://trac.macports.org/ticket/54800#comment:10>
MacPorts <https://www.macports.org/>
Ports system for macOS