[MacPorts] #60590: rsync.macports.org and other services at FAU are down

[MacPorts]

#60590: rsync.macports.org and other services at FAU are down
-----------------------------+---------------------
  Reporter:  herbygillot     |      Owner:  admin@…
      Type:  defect          |     Status:  new
  Priority:  High            |  Milestone:
 Component:  server/hosting  |    Version:
Resolution:                  |   Keywords:
      Port:                  |
-----------------------------+---------------------

Comment (by danielluke):

 Replying to [comment:16 ryandesign]:
 > Replying to [comment:14 danielluke]:
 > > Yeah, I was suggesting that we have FAU and maybe one other mirror
 rsync from the private server.
 >
 > We could do that.  It would help make distfiles and packages available
 in the event that FAU is down, since MacPorts tries multiple servers when
 it cannot find distfiles or packages. It would not help much for the ports
 tree, since users must manually configure MacPorts for which rsync server
 they want to use. Users would have to know which of our other mirrors is
 the "second master" that you're proposing.

 We could solve that with changes to base or by updating the DNS record to
 point to only active servers (there are providers which offer this as part
 of DNS load balancing).

 > > We could reconfigure the CDN to use the 'other' mirror as origin
 (manually, I guess if they don't offer a way to failover). One way to
 handle that would be to repoint DNS as you suggest.
 >
 > We only have the arrangement with FAU that allows us to use them as our
 CDN origin. We would have to contact the other mirror administrators and
 ask them if such an arrangement would be possible with them as well.

 Yes, that's the suggestion - that we get (at least) one other mirror that
 could act as our origin.

 > > I guess if we're going to have to manually reconfigure, we could
 configure a second site to only rsync from the private server if FAU is
 down.
 >
 > So you're suggesting that this one "second master" should modify their
 mirroring script so that it checks if FAU is up, if it is then mirror from
 FAU, and if not then mirror from the private server? What criteria would
 they use to determine when FAU is up? Right now their rsync server is
 responding, but according to their web site their server is unreliable due
 to the RAID controller failure so it may respond sometimes and other times
 not.

 The key in that sentence was 'manually reconfigure' - this would only
 necessary in the case where private server bandwidth constraints prevent a
 second mirror from syncing. Something like checking for the 'freshness' of
 the files on the primary + some flap dampening would work (even in this
 case), but it would be nicer if we didn't need to be extra clever with
 failover logic.

 > > Do you know what the current disk space + bandwidth requirements are?
 >
 > Disk space hovers around 1TB. Currently a little less because I recently
 ran the [ticket:56181 cleanup script]. Sometimes more if I haven't run the
 cleanup script for awhile. I don't know how much bandwidth they should
 expect to be used.
 >
 > > Do we need to recruit one or more sites that could act as CDN origins?
 >
 > Such a need has not arisen until now. We could certainly ask around.

 Probably best if one of our existing mirrors can add this - if not, I can
 double-check and see if it'll be OK for me to do this on a box I have co-
 located on a gige connection - it would be helpful to get some traffic
 statistics to determine if I can or not (alternatively, I may have another
 box available in a different location in a few months with different
 operating constraints that could work - but it won't help us now).

-- 
Ticket URL: <https://trac.macports.org/ticket/60590#comment:23>
MacPorts <https://www.macports.org/>
Ports system for macOS