[MacPorts] #59286: Mirror distfiles immediately after a commit

[MacPorts]

#59286: Mirror distfiles immediately after a commit
---------------------------+---------------------
 Reporter:  ryandesign     |      Owner:  admin@…
     Type:  enhancement    |     Status:  new
 Priority:  Normal         |  Milestone:
Component:  buildbot/mpbb  |    Version:
 Keywords:                 |       Port:
---------------------------+---------------------
 Thought we had a ticket for this already but can't find it.

 When GitHub notifies our buildbot 0.8 setup of a commit, it schedules a
 portwatcher build for each OS version. One of the steps of portwatcher is
 to trigger a mirror job.

 This arrangement causes 2 problems:

 1. A mirror job is scheduled for each OS version. We only need it done
 once because the mirror job already loops over each OS version to download
 any files specific to that OS version. Scheduling one mirror job for each
 OS version wastes time at best, and at worst can cause extraordinary
 amounts of unnecessary network traffic for the buildbot and the servers it
 contacts; see #57343.
 2. Files are not mirrored until right before they're built. This is a
 problem if all of the builders for 10.9+ are busy. For example, right now,
 all builders except one have been busy for days building all the p5.30
 ports. Commits for other port updates happened after that, but they are
 waiting, and so their files have not been mirrored. Meanwhile, users of
 10.6-10.8 will learn of those updates when they `port sync`, but they
 might not be able to upgrade their ports if upstream uses restrictive SSL.
 Our 10.6-i386 builder is behind the others and has not yet reached the
 p5.30 ports commit, so it is still working through older updates. But
 because the buildbot updates the ports tree to current before each build,
 it already knows about the sqlite3 update, and tries to build it as a
 dependency of other ports, but fails to fetch it, so I have had to take
 that builder offline.

 We should do things the other way around. On GitHub notification of a new
 commit, we want to schedule one mirror job. When that mirror job is done,
 it should trigger (and ''not'' wait for) a portwatcher build for each OS.

 Portwatcher needs to be enhanced to be able to distinguish between being
 triggered from a GitHub notification and being forced from the web
 interface. When forced from the web interface, portwatcher should trigger
 a mirror job as it does now. When notified from GitHub, it should not.

 I know work has already been done on a buildbot 2 setup. Does this work
 already address this issue?

-- 
Ticket URL: <https://trac.macports.org/ticket/59286>
MacPorts <https://www.macports.org/>
Ports system for macOS