Built archive not uploaded

[Ryan Schmidt]

On Mar 27, 2021, at 20:51, Jason Liu wrote:

> On Sat, Mar 27, 2021 at 1:44 PM Mojca Miklavec wrote:
> 
>> I didn't try to look into details, but note that the 10.15 builder has
>> more than 500 jobs in the backlog, so it might also be just a matter
>> of time.
> 
> Thanks for pointing this out... I had no idea that the 10.15 builder had such a huge backlog of pending builds.

It was offline for a month and is still catching up. First the SSD holding the 10.15, 10.9 and 10.6 x86_64 workers died. I replaced the SSD and restored the 10.9 and 10.6 x86_64 workers but then I had a minor hand injury that made it uncomfortable to work on the computer. That healed, and then we had a winter storm during which the servers were offline either because there was no power or because the power company was asking everyone to conserve power. After power was restored, the buildmaster's SSD died and I worked on replacing and restoring that. And then finally I got the 10.15 worker back online. It has finished the builds for the commits that happened during the downtime, but has not yet finished the builds for the commits that happened subsequently. There are currently 493 portwatcher builds in queue. The next one in the queue was scheduled 270 hours ago. Each portwatcher build can expand to zero or more portbuilder builds, so there are probably more than 493 ports to be built before we're caught up.