[MacPorts] #54247: buildbot buildmaster is sometimes very slow to respond

[MacPorts]

#54247: buildbot buildmaster is sometimes very slow to respond
----------------------------+---------------------
 Reporter:  ryandesign      |      Owner:  admin@…
     Type:  defect          |     Status:  new
 Priority:  Normal          |  Milestone:
Component:  server/hosting  |    Version:
 Keywords:                  |       Port:
----------------------------+---------------------
 The [https://build.macports.org buildbot buildmaster] is sometimes
 extremely slow to respond. We use nginx as a reverse proxy in front of the
 buildmaster, and buildmaster's slowness sometimes exceeds nginx's default
 proxy timeouts, causing nginx to respond with a `502 Bad Gateway` or `504
 Gateway Time-out` error. Not only does this cause developers to be unable
 to use the buildbot web interface at times, it also prevents GitHub from
 successfully delivering push notifications, so some ports might not be
 getting built.

 Not sure whether the notifications aren't getting to the buildmaster, or
 whether the buildmaster just isn't responding in time for GitHub to
 consider it a success, but I suspect the latter.

 Not sure whether it's the `proxy_connect_timeout` or `proxy_read_timeout`
 or `proxy_send_timeout` or a combination of these but I've increased them
 all, so now we shouldn't see any more 502 or 504 errors; the requests
 should complete, if given enough time. This hasn't helped GitHub
 successfully send its push notifications. GitHub seems to have its own
 built-in timeout for sending these notifications.

 I've gone through the entire history of GitHub push notifications up to
 now and redelivered those that failed, so any ports that previously
 weren't built because of this should now be built.

 I suspect the cause is that the RAID the buildmaster is on is too slow. It
 seems quick enough when tested with Blackmagic Disk Speed Test, but I have
 a feeling that the buildmaster workload plus the periodic syncing via
 rsync makes random disk accesses slow. When the web interface is
 responding slowly, accessing the server over ssh also feels sluggish.

 The server does have an unused Apple SSD. I could try moving the
 buildmaster folder to the SSD; that would be fairly easy to do and would
 let us know whether disk speed is a factor. Another option is to move the
 entire OS and everything except the rsync directory to the SSD. That's
 more difficult to do and will involve more downtime, and the reason I
 hadn't done so initially is that I wanted the assurance of a RAID for our
 critical infrastructure. Some parts of macOS Server, such as the Caching
 Server, also seem particularly unhappy about storing their data on a disk
 that is not the startup volume, and default back to storing their data on
 the startup volume after a restart. But the caching service uses less disk
 space than I had anticipated so there should be room to store that data on
 the SSD.

--
Ticket URL: <https://trac.macports.org/ticket/54247>
MacPorts <https://www.macports.org/>
Ports system for macOS