Build servers offline due to failed SSD

[Ryan Schmidt]

On Mar 7, 2021, at 00:20, Dave C wrote:

> Isn’t SSD a bad choice for server duty?

My opinion is that it is a good choice in terms of performance. When I first up this incarnation of our buildbot system in 2016 I had the workers running on SSDs so that builds would be fast (our previous buildbot setup at Apple's macOS forge used a very expensive very-many-hard-disk RAID; we were in no position to purchase any equivalent type of hardware once we left macOS forge) and I had the master and distfiles/packages storage on a hard disk RAID for reliability. The specific RAID that I have turned out to be too slow. Web requests could take many seconds to respond. GitHub Web Hooks being delivered to the server could be marked as failed because GitHub didn't always wait long enough for our server to respond. This was unsatisfying so I moved the buildmaster to an old SSD while keeping the large files on the RAID. This was much faster, though not as fast as if I had used a new SSD, which is what I will ultimately be using. For now, the buildmaster is temporarily running off a USB hard drive and is slow as molasses. This is a terrible choice but all drive bays are already occupied by the RAID.

All of the SSDs we used for the workers have failed as well, 2 last year and the last one last month. In response to these failures, someone else also suggested that we should not use SSDs. I've run one of the workers off of three independent hard disks for the past year, and my opinion is that the performance and power consumption of SSDs is much better and I will switch the hard disk-based worker back to an SSD in the future. You can read this discussion here:

https://trac.macports.org/ticket/60178