build.macports.org down?

[Ryan Schmidt]

On Jan 19, 2021, at 18:17, Ryan Schmidt wrote:

> On Jan 19, 2021, at 14:22, Christopher Nielsen wrote:
> 
>> This isn’t urgent - I was simply interested in checking the build backlog - but can’t get a response from build.macports.org.
>> 
>> Earlier I was seeing a “Bad Gateway” message via web browser, now it’s simply timing out with a blank page.
> 
> The buildmaster's sqlite database has become corrupted. I have stopped the buildmaster process so you will now see the "Bad Gateway" message if you visit the page. I will investigate if we can salvage the database. We had this problem before in February 2020 and in May 2020.
> 
> https://lists.macports.org/pipermail/macports-dev/2020-May/041882.html
> 
> At those times, we deleted the database and allowed the buildmaster to create a new one. This does not affect the history of completed builds, which are not stored in the database, but we would lose the list of scheduled builds and would have to reschedule them. At this time there was a huge backlog of scheduled builds due to the recent jpeg-to-libjpeg-turbo switchover plus an additional backlog before that on the 10.14 and 11 x86_64 builders, so it will be a bit of work to reschedule everything.
> 
> I also noticed the mprsyncup job regenerating all portindexes, which is unusual. Some of the PortIndex.quick and PortIndex.json files that got generated have 0 size, which is wrong. We had this problem before in November 2020.
> 
> https://trac.macports.org/ticket/61507
> 
> Previously when we've had these problems I thought a high server uptime contributed to the problem. But the uptime now is only 13 days. Still, I'll begin by restarting the server and running disk first aid.

Disk first aid showed no major problems but buildbot's sqlite3 database was unrecoverable so we'll start over with a new database. I'll be scheduling builds for anything that was in the queue due to commits, but if anyone had put any forced builds into the queue manually those are lost so you should do it again if those builds are still needed.