build.macports.org unavailable again

[Joshua Root]

On 2013-12-8 00:46 , Ryan Schmidt wrote:
> 
> On Dec 1, 2013, at 17:01, Joshua Root wrote:
> 
>> Are the cleaners coming in every weekend and unplugging the server or
>> something? :)
> 
> It’s unavailable again, and it is the weekend again, which is either a coincidence or a relevant diagnostic…
> 
> But Joshua I’m wondering if the “all ports” build you’re initiating is just too much for the new Mavericks buildbot to handle all at once and it’s bringing down the rest of the system — before the December 1 outage, the build had been running non-stop for 8 days; I don’t know how far it got in that time. Or possibly one specific port is causing the problem, and it just takes awhile to reach it.

It got further on the second run so I doubt it's a specific port. Ports
that were already built are skipped, and ports that failed previously
are skipped during 'all' runs unless the portfile has been modified since.

The buildmaster went down, not just the slave. (Again, there is no
"Mavericks buildbot", there is a Mavericks buildslave.) It hadn't got to
uploading any files yet so the only effect the slave could have on the
master is via log files and status reporting. The log is big, but it
seems unlikely it would be dumb enough to try to load the whole thing
into memory at once. We never had problems previously with logs from
full runs that would have been bigger than the partial runs here.

> During the build, none of the archives that are being built are being uploaded to the packages server; it’s crummy to have an archive built but then not uploaded and made available to users until over a week later.
> 
> Instead of building all ports, maybe we could have the buildbot build smaller subsets of ports. That way the packages would be available to users sooner. Can we use normal port selectors with the buildbot? For example, only ports with names beginning with letter a (“name:^a”), or only ports in category python (“category:python”).

There isn't support for that currently. When building everything it's
important to do it in dependency order as per genportlist.tcl (used
internally by mpab when doing 'all'), so that ports can be skipped when
a dependency failed, and also so the progress info is more accurate. You
could generate that list locally and paste segments of it into the force
build form, but the problem is subports. The portlist is normally
generated by looking which portfiles were modified by each commit and
adding their parent directory to the list. When you specify a portlist
manually that is still what it's meant to be, a list of portdirs rather
than port names. If you include something else, like the name of a
subport, it doesn't work right (generally in the form of building the
subports multiple times). If you remove all the subports from the list,
you won't always get the correct dependency order.

Anyway, it was very close to the end last time, so I don't see why it
wouldn't finish relatively quickly now.

- Josh