Building all ports when setting up a new buildbot builder

[Joshua Root]

On 2014-12-2 21:57 , Ryan Schmidt wrote:
> So after 99 hours, the Yosemite builder failed building "all" ports, with a memory error:
> 
> https://build.macports.org/builders/buildports-yosemite-x86_64/builds/71/steps/compile/logs/err.text
> 
> I think we should refrain from trying to build all 20,000 ports at once. It takes over a week anyway. We should instead build smaller sets of ports at a time. Since archives are not uploaded to the packages server until the entire set of ports has been built, building smaller sets of ports will let some packages become available sooner.

It's also less efficient because they are not built in dependency order,
so skipping ports whose dependencies failed is affected. If a lot of
ports depend on something that takes a long time to build and is failing
partway through, it can really add up.

(BTW, you do realise there are already quite a lot of archives for
Yosemite uploaded?)

> Here are some reasonably-sized chunks we could have the builder build:
> 
> $ port echo name:^[0-9a-d]|wc -l
>     1120
> $ port echo name:^[e-h]|wc -l
>     1316
> $ port echo name:^[i-l]|wc -l
>     1319
> $ port echo name:^[m-o]|wc -l
>     1005
> $ port echo name:^[q-s]|wc -l
>     1115
> $ port echo name:^[t-z]|wc -l
>     1078
> $ port echo name:^p[a-z]|wc -l
>     5761
> $ port echo name:^p[0-9]|wc -l
>     7362
> 
> I am assuming that if we set the portlist to e.g. "name:^[0-9a-d]" instead of "all" that it would work.

No, it won't. You can't use regexes and it's not a list of port names,
it's a list of portdirs. All subports of the portfile in each listed
portdir are built.

Doing multiple smaller runs isn't a bad idea, but it's not as simple as
you seem to think. It requires significant additions to multiple parts
of the system.

- Josh