Building all ports when setting up a new buildbot builder

[Ryan Schmidt]

On Dec 2, 2014, at 8:39 AM, Joshua Root wrote:

> On 2014-12-2 21:57 , Ryan Schmidt wrote:
>> So after 99 hours, the Yosemite builder failed building "all" ports, with a memory error:
>> 
>> https://build.macports.org/builders/buildports-yosemite-x86_64/builds/71/steps/compile/logs/err.text
>> 
>> I think we should refrain from trying to build all 20,000 ports at once. It takes over a week anyway. We should instead build smaller sets of ports at a time. Since archives are not uploaded to the packages server until the entire set of ports has been built, building smaller sets of ports will let some packages become available sooner.
> 
> It's also less efficient because they are not built in dependency order,
> so skipping ports whose dependencies failed is affected. If a lot of
> ports depend on something that takes a long time to build and is failing
> partway through, it can really add up.

True, building smaller sets of ports is less efficient, because some ports will be encountered more than once, and it will take longer to get the complete set of ports built. However, smaller batches of ports are more likely to succeed, less likely to run into resource limitations, and it will result in some ports' binaries being published earlier, which would be better than waiting a month or however long it takes before building "all" ports finally runs without error. Each time we tell it to build "all" ports it has less to do since some ports succeeded last time, so we would eventually reach a point where "all" succeeds, it just feels like a messy solution, and a time-consuming one.


> (BTW, you do realise there are already quite a lot of archives for
> Yosemite uploaded?)

Yes, I am aware of that. But we are apparently still trying to build "all" ports on that builder, and it is failing, and I was pointing that out, and trying to suggest a solution.


>> Here are some reasonably-sized chunks we could have the builder build:
>> 
>> $ port echo name:^[0-9a-d]|wc -l
>>    1120
>> $ port echo name:^[e-h]|wc -l
>>    1316
>> $ port echo name:^[i-l]|wc -l
>>    1319
>> $ port echo name:^[m-o]|wc -l
>>    1005
>> $ port echo name:^[q-s]|wc -l
>>    1115
>> $ port echo name:^[t-z]|wc -l
>>    1078
>> $ port echo name:^p[a-z]|wc -l
>>    5761
>> $ port echo name:^p[0-9]|wc -l
>>    7362
>> 
>> I am assuming that if we set the portlist to e.g. "name:^[0-9a-d]" instead of "all" that it would work.
> 
> No, it won't. You can't use regexes and it's not a list of port names,
> it's a list of portdirs.

I see. But somehow the special keyword "all" is being handled; I assumed it was being passed on to `port` and that we could use any of the port-selection methods `port` allows. Is there a reason why it was not done that way?


> All subports of the portfile in each listed
> portdir are built.

Sure.


> Doing multiple smaller runs isn't a bad idea, but it's not as simple as
> you seem to think. It requires significant additions to multiple parts
> of the system.

I see.


On Dec 2, 2014, at 8:40 AM, Daniel J. Luke wrote:

> this seems like a reasonable workaround, but it might be nice if we could make building 'all' ports work

It seemed to be an out of memory error relating to stderr and stdout. I'm not surprised that the entire combined debug output of building all 20,000 ports might be a bit too much data to process all at once.


> (and maybe also set things up to upload archives as they become available?)

I don't know what that would involve, having not looked at the buildbot infrastructure that much. I just know that from the buildbot web interface, it's clear that uploading archives is currently implemented as a separate step, after building. (In fact, gathering archives is another separate step, currently.) So putting the gathering and uploading steps into the building step would involve rearranging some code.