buildbot build time

[Craig Treleaven]

> On Oct 25, 2017, at 7:45 PM, Mojca Miklavec <mojca at macports.org> wrote:
> 
> On 25 October 2017 at 16:23, Ryan Schmidt wrote:
>> On Oct 25, 2017, at 03:28, db wrote:
>>> On 25 Oct 2017, at 07:35, Mojca Miklavec wrote:
>>>> On 23 October 2017 at 10:40, db wrote:
>>>>> On 9 Oct 2017, at 12:46, Mojca Miklavec wrote:
>>>>>> That info is easy to read and collect from the buildbot. The keyword is: "to collect".
>>>>> I checked https://build.macports.org/json/help, but couldn't find it.
>>>> For example:
>>>> - https://build.macports.org/json/builders/ports-10.13_x86_64-builder/builds/10627?as_text=1
>>>> which corresponds to
>>>> - https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/10627
>>>> 
>>>> The idea would be to write a script to iterate through all numbers
>>>> from 1 on and store results somewhere.
>>>> 
>>>> If you want the time estimate specifically, here might be the relevant
>>>> information:
>>>> 
>>>>    "name": "install-port",
>>>> ...
>>>>    "times": [
>>>>      1508901749.648058,
>>>>      1508901850.387383
>>>>    ]
>>>> 
>>>> I guess the difference represents the seconds spent building the port.
>>> 
>>> Thanks. I was searching for "to collect" there and in base, to no avail.
> 
> "To collect" was not meant as a literate strings to search for, but as
> the main action that needs to be done.
> 
>>> Those are the start/end epochs that I need to subtract, for a single port.
>>> 
>>> How can I correlate portname to buildnumber?
> 
> You cannot at the moment.
> 
> What I envision is a script that would run once from number 1 on and
> then either daily or hourly to update for new builds. The script would
> then construct a database of builds and do some simple visualisation.
> The most important part would be to ask some simple app to return data
> about one particular port. I would expect it to return the list of all
> builds (including URLs) for that particular port, but most
> importantly, for each buildslave:
> - whether the last build succeeded or failed
>  - (if failed, whether it was a dependency or the port installation
> that failed)
> - commit timestamp / shasum & version of the port
> - whatever else seems relevant, potentially including the build time
> and size of the package
> 
>> I don't think scraping information from buildbot json files is the right way to collect this data. If we want to collect it, the buildbot scripts could be modified to do so.
> 
> I don't see any big difference. The important part would be to do it
> in the first place. How to do it most efficiently can be a subject for
> discussion, but parsing json files with a library seems the easiest
> approach to me, in particular because we are currently unable to
> modify the buildbot script to return us data about packages built in
> the past. We can always modify the build scripts later.
> 
>> But as I said before, I don't think the information is useful to collect.
> 
> Build time doesn't seem *the most* interesting part of information,
> but it is useful to know in any case.
> 
>> Besides the fact that the buildbot worker machines have different hardware specs than user hardware, not all of the buildbot workers run on the same hardware. A build will go faster on our High Sierra worker than on the Sierra worker, because the High Sierra worker has a faster processor. In addition, the buildbot workers are virtual machines, with multiple VMs running on a single host. So a build may be faster or slower depending on whether the other VMs that run on the same host are busy or not.
>> 
>> If all you want to do is display on a web site somewhere that a particular build of a port took a particular amount of time on our buildbot workers, ok. But I wouldn't try to use this information to, for example, modify MacPorts base to have it offer the user a prediction of how long the port will take to build on their system, because I don't think we can accurately predict that.
> 
> I would definitely not use it for that purpose. But having *some
> statistics* about which ports built and failed on which builder would
> be super super useful. What I miss most is quickly finding the build
> where a particular port failed to build. At the moment it's nearly
> impossible to find it.

If we’re going to do this, can we be a little more precise about “failed” builds?  

1) Any port that relies on a non-default variant “fails” on the buildbots.  All of my myth* ports will ALWAYS show a “failed” status on ALL of the buildbots for this reason.   I would rather that users were told that the build was “not attempted”.  Telling them that build “failed" isn’t helpful.

2) Any port that needs C++11 “fails” on the existing 10.7 and earlier buildbots.

3) Various ports have a pre-fetch block that checks the OS version and “fails” if it is not supported.

Assuming the above can be fixed, is the following a mock-up of the data we’d collect?  Each commit to the ports tree triggers builds on the then-current fleet of builders:

Port:    blurfl 
Verson:  2.54.1_0
Commmit: https://github.com/macports/macports-ports/commit/6b120d678245ba6f14a5a364f83c3c5970f762af

Buildbots:
Builder     Build ID    Build (minutes.seconds)
X86-10.13   nnnnnnnn        1.20
X86-10.12   nnnnnnnn        2.20
X86-10.11   nnnnnnnn        2.00
X86-10.10   nnnnnnnn        4.50
X86-10.9    nnnnnnnn        2.40
X86-10.8    nnnnnnnn        fail
X86-10.7    nnnnnnnn        not supported
X86-10.6    nnnnnnnn        not supported
PPC-10.5    nnnnnnnn        not supported

Craig