buildbot build time

[Ryan Schmidt]

> On Oct 25, 2017, at 18:45, Mojca Miklavec <mojca at macports.org> wrote:
> 
> On 25 October 2017 at 16:23, Ryan Schmidt wrote:
>> On Oct 25, 2017, at 03:28, db wrote:
>>> On 25 Oct 2017, at 07:35, Mojca Miklavec wrote:
>>>> On 23 October 2017 at 10:40, db wrote:
>>>>> On 9 Oct 2017, at 12:46, Mojca Miklavec wrote:
>>>>>> That info is easy to read and collect from the buildbot. The keyword is: "to collect".
>>>>> I checked https://build.macports.org/json/help, but couldn't find it.
>>>> For example:
>>>> - https://build.macports.org/json/builders/ports-10.13_x86_64-builder/builds/10627?as_text=1
>>>> which corresponds to
>>>> - https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/10627
>>>> 
>>>> The idea would be to write a script to iterate through all numbers
>>>> from 1 on and store results somewhere.
>>>> 
>>>> If you want the time estimate specifically, here might be the relevant
>>>> information:
>>>> 
>>>>    "name": "install-port",
>>>> ...
>>>>    "times": [
>>>>      1508901749.648058,
>>>>      1508901850.387383
>>>>    ]
>>>> 
>>>> I guess the difference represents the seconds spent building the port.
>>> 
>>> Thanks. I was searching for "to collect" there and in base, to no avail.
> 
> "To collect" was not meant as a literate strings to search for, but as
> the main action that needs to be done.
> 
>>> Those are the start/end epochs that I need to subtract, for a single port.
>>> 
>>> How can I correlate portname to buildnumber?
> 
> You cannot at the moment.
> 
> What I envision is a script that would run once from number 1 on and
> then either daily or hourly to update for new builds. The script would
> then construct a database of builds and do some simple visualisation.
> The most important part would be to ask some simple app to return data
> about one particular port. I would expect it to return the list of all
> builds (including URLs) for that particular port, but most
> importantly, for each buildslave:
> - whether the last build succeeded or failed
>  - (if failed, whether it was a dependency or the port installation
> that failed)
> - commit timestamp / shasum & version of the port
> - whatever else seems relevant, potentially including the build time
> and size of the package
> 
>> I don't think scraping information from buildbot json files is the right way to collect this data. If we want to collect it, the buildbot scripts could be modified to do so.
> 
> I don't see any big difference. The important part would be to do it
> in the first place. How to do it most efficiently can be a subject for
> discussion, but parsing json files with a library seems the easiest
> approach to me, in particular because we are currently unable to
> modify the buildbot script to return us data about packages built in
> the past. We can always modify the build scripts later.

True, modifying the build scripts wouldn't help collect data about past builds. We would need to process past builds separately.


>> But as I said before, I don't think the information is useful to collect.
> 
> Build time doesn't seem *the most* interesting part of information,
> but it is useful to know in any case.
> 
>> Besides the fact that the buildbot worker machines have different hardware specs than user hardware, not all of the buildbot workers run on the same hardware. A build will go faster on our High Sierra worker than on the Sierra worker, because the High Sierra worker has a faster processor. In addition, the buildbot workers are virtual machines, with multiple VMs running on a single host. So a build may be faster or slower depending on whether the other VMs that run on the same host are busy or not.
>> 
>> If all you want to do is display on a web site somewhere that a particular build of a port took a particular amount of time on our buildbot workers, ok. But I wouldn't try to use this information to, for example, modify MacPorts base to have it offer the user a prediction of how long the port will take to build on their system, because I don't think we can accurately predict that.
> 
> I would definitely not use it for that purpose. But having *some
> statistics* about which ports built and failed on which builder would
> be super super useful. What I miss most is quickly finding the build
> where a particular port failed to build. At the moment it's nearly
> impossible to find it.

I agree with all of that. I had not considered that those issues might also be addressed by this process.