buildbot build time
ctreleaven at cogeco.ca
Thu Oct 26 01:09:43 UTC 2017
> On Oct 25, 2017, at 7:45 PM, Mojca Miklavec <mojca at macports.org> wrote:
> On 25 October 2017 at 16:23, Ryan Schmidt wrote:
>> On Oct 25, 2017, at 03:28, db wrote:
>>> On 25 Oct 2017, at 07:35, Mojca Miklavec wrote:
>>>> On 23 October 2017 at 10:40, db wrote:
>>>>> On 9 Oct 2017, at 12:46, Mojca Miklavec wrote:
>>>>>> That info is easy to read and collect from the buildbot. The keyword is: "to collect".
>>>>> I checked https://build.macports.org/json/help, but couldn't find it.
>>>> For example:
>>>> - https://build.macports.org/json/builders/ports-10.13_x86_64-builder/builds/10627?as_text=1
>>>> which corresponds to
>>>> - https://build.macports.org/builders/ports-10.13_x86_64-builder/builds/10627
>>>> The idea would be to write a script to iterate through all numbers
>>>> from 1 on and store results somewhere.
>>>> If you want the time estimate specifically, here might be the relevant
>>>> "name": "install-port",
>>>> "times": [
>>>> I guess the difference represents the seconds spent building the port.
>>> Thanks. I was searching for "to collect" there and in base, to no avail.
> "To collect" was not meant as a literate strings to search for, but as
> the main action that needs to be done.
>>> Those are the start/end epochs that I need to subtract, for a single port.
>>> How can I correlate portname to buildnumber?
> You cannot at the moment.
> What I envision is a script that would run once from number 1 on and
> then either daily or hourly to update for new builds. The script would
> then construct a database of builds and do some simple visualisation.
> The most important part would be to ask some simple app to return data
> about one particular port. I would expect it to return the list of all
> builds (including URLs) for that particular port, but most
> importantly, for each buildslave:
> - whether the last build succeeded or failed
> - (if failed, whether it was a dependency or the port installation
> that failed)
> - commit timestamp / shasum & version of the port
> - whatever else seems relevant, potentially including the build time
> and size of the package
>> I don't think scraping information from buildbot json files is the right way to collect this data. If we want to collect it, the buildbot scripts could be modified to do so.
> I don't see any big difference. The important part would be to do it
> in the first place. How to do it most efficiently can be a subject for
> discussion, but parsing json files with a library seems the easiest
> approach to me, in particular because we are currently unable to
> modify the buildbot script to return us data about packages built in
> the past. We can always modify the build scripts later.
>> But as I said before, I don't think the information is useful to collect.
> Build time doesn't seem *the most* interesting part of information,
> but it is useful to know in any case.
>> Besides the fact that the buildbot worker machines have different hardware specs than user hardware, not all of the buildbot workers run on the same hardware. A build will go faster on our High Sierra worker than on the Sierra worker, because the High Sierra worker has a faster processor. In addition, the buildbot workers are virtual machines, with multiple VMs running on a single host. So a build may be faster or slower depending on whether the other VMs that run on the same host are busy or not.
>> If all you want to do is display on a web site somewhere that a particular build of a port took a particular amount of time on our buildbot workers, ok. But I wouldn't try to use this information to, for example, modify MacPorts base to have it offer the user a prediction of how long the port will take to build on their system, because I don't think we can accurately predict that.
> I would definitely not use it for that purpose. But having *some
> statistics* about which ports built and failed on which builder would
> be super super useful. What I miss most is quickly finding the build
> where a particular port failed to build. At the moment it's nearly
> impossible to find it.
If we’re going to do this, can we be a little more precise about “failed” builds?
1) Any port that relies on a non-default variant “fails” on the buildbots. All of my myth* ports will ALWAYS show a “failed” status on ALL of the buildbots for this reason. I would rather that users were told that the build was “not attempted”. Telling them that build “failed" isn’t helpful.
2) Any port that needs C++11 “fails” on the existing 10.7 and earlier buildbots.
3) Various ports have a pre-fetch block that checks the OS version and “fails” if it is not supported.
Assuming the above can be fixed, is the following a mock-up of the data we’d collect? Each commit to the ports tree triggers builds on the then-current fleet of builders:
Builder Build ID Build (minutes.seconds)
X86-10.13 nnnnnnnn 1.20
X86-10.12 nnnnnnnn 2.20
X86-10.11 nnnnnnnn 2.00
X86-10.10 nnnnnnnn 4.50
X86-10.9 nnnnnnnn 2.40
X86-10.8 nnnnnnnn fail
X86-10.7 nnnnnnnn not supported
X86-10.6 nnnnnnnn not supported
PPC-10.5 nnnnnnnn not supported
More information about the macports-users