Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

On 24 April 2018 at 00:45, Vishnu wrote:
>
>> (c.3) Plan the API to get the data from the database in JSON format
>> (so that someone else could write an independent app with the same
>> display functionality).
>> More in a separate email, I guess.
>
> (c.3) You are talking about Build History data ..right?

No. After the proposal deadline was already over, Bradley pointed out
that it would be really helpful if the website would also *serve* the
API rather than just read if from elsewhere.

That is: the fact that Buildbot has a pretty nice API (version one has
a way more extensive one than version 0.8) allows us to easily access
some of its data, otherwise we might have needed to parse raw HTML.

When you create a view to, say, display all ports in the math
category, you would probably serve the page via something like
    https://www.macports.org/ports/list/category/math
which would render a nice website with hyperlinks to all ports in this
category. It would be cool if the website would also allow something
like
    https://www.macports.org/api/list?category=math
that would basically provide the same piece of information, only in
json format instead of rendered html.

Note that I'm making up the exact URLs.

The rationale would be that anyone who might need some basic
information about the ports could easily access the data. What Bradley
envisioned in the long run is that provided how much the frontend
technologies change, if we had a solid database and API, anyone could
simply create a one-page javascript file at some point in future and
get an awesome site without having to touch the backend. Or maybe we
would need part of this information to render views on the buildbot.
Or perhaps "port arethereanyknownproblems python27" could at some
point in future tell the user whether one can expect a failure etc.

I imagine that this should be possible to without too much additonal
effort, but we can see.

By far the most important part of the website is to have a solid
database and data collection strategies in the background to allow
efficiently querying various data. Even if the frontend won't be
perfect just yet, that's always possible to change later.

You will certainly need to fetch and interpret various JSON data from
various sources. This addition would be about being able to generate
one. But we can still discuss that.

> There is already a JSON API for it as you said..so cant we use it? (
> https://build.macports.org/json/help )

Yes, we can (and will) use that one. But we would need to create one ourselves.

See also http://docs.buildbot.net/latest/developer/rest.html for the
latest Buildbot API.

> (e.1) I think when i will be changing Buildbot to update the database after
> every build ..Then i would need Tcl.

Not really. Buildbot itself is written in python. There's no Tcl
involved in that step.
One thing about the buildbot: we need to be careful not to invest too
much effort into buildbot 0.8 because plenty of stuff changes with
buildbot 1.x. It's certainly vital to be able to collect build
statistics for the old builds, and the script can run once per hour or
so, but when it comes to interaction, we should probably check if we
could perhaps update our own buildbot to version 1.1 before starting
implementing too meany features that are specific to one particular
buildbot version.

> Also i have my Semester End exams from 27 April - 7 may.
> So maybe i would not be able to reply on time.

Good luck with your exams. Do them well, in order to get more free
time and less worries during the summer :)

In this case I suggest that you try to get at least a working MacPorts
installation before exams and we get back to planning with full speed
after your exams are done.

Mojca