Gsoc 18 Project | Collect build statistics

[Vishnu]

" I just wanted to say that in the long term it's more important to make
sure that new builds are stored soon after they finish, but it's
probably much easier to start with historical data (and it's also
super useful in the short term), so that you have immediately
something to work with. "

yes i definitely agree with you.It would be best to first work upon
history.That will allow me to understand the code more.

Can you help me with where does that json api work from?where is the code
that sends request to the backend.To get that data.
Once i get to know that it will simplify my job to send request to backend
for port

Also is there any existing method to show all ports of a maintainer?
or to show all ports of a category?say python?


It would be best if i start working gmy gsoc with 3 most important things .
1) Basic Port information through portindex2json
2) Port Build history. It would use JSON API
3) Buildbot code updation to send updates to the logs.


And as you said logs would be ineffective to store .For history of builds
of the port.I think logfile would be the bestway to store history.
And regarding the syntax . I was thinking of storing the build history in
json format in one line in log.txt of a particular port.

Also mojca can i know your working hours.or atleast the timezone where you
live.
I will try to do most of the work on proposal during that time.Because only
one day is left and a lot to change.

Thanks
Vishnu





On 26 March 2018 at 05:46, Mojca Miklavec <mojca at macports.org> wrote:

> Dear Vishnu,
>
> On Mon, Mar 26, 2018 at 2:58 AM, Vishnu wrote:
> >
> > Hi,
> >
> > I wanted some help in build bot summary.
> > I am not sure i completely understand what's happening here.
> > Could you please explain me about buildbot summary more?
> >
> > what i understand is :
> > There can be n number of builds for each port.
> > So how am i supposed to get the information of what all builds have
> > already been done for that particular port.Where can i find that history
> > from? from build.macports.org ?
>
> Yes, that information can be obtained from build.macports.org.
>
> > First and foremost I think first i have to implement the Buildbot idea
> > .That would create logs.
> > But for that idea..how to get all the exiting build history of
> ports?Build
> > history such as timestamp,os,Succesful or not?
> > Where to get exclusive information about a port.
>
> You cannot get that at the moment.
> That's what the whole idea is about: to implement the functionality
> that will let developers query a single port.
>
> Take a look at
>     https://build.macports.org/builders/ports-10.13_x86_64-builder
>
> You'll see a list of builds. The one currently being built is
>     https://build.macports.org/builders/ports-10.13_x86_64-
> builder/builds/21976
> for the port "sumo".
>
> The task is then to go back to the history by decreasing that number
> (21976) one by one and fetching information from each individual build
> until you reach number 1 or the first build where the information is
> no longer available. I believe the build logs get deleted after cca.
> 10.000 builds and we have 9 builders, so you would end up querying
> roughly 100k entries. That could take a couple of days and might need
> some coordination with Ryan, but you can start with as little as two
> entries per builder.
>
> This information can be queried via JSON interface
> (https://build.macports.org/json/help). You can infer OS version from
> the build slave name. Timestamp, port name, whether the build was
> successful etc. are all available in that JSON API, but there might be
> some information missing which you can only get by reading additional
> files/logs.
>
> For example,
>     https://build.macports.org/builders/ports-10.13_x86_64-
> builder/builds/21973/steps/install-port/logs/files
> will give you the list of files installed by the package. I don't know
> if we want to store that particular information or not.
> And
>     https://build.macports.org/builders/ports-10.13_x86_64-
> builder/builds/21973/steps/gather-archives/logs/stdio
> might give you information about whether the tarball was uploaded or not.
>
> It might be necessary to adapt/improve the buildbot setup to make some
> variables more easily accessible. At the moment it's pretty cumbersome
> to extract the version information, for example. Assigning the correct
> commit that lead to this build would also need some additional work.
>
>
> I just wanted to say that in the long term it's more important to make
> sure that new builds are stored soon after they finish, but it's
> probably much easier to start with historical data (and it's also
> super useful in the short term), so that you have immediately
> something to work with.
>
>
> The part "A" (buildbot idea) I initially mentioned was meant to
> simplify such queries. If we extended the buildbot (configuration),
> this might enable querying a single port (making the need for a
> separate website like the one in proposal slightly less urgent, but
> not irrelevant as we still want installation statistics etc.).
>
> Mojca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20180326/920ad755/attachment-0001.html>