Gsoc 18 Project | Collect build statistics

[Vishnu]

Also i wanted to know.Like if a new port is created .Buildbot is
compulsorily run right?
and if it is deleted.There is no need for buildbot to run.So buildbot wont
send updates to the log that port has been deleted.
In that case i want to know how / where is the port deleted from.What code ?
Because i would have to do changes there.That as soon as port is deleted
log of that port is updated.

" Buildbot master and this website will live on a different server. How
exactly do you plan to update the log file? "
In that case. I would need to make a mediator(present on same server as
that of website) of some sort. Wherein requests would be sent to that
mediator. And that mediator would inturn write in the log files.

or i think that would be hectic.
It would be best to have the logs on the Buildbot master side.Because every
time buildbot is used it can easily update the log at that time only.No
need for continuously sending stuff to mediator as i previously planned
above.
As at max the logs(history) required by the web app is 20 logs.It can make
a request to the backend to get them.

I think this would be better.

Also is backend and your website lives on different servers ?
How does the interaction happen ?

On 26 March 2018 at 15:09, Vishnu <vishnum1998 at gmail.com> wrote:

> " I just wanted to say that in the long term it's more important to make
> sure that new builds are stored soon after they finish, but it's
> probably much easier to start with historical data (and it's also
> super useful in the short term), so that you have immediately
> something to work with. "
>
> yes i definitely agree with you.It would be best to first work upon
> history.That will allow me to understand the code more.
>
> Can you help me with where does that json api work from?where is the code
> that sends request to the backend.To get that data.
> Once i get to know that it will simplify my job to send request to backend
> for port
>
> Also is there any existing method to show all ports of a maintainer?
> or to show all ports of a category?say python?
>
>
> It would be best if i start working gmy gsoc with 3 most important things .
> 1) Basic Port information through portindex2json
> 2) Port Build history. It would use JSON API
> 3) Buildbot code updation to send updates to the logs.
>
>
> And as you said logs would be ineffective to store .For history of builds
> of the port.I think logfile would be the bestway to store history.
> And regarding the syntax . I was thinking of storing the build history in
> json format in one line in log.txt of a particular port.
>
> Also mojca can i know your working hours.or atleast the timezone where you
> live.
> I will try to do most of the work on proposal during that time.Because
> only one day is left and a lot to change.
>
> Thanks
> Vishnu
>
>
>
>
>
> On 26 March 2018 at 05:46, Mojca Miklavec <mojca at macports.org> wrote:
>
>> Dear Vishnu,
>>
>> On Mon, Mar 26, 2018 at 2:58 AM, Vishnu wrote:
>> >
>> > Hi,
>> >
>> > I wanted some help in build bot summary.
>> > I am not sure i completely understand what's happening here.
>> > Could you please explain me about buildbot summary more?
>> >
>> > what i understand is :
>> > There can be n number of builds for each port.
>> > So how am i supposed to get the information of what all builds have
>> > already been done for that particular port.Where can i find that history
>> > from? from build.macports.org ?
>>
>> Yes, that information can be obtained from build.macports.org.
>>
>> > First and foremost I think first i have to implement the Buildbot idea
>> > .That would create logs.
>> > But for that idea..how to get all the exiting build history of
>> ports?Build
>> > history such as timestamp,os,Succesful or not?
>> > Where to get exclusive information about a port.
>>
>> You cannot get that at the moment.
>> That's what the whole idea is about: to implement the functionality
>> that will let developers query a single port.
>>
>> Take a look at
>>     https://build.macports.org/builders/ports-10.13_x86_64-builder
>>
>> You'll see a list of builds. The one currently being built is
>>     https://build.macports.org/builders/ports-10.13_x86_64-build
>> er/builds/21976
>> for the port "sumo".
>>
>> The task is then to go back to the history by decreasing that number
>> (21976) one by one and fetching information from each individual build
>> until you reach number 1 or the first build where the information is
>> no longer available. I believe the build logs get deleted after cca.
>> 10.000 builds and we have 9 builders, so you would end up querying
>> roughly 100k entries. That could take a couple of days and might need
>> some coordination with Ryan, but you can start with as little as two
>> entries per builder.
>>
>> This information can be queried via JSON interface
>> (https://build.macports.org/json/help). You can infer OS version from
>> the build slave name. Timestamp, port name, whether the build was
>> successful etc. are all available in that JSON API, but there might be
>> some information missing which you can only get by reading additional
>> files/logs.
>>
>> For example,
>>     https://build.macports.org/builders/ports-10.13_x86_64-build
>> er/builds/21973/steps/install-port/logs/files
>> will give you the list of files installed by the package. I don't know
>> if we want to store that particular information or not.
>> And
>>     https://build.macports.org/builders/ports-10.13_x86_64-build
>> er/builds/21973/steps/gather-archives/logs/stdio
>> might give you information about whether the tarball was uploaded or not.
>>
>> It might be necessary to adapt/improve the buildbot setup to make some
>> variables more easily accessible. At the moment it's pretty cumbersome
>> to extract the version information, for example. Assigning the correct
>> commit that lead to this build would also need some additional work.
>>
>>
>> I just wanted to say that in the long term it's more important to make
>> sure that new builds are stored soon after they finish, but it's
>> probably much easier to start with historical data (and it's also
>> super useful in the short term), so that you have immediately
>> something to work with.
>>
>>
>> The part "A" (buildbot idea) I initially mentioned was meant to
>> simplify such queries. If we extended the buildbot (configuration),
>> this might enable querying a single port (making the need for a
>> separate website like the one in proposal slightly less urgent, but
>> not irrelevant as we still want installation statistics etc.).
>>
>> Mojca
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20180326/db023000/attachment.html>