Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Hi,

Please keep any technical discussion (that doesn't include
private/sensitive data) on the mailing list.

On 25 March 2018 at 00:42, Vishnu wrote:
>
> And regarding the individual port page.
> I was thinking whenever someone opens the port page.The content is generated
> at that time and displayed.

True.

> So everytime port page is opened my scraper would scrape the build summary.

No way. You would jam both your website and the buildbot that way.
The build logs get deleted after a while (I think after 10k builds and
a single revbump of perl version in a single commit might trigger more
than 1000 builds), but more importantly you have absolutely no idea
where to look for the builds (unless the buildbot idea gets
implemented) which is why we put that idea on the list in the first
place.

Also, you cannot consult each user who opted in to participate at
statistics collection to tell you whether they have a particular port
installed each time when some user clicks on the website ...

> Or are you saying to create such a framework that will scrape all the ports
> only once(in forever) and create a port log?

Before it gets deployed it needs to collect information about all
ports (once), but then it needs to keep monitoring everything, after
each commit and after each build, and update the data accordingly.

> And whenever a new build is done Base code would add to the website log of
> the port .?
>
> Also how to know about binary package status?

See an example where a package has been uploaded:
    https://build.macports.org/builders/ports-10.11_x86_64-builder/builds/51073/steps/gather-archives/logs/stdio
and an example where there were "issue":
    https://build.macports.org/builders/ports-10.9_x86_64-builder/builds/51949/steps/gather-archives/logs/stdio
You should of course also consult a page like
    http://packages.macports.org/python27/
and make sure that both sources match. For old files on
packages.macports.org you won't have the logs, but it's helpful to
keep the information on the website. The most important piece of
information is when an old version built and the new one no longer
does, or only builds on a different os version etc.

Mojca