Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 24 March 2018 at 23:55, Vishnu wrote:
> Thanks Mojca for your prompt reply.
> 1) description, version,maintainer, homepage, variants,
> dependencies,dependent ports,etc
> So for information of each port, my website has to scrape data from such
> pages :
>
> https://github.com/macports/macports-ports/blob/master/python/py-amqplib/Portfile
>
> is this correct?

No. The information is already "scraped" by PortIndex. This is what
portitndex2json script does:
    https://github.com/macports/macports-contrib/tree/master/portindex2json

{
    "subports"         : "py27-amqplib",
    "portdir"          : "python/py-amqplib",
    "description"      : "Simple non-threaded Python client library for AMQP.",
    "homepage"         : "https://code.google.com/p/py-amqplib/",
    "epoch"            : "0",
    "platforms"        : "darwin",
    "name"             : "py-amqplib",
    "depends_lib"      : "port:py27-amqplib",
    "license"          : "LGPL-2.1+",
    "long_description" : "{Simple non-threaded Python client library
for AMQP.}",
    "maintainers"      : "stromnov openmaintainer",
    "version"          : "1.0.2",
    "categories"       : "python devel",
    "revision"         : "0"
}

But the script for conversion should be improved to provide the data
in nicer format.

It would be helpful if you could run macports base and portindex on
your machine.

> 2) Build Summary:
>
> Would scraping buildbot history pose a problem?

If you don't parallelize the jobs and read full logs, I hope not. This
would only have to be done once anyway. If the server would suffer,
some artificial time delay could be added between queries.

> and "  modify the buildbot setup to submit data to the server. "
> isn't it aldready submitting data to the server?
> if not how is the buildbot history currently generated?

Buildbot master has the build data (in a completely unorganised-for-us
way) of course. But it would be better if the build master would
notify the website once each build is done rather than the website
trying to neg the buildbot for updates every few minutes.

Mojca