Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 24 March 2018 at 22:50, Vishnu wrote:
> Well that's really nice.
> I'll be emailing you the proposal very soon.
> I would really like to work on creating dynamic websites for each port with
> one framework.
> I think that is what you are also working on currently.Or is it something
> else?(Umesh Sir)
>
> I would like to go with option B.
>
> What i understand is .Basically, i will be creating a dynamic
> website/framework (use some scripting language) to generate/gather all the
> following information for any port that is requested.
>
>
> - all the information about the port (description, version,
> maintainer, homepage, variants, dependencies, dependent ports, ...)
> - build summary (as scraped from buildbot history): which version was
> built at what time on which os, successful or not, ...
>   - including whether or not a binary package exists
> - installation statistics (from users who opted in and send a json
> file every now and then) with some graphs, similar to the stats page
> above
> - results of "livecheck": whether there's a newer version of the port
> available
> - short git log with links to the last few changes to the Portfile
> - links to trac tickets for that port
>
> https://trac.macports.org/query?0_port=python27&0_port_mode=%7E&0_status=%21closed
> - (maybe links to pull requests for that port)
>
> Is my understanding of the project correct?
>
> Now to generate:
> 1) description, version,maintainer, homepage, variants, dependencies,
> dependent ports,etc:
> I will need to update this script
> https://github.com/g5pw/macports-port-tree made by Aljaz for the remaining
> missing information.because not all information is produced by this tool.

This source of information PortIndex which can be generated from
https://github.com/macports/macports-ports (that's part of MacPorts
installation process, it should be possible to get it under linux as
well, but I didn't try, we also mirror it).

PortIndex is some data structure in Tcl which can be converted into
JSON with an existing script postindex2json, but ideally the Tcl
script should be fixed to generate data in a better format.
This data would then need to be imported at the beginning, but the
application should keep accepting updates about new or deleted ports.

> 2) Build Summary: This can be generated by scraping through buildbot
> history.

For historic data. For the future it would be better to modify the
buildbot setup to submit data to the server.

> 3) Installation Statistics: Take help from Stats web app(Gsoc 2011 project)
>
> 4)Results of livecheck: I am not sure how to get that. i'll be needing help
> to how to get this.

Sample output from the package manager:

> port -v livecheck *
Bear seems to be up to date
CImg seems to have been updated (port version: 1.7.9, new version: 2.2.1)
CoreOSMakefiles seems to be up to date
Error: cannot check if Djinn was updated (regex didn't match)
Drizzle seems to have been updated (port version: 30, new version: 31)
FlowUI seems to be up to date
GASNet seems to have been updated (port version: 1.26.3, new version: 1.30.0)
GitX seems to be up to date
HaXml seems to have been updated (port version: 1.20.2, new version: 1.23.3)
Error: livecheck failed for LibVNCServer: extracted version '0.4' is
older than livecheck.version '0.9.9'
LucenePlusPlus seems to be up to date
Error: livecheck failed for MongoObjCDriver: extracted version 'test2'
is older than livecheck.version '1.0.6'
Error: cannot check if PlasmaShop was updated (regex didn't match)
ResIL seems to have been updated (port version: 1.7.9, new version: 1.8.3)
Error: cannot check if SilverCity was updated (regex didn't match)
SourceKitten seems to have been updated (port version: 0.17.6, new
version: 0.19.1)
Error: cannot check if VillainousStyle was updated (regex didn't match)
aap seems to be up to date

> 5) Link to track tickets : Can be done using scripting,html.
> 6)Pull request : Can be done.

Mojca