Gsoc 18 Project | Collect build statistics

[Vishnu]

Thanks Mojca for your prompt reply.
1) description, version,maintainer, homepage, variants, dependencies,dependent
ports,etc
So for information of each port, my website has to scrape data from such
pages :

https://github.com/macports/macports-ports/blob/master/python/py-amqplib/Portfile

is this correct?

2) Build Summary:

Would scraping buildbot history pose a problem?

and "  modify the buildbot setup to submit data to the server. "
isn't it aldready submitting data to the server?
if not how is the buildbot history currently generated?


On 25 March 2018 at 04:01, Mojca Miklavec <mojca at macports.org> wrote:

> On 24 March 2018 at 22:50, Vishnu wrote:
> > Well that's really nice.
> > I'll be emailing you the proposal very soon.
> > I would really like to work on creating dynamic websites for each port
> with
> > one framework.
> > I think that is what you are also working on currently.Or is it something
> > else?(Umesh Sir)
> >
> > I would like to go with option B.
> >
> > What i understand is .Basically, i will be creating a dynamic
> > website/framework (use some scripting language) to generate/gather all
> the
> > following information for any port that is requested.
> >
> >
> > - all the information about the port (description, version,
> > maintainer, homepage, variants, dependencies, dependent ports, ...)
> > - build summary (as scraped from buildbot history): which version was
> > built at what time on which os, successful or not, ...
> >   - including whether or not a binary package exists
> > - installation statistics (from users who opted in and send a json
> > file every now and then) with some graphs, similar to the stats page
> > above
> > - results of "livecheck": whether there's a newer version of the port
> > available
> > - short git log with links to the last few changes to the Portfile
> > - links to trac tickets for that port
> >
> > https://trac.macports.org/query?0_port=python27&0_port_
> mode=%7E&0_status=%21closed
> > - (maybe links to pull requests for that port)
> >
> > Is my understanding of the project correct?
> >
> > Now to generate:
> > 1) description, version,maintainer, homepage, variants, dependencies,
> > dependent ports,etc:
> > I will need to update this script
> > https://github.com/g5pw/macports-port-tree made by Aljaz for the
> remaining
> > missing information.because not all information is produced by this tool.
>
> This source of information PortIndex which can be generated from
> https://github.com/macports/macports-ports (that's part of MacPorts
> installation process, it should be possible to get it under linux as
> well, but I didn't try, we also mirror it).
>
> PortIndex is some data structure in Tcl which can be converted into
> JSON with an existing script postindex2json, but ideally the Tcl
> script should be fixed to generate data in a better format.
> This data would then need to be imported at the beginning, but the
> application should keep accepting updates about new or deleted ports.
>
> > 2) Build Summary: This can be generated by scraping through buildbot
> > history.
>
> For historic data. For the future it would be better to modify the
> buildbot setup to submit data to the server.
>
> > 3) Installation Statistics: Take help from Stats web app(Gsoc 2011
> project)
> >
> > 4)Results of livecheck: I am not sure how to get that. i'll be needing
> help
> > to how to get this.
>
> Sample output from the package manager:
>
> > port -v livecheck *
> Bear seems to be up to date
> CImg seems to have been updated (port version: 1.7.9, new version: 2.2.1)
> CoreOSMakefiles seems to be up to date
> Error: cannot check if Djinn was updated (regex didn't match)
> Drizzle seems to have been updated (port version: 30, new version: 31)
> FlowUI seems to be up to date
> GASNet seems to have been updated (port version: 1.26.3, new version:
> 1.30.0)
> GitX seems to be up to date
> HaXml seems to have been updated (port version: 1.20.2, new version:
> 1.23.3)
> Error: livecheck failed for LibVNCServer: extracted version '0.4' is
> older than livecheck.version '0.9.9'
> LucenePlusPlus seems to be up to date
> Error: livecheck failed for MongoObjCDriver: extracted version 'test2'
> is older than livecheck.version '1.0.6'
> Error: cannot check if PlasmaShop was updated (regex didn't match)
> ResIL seems to have been updated (port version: 1.7.9, new version: 1.8.3)
> Error: cannot check if SilverCity was updated (regex didn't match)
> SourceKitten seems to have been updated (port version: 0.17.6, new
> version: 0.19.1)
> Error: cannot check if VillainousStyle was updated (regex didn't match)
> aap seems to be up to date
>
> > 5) Link to track tickets : Can be done using scripting,html.
> > 6)Pull request : Can be done.
>
> Mojca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20180325/31883a05/attachment-0001.html>