[GSoC 2019] Web App Project Update

[Mojca Miklavec]

Hi,

This is just to say a huge thank you to everyone who volunteered to
contribute. In just a week we collected statistics from nearly half as
many machines compared to our existing (semi-official) site. At the
moment the site is only showing OS version distribution, but we expect
other data to be shown soon (ideally in about two weeks).

Thank you very much,
    Mojca

(And of course feel free to provide feedback about the rest of the site.)

On Sat, 8 Jun 2019 at 14:52, Mojca Miklavec wrote:
>
> Dear MacPorts users,
>
> I would warmly welcome everyone to check what has been done so far,
> provide feedback, but in particular volunteer to run
>     sudo port install mpstats-gsoc
> to help us get more statistics ready by the time Arjun starts working
> on that part (about two weeks from now; it would be really cool to
> have more than one month worth of statistics from a sufficient number
> of users well before the GSOC ends to be able to know whether we'll be
> heading in the right direction).
>
> The detailed statistics is not there yet, but you can probably guess
> whether your statistics submission has been successful by checking the
> following page:
>     https://frozen-falls-98471.herokuapp.com/statistics/
> right before and after you install the port. You can also manually run
>     /opt/local/libexec/mpstats-gsoc submit
> or
>     /opt/local/libexec/mpstats-gsoc show
> if you just want to see what gets submitted.
>
> Thank you very much,
>     Mojca
>
> (I CC-ed macports-users to get some more visibility and hopefully get
> more volunteers to submit their statistics, but any further discussion
> probably only makes sense on macports-dev. Even though ... one of the
> student contributors complained that we didn't post anything about
> GSOC to the macports-announce ...)
>
> On Fri, 7 Jun 2019 at 22:21, Arjun Salyan wrote:
> >
> > Hi,
> >
> > I am working on the web app to collect build and installation statistics and show information about ports, maintainers etc.
> >
> > Repository: https://github.com/macports/macports-webapp
> >
> > We have deployed a demo on Heroku: https://frozen-falls-98471.herokuapp.com . Any feedback and feature requests on this are very welcome.
> >
> > Currently we are working on build history and port information. Port information is fetched from Portindex after converting it to JSON using portindex2json.tcl and build history is fetched from JSON API of buildbot, it is planned to be kept updated using HttPStatusPush reporter on Buildbot.
> >
> > Keeping the port informtation up-to-date is a challenge given the large size of the portindex. We are looking at two aspects:
> > - Incremental updates using difference between two port indices [1]
> > - Full updates to completely sync the database with latest portindex
> >
> > Full updates are dead slow on an Amazon RDS free tier instance, and probably it won't be feasible on any externally hosted database. On a local database, full updates take more than 5 minutes.
> >
> > For installation statistics, we have setup a temporary port (mpstats-gsoc) which would submit stats to the demo app. This is important becuase it would give us enough data to write proper views and do testing. Please consider installing mpstats-gsoc.
> >
> > We are looking to deploy the app on MacPorts' server and host it on a macports.org subdomain using Docker.
> >
> > Thank you for your time, any suggestions would be very helpful.
> >
> > [1] https://github.com/macports/macports-base/pull/121
> >
> > Arjun