GSoC 2019 [Collect build statistics]

[Mojca Miklavec]

Dear Arjun,

I'm CC-ing the list as that's potentially interesting to others as well.

On Tue, 14 May 2019 at 19:45, Arjun Salyan wrote:

> Summary of today's discussion:
>
> 1. *Repository: *The existing repository for the web app would be moved
> from the staging area to the official MacPorts organisations's
> repositories. And then we will setup issues and milestones (after making
> changes to the timeline, if needed). The previous macports-webapp
> repository (from last year) can be key separate to the new one by making
> some name changes.
>
> 2. *Docker Containers: *The infra team has suggested that for initial
> deployment of the app we make a docker container so that our app would not
> pose any security issues to rest of the server.
>
> 3. *mpstats: *We can add a new port (mpstats 2.0 or some other name) to
> submit statistics to our app, so that when we start working on stats part-
> we will have some data to test the setup. The newer version would drop
> submission of "inactive ports" and "gcc".
>
> 4. *Buildbot*: I will be learning buildbot more thoroughly.
>

Apart from
    https://github.com/macports/macports-infrastructure/tree/master/buildbot
see also
    https://github.com/rajdeepbharati/macports-buildbot/

Students with similar projects are certainly encouraged to follow (and
provide feedback) to the other similar projects.


> 5. *Database Design: *It would be better to store all weekly submissions
> instead of just the latest one in the base table (the one which is used to
> populate all other tables).
>
> 6. *Variants: *A low priority task. We could pickup default variant for
> any port from our database of port info.
>
> I hope I haven't missed anything important.
>
> I have set up the app to receive submissions from mpstats at:
> https://frozen-falls-98471.herokuapp.com/statistics/submit/ . I just
> replaced the url in mpstats and it is working correctly.
>
> Thank You
>

I forgot to mention another super important point. You should use this time
to get familiar with writing:

   - *Unit tests*: You already created an empty file, but it's about high
   time to populate it with some real tests, like preparing a very simple set
   of portfiles, generate json, enter them to database and check whether all
   the tables are exactly as we want them to be (in particular with respect to
   authors, categories, ...), and then some checks for subsequent port
   updates, ... Those tests could also run on Travis (or some other service).

Ad:

   - *Database design*: prepare the design (for statistics submissions in
   particular). My suggestion was for the port installation statistics to
   store something like (submission_id, user_uuid, timestamp, os_version, ...)
   in one table, and something like (submission_id, port, version,
   is_requested, <something with variants>) to another table, make a view
   joining the two, and then be able to answer queries like "how many distinct
   users had port 'python27' installed between 2019-05-01 and 2019-05-31" with
   some advanced sql statements (I hope they are still supported by Django),
   for arbitrary dates etc. Results could of course later be cached for faster
   rendering. This might need some playing around.

Mojca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20190515/e2acb1ad/attachment-0001.html>