Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 15 May 2018 at 05:46, Vishnu wrote:
> Hi
>
> Yes I was laos thinking about it for some time.
>
> Please someone from infra team create another repository.

This would also remove all the tickets etc., so I don't think it is a
super good idea.

Git is flexible enough to allow force pushing to a branch. We should do that.
I tried to squash some commits, but the history is too strange
(squashing probably doesn't work well across multiple branches) and I
seriously believe that starting from scratch would be the best thing
to do at this point. But we should not create a new repo, simply use
the existing one.

In the meantime, *please* keep working normally on your own copy (the
repository on github under your username) on the tasks, don't wait for
the repository cleanup, else you will run out of time.

> Things I will be doing:
>
> - Limit testing.json to 1000 ports only.

If you put such a file to a repository for testing purposes (for
running unit tests), it should have 10 entries rather than 1000. If
you want the real/realistic data to start with, this would better be
done by putting a script into repository that fetches the real
PortIndex from our server (or simply runs portindex on your machine),
runs portindex2json and then loads that file. Or you can simply have a
local copy of the file, or create another repository with temporary
testing files. 15 MB files shouldn't be part of source repository.

> - Implement Categories table and the modified port table.

Yes, please do this.

> - Implement some sort of test atleast very trivial but will
> - Did not index the port table so far.But will now index the table.
>
>
>
> Things discussed in the last meeting:
>
> Implement these as well
> - Implement ability to mark deleted ports in the port table itself
> - For every port just display the last 5 - 10 commits on the ports

Sure, but this can be done towards the end of the project, no need to
rush into this now. Let's primarily make sure that we have a fully
functional portindex and installation statistics working properly.

> - Just mention the version everywhere. No portversion id.
> - When build started and ended.
>
> Future Goals:
> - implment obsolete
> - Fix portindex2json, atleast maintainers
> - Differential feature to be in stretch goals
> - Port History
> - Port Version table
>
> Also I think finalising the database ticket will be for long.As we are
> continusally evolving it so... Probably the milestone to create Database
> structure within the week may not be possible.

The milestone doesn't mean that you need to create a database that
will serve everyone for eternity. It means that:
- primarily ourselves, but ideally also a wider community needs to be
satisfied with the overall design proposal (we are nearly there)
- that the design should cover most of the functionality planned for
the summer, even if some details change later
- that you ideally have the port index fully working, utilizing all
the fields from portindex json file, but in particular have a sound
understanding of how to efficiently join tables (read as: this mostly
covers support for categories and maintainers, even if maintainers
don't cover 100% of weird cases with discrepancies in our index) and
that you are able to update the tables from a new portindex file
without exploding the database :)

Once this is done, the ticket would be closed and you would end up
with a document that will serve as a reference for the future /
documentation. You are of course free to make further changes to the
database afterwards, but at least the plan should be sound and we need
to be on the same page about what things should go in there and which
ones not.

> Mojca you said something about static files.
> I think we should have all Javascript also in a separate individual file.
> That's why in a seperate individual file.

Sure, javascript goes to a separate file, but the fake data table
should not be there.

Mojca