Gsoc 18 Project | Collect build statistics

[Rainer Müller]

On 2018-05-12 17:27, Mojca Miklavec wrote:
> I would in fact really really like to see a functionality like
>     portindex --mode=diff --format=json
> that would create a table like
> 
> {
>     new = [...],
>     removed = [...],
>     updated = [...]
> }
> 
> which would only provide the difference since the last run of
> portindex. and then we would only push this data to the database.
> Since portindex already knows which files are new, this doesn't sound
> impossible to implement. This sounds a lot more efficient that
> transmitting full data, but sure ... full data can be done as well. I
> assume that the command which updates the ports would need to mark all
> ports as "potentially deleted" before reading and storing the data.

Something like this would be nice-to-have. That could also be used as a
way to present to the user what has changed since the last time they ran
a selfupdate/sync command.

However, incremental updates can be error-prone as they can get out of
sync, even if only a single update was missed for whatever reason.
Therefore, the functionality to transfer the complete portindex should
always be available in case we need to resync the database from scratch.
Transmitting an incremental update can still be added as an optimization
later.

Overall, we did not experience any problems with the database updates
that always transfer the full 15 MB of SQL. Why exactly do we expect
problems with 15 MB of JSON...?

Rainer