Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

Dear Vishnu,

Sorry for being slightly out-of-sync (sending emails I wrote quite a
while ago and in the meantime no longer being up-to-date with other
responses).

On 12 May 2018 at 17:26, Vishnu wrote:
> Hi
>
> "The PortIndex file itself is generated with the 'portindex' command,
> which only updates the data for Portfiles that have a modification time
> that is newer than the existing PortIndex file."
>
> So where can i see that code that checks the modification time?

https://github.com/macports/macports-base/blob/master/src/port/portindex.tcl

> So suppose
> Our ports tree contains three ports A, B, C last modified time : 1:00 UTC
> present port index:
> [ A
>   B
>   C
> ] @ 1:00 UTC
>
> There was a edit in Port A @ 1:30 UTC
> so something will check all the port files modification time and if
> relatively newer it will update the port index.
>
> so new  port index:
> [ A
>   B
>   C
> ] @ 1:30 UTC
>
>
> "The portindex2postgres.sql script merely converts the PortIndex file
> from the custom format based on Tcl lists to SQL. The output will always
> contain SQL statements with the full data for every port."
>
> Then after the port index is updated this script runs and flushes the db
> then fills it with new portindex data?
> or just updates the db with the modified port data?

I don't know the answer without looking at it first (you can check
what the code does yourself), but from the perspective of your app
development this is irrelevant.

> So after every commit the entire 15mb portindex being processed and data
> being uploaded to the db.
> I think this is very inefficient.
> rather it would be best if after every commit the buildbot/ webhook updates
> the database itself.

I totally agree that it might make a lot more sense to submit just
differential data, but my suggestion would be to first implement the
functionality and optimise it at the end if there will be time. No
user will suffer from suboptimal data transfer (even if that database
only gets updated once per day, it will still be perfectly
acceptable), while nobody will benefit from spending a lot of time
figuring out how to properly implement a super efficient solution and
then running out of time to actually finish the app.

Let's just work with what we have for now.

Mojca