Gsoc 18 Project | Collect build statistics

[Mojca Miklavec]

On 8 May 2018 at 01:12, Vishnu wrote:
> Hi
>
> I will work on all the points mentioned today..
>
> But i have been trying many different ways.
> To get 20k ports onto the database.. The site is crashing.. Timeout error.
> Or the site cant be reached.
> Maximum i got till 9k ports successfully entered.
>
> Tried many ways..
>
> If you have any suggestions do let me know.

I was pretty sure that we would reach the database limit here.

This site
    https://devcenter.heroku.com/articles/heroku-postgres-plans
says that the limit is 10k rows which is not anywhere near enough and
fully consistent with your observation. The plan that lets you have
10M rows is 9 USD per month which I gladly pay if that would solve the
issues (I'm not sure if those are the only ones, we might not have
sufficient memory etc.)

(I also remembered that I might have access to create a clean virtual
machine, but I need a couple of days to physically reach a place where
I have the additional info. That would require setting up the whole
machine manually of course, but we need to do that for the final
deployment anyway.)

> I think i have to break the data entry in separate chunks and do it. But not
> sure how to do that.

In any case Heroku will apparently not let you import the full
database anyway until we switch to a payed account. If you still have
troubles with performance issues afterwards, this also won't help.
What you could do is temporarily copy or move a few folders (say,
math, science and python) and run portindex just on those. This will
give you a smaller number of ports to work with, but still sufficient
to figure out what other issues you'll need to deal with (port listing
will have to be paginated etc.)

It's a good exercise to know what the limitations are. When you play
with new features and database design, it's always easier to play
with, say 10-100 ports than with full 15 MB portindex file.



Please check if you have commit access to macports-webapp now and try
to start putting there the documentation, plan, database design,
sample html page(s) etc. The repository needs to have a clear
README.md describing its purpose + pointers to additional
documentation inside the repository with more details, like database
design etc.

Once you transfer the database scheme into some markdown tables (you
can use any alternative way as long as it's easy for you to modify and
easy to review), we'll discuss whether anything else needs
modifications.

Mojca