Gsoc 18 Project | Collect build statistics

[Jackson Isaac]

On Tue, May 8, 2018 at 10:47 AM, Mojca Miklavec <mojca at macports.org> wrote:
>
> On 8 May 2018 at 01:12, Vishnu wrote:
> > Hi
> >
> > I will work on all the points mentioned today..
> >
> > But i have been trying many different ways.
> > To get 20k ports onto the database.. The site is crashing.. Timeout error.
> > Or the site cant be reached.
> > Maximum i got till 9k ports successfully entered.
> >
> > Tried many ways..
> >
> > If you have any suggestions do let me know.
>

We can even try firebase for the DB backend and host the frontend on heroku.

Firebase free version gives 1GB of storage and 10GB of bandwidth per month.
I guess for testing out the app on complete portindex we could give it a try.

https://firebase.google.com/pricing/

Then again we would have some kind of limitations over here in long run.

>
> I was pretty sure that we would reach the database limit here.
>
>
> This site
>     https://devcenter.heroku.com/articles/heroku-postgres-plans
> says that the limit is 10k rows which is not anywhere near enough and
> fully consistent with your observation. The plan that lets you have
> 10M rows is 9 USD per month which I gladly pay if that would solve the
> issues (I'm not sure if those are the only ones, we might not have
> sufficient memory etc.)
>

9USD is only for the DB though. We might need to pay for the compute separately.

Looks like free VMs come with only 512MB RAM.

>
> (I also remembered that I might have access to create a clean virtual
> machine, but I need a couple of days to physically reach a place where
> I have the additional info. That would require setting up the whole
> machine manually of course, but we need to do that for the final
> deployment anyway.)
>
> > I think i have to break the data entry in separate chunks and do it. But not
> > sure how to do that.
>
> In any case Heroku will apparently not let you import the full
> database anyway until we switch to a payed account. If you still have
> troubles with performance issues afterwards, this also won't help.
> What you could do is temporarily copy or move a few folders (say,
> math, science and python) and run portindex just on those. This will
> give you a smaller number of ports to work with, but still sufficient
> to figure out what other issues you'll need to deal with (port listing
> will have to be paginated etc.)
>
> It's a good exercise to know what the limitations are. When you play
> with new features and database design, it's always easier to play
> with, say 10-100 ports than with full 15 MB portindex file.
>

This plan sounds good. +1

-- 
Jackson Isaac