Port update outdated problem.

[Clemens Lang]

Hi,

>> OK. BTW, isn't SQLite a protocol that doesn't really scale well? I'm far from an
>> expert on this kind of subject, but I recently followed advice on the
>> kdepim-user ML to migrate my akonadi db to mysql (mariadb, as is the default
>> for port:akonadi), and performance has become way better.
>> The usage scenario is probably quite different though.
> 
> SQLite is nice in that it does not require a separately running server process.
> MySQL (MariaDB) and PostgreSQL are full database servers that require a
> database administrator to manage them. We don't want every MacPorts user to
> have to become a database administrator.

SQLite works well enough for the small amount of data we're keeping. This isn't a
database with millions of records and gigabytes of data (like your mailbox might
be). In fact, SQLite scales well way beyond our needs.


>> Just to clarify though: I typically tend to run selfupdate/upgrades when I start
>> getting the warning about the repo being outdated, and then set the process to
>> a very low priority (but keep the number of jobs equal to the number of cores I
>> have). And do other stuff that usually doesn't require a lot of resources.
>> When I still had "only" 8Gb of memory I often saw messages about emergency pages
>> being used during the upgrade process (in kernel.log). The other day I had a
>> parallels VM with 4Gb RAM open, and at some point got low disk warnings because
>> 19Gb or so of swap files had consumed most of my remaining free space.

That might be caused by the builds MacPorts is running.


>> That's on 10.6, where I have indeed seen clang-3.4 consume huge amounts of
>> memory on a single file, AFAIK that compiler wasn't being used. However, if
>> port upgrade outdated uses a single tclsh process for the whole procedure, and
>> it's this process that handles (inefficient) db stuff, that could cause its
>> memory footprint to explode.

Most of the stuff this tclsh process does is not written in a language that
requires manual memory management. Tcl code doesn't leak memory.


> It's certainly possible there is a memory leak in MacPorts base somewhere.

It's possible that there is a memory leak in the parts of MacPorts written in C. A lot
of the code is old and hasn't seen maintenance or re-factoring in a while. I'm pretty
confident the parts I've looked at (registry, darwintrace, tracelib and curl parts of
pextlib) are leak-free, but that doesn't mean that other parts are.

-- 
Clemens Lang