[MacPorts] #70556: "base" seemingly locking up for sometimes very long

[MacPorts]

#70556: "base" seemingly locking up for sometimes very long
--------------------------+--------------------
  Reporter:  RJVB         |      Owner:  (none)
      Type:  enhancement  |     Status:  new
  Priority:  Normal       |  Milestone:
 Component:  base         |    Version:
Resolution:               |   Keywords:
      Port:               |
--------------------------+--------------------

Old description:

> I'm not the only one who has noticed or thought at times that the `port`
> command "hung" because apparently nothing was happening e.g. at the end
> of a command. I've also already had a registry corruption at least once
> in the past because I had interrupted (^C) a command that to the best of
> my recollection wasn't even one that should have touched the registry.
>
> I know I'm also not the only one who has observed the "write-ahead log"
> `registry.db-wal` file getting larger than the db itself, and sometimes
> just persisting until for some unclear reason it disappears (which
> usually means one of the aforementioned "freezes").
>
> I have a custom script written in Tcl that uses code from "base" to check
> a newly generated `destroot` tree for conflicts, new files, updated vs.
> touched files or files that will go missing compared to the currently
> installed version/variant of a port. This command routinely takes a
> comparatively long time to start up the 1st time after a `port destroot`
> command (curiously more noticeably on my Mac than on my significantly
> less powerful Linux set-up).
>
> That all comes across as a bit sub-optimal:
> - long operations should be announced and if possible (optionally)
> provide a progress indicator
> - leaving db operations "in limbo" in the -wal file seems like a recipe
> for disaster (esp. if they're not just backups)
>
> All in all it sounds like `port` is missing the equivalent of the `fsck`
> and/or `vacuum` maintenance commands provided by KDE PIM's akonadictl
> driver (KDE Mail stores emails, filters etc. in -by default an sqlite3-
> database).
>
> Given how large the registry can become and how well it compresses
> (better than 2x with just LZ4 online compression on ZFS) it seems like an
> optimisation command would be welcome too.
>
> I've already tried to use the maintenance/optimisation commands in
> `sqlitebrowser` but that utility warns about custom tables that made be
> abort the attempt.
>
> On a related note: it wouldn't hurt to have an automatic backup mechanism
> that is removed after succesful and total completion of an operation
> except in case of a DB version update. I'm also not the only person who
> found out the hard way that you can't (always?) roll back from a MacPorts
> "base" update, and that DB updates can happen unexpectedly because of
> operator error e.g. after creating a secondary MacPorts install.

New description:

 I'm not the only one who has noticed or thought at times that the `port`
 command "hung" because apparently nothing was happening e.g. at the end of
 a command. I've also already had a registry corruption at least once in
 the past because I had interrupted (`^C`) a command that to the best of my
 recollection wasn't even one that should have touched the registry.

 I know I'm also not the only one who has observed the "write-ahead log"
 `registry.db-wal` file getting larger than the db itself, and sometimes
 just persisting until for some unclear reason it disappears (which usually
 means one of the aforementioned "freezes").

 I have a custom script written in Tcl that uses code from "base" to check
 a newly generated `destroot` tree for conflicts, new files, updated vs.
 touched files or files that will go missing compared to the currently
 installed version/variant of a port. This command routinely takes a
 comparatively long time to start up the 1st time after a `port destroot`
 command (curiously more noticeably on my Mac than on my significantly less
 powerful Linux set-up).

 That all comes across as a bit sub-optimal:
 - long operations should be announced and if possible (optionally) provide
 a progress indicator
 - leaving db operations "in limbo" in the -wal file seems like a recipe
 for disaster (esp. if they're not just backups)

 All in all it sounds like `port` is missing the equivalent of the `fsck`
 and/or `vacuum` maintenance commands provided by KDE PIM's akonadictl
 driver (KDE Mail stores emails, filters etc. in -by default an sqlite3-
 database).

 Given how large the registry can become and how well it compresses (better
 than 2x with just LZ4 online compression on ZFS) it seems like an
 optimisation command would be welcome too.

 I've already tried to use the maintenance/optimisation commands in
 `sqlitebrowser` but that utility warns about custom tables that made be
 abort the attempt.

 On a related note: it wouldn't hurt to have an automatic backup mechanism
 that is removed after succesful and total completion of an operation
 except in case of a DB version update. I'm also not the only person who
 found out the hard way that you can't (always?) roll back from a MacPorts
 "base" update, and that DB updates can happen unexpectedly because of
 operator error e.g. after creating a secondary MacPorts install.

--

Comment (by ryandesign):

 As of MacPorts 2.0.0, it does vacuum the registry; see
 [7d950c17e4be1c12ea13d487997806e5885e8f20/macports-base].

 It used to do it after every operation but this was slow every time. As of
 MacPorts 2.9.0, it only vacuums if it's worth it, so now it's only slow at
 those times; see [c82a2699117181b1fedd490c62a3efffca7c8947/macports-base].

-- 
Ticket URL: <https://trac.macports.org/ticket/70556#comment:5>
MacPorts <https://www.macports.org/>
Ports system for macOS