registry.db-journal file location

[Rainer Müller]

On 2018-02-05 11:08, René J.V. Bertin wrote:
> I have a test system on which certain operations, esp. uninstalling, block for up to over a minute after they appear to be done (= after the last debug info was printed). This doesn't just block the port command in question, it also blocks (almost?) any other port command I may execute, even those that probably don't need the registry like `port work`, `port clean` etc. I think this was a new "feature" that appeared somewhere in 2.4.99 .
> 
> There is disk activity during that period, I presume related to registry reorganisation, and yesterday I noticed a registry.db-journal file in the registry directory which was there and updated until the ongoing uninstall command finally exitted.
> 
> This particular system has MacPorts installed on a ZFS pool with lz4 compression and copies=2. I think this expensive operation could be sped up by using /tmp or $TMPDIR for the temporary registry file(s) - a change that should be transparent for regular users but potentially give a significant gain for more advanced set-ups where the temp directories are on much faster storage than that which holds the MacPorts prefix (or just a different drive).

The whole point of using SQLite is to guarantee a consistent state. The
journal ensures consistency of the data, therefore the journal has to be
kept right next to the database file itself. If it were on volatile
storage like tmpfs, it would be lost on interruption/crash/etc.

> I did a quick search for "journal" but the word doesn't appear to occur in the entire MacPorts src tree. Does that mean the use of temporary db/journal files is entirely internal to libsqlite3?

https://sqlite.org/tempfiles.html#rollback_journals

Maybe switching to write-ahead log would increase the SQLite performance
in your setup. To try this, you should be able to convert your registry
database to this mode without making any changes to base itself.

https://sqlite.org/wal.html#activating_and_configuring_wal_mode


Another thing causing this could be the VACUUM when MacPorts exits. If
changes were made to the registry, base always run a VACUUM on the
database when detaching. This is meant to reduce fragmentation and save
disk space (by default the file size of a SQLite database never
shrinks), but of course it causes some disk activity.

As far as I remember, when we tested it, it did not take significant
time, but depending on your setup, this might be the expensive operation.

https://github.com/macports/macports-base/blob/9606edb0b27814d5eb31525806e3253aad62fffe/src/registry2.0/registry.c#L129

However, you are not supposed to see VACUUM after read-only actions such
as 'port work' or similar as it is guarded by a flag whether changes
have been made.

Rainer