registry.db-journal file location

[René J.V. Bertin]

On Monday February 05 2018 15:34:52 Rainer Müller wrote:

Hi,

>journal ensures consistency of the data, therefore the journal has to be
>kept right next to the database file itself. If it were on volatile
>storage like tmpfs, it would be lost on interruption/crash/etc.

I'd guess that the main point of using a journal is that in case of a catastrophe you only use the journal, so the new information. Most of the time that should be possible to overcome the effects of that for a registry like the one in MacPorts.
FWIW, ZFS can be configured to put its journal on a dedicated (and fast) log device, and so can HFS, AFAIK.
But evidently I wasn't proposing to put the journal on volatile storage, just on faster storage (or if you want, local storage, if you put $prefix on a NAS.
(Wait, I was ... /tmp and $TMPFS are cleared on reboot. Hmmm...)

>in your setup. To try this, you should be able to convert your registry
>database to this mode without making any changes to base itself.

I presume that I can always keep a backup copy (= this affects only the registry.db file)?

>database when detaching. This is meant to reduce fragmentation and save
>disk space (by default the file size of a SQLite database never
>shrinks), but of course it causes some disk activity.

That certainly explains why the effect is so much more noticeable after uninstalling or a reinstall (upgrade --force).
You're reducing fragmentation in the DB with that operation, but could well be increasing on-disk fragmentation.

>As far as I remember, when we tested it, it did not take significant
>time, but depending on your setup, this might be the expensive operation.

Can you remember how large the test registry was (or how many ports installed, if that's more determinant)? 
On the main machine, I'm at approx. 260Mb for the registry; on the test rig that's only 95Mb (24Mb compressed). 
If the VACUUM does indeed do defragmentation by moving data to fill empty space left after uninstalling this certainly explains what I'm seeing. That's not "some disk activity", that's quite a bit of disk activity. For safe defragging you first create a contiguous copy of a file/record, then compare the copies, remove the old one, copy back the file to its new location and do a 2nd verification before removing the temp. copy.

I think I've already seen discussions whether it's reasonably safe to disable the sqlite journal on a filesystem like ZFS which already provides its own comparable protection, and that the answer was probably no despite the potentially costly redundancy.

>However, you are not supposed to see VACUUM after read-only actions such
>as 'port work' or similar as it is guarded by a flag whether changes
>have been made.

Indeed, I'm not. I'm seeing the vacuming effects when I do a `port work` or similar AFTER a port uninstall. In the example I gave, I have to remember to do the port uninstall after starting the fetch,checksum,extract,patch steps if I do not want to waste time waiting to get control back.

It's not a huge deal, but having to wait a minute here, a minute there does grow old, esp. if you just want to launch a lengthy upgrade operation and go do something else in the meantime (and really esp. if that something else is for some*one* else, like feeding the cats or doing the dishes ;) )

R.