registry backup (and upgrade --force)

[Clemens Lang]

Hi,

On Sun, Apr 03, 2016 at 05:56:49PM +0200, René J.V. Bertin wrote:
> >How did you presume that? Are you sure your assumption is correct?
> 
> The only sensible feedback I got was from the CrashReporter after I
> sent a SIGTERM (^]) which showed a backtrace into sqlite fsck
> functions. I also think I had something similar before, and like the
> previous time, restoring the registry.db file from backup unblocked
> the situation.
> Now, maybe the file wasn't "corrupt" but in some other state, but as
> far as I'm concerned the net result was the same ;)

Sounds like SQLite was doing its job and failed at it. There's nothing
we can do from a MacPorts PoV on SQLite failing to recover. Is there
anything odd about your configuration that could cause problems with
SQLite?

> > That would still leave a time window where your registry doesn't
> > match what's on disk. You wouldn't loose all data, but your data
> > would now be faulty. I'd argue that's not much of an improvement.
> 
> Does it? If the operation that would introduce new data fails and
> leaves the file in some unknown state, can you really say new data has
> been recorded?

You suggested having a cron job that makes a backup. If an operation was
successful after said backup, the state has already changed and going
back to the backup will loose data. The only other alternative is making
a backup copy before each and every operation that modifies the
database.

> It's not unlike what journaled and COW filesystems do (as far as I
> understand those). Commit the new state to some transitional registry,
> and make that current when you know the commit has succeeded. If that
> operation fails, you only lose whatever you were trying to change,
> just like in the scenario I had in mind - and in our case that should
> typically be an operation that'll be easy enough to repeat.

We're using a databse precisely to get this behavior. We shouldn't
second-guess the database but fix the reason that causes the database to
not provide this behavior.

> Yeah, I was thinking of that too. The problem is of course that's
> something you think about *after* hitting ^C when you realise you just
> launched something that also carries the risk of borking your whole
> install. Not easy to time that ^C (maybe I should get used to using ^Z
> instead).

Which is why we're adding signal handling to fix that.

> I was very specifically thinking of a mechanism that maintains a
> backup of the last known valid state. I know I mentioned an offline
> procedure, but that was with launchd's file-watching feature in mind
> which would trigger the copy after each change to the file.

It's an SQLite database, each transaction will trigger a change to the
file. That would cause a lot of backups to be created. Additionally, you
cannot safely copy a consistent representation of a database without
using the database access layer (on standard OS X filesystems), so this
achieves nothing: How would you ensure that the database isn't being
modified while you copy it?

> > I agree, upgrade --force should imply only rebuilding the given
> > ports. A patch would be nice.
> 
> Any idea if that's going to require a lot of changes or rather
> something that's as trivial as it sounds like it should be?

Josh's answer sounds like it's not that much effort. I don't know,
however, I'm not very familiar with that part of MacPorts base.

-- 
Clemens