[MacPorts] #60590: Macports mirrors are down?

[Clemens Lang]

Hi,

On Sat, Jun 06, 2020 at 12:57:19PM +0200, Ruben Di Battista wrote:
> I'm probably saying something completely wrong, but can't you leverage
> the CI system and buildbots to create a Github release that includes
> the PortIndex and whatsoever and then download it from Github servers
> via http? What am I missing here?

The ports tree changes multiple times a day. We currently build a new
PortIndex every few hours, and the way we sync things (PortIndex
together with the ports tree), we automatically ensure that the
PortIndex matches the files you have on disk. Additionally, we sync the
file modification time, so even if the PortIndex does not match exactly,
we can easily compute on the client side which ports have not been
indexed yet.

This is not as simple with Git. Git sets the mtime of modified files to
the date at which you did a checkout on them, not the time when they
were committed. The portindex command, however, compares the PortIndex
mtime with the mtimes of the portfiles.

The unsolved problems are:
1. Do we really want to create a new GitHub release every few hours?
2. How does MacPorts know which ports are correctly indexed in a
   downloaded PortIndex, and which ones were added afterwards and thus
   need to be re-indexed locally?
3. How do we ensure efficient transfer of the PortIndex? The PortIndex
   is currently ~15M, rsync efficiently only copies the delta for us.
   With hosting a precompiled PortIndex on GitHub, how do we avoid
   downloading the entire PortIndex on every sync?

I guess the solution for (2) could be to include the Git hash that was
indexed, compute the modified files between that hash and HEAD, touch
them, and re-run portindex.

I don't have a good suggestion for (3) yet. It might be possible to
generate daily PortIndex diffs, download all of those between the time
of the last update and now and apply them in-order. Such a mechanism
would have to work with any local modifications users might have in
their tree, though. As soon as the size of the diffs is larger than just
re-downloading the entire thing, we could switch to that.

Technically, all of this is doable. However, somebody actually needs to
sit down and implement it.

-- 
Clemens