port sync performance

[Rainer Müller]

On 2018-03-25 16:24, Perry E. Metzger wrote:
> Now that I'm a project member, I've noticed that the performance of
> "port sync" is vastly lower than the performance of "git pull",
> probably because rsync is much slower than git.
> 
> 1) Does anyone know why this is? Is git doing something smarter? Is
> it just that "git pull" gets the benefit of github's CDN?

We want to distribute the ports tree with a signature that is verified
on users' machines. Therefore, the rsync actually downloads an
uncompressed tarball and a signature file. After verifying the signature
against /opt/local/etc/macports/pubkeys.conf, the tarball is then extracted.

The reason why it is implemented that way is that verifying a signature
for each file in the tree would be a lot slower than verifying
everything at once in a tarball. As the tarball is uncompressed, it
still allows the rsync algorithm to reuse parts the user already has
locally.

An alternative to this could be to create tags with signatures attached
in the git repository and rely on git to verify the integrity. However,
we cannot use GPG, as it is not available on macOS by default.

> 2) Would it be interesting to offer people the ability to use git for
> synchronization instead of rsync?

In fact we do for those that cannot use rsync. But of course these users
would no longer have signature verification. Also, the git ports tree
with full history is about twice the size.

https://trac.macports.org/wiki/howto/SyncingWithGit

We should work on improving this. The setup of syncing from Git could be
a lot simpler. For example, we could simplify this a lot by adding a
"git" flag to sources.conf:

  https://github.com/macports/macports-ports.git [default,git]

This should then automatically make a new clone at
${prefix}/var/macports/sources/github.com/macports/macports-ports/ or
update an existing repository at this path.

Rainer