Build Reproducibility Workshop Report

[Clemens Lang]

On Wed, Dec 09, 2015 at 05:21:09PM -0800, Ryan Schmidt wrote:
> Apple hardware is available. Virtualizing OS X on Apple hardware is no
> problem; we do it today with our existing buildbot builders.
> [...]
> I'm open to changes in the buildbot builder setup or replacing it, if
> there's something better.

That's good news, but my impression is that our buildbot setup is
currently already quite heavily loaded. Do you think it could handle
building every port twice? How much headroom do we have?


> > To fix the timestamp issues, I am looking for a suitable value to use as
> > SOURCE_DATE_EPOCH and then add a find statement before creating the
> > archive that will put an upper mtime limit on all files to be packaged.
> > I am not yet sure what a good (reproducible!) timestamp might be:
> > - The Portfile mtime would be perfect, but is not preserved by
> >   Subversion, so we cannot rely on it. It is preserved by our rsync
> >   sync, but the mtime in that is probably meaningless since it's the
> >   one generated on the rsync server during svn update.
> 
> And it differs between different rsync mirrors, because they mirror at
> different times. We should examine our mirroring strategy and fix it
> so the mirrors are true copies, including metadata like timestamps.

Yes, mirrors should sync with rsync -t. Isn't this standard practice for
most mirrors already?


> I've thought about doing this at least for the MacPorts installer
> itself, to make it easier for users to install MacPorts to other
> prefixes. However, I figured demand for this was low, since in its
> current state this would make that prefix ineligible for binaries.
> Extending this strategy to all ports is probably more difficult as I
> suspect a large number of edge cases.

The message here is that the number of edge cases doesn't seem to be as
large as we might think, since the simple approach that covers the most
common places to encode paths works reasonably well for Homebrew.

> I would suggest that the prefix for a buildbot builder in this
> hypothetical scenario should be a long random string such that it is
> highly unlikely to occur within any project's source, whereas the
> string "/opt/local" occurs in many projects' sources and we might
> inadvertently replace something we shouldn't replace. On the other
> hand, continuing to use /opt/local as the build prefix might fix
> undiscovered prefix portability problems in some ports. However, a
> long prefix would have the advantage that a port that doesn't obey our
> LDFLAGS and thus doesn't include -headerpad_max_install_names would
> nevertheless have enough space in the library for the user's
> presumably shorter custom prefix.

I don't think that we should change the default prefix for the archive
builds. There's a simple reason for that: If we can not make all
packages relocatable, we still want binaries to be available for our
default prefix.


> I think the problem is that the rsync server is where the portindex is
> generated. Since use-commit-times=yes is such an obvious Subversion
> feature, there must be a good reason why it hasn't been enabled on the
> server yet.

Yes, for this to work correctly use-commit-times=yes would have to be
enabled on the rsync server. My guess is that it isn't enabled because
it isn't the default and nobody bothered changing settings from the
default.

-- 
Clemens