revision control downloads

[Rainer Müller]

On 22/03/2022 22.23, Ryan Schmidt wrote:
> On Mar 22, 2022, at 13:08, Daniel J. Luke wrote:
> 
>> On Mar 21, 2022, at 9:20 PM, Ryan Schmidt wrote:
>>> Ports that fetch their sources from a revision control system do not enjoy the protection of checksums. Although ports that fetch source from a revision control system specify which tag or commit hash to fetch, it is conceivable that a developer with sufficient access to that repository could delete an old tag and replace it with a new tag of the same name that contains different software. This is one of the reasons why we recommend ports fetch using distfiles, and the vast majority do. We might consider recommending that ports that fetch directly from a git repository (fetch.type git) never use a tag, and always use the commit hash corresponding to that tag, since replacing the contents of the repository while keeping the same sha1 hash is, as far as I know, still impossible in the general case. (Yes, it is possible to engineer a sha1 collision, but only if you can carefully control both the old and new files. Generating a sha1 collision against some existing old file is a very different matter.)
>>
>> I haven't reviewed the current literature but one thing is certain - attacks only get better. We should strongly discourage any use of revision control as the source.
> 
> As I said we do already and have for a long time strongly discouraged using revision control systems for downloads.

As these days most web frontends allow to download a tarball, we could
also think about discouraging fetching from VCS completely.

>> We could shorten the window where a person could get sources that don't match what the maintainer expected with a little infrastructure magic. Roughly in order of presumed level of effort:
>>
>> - provide a place for maintainers to manually upload distfiles that they can generate from a source checkout
>> - provide some port command for maintainers to upload distfiles (as a convenience for the above upload process)
>> - have a process that automatically generates distfiles from the checked out source and puts it on our mirrors (builder could probably do this since it's already going to check out the source), then have base look for the distfile on our mirrors first (even if the portfile specifies to check out from a revision control system)
> 
> Rainer did some work on automating the creation of distfiles from revision control systems but it has not been touched in some years. Since Rainer hasn't been very active here lately maybe someone else would like to pick it up and finish the work.
> 
> https://github.com/macports/macports-base/tree/vcs-fetch

Right, what Daniel describes in the last point is basically what is
implemented in the vcs-fetch branch. I think the compare view is the
better starting point for the code:

https://github.com/macports/macports-base/compare/vcs-fetch

MacPorts will fetch from a VCS, but creates a reproducible tarball with
bsdtar. This allows to mirror a source tarball for all of fetch.type
git/svn/hg/bzr/cvs if they point to a fixed revision as they should. The
checksum of this tarball is in the Portfile as usual.

The general idea is that the existing mirroring job would create such a
reproducible tarball from the VCS that is distributed normally. On the
user end, fetch looks for a tarball on the mirrors first before
attempting a fetch from VCS.

For fetch.type git it also implements the additional feature to also
download submodules, which has been a requirement that could not be
specified with the fetch.* options [1].

The problem is that this branch has become stale, fully attributed to my
lack of interest and shift in priorities. This feature was started back
in 2016 at the MacPorts Meeting together with Mojca, but according to
the git log I last worked on it in 2018 ...

I am fairly confident the vcs-fetch branch could be rebased onto the
current master. I really don't mind if someone else wants to pick this
up, no, I would rather be glad to see this finished.

Rainer

[1] https://trac.macports.org/ticket/50708