MacPorts caching of distfiles [was Re: ntfs-3g 1.1120 source missing]

[Ryan Schmidt]

On Feb 25, 2008, at 15:34, Rainer Müller wrote:

> Jordan K. Hubbard wrote:
>
>> Because given the $portname-$portversion naming of 99% of the
>> distfiles (and the unique names of the 1% that are left), I just  
>> don't
>> see collisions as a problem.  I don't see it as a problem with the
>> FreeBSD ports collection either, and they're almost 3X bigger than we
>> are.
>
> I didn't know they are using such a flat namespace.

I've looked for old distfiles on such mirrors before, and it's a PITA  
to browse directories of thousands of distfiles, especially when that  
server is on the other side of the world on apparently a 28.8 modem  
connection and it takes 42 minutes to download just the HTML index.

>>> Why are you against adding more logic to the fetch code?
>>
>> Because it's always "someone" who gets to add it and that someone
>> hasn't volunteered yet.   If you're volunteering, then I withdraw  
>> most
>> of my objection.
>>
> [...]
>> I guess this just doesn't bother me as much as it evidently bothers
>> you.  I like the notion of a single URL which points to all the
>> distfiles.   Again, however, if you're volunteering to do the work in
>> MacPorts then you're supporting your proposal in the only way that
>> really counts and I'm happy to withdraw my objection.
>
> I didn't want to volunteer until we got to a conclusion. But of  
> course I
> would volunteer to improve our fetch code to include this
> http://distfiles.macports.org/<distname>/<file> scheme.

I would be in favor of having the URL space of distfiles.macports.org  
match the distfiles directory exactly. That would be

http://distfiles.macports.org/<dist_subdir>/<file>

It should not be difficult to modify the fetch phase to download  
files from such a URL.

This would accommodate all distname collisions that the port author  
has made accommodation for in the portfile, which could be sufficient.

This also makes it very easy to set up http://distfiles.macports.org/  
-- just point it at the server's distfiles directory and you're done.

> The fetch code is not the difficult part for this, the hard part is to
> get the fetching done for every platform/variant.

Fetching for every platform and variant is fairly straightforward as  
I explained in my lengthy earlier email on macports-dev (which is  
where this conversation should again be exclusively taking place).  
Simply fetch once for the port itself, then fetch again for each  
variant reported by "port variants", e.g.:

port fetch smlnj
port fetch smlnj +universal
port fetch smlnj +powerpc
port fetch smlnj +i386

What I'm not sure yet how to solve is the portfiles that fetch  
different files based on something other than a platform or variant  
selector.