MacPorts caching of distfiles [was Re: ntfs-3g 1.1120 source missing]

[Jordan K. Hubbard]

On Feb 22, 2008, at 7:11 PM, Rainer Müller wrote:

> One of these things would be how we would push files on the mirror?  
> If we keep master_sites we could use something like `port fetch all'  
> in a cronjob or in a post-commit hook. But that will have the  
> problem that it currently port fetch will not remove old distfiles.  
> So this would need improvement.
> Alternatively committers would have to upload there files manually  
> which would be more work for them and also more error-prone.

1. I think doing a "port fetch all" is just fine - it will skip the  
fetch step for ports which already have their distfiles in place (we  
can even add a little extra logic to stick them somewhere other than  
in ${prefix}/var/macports/distfiles for the mirroring case) and  
doesn't involve a lot of extra work on our part.  The only thing we'll  
need to make sure of is that our timeout logic is genuinely robust  
since we don't want a single hanging fetch to scrog the entire mirror  
run, ditto for our failed fetch logic - we don't want partial  
distfiles to get left lying around, but both of those are things we  
should have working well anyway.

2. I see no need to delete stale distfiles.  People will have older  
versions of the ports (not everyone keeps macports constantly up to  
date) and we still need to cache old distfiles for old ports (it's  
also useful for checking older versions of a port out of svn and still  
being able to use them, say if a new version has completely failed to  
work in an unexpected way).  We have the disk space - there's no  
pressing need to garbage collect the distfiles with any urgency.

- Jordan