Simplify sourceforge fetch group

[Ryan Schmidt]

Joshua (and others), what should we do about this situation?

Should we:

a) accept that we are not currently benefiting from the large list of mirrors, and remove them
b) fix the distfile paths in all ports using the sourceforge fetch group so that we once again can fetch without redirects
c) both of these?

As explained below, how we are currently dealing with sourceforge fetching is not correct and needs to be changed.


On Jun 13, 2011, at 13:57, Ryan Schmidt wrote:

> On May 23, 2011, at 14:00, Joshua Root wrote:
> 
>> The original reason (r21768) for having more than just the redirector
>> was in case it goes down. The main downside to removing the rest now is
>> that the ping time for the redirector is not the same as that of the
>> mirror it redirects to, but it will be compared to those of the MacPorts
>> mirrors during sorting.
> 
> I don't think we have the ping time advantage now anyway. I think the vast majority of our ports that use SourceForge for downloads assume the old "flat" namespace of files, for example:
> 
> $ port distfiles expat
> --->  Distfiles for expat
> [expat-2.0.1.tar.gz] /opt/local/var/macports/distfiles/expat/expat-2.0.1.tar.gz
> md5: ee8b492592568805593f81f8cdf2a04c
> sha1: 663548c37b996082db1f2f2c32af060d7aa15c2d
> rmd160: d31bcb152adaff9e358968be1ea901e1b4ed3b2f
>  http://downloads.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://aarnet.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://biznetnetworks.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://cdnetworks-us-1.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://cdnetworks-us-2.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://cdnetworks-kr-1.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
>  http://cdnetworks-kr-2.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
> 
> [snip]
> 
> However, SourceForge reorganized their files into folders some time ago. And requesting any of the URLs above results in multiple redirects: first back to the main SourceForge redirector, and then on to a mirror of its choosing:
> 
> $ curl -L -I http://cdnetworks-us-1.dl.sourceforge.net/expat/expat-2.0.1.tar.gz
> HTTP/1.1 302 Found
> Date: Mon, 13 Jun 2011 18:52:14 GMT
> Server: Apache
> Location: http://downloads.sourceforge.net/expat/expat-2.0.1.tar.gz?download&failedmirror=cdnetworks-us-1.dl.sourceforge.net
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
> 
> HTTP/1.1 301 Moved Permanently
> Access-Control-Allow-Origin: *
> X-Powered-By: PHP/5.2.9
> Location: http://downloads.sourceforge.net/project/expat/expat/2.0.1/expat-2.0.1.tar.gz?download=&failedmirror=cdnetworks-us-1.dl.sourceforge.net
> Content-type: text/html
> Date: Mon, 13 Jun 2011 18:52:15 GMT
> Server: lighttpd/1.4.26
> 
> HTTP/1.1 302 Found
> Access-Control-Allow-Origin: *
> X-Powered-By: PHP/5.2.9
> Content-Disposition: attachment; filename="expat-2.0.1.tar.gz"
> Location: http://softlayer.dl.sourceforge.net/project/expat/expat/2.0.1/expat-2.0.1.tar.gz
> Content-type: text/html
> Date: Mon, 13 Jun 2011 18:52:15 GMT
> Server: lighttpd/1.4.26
> 
> HTTP/1.1 200 OK
> Date: Mon, 13 Jun 2011 18:52:15 GMT
> Server: Apache
> Last-Modified: Thu, 07 Jun 2007 02:37:51 GMT
> ETag: "25e240f7-6cff8-43247cd0b9dc0"
> Accept-Ranges: bytes
> Content-Length: 446456
> Connection: close
> Content-Type: application/x-gzip
> 
> So all that our list of mirrors is doing is introducing one extra redirect (if the file is found), or dozens more redirects and DNS lookups (if the file is not found on any SourceForge server). And sometimes causing users checksum errors when mirrors are taken offline (see #29810).