where do feature requests go?

[Ryan Schmidt]

On Jun 11, 2007, at 02:05, paul beard wrote:

> Still wondering why my distfiles have to come from so far away.

Because nobody has programmed (or discussed) an alternative.

I'm Cc'ing the -dev list, where discussions of this nature  
(developing MacPorts) should probably take place. When you reply,  
please edit -users out of the addressees and let's continue this  
discussion only on -dev.

> I would create a more formal feature request but I can't access  
> anything at MacOSForge.

Clearly that's a separate issue, but we should address that too.  
Please tell us (on -users) what problems you're having.

Before we file a feature request in Trac, though, we should discuss  
on the mailing list what the feature should do.

> Is there some way for the lists of mirrors to be associated with  
> the user's timezone (gdate +%Z value) so that the load can be  
> distributed around more evenly, users get distfiles faster, and  
> mirror sites at the top of the list don't get pounded?
>
> {prefix}/share/darwinports/resources/port1.0/fetch/mirror_sites.tcl
> Actually for some of these, it could be done by hemisphere or  
> continent ;-)
>
> set portfetch::mirror_sites::sites(gnome) {
>     http://mandril.creatis.insa-lyon.fr/linux/gnome.org/
>     http://mirror.aarnet.edu.au/pub/GNOME/
>     http://ftp.unina.it/pub/linux/GNOME
>     http://fr.rpmfind.net/linux/gnome.org/
>     http://fr2.rpmfind.net/linux/gnome.org/
>     http://ftp.acc.umu.se/pub/GNOME/
>     http://ftp.belnet.be/mirror/ftp.gnome.org/
>     http://ftp.linux.org.uk/mirrors/ftp.gnome.org/
>     http://ftp.nara.wide.ad.jp/pub/X11/GNOME/
>     http://ftp.gnome.org/pub/GNOME/
>     ftp://mirror.pacific.net.au/linux/GNOME
>     ftp://ftp.dataplus.se/pub/GNOME/
>     ftp://ftp.dit.upm.es/pub/GNOME/
>     ftp://ftp.no.gnome.org/pub/GNOME/
>     ftp://ftp.isu.edu.tw/pub/Unix/Desktop/GNOME/
>     ftp://ftp.nara.wide.ad.jp/pub/X11/GNOME/
>     ftp://ftp.chg.ru/pub/X11/gnome/
>     ftp://ftp.kddlabs.co.jp/pub/GNOME/
>     ftp://ftp.dti.ad.jp/pub/X/gnome/
> }

I think the suggestion has merit in principle, though I wouldn't  
necessary do it by time zone.

SourceForge has implemented something like this, where you access one  
central URL to download anything, and it redirects you to a nearby  
mirror, based on your IP. I don't know if they do it by country or  
what, but I know there are databases (free ones, even) that will tell  
you what country an IP address is in, so that's a reasonable way to  
do it.


One solution would be to have a script (like a PHP script*) available  
on www.macports.org which we could call to download files from our  
various master_sites groups (like the gnome group you've quoted  
above) with a URL like:

http://example.macports.org/download/from/gnome/sources/pango/1.16/ 
pango-1.16.4.tar.bz2

http://example.macports.org/download/from is the download script
gnome is the group
sources/pango/1.16/pango-1.16.4.tar.bz2 is the path used on servers  
in that group

In this scenario, the list of gnome mirrors (and other groups from  
mirror_sites.tcl) would migrate to this "download" script, which  
would also know (or automatically find out) what country each mirror  
is in. And for each incoming request, it would find out the visitor's  
IP and direct them to one of the mirrors in their country.

We obviously also need a solution for when there is no mirror in the  
visitor's country. We probably need a table that says which countries  
are near which other countries. Maybe such data can also be  
downloaded for free. If we can even just get lat/long coordinates for  
each country, MySQL 5 has support for spacial queries which may help  
us find other nearby servers.

I guess this would be one aspect where doing it by time zone might  
simplify things... Finding out the visitor's time zone is easy (on  
the client side), and finding out what time zones are near other time  
zones is also easy. Finding out which time zone a given server is in  
might also be possible... though I can't off the top of my head think  
of an automated way... The headers the servers return seem to always  
be GMT...

Also, I'm not sure how much load this would add to www.macports.org.  
And I'm not sure if it would be seen as a bad thing that the mirror  
list becomes a server-side thing, not a client-side thing. Though to  
me it seems like a reasonable good idea.


As expressed above, the idea would only handle the common  
master_sites groups. It wouldn't deal with other ports that define  
several master_sites.

A different form of the above idea would be a download URL that just  
says which port and which file to download, with a URL like:

http://example.macports.org/download/pango/pango-1.16.4.tar.bz2

The "download" script would read the portfile and get its list of  
master_sites, so this would work both for portfiles that use  
master_sites groups and those that define the sites individually. For  
each master_site, determine its IP, and from that, its country (and  
cache this). Get the visitor's IP, and thus country, compare, find  
match, redirect for download. Same caveat as above regarding visitors  
that don't have a mirror in their country.


There's probably a way to turn this around and do it a more  
"MacPorts"-like way where the country data is either embedded into  
the PortIndex or into another file that like the PortIndex gets  
automatically updated at some interval. This way clients aren't  
constantly pinging the MacPorts web site to get each file's download  
location; they'd already know.


A less-automated solution could be that for each master_site defined  
in master_sites.tcl and for each master_site defined in each  
portfile, one must specify the country code. The port command can  
then determine what country the user is in (e.g. by pinging a simple  
script on the server which returns the country code -- e.g. http:// 
example.macports.org/country.php -- and this can be cached on the  
client side for some interval -- or since we only support Mac OS now,  
we can find a way to look up wherever the Date & Time system  
preferences stores its data) and see which mirror to download from.

I think I don't like this option because it places too great a burden  
on the port authors to determine and specify each master_site's  
country. Since we can figure it out, we should do so, and not make  
the port author do it.


I'm just throwing ideas out there. Any others? Any comments on my  
ideas? What's good, what's bad?


* I just say PHP script because I like to write PHP scripts. It could  
of course be in any language.