The fetch command in Portfiles

Mon Jul 9 20:58:26 PDT 2007

On Jul 9, 2007, at 22:00, Brent Fulgham wrote:

> On Jul 9, 2007, at 8:26 AM, Ryan Schmidt wrote:
>
>> On Jul 9, 2007, at 09:45, Doug McCorkle wrote:
>>
>>> The correct fetch.ignore_sslcert command solved my problems. In  
>>> regards to the args command I thought I might be able to pass  
>>> command line args to the utility that downloads packages. In  
>>> MacPorts what is used to download packages? Is it curl?
>>
>> It is curl, but I don't think there's a way to pass arbitrary  
>> arguments to it. I'm not even sure whether it's called as a  
>> command-line utility or used as a library. I think if you need  
>> some special functionality to download something, then that  
>> functionality should be built into MacPorts base and made  
>> available via the fetch object, like fetch.ignore_sslcert.
>
> Given that some sites prohibit anything that's not a normal user  
> agent, it would be really useful to be able to set an argument for  
> setting the user agent to Mozilla or something similar.
>
> As an example, Cenon can't be downloaded normally:
>
> Dwight:~ brent$ curl  http://www.vhf-group.com/vhfInterservice/ 
> download/source/Apple/Cenon-3.82.tar.bz2
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <HTML><HEAD>
> <TITLE>403 Forbidden</TITLE>
> </HEAD><BODY>
> <H1>Forbidden</H1>
> You don't have permission to access /vhfInterservice/download/ 
> source/Apple/Cenon-3.82.tar.bz2
> on this server.<P>
> </BODY></HTML>
>
> But using the useragent argument everything is fine:
>
> Dwight:~ brent$ curl -A Mozilla http://www.vhf-group.com/ 
> vhfInterservice/download/source/Apple/Cenon-3.82.tar.bz2
> BZh91AY&SY?!????????????????????
> ?? f?a
> ?ڎ???j????,??V?T?P???1??W?^? hր???li^???W?zU?\?|!?1Ka?? 
> ZU*K`-????????wf?????l͙T'=???=>??蘆3z?sEu?`?]z4GPMJ֪??Fc7???@
> [.. etc ...]

Hmm. We have a "cenon" port already. And you're right, it fails to  
download the archive. It looks like they only deliver the 403  
response if the user agent contains "curl". Other random strings work  
fine. I would consider the www.vhf-group.com web server broken. They  
should not be preventing us from downloading their files just because  
our user agent contains "curl".

Presumably the cenon port worked properly when it was most recently  
committed, so they must have broken their server some time after that.

> Adding a "fetch.user_agent" option would be a huge help.

I'm inclined not to like that idea. I'm not sure if I can explain why.

We had a user agent problem before, where a server delivered a silly  
response if no user agent was set:

http://trac.macports.org/projects/macports/ticket/7563

This was fixed with MacPorts 1.4.0 when we started sending the user  
agent "MacPorts libcurl/[libcurl version]". There was a request in  
the log message for r22771 that if someone knows how to put the  
MacPorts version number in there, we should do that too, and I agree.

Perhaps it's time to advertise ourselves as Mozilla. Something like:

Mozilla/5.0 MacPorts/1.5.0

It's a bit silly, I know, but I like it better than allowing port  
authors to override the user agent.

The best solution, of course, would be for them to fix their server.  
Have you already asked them to?