Buildfarm try

[Ryan Schmidt]

On Oct 17, 2007, at 15:53, Simon Ruderich wrote:

> On Sun, Oct 14, 2007 at 10:32:20PM -0500, Ryan Schmidt wrote:
>
>> Very interesting and useful! Some random comments:
>>
>> I hope you will regenerate this occasionally, at least those that are
>> failing right now. How long did it take to generate that list?
>
> It took my MacBook about 2-3 days which is quite long. I guess the  
> whole port
> tree would need at least a week, maybe (much) more.

I figured it would take quite awhile.


>> It would help to know the date/time when you encountered each  
>> failure,
>> and/or the revision of the portfile that was used.
>
> Yes, this would be indeed very nice.
>
>> You may want to add a table of contents at the top so we can jump to
>> specific ports.
>
> Good idea, I will add this in the future.
>
>> You may want to sort the ports in some way. It currently seems to be
>> somewhat alphabetical, but with other ports in between (maybe
>> dependencies?).
>
> I thought it's already sorted alphabetical. Maybe the problem is  
> that ports
> with a uppercase name are sorted in an own group at the top, then all
> lowercased ports.

Well, build.html currently lists failures in this order:

ArpSpyX
gnustep-base
DesktopManager
gnustep-base
gnustep-base
GNUMail-Aqua
HandBrake
gnustep-base
ID3
gnustep-base
qt3-mac
NotificationWatcher
gnustep-base

You see my confusion.


>> In build.html you have gnome-session, gedit, gnu-crypto, and  
>> gnustep-base
>> listed several times. Probably others as well.
>
> Yes, this is a problem with the parser I wrote to get the ports. I  
> hope I can
> improve it.
>
>> I fixed iksemel's fetch failure. Note that you had iksemel listed
>> build.html. You have other fetch failures listed in build.html
>> (DarwinPortsStartup, cook, p5-email-address, etc.).
>
> Thanks, I removed it from build.html.
> This is also a parser problem, sorry.
>
>> Lots of ports complaining about /opt-devel/bin/python2.4 being  
>> missing.
>> Aren't those ports in the python portgroup? Wonder what's going on  
>> there.
>
> Not sure either. I will look into it.
>
>> Some of your failures are suspicious. For example:
>>
>> php5 does not fail under normal circumstances, so I'd like to know  
>> how your
>> build farm is operating.
>>
>> You report a gettext/libintl build failure for grep that I am  
>> unable to
>> reproduce. The portfile includes the --disable-nls flag so I  
>> cannot imagine
>> why it would be trying to link against gettext/libintl.
>>
>> The reported build failure for ufraw ("Image error: /opt-devel/bin/ 
>> dcraw is
>> being used by the active dcraw port") is curious since ufraw does  
>> not depend
>> on dcraw (not even indirectly). Why was dcraw installed? I would  
>> think a
>> build farm would need to build each port in a clean system.
>
> As this was my first test I just executed "sudo port install all"  
> and wrote a
> small script which parsed the output of this command. I know see  
> how limited
> this approach was, but I'm already working on a better one.
>
> A big problem for example is that some fetches are not failing  
> after a timeout
> even if the server is not responding. I'm not sure how to fix this  
> as an own
> timeout in my buildfarm would stop long downloads.

MacPorts uses curl to download. Curl has options that can be used to  
consider the download failed if the download speed drops below some  
threshold for some period of time. If MacPorts is not currently using  
this option, perhaps it should. It would alleviate this problem.  
Here's how it's done on the command line (from "man curl"):

        --connect-timeout <seconds>
               Maximum  time  in  seconds  that you allow the  
connection to the
               server to take.  This only limits  the  connection   
phase,  once
               curl  has  connected this option is of no more use.  
See also the
               -m/--max-time option.

        -y/--speed-time <time>
               If a download is slower than speed-limit bytes per  
second during
               a speed-time period, the download gets aborted. If  
speed-time is
               used, the default speed-limit will be 1 unless set  
with -y.

               This option controls transfers and thus  will  not   
affect  slow
               connects  etc.  If this is a concern for you, try the  
--connect-
               timeout option.

        -Y/--speed-limit <speed>
               If a download is slower than this given speed, in  
bytes per sec-
               ond, for speed-time seconds it gets aborted. speed- 
time  is  set
               with -Y and is 30 if not set.


> My current idea is to get a list of all ports ("port list") and  
> then build
> each port after another. Before each new build all other ports are  
> removed
> ("port uninstall installed"). The output of port is parsed and then  
> based on
> this saved in a database which is then used to create the html files.
>
> It would also be good if maintainers could mark a port fixed so  
> it's removed
> from the failure list.
>
> But the best thing would be if the buildfarm is linked to the svn  
> tree and
> just tries to rebuild new or failed ports so we always know which  
> ones are
> failing. I already have some ideas for this and will try to  
> integrate it in my
> new approach I'm currently working on.

Yes, that would be better.


> The only problem is that some ports (like gtk2) need a long time to  
> build so
> maybe they shouldn't be removed after each port build.
>
> What do you think of my ideas?

Some ports do take a very long time, and some ports have very many  
dependencies. To start with, maybe it would be good to limit your  
build farm to ports that don't have so many dependencies (including  
indirect dependencies). Just so it doesn't take forever, and so that  
we start learning about the easier-to-fix failures.