Is it time to start regression testing yet?

[William Siegrist]

On Jun 6, 2009, at 12:43 PM, Jordan K. Hubbard wrote:

> I know that the word "packaging" is kind of a dirty word in MacPorts- 
> land (perhaps largely due to the fact that certain people just won't  
> stop harping about it :-), so maybe it's time for a new(er) topic in  
> an old conversation:  Testing.
>
> Since a picture is supposed to be worth a thousand words, let me  
> also refer to the picture below for justification as to why we  
> should be worrying more about testing.  Ports are accumulating at  
> the fairly steady rate of 800-1000 a year, and it's also fair to say  
> that individual ports are getting more complex.  What started as a  
> fairly simplistic attempt at key/value pairs in Tcl has since grown  
> Groups, variants and, in some cases, fairly non-trivial tcl code in  
> individual Portfiles, and all of that begs the question:  Given all  
> the complexity involved, how many of these almost 6000 ports  
> actually work at any given time?  Anyone have an accurate number?    
> Anyone?  Beuller? No worries, it was a purely rhetorical question to  
> which I already know the answer:  We have no idea, though we  
> certainly hope that users will report breakage in a fairly timely  
> fashion so we can fix things as they come up, and if there are no  
> users of a port to report errors, then who really cares if it's  
> broken?  We then proceed to the rather circular argument of  
> justifying the existence of ports which don't currently work but are  
> kept around purely on the argument that they *might* at some point  
> in the future.
>


I suggest we setup a buildbot master at build.macports.org and have it  
(at least initially) test building/installing/testing base and ports  
after each commit. I think for an initial effort, in light of all  
earlier attempts failing, we should start small and just do default  
variants.

-Bill