Is it time to start regression testing yet?

[Jeremy Huddleston]

On Jun 6, 2009, at 13:31, Jordan K. Hubbard wrote:

> What I'm essentially proposing is that a testing harness be built  
> which:
>
> 1. Iterates through all Portfiles on the system, save those  
> explicitly marked "Broken" (which will be periodically swept and  
> marked for extermination after a short jury trial).
> 2. For each Portfile, introspects it for any and all explicit  
> variants and then does n builds, where build #1 is "the default  
> configuration" and subsequent builds are for each of the variants it  
> advertises.

This part feels a bit tricky to me.  There are many ports which  
expect, "if I have variant X, then all my dependencies were built with  
variant X as well."  Should we actually put in test code to sanity  
check these configurations?  I argue that effort would be better spent  
developing variant-based dependencies (yes, this is my Gentoo  
background speaking here...)

Furthermore, this 'n' tends to grow quite quickly when you consider  
that a port should be tested not only with all permutations of its  
variants but all permutations of its dependencies variants as well!   
This testing is something that I believe is completely infeasible at  
this moment.

I would argue that as a starting point, we need to define a few sets  
of common configurations and test just those variants for all ports.   
Suggestions:

1) default
2) +no_x11 +quartz
3) +aac +mp3 +mad +lame +ogg +vorbis +theora +x264  
+all_the_media_variants_you_can_think_of
4) some kind of "web-server" variant set... like +apache +mysql +php
...