Is it time to start regression testing yet?

[Jordan K. Hubbard]

On Jun 6, 2009, at 12:50 PM, Jeff Johnson wrote:

> What is the actual connection between "testing" and the graph you  
> have included?

Um.  OK, sure, I'll spell it out.

1. We have a lot of ports.  The number of ports we have is growing at  
3-4 digit rates a year.
2. The essential mission of any given port is to build software and,  
one hopes, install it in a relatively canonical fashion.
3. How are we testing that the above proposition is actually true for  
any given port?

What I'm essentially proposing is that a testing harness be built which:

1. Iterates through all Portfiles on the system, save those explicitly  
marked "Broken" (which will be periodically swept and marked for  
extermination after a short jury trial).
2. For each Portfile, introspects it for any and all explicit variants  
and then does n builds, where build #1 is "the default configuration"  
and subsequent builds are for each of the variants it advertises.
3. A summary report is generated based on each port which (in the  
human-readable form), looks something like this:
	Port:	scabtaculous
	Version:	1.0
	<whatever other summary info about port is interesting:
	Build: Default
		Status:  FAIL
		Where: install
		Link: SomeURLToFail
	Build: +opengl
		Status: SUCCESS
	Build: +postgres
		Status: FAIL
		Where: Dependent port postgres
		Link: URLToPostgresFail

Etc.