Is it time to start regression testing yet?

[Ryan Schmidt]

On Jun 8, 2009, at 15:49, Scott Haneda wrote:

> On Jun 8, 2009, at 2:18 AM, Ian Eiloart wrote:
>
>> So, and automated test suite would (a) get errors diagnosed and  
>> fixed quicker, (b) reduce the number of errors as a result, and  
>> (c) give us a way of flagging them before a user starts a 20  
>> minute build process.
>
> I have been reading this thread, looks like the current project I  
> looked at with the web listing is really nice.
>
> Curious, what if MacPorts were to implement a sort of CrashReporter  
> type feature.  Granted, this is not a crash, but a system where the  
> output of the build errors were sent somewhere, into a database,  
> where they could be looked at.

MacPorts does not log build errors, or in fact anything. That's what  
the logging proposal I pointed to in my previous mail is all about.  
Once that is implemented, any number of things could happen with  
those logs, as you suggested.


> I can see a few other values to this as well, and I believe there  
> are a number of simple open tools out there that would facilitate  
> making this easy to do.
>
> The nice thing as I see it, is end user support.  If MacPorts has  
> an error, the app could present them a message "Sorry we could not  
> build port x, to date, there have been x build errors with this  
> portfile.  We have assigned this build error an id of x.  If you  
> would like to follow or start a discussion of this issue, please  
> follow this link: http://eample.com?id=x.  Users may have had  
> similar issues, which you can see at http://example.com?id=x&porfile=y
>
> These build errors could be auto posted to a mailing list, put into  
> trac, whatever was wanted really.  The ability to help a user  
> along, in order to get them to help out MacPorts, may be something  
> worth looking into.

I think the idea of a central build and testing server is the best  
part of this project to start on. Users can and do have nonstandard  
things on their machines which can cause builds to fail, and an  
automated build log posting wouldn't contain such information; as  
you've no doubt seen in mailing list discussions, it often has to be  
dragged out of the user that they have things installed in /usr/local  
or /sw or they are using an outdated version of Xcode. If we get a  
central testing server, extending that to packaging and distributing  
binaries should be straightforward. And once we are distributing  
binaries, there's much less reason for regular users to ever build  
from source and thus to ever encounter build errors.


> It is also a very good way to get an exact idea of what ports are  
> being installed.  You can then see that port x has been installed  
> 400 times, 390 of those with no issues, 10 of those with some issue.

I proposed this two years ago but was shot down because this was  
considered an invasion of privacy and people didn't want MacPorts  
"phoning home". I had wanted to have a nice status display on the  
MacPorts homepage showing which ports were the most popular, for  
example. I see their point, but I'm glad to re-open the topic if  
attitudes have changed.