Is it time to start regression testing yet?

[Scott Haneda]

On Jun 8, 2009, at 2:46 PM, Ryan Schmidt wrote:

> On Jun 8, 2009, at 15:49, Scott Haneda wrote:
>
>> On Jun 8, 2009, at 2:18 AM, Ian Eiloart wrote:
>>
>>> So, and automated test suite would (a) get errors diagnosed and  
>>> fixed quicker, (b) reduce the number of errors as a result, and  
>>> (c) give us a way of flagging them before a user starts a 20  
>>> minute build process.
>>
>> I have been reading this thread, looks like the current project I  
>> looked at with the web listing is really nice.
>>
>> Curious, what if MacPorts were to implement a sort of CrashReporter  
>> type feature.  Granted, this is not a crash, but a system where the  
>> output of the build errors were sent somewhere, into a database,  
>> where they could be looked at.
>
> MacPorts does not log build errors, or in fact anything. That's what  
> the logging proposal I pointed to in my previous mail is all about.  
> Once that is implemented, any number of things could happen with  
> those logs, as you suggested.

Ok, understood.  I assumed that the -d we saw on screen, somewhere,  
was logged, and just cleaned up.  Then again, I should not have assume  
that, since I have never seen anyone here ask for a copy of this log.   
Sounds like a great idea to implement.

>> I can see a few other values to this as well, and I believe there  
>> are a number of simple open tools out there that would facilitate  
>> making this easy to do.
>>
>> The nice thing as I see it, is end user support.  If MacPorts has  
>> an error, the app could present them a message "Sorry we could not  
>> build port x, to date, there have been x build errors with this  
>> portfile.  We have assigned this build error an id of x.  If you  
>> would like to follow or start a discussion of this issue, please  
>> follow this link: http://eample.com?id=x.  Users may have had  
>> similar issues, which you can see at http://example.com?id=x&porfile=y
>>
>> These build errors could be auto posted to a mailing list, put into  
>> trac, whatever was wanted really.  The ability to help a user  
>> along, in order to get them to help out MacPorts, may be something  
>> worth looking into.
>
> I think the idea of a central build and testing server is the best  
> part of this project to start on. Users can and do have nonstandard  
> things on their machines which can cause builds to fail, and an  
> automated build log posting wouldn't contain such information; as  
> you've no doubt seen in mailing list discussions, it often has to be  
> dragged out of the user that they have things installed in /usr/ 
> local or /sw or they are using an outdated version of Xcode. If we  
> get a central testing server, extending that to packaging and  
> distributing binaries should be straightforward. And once we are  
> distributing binaries, there's much less reason for regular users to  
> ever build from source and thus to ever encounter build errors.

I keep forgetting that MacPorts aims to distribute binaries.  Once  
that happens, things will indeed be much simpler.  As to the user  
having things in /sw and /usr/local, I was thinking along the lines of  
having some additional things in the output, such as listings of those  
file paths, as well as echoing out PATH and whatever else may be needed.

>> It is also a very good way to get an exact idea of what ports are  
>> being installed.  You can then see that port x has been installed  
>> 400 times, 390 of those with no issues, 10 of those with some issue.
>
> I proposed this two years ago but was shot down because this was  
> considered an invasion of privacy and people didn't want MacPorts  
> "phoning home". I had wanted to have a nice status display on the  
> MacPorts homepage showing which ports were the most popular, for  
> example. I see their point, but I'm glad to re-open the topic if  
> attitudes have changed.

I think we should re-open it.  I do not see this as an invasion of  
privacy, and it could very well be turned off, or requested on demand:
Report stats to MP [yes] [no] [never again]

Every port downloads a file from somewhere, the maintainer of that  
software already has stats on how many downloads of that software  
there are.  I am not sure I see any harm in this at all.

Now, my suggestion to send back a list of /sw and output of PATH and  
other environment variables, that is probably going to freak some  
people out for sure.  There are sane ways of doing it, sane ways to  
mask what is private for sure and what is not to be sent.  In the end,  
I think it is a question of how well can end users problems be  
supported.

I can only guess, but the smallest percentage of port build issues  
make it to the mailing list. I bet a similar amount make it into  
trac.  Most people fail, see it as an error, and then go about finding  
some other way to solve their problem.  At the very least, the stats  
would allow a buggy port to be noticed, and pulled.  I see a lot of  
plusses, a few minuses, which could be hashed out and solved in a way  
that makes most people happy.

Thanks for your input on this.
-- 
Scott * If you contact me off list replace talklists@ with scott@ *