Is it time to start regression testing yet?

[C. Florian Ebeling]

On Sat, Jun 6, 2009 at 9:43 PM, Jordan K. Hubbard<jkh at apple.com> wrote:
> I know that the word "packaging" is kind of a dirty word in MacPorts-land
> (perhaps largely due to the fact that certain people just won't stop harping
> about it :-), so maybe it's time for a new(er) topic in an old conversation:
>  Testing

This has also been my view, and I have secretly been working on a
build server I call Portmill, which polls the svn, and builds every
port that changes. Results get posted to a web app which presents
them, together with build logs (tails) for the ones that fail. This
all is in a reasonably usable state just now, and I have spent the day
adding a feed and links to trac changesets and such convenience stuff.
The builds happen on my laptop machine right now, and the web app is
on a small server slice.

Internally this relies heavily on Bryan's MPAB scripts, which I have
adopted a bit to my needs, e.g. to be able to sync the ports tree and
bring up a shell inside the chroot to look after the situation for
debugging.

Wrapping around MPAB is a ruby program 'mpbuildserver', which
periodically looks into the svn log, pulls changes found there if
there are any, and builds them inside the chroot environment.

The source is all online visible, and you can find it here:
http://trac.macports.org/browser/contrib/mpab -- MacPorts Autobuild,
the chroot environment
http://github.com/febeling/mpbuildserver -- the actual buider process
http://github.com/langalex/portmill  -- the web frontend on the results

There is nothing that would stop us from running the mpbuildserver on
several different machines, potentially with a range of different OS
versions, and we would get a picture of how well the most recent
changes work.

Obviously only building the changed port is not the whole story,
because it does not reveal if dependents break through new versions,
and this might in fact be a more common case of failure.

But I guess only an minority of maintainers have a set of machines at
their hands which covers all the OS versions which are theoretically
supported, and that would be covered by what I have now. So this setup
offers something valuable already.

Currently the web frontend is quite limited in features and slow
because of simplistic implementation in certain parts. (The list view
load the build logs 8). But we can probably get a feel for which
benefits one can expect from such a system. I hope that the direct
feedback helps quite a bit, but we will see. Of course I'd be happy to
relocate everything to a macports or apple server.

Another thing that has to be looked after is ease of deployment. This
is quite slick for the web app, by virtue of being a
rails+capistrano+passenger app, but the build
box/buildserver/buildslave part is a bit rough as of yet. I have
discussing this with Bryan lately. Bryan also helped with ideas with
problems I had with registry state, so that the chroot now runs trunk,
which works better that 1.7.1.

Another milestone I see would be to move the svn polling back onto the
server, so that it creates a build queue of sorts. That queue could
then be filled in more than one way, and, depending on the number of
available build machine, also build dependents of changed ports. Maybe
also running all combinations of variants.

So, that's Portmill :) Hope you like it and see some use in it. As
soon as the installation of the build box part is smoothed out a bit I
would invite volunteers to host build boxes. Or there we could get
even Apple help and use always-on machines in a data-center.

Cheers,
Florian


-- 
Florian Ebeling
Twitter: febeling
florian.ebeling at gmail.com