port install efficiency issue

[Darren Weber]

Is it possible to create a distributed build system that uses Xgrid, to
allow all macport users the option of adding their machine to a distributed
macports build system?  In effect, every time anybody on this grid has to
build a package from source, some kind of meta-package monitor can detect
whether the build and install was successful and then make an inquiry of a
central managment system as to whether or not this build should be added to
the mirrors for binary distributions.  If this inquiry was made before the
build and a binary is available somewhere, the system simply downloads the
binary (perhaps a torrent system).  If the binary is not available, the
build proceeds.  If it's successful, the outcome of the build is added to
the binary downloads available (ie, uploaded and distributed to mirrors to
become an effective part of a torrent download system).  Also, perhaps the
Xgrid system can be used to create a distributed build system, regardless of
what packages a user installs, but make sure that it uses idle systems (a
lot like the scheduled builds of older days).

For example:
http://cmgm.stanford.edu/~cparnot/xgrid-stanford/
http://www.apple.com/downloads/dashboard/status/xgridstanford.html



On Sun, Mar 22, 2009 at 5:54 PM, Joshua Root <jmr at macports.org> wrote:

> Scott Haneda wrote:
> > Can we talk more about this?  I have the ability to host such a build
> > farm.  Now, I could not host one machine, of every architecture, of
> > every OS, I just do not have the room in colocation.
> >
> > I do have quite a bit of room if we go 1U though.  So 2 1U machines, a
> > PPC and a Intel, and I would imagine, that PPC machine could go away a
> > lot sooner than we all think.  Mac Mini's could take it further, since
> > they are so small, 8 of them can fit on a shelf and occupy no more than
> > a few U's of space.  The damn power bricks are more an issue than
> > anything else.  There have to be PPC Mini's out there to be had.
> >
> > As long as the various OS versions could be virtualized, so we could
> > have 10.3, 10.4, 10.5, 10.6 and forward virtualized on each machine, it
> > would not at all be hard to come up with a authentication routine to
> > allow builds to happen on whatever virtual interface you want.
> >
> > I have the Ip space to spare, so each virtual machine could have it's
> > own connection space, or we could do some simple dhcp pooling.  Static
> > IP's are something I was alloted a /24 of, and do not see giving up 20
> > or so of them being an issue.
> >
> > Am I looking at this wrong, or would this be helpful?  Is this too much
> > reliance on an outside party for core ports features?  I have had access
> > to this data center for 9+ years or so, I do not plan on giving up that
> > access at any time soon.  The bandwidth requirements are minimal, it
> > would go unnoticed.
>
> Thanks for the offer. I think it's safe to say, though, that finding the
> hardware for a build farm will be a relatively minor issue.
>
> The main thing that is needed is the software. Server-side, to build all
> the ports in some kind of intelligent order and copy the archives to
> somewhere from which they can be downloaded by the client, with error
> reporting; and to keep doing this as needed when new versions and new
> ports are added.
>
> Client-side, to check for the presence of an archive with the desired
> version/variants on the server, and to download and unarchive it instead
> of building from source when possible.
>
> - Josh
> _______________________________________________
> macports-users mailing list
> macports-users at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/macports-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/macports-users/attachments/20090322/ab36d46f/attachment.html>