Proposal for a new repository layout

[Juan Manuel Palacios]

On Apr 25, 2007, at 7:25 AM, Paul Guyot wrote:

> On Apr 25, 2007, at 9:02 AM, Ryan Schmidt wrote:
>
>>> 	I propose we start with those two trees and fill them up as  
>>> necessary, the first one being the existing trunk/dports which  
>>> would *have* to stay in sync with base/branches/release_x_y and  
>>> the second being a testbed for Portfiles that need unreleased  
>>> features and syntax to function (ports would be added to it as  
>>> needed, no need to mirror every single category and port in ports/ 
>>> released if the corresponding Portfile doesn't pack anything  
>>> different). I would like to make clear that our released Vs.  
>>> testing split would be with respect to our *Portfiles* and not to  
>>> the projects we port; that is, emacs and emacs-devel would still  
>>> be in the released tree as long as their Portfiles are in sync  
>>> with MacPorts released code. This is contrary to the a la Fink  
>>> "stable" Vs. "unstable" model, which we all know condemns stable  
>>> to always be incredibly out of date and pushes everyone to use  
>>> the unstable tree, while alerting users their computers could  
>>> catch fire.
>>
>> Having "released" and "testing" ports trees sound like a good idea.
>
> I disagree. What's the point of putting portfiles in testing?
> Who's going to move them to release when they are ready?
> How installation will pick them?


	The testing ports tree would just be a facility offered by the  
project to developers who, like you, develop new features in trunk  
and/or bug fixes and, naturally, want and need to test them in  
Portfiles. Such Portfiles break an MP installation based on released  
code as we've seen one too many times already, so we can't have them  
in the shipping ("released") ports tree as they are useless there.  
Here's where my "testing" tree proposal comes in, it's name (or  
whatever is more descriptive of its purpose) being simply a label  
claiming "these portfiles are meant to work with trunk, so don't  
complain if you get errors". As I explained in my original message,  
this tree would not be some kind of Secret Service Agent keeping a  
detailed eye on each of our Portfiles (that is, "not another road  
block to get things into users hands"), some kind of prerequisite  
every single Portfile would have to meet and go through before being  
considered "stable" and moved to the "released" tree after some  
approval is granted.... no, not at all, not at all what I meant.

	On the contrary, consider "testing" as a tree open to our entire  
developer community for testing MacPorts itself, putting in Portfiles  
therein whatever you want and need, thus lifting every single  
restriction you may ever encounter with Portfiles in "released". We  
would not ship the "testing" tree by default, just make it available  
for those knowledgeable enough to configure their sources to fetch  
it. We would not (or at least that's not my intent) hold Portfiles in  
"testing" with different (newer) versions for the project they  
install with respect to the corresponding Portfile in "released", the  
difference between the trees simply being the features each Portfile  
sports: emacs and emacs-devel would both be in "released" as long as  
they both work with a released version of MP; one of them, whichever,  
could be duplicated into the "testing" tree to test a new *MacPorts*  
feature. I can't stress this enough, this is not an a la Fink  
"stable" Vs. "unstable" trees proposal.

	Also, the "testing" tree would not mirror "released" either with  
respect to every single category and portdir in the latter, they  
would simply be created in the former as needed. If you want to test  
a new MP feature in one of your Portfiles you would be free to do  
whatever you need in the "testing", and otherwise there would be no  
need to add it there, it would simply live in "released", as every  
single other Portfile.

	In short, "released" is our existing dports tree renamed, "testing"  
is where anyone would be free to play in an open fashion. If you keep  
a local tree to play with new features in trunk, your testing would  
be limited to you and anyone you send your development Portfiles to;  
if you put them in dports things break, as they already have; if you  
make them available to everyone through an alternate tree where you  
can feel free to play as you like, wouldn't that, at least  
potentially, translate into wider testing?

	To put it in other words, nothing of what we now have would  
change..... I'm just proposing we *also* offer something new, an  
alternate and public tree for the purpose of testing new MP features  
and syntax.

>
> Releases are already a big pain, which explains why we have them so  
> infrequently.


	Not recently,we've done 4 releases in less than a month. I am aware  
I personally wasn't too convinced about it in the first place, but I  
didn't oppose it either and your point that we could/should do more  
frequent releases was proven.

> Making this more pain will only make things worse.


	Nothing of what I am proposing here slows down the release process  
in any way. As already explained, the "testing" tree is not a  
requirement our Portfiles have to go through to only then cascade  
down into "released", so why would it make things worse?

>
> I think we should postpone the versioning of the Portfile APIs and  
> condition it to the existence of a remote repository (through mpwa  
> for example).


	The versioning of the API is something that could help us in many  
ways, I believe, but I agree this could be a different thread and  
that it could be postponed for various reasons (though I don't  
understand the ones you explain here). As for mpwa, I admit that I  
haven't done my homework in looking into it, but other than that I  
don't have a single idea about its roadmap. Tomorrow? Next month?  
MacPorts 2.0? What I'm proposing is something simple and easy that  
can help us *now* solve a problem we've had for long.

>
> And until we have continuous integration, I'm in favor of simply  
> dropping the very idea of releases.


	As I said at one point already, I get the feeling you forget what a  
pain the days of users using ToT were, how reduced our user base was  
because of that among other reasons and how difficult providing  
support was. I have to be adamant about this, I am completely opposed  
to the idea of dropping releases. Do we need to streamline our  
release process even further....? Yes, I agree and I would like to  
see it happen. Should someone quicker than me step in to do releases?  
I wouldn't be opposed to that... and, in fact, wasn't, as James  
already did. Dropping releases as some kind of panacea for our  
problems? Couldn't disagree more!

> The 1.4.x series show that base/ is globally extremely stable, that  
> major failures are introduced by active developers who are able to  
> fix them very quickly. It also proved how wrong the release  
> candidate workflow is.


	I disagree that the release candidate workflow is "wrong". That it  
doesn't serve us in the best way possible because we don't have a  
large enough testing community? I agree to that, but that does not  
mean the workflow is inherently flawed. Hey, testing MacPorts is in  
itself a difficult task, as it implies to some extent testing as many  
ports as possible, and that's a problem we're still to solve, albeit  
a somewhat different one.

> Active users who might test release candidates and trunk/ do not  
> run 10.3 (this means that testing time is infinite), while we are  
> able to fix bugs in base/ and ship fixes within 48 hours (this  
> means the update time is epsilon).


	Testing on Panther is difficult and a pain, yes. I was practically  
the only 10.3 committer and tester and, unfortunately, not any longer  
because I recently upgraded to Tiger. I'm getting the feeling we'll  
need to start thinking about dropping Panther support sooner than  
latero, as many Open Source projects already have. This is, however,  
a different problem, I believe.

>
> And I definitely do not understand the interest of changing the  
> repository layout, except for the joy of requiring every developer  
> who has uncommitted code to move their files and fix their code  
> manually to adapt to the new layout and names.


	trunk/base makes no sense for us, that's an svn adaptation of  
something we inherited from the old cvs layout that does not apply  
any longer. We're only versioning base, not our ports nor www nor  
docs, so why have all that in trunk? It's confusing at best. I say  
pull 'em out of there and put them elsewhere, as already suggested.  
We'd end up having only trunk/base, but then again, we're only  
versioning base! So why have {trunk,branches,tags} only for base?  
Makes more sense to have base/{trunk,branches,tags}, thus emphasizing  
base is the only thing we version. Am I an order freak? Yes, I am,  
that's no secret. But then again, I think we can all agree order is  
only beneficial. Can we envision the need of ever versioning anything  
else? Yes? OK, lets keep trunk/{base,something_else} and  
{branches,tags} parallel to trunk. If not... what's the use of the  
now somewhat gratuitous extra (and loner) "base" directory inside  
trunk to get to our sources? I don't see it.

	No, my motivation is not disturbing developers simply for the joy of  
it, not by far.

>
> Juan, I realize that I'm just proposing to simply cancel most of  
> your job within the project. And I realize this is partly why you  
> are so reluctant from dropping releases


	You are assuming that, I did not say nor imply that anywhere, and  
you're wrong. I am reluctant to dropping releases for a myriad of  
other, valid, reasons.

> you consider your job as the member of portmgr responsible for  
> releases is to be conservative against us, the liberal developers.


	One of my responsibilities is to insure our shipping product meets  
at least some quality standard, not to counterbalance anyone per se.

> Well, from my liberal developer's point of view, your contribution  
> to the project is mostly beyond this perceived job and your GSoC  
> proposal seemed much more useful for the project than this new  
> repository layout idea. But this is just my opinion, and I do not  
> mean to dictate anyone's behavior.


	I do not either, I appreciate highly all the work you do for the  
project and would never try to block or oppose it. On the contrary,  
what I'm trying here is to find better ways to streamline the  
workflow for our entire developer community, within certain  
parameters (as open as possible) that allow us to ensure necessary  
traits like quality control, versioning base being one of them.

>
> Paul


-jmpp


PS: Indexing the "testing" tree is a secondary problem that can be  
solved easily with minor tweaks to the IndexRegen.sh script.