An idea for distributing ports

[James Berry]

On Apr 5, 2007, at 3:00 AM, Jordan K. Hubbard wrote:

>
> On Apr 4, 2007, at 3:14 AM, Randall Wood wrote:
>
>> The idea is this:
>>
>> Stable and Unstable Ports trees available via both rsync and http
>>
>> Port currently works by grabbing info out of the port registry and  
>> PortIndex file(s) on the local system and by looking at Portfiles  
>> in the source trees whenever it is going to change the state of  
>> the system by installing/uninstalling/activating/deactivating/ 
>> whatever a port.
>
> I think this doesn't go far enough from where we currently are to  
> justify the work involved.   I'm much more enthusiastic about  
> taking Kevin's "remote index" idea to its logical conclusion and  
> also, rather than thinking of things in terms of "stable and  
> unstable branches", go to the more powerful model of port  
> attributes which allows for "stability" to be just one possible  
> search/selection criteria.

Thanks for the recap Jordan. And in case anybody is curious, this is  
pretty the outline of what I've been working on with my mpwa  
(MacPorts Web App) prototype:

	- "port submit" to submit a portpkg to the database which gives it a  
unique pkgid and http: url.
		(A portpkg is simply a bundled up port directory, plus any  
additional meta information we need).

	- A url that may be used to request a particular (previously  
submitted) portpkg.

	- The webapp will also generate html UI to describe and index ports.  
For each port, there is a set of portpkgs for the port, each of which  
will include additional meta information such as:
		- submitter, date, etc.
		- version, epoch, revision
		- variants, dependencies
		- tags
		- Votes (thumbs up/down)
		- Build and/or install reports (go/no-go status, logs from failed  
builds)

	- Jordan describes attributes, I'm thinking tags, since they seem  
more comfortable in these web 2.0 days, but I think they're basically  
the same ;). Some tags might be restricted (perhaps "stable"); most  
will be accessible by all. Tags should also replace the concept of  
category. But the idea is that you can search and/or browse for ports  
by tag, and you might also be able to limit installs to only ports  
that match certain tags, like "stable", or whatever.

	- Some sort of port dependency closure solver interface. So I can  
submit a query that says something like "I've got these ports, with  
these variants installed, and I'd like to end with these ports, with  
these variants, and perhaps restricted only to the latest versions of  
ports with these tags: now tell me what actions need to be taken, in  
which order, to achieve that state." And the answer would come back  
as a set of actions to be take: "remove these ports, install these  
ports/variants (and here are their urls) (and here are their  
binaries)", or "you can't do that, Dave, because you asked for stable  
ports only and version 3.6 of this port is only available without  
that tag". The nice part about doing this over the net is that the  
database of potential portpkgs to install will likely grow far bigger  
than you want to keep locally. The downside, of course, is that you  
need net reachability in order to perform the action. A middle ground  
might be that you could perform the solver query and fetch the needed  
set of portpkgs while online, and do the actual install while  
offline. The reality of MacPorts today, of course, is that you mostly  
need to be online to install anything anyway, since src needs to be  
fetched.

Anyway, those are the ideas I'm playing with. The app I've started  
writing is in Ruby on Rails, and I'm thinking the database will be  
MySQL to contain port/portpkg information, metadata, and dependency  
information.

I'm hoping that some pieces of this may come from GSoC:

	- Dependency analysis given a portpkg (what are the dependencies by  
given each variant).

	- New dependency expressions: a dependency might be expressed onto a  
particular port, plus variants, plus other metadata restrictions:  
version >= 2.5; variants requires (+server, -client); variants  
conflicts (apache); tags has (foo,bar); submitter in (jberry,mww);  
etc. Syntax tbd.

	- Dependency solver

	- Reporting of build status/logs back to the webapp.

There's some design information here, though it's pretty fluid still:  
http://trac.macosforge.org/projects/macports/browser/users/jberry/ 
mpwa/doc/design.txt.

Oh, and I guess it goes without saying that once we build the  
infrastructure do all that, we can also begin to do autobuilds of  
things and to add binaries for portpkgs.

James.




>
> For those who didn't follow the whole remote index discussion way  
> back when it happened, the general concepts were as follows:
>
> 1. dports/ as a hierarchy of directories and files goes away.   It  
> is replaced by a flat "database" (at least conceptually - the back- 
> end implementation of this database can be done any number of ways,  
> from the crude to the impressive) of port blobs, where a "blob" is  
> basically everything that currently exists in a port directory  
> (Portfile + files/ dir) plus a set of attributes.  The most obvious  
> attributes to start with would be name, version and category, but  
> of course there could be any number (including "stable", "devel" or  
> even "unreviewed"), limited only by pragmatism and imagination.
>
> 2. sources.conf lists one or more database sources, by http: URL.   
> Since pretty much everyone's firewall allows http access, that gets  
> around that particular problem, though other types of URLs could of  
> course be supported (including, of course, a local database for  
> disconnected operation).
>
> 3. Assuming that a database can be contacted, "port install foo"  
> then becomes:
> 	3a). Dispatch a search query for foo
> 	3b). foo is found (let's assume that) and its blob is sent back.
> 	3c). The blob is unpacked and port walks in there to deal with the  
> Portfile (and files/) just as it does today.  For all dependencies,  
> of course, repeat steps 3a-3c recursively as needed.
> 	3d). Clean up and remove the blob when finished.
>
> 4. Now, since we have a database, other things become possible,  
> such as "port submit".   Someone who wants to create a new port and  
> make it available to everyone could, conceivably, go write that  
> port, cd into its directory and say "port submit foo".  This would  
> cause foo to be bundled into a blob and shipped off to the remote  
> database, where it would probably (as a matter of policy) be tagged  
> with the "unreviewed" attribute.   Anyone else with read access to  
> this database could grab that port immediately thereafter and try  
> it for themselves, so the database becomes a collaborative tool as  
> well.
>
> It's purely a policy decision, of course, but one could even allow  
> for multiple "foos" to exist at the same time, adding some sort of  
> disambiguating attribute, so that several people could submit foo  
> ports and let the reviewers pick the best for stamping "reviewed".   
> We've had people collide in the tree before (usually submitting the  
> same port in two different categories), this would simply make that  
> less traumatic.  Needless to say, another policy for "port submit  
> foo" could also be to emit an error if a foo already existed and  
> force the submitter to disambiguate it if they really had a good  
> reason for submitting another port of foo.  However we want to go,  
> it's fairly simple to implement the policy once all the other work  
> is done.
>
> Needless to say, if the database is sufficiently powerful to allow  
> for multiple versions of foo to co-exist (and I see no reason why  
> it wouldn't be), the need to maintain dports under svn also goes  
> pretty much by the wayside.  The database IS an SCM tool unto  
> itself at that point.
>
> Also pretty much needless to say, a database means that the index  
> is updated whenever something is submitted or has its attributes  
> changed, so there's no need to manually recreate and check in a new  
> PortIndex.
>
> - Jordan
>
> _______________________________________________
> macports-dev mailing list
> macports-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo/macports-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.macosforge.org/pipermail/macports-dev/attachments/20070405/90695ba3/attachment.html