GSoC 15 - Dependency Calculation using SAT

[Clemens Lang]

Hi,

----- On 17 Mar, 2015, at 14:43, Jackson Isaac jacksonisaac2008 at gmail.com wrote:

> As you know the application period has started. Can you please help me
> out with some things regarding the proposal.

Certainly, I'd be happy to :)


> I have built the latest Macports (i.e 2.3.99 from source) and as you
> said I can see the confirmation before installing dependencies and
> also it looks much cleaner now.
> 
> Also I went through the source code. The dependency is build in
> src/macports1.0/macports.tcl src/macports1.0/portdepends.tcl and
> src/macports1.0/macports_dlist.tcl if I am not wrong or missing out
> anything else.

Nope, that's correct. The code is old, but works fine.


> I have also gone through the CUDF at
> http://www.mancoosi.org/cudf/primer/ and also
> http://docs.codehaus.org/display/MAVEN/SAT+Based+Dependency+Resolution.
> 
> Does using CUDF mean that our portfile format will have to be modified
> (there is not much syntax difference, we might be able to interpret
> the old portfiles to a cudf file format while running dependency
> check.

I'm not particularly attached to actually using CUDF. I sounded like a
good idea when I last checked it, but it largely depends on the SAT
solver we'd use in the end. I'm not sure the current CUDF-solvers have
good support for specifying your own objective function to optimize.

But anyway, regardless of the format we actually use for the SAT solver,
a problem that might come up is actually converting Portfiles into a
representation that can be used. As you correctly stated, the results
of both formats are equivalent enough so that the actual conversion
isn't hard. However, to generate accurate dependency information, we'd
usually have to execute the Portfiles, and that can really slow the
whole process down quite a bit, which we should avoid. From what I've
seen, dependency resolution methods based on SAT solving expect the
complete universe of available packages and their dependency relations
plus a list of packages that should be installed after the current
action finished. Generating the complete list with accurate dependency
info by executing the Portfiles would take ~15 minutes and is thus
out of the question.

We do have a cache of the information, which we call the port index.
However, since Portfiles can execute arbitrary code, some Portfiles
may currently depend on the behavior that they are being evaluated
during dependency calculation (e.g. by changing their depends_lib
depending on the variant selection the user has given). We should
figure out a solution for that, for example by
 (a) somehow adding the effect of a variant on dependency relations
     to the portindex, so we could tell what the dependencies of
     foobar +bar +baz were without actually running the Portfile.
 (b) Pre-generating a universe description file (e.g. in CUDF
     format) and only updating it with accurate dependency
     information of the port we're currently installing and its
     dependencies. That would however, defeat the point of the
     whole exercise, because we'd still have to figure out the
     complete dependency tree before using a SAT solver to, well,
     figure out the dependency tree.
 (c) Pre-generate the universe description and find a different
     solution to deal with variants. There are a couple of
     approaches that might work here, such as doing fixpoint
     iteration of (1) figuring out dependencies using SAT solving,
     (2) updating the dependency information of these ports taking
     variants into account, (3) repeat until stable.

I'd actually prefer (a), to be honest. But please plan some time for
further discussion and design on this topic. Please don't hesitate to
ask if what I've just told you was unclear or way too complicated,
and I'll try to improve my writing ;-)


> Also the Phasing out xcode dependency resides in the diagnose.tcl
> file, maybe we can work on that too when we are done with the
> Dependency calculation.

I doubt that we'd actually find time for that, especially given the
problems I've outlined above. I'd rather put any leftover spare time
into writing a couple of test cases to verify that our dependency
solving works as it should. Sounds good?


Let me know if you have any further questions.

-- 
Clemens Lang