GSoC 15 - Dependency Calculation using SAT

[Mihai Moldovan]

On 19.03.2015 11:16 AM, Jackson Isaac wrote:
> On Thu, Mar 19, 2015 at 3:25 AM, Clemens Lang <cal at macports.org> wrote:
>> Hi,
>>
>> ----- On 17 Mar, 2015, at 14:43, Jackson Isaac jacksonisaac2008 at gmail.com wrote:
>>
>>> As you know the application period has started. Can you please help me
>>> out with some things regarding the proposal.
>> Certainly, I'd be happy to :)
>>
>>
>>> I have built the latest Macports (i.e 2.3.99 from source) and as you
>>> said I can see the confirmation before installing dependencies and
>>> also it looks much cleaner now.
>>>
>>> Also I went through the source code. The dependency is build in
>>> src/macports1.0/macports.tcl src/macports1.0/portdepends.tcl and
>>> src/macports1.0/macports_dlist.tcl if I am not wrong or missing out
>>> anything else.
>> Nope, that's correct. The code is old, but works fine.
>>
>>
>>> I have also gone through the CUDF at
>>> http://www.mancoosi.org/cudf/primer/ and also
>>> http://docs.codehaus.org/display/MAVEN/SAT+Based+Dependency+Resolution.
>>>
>>> Does using CUDF mean that our portfile format will have to be modified
>>> (there is not much syntax difference, we might be able to interpret
>>> the old portfiles to a cudf file format while running dependency
>>> check.
>> I'm not particularly attached to actually using CUDF. I sounded like a
>> good idea when I last checked it, but it largely depends on the SAT
>> solver we'd use in the end. I'm not sure the current CUDF-solvers have
>> good support for specifying your own objective function to optimize.
>>
>> But anyway, regardless of the format we actually use for the SAT solver,
>> a problem that might come up is actually converting Portfiles into a
>> representation that can be used. As you correctly stated, the results
>> of both formats are equivalent enough so that the actual conversion
>> isn't hard. However, to generate accurate dependency information, we'd
>> usually have to execute the Portfiles, and that can really slow the
>> whole process down quite a bit, which we should avoid. From what I've
>> seen, dependency resolution methods based on SAT solving expect the
>> complete universe of available packages and their dependency relations
>> plus a list of packages that should be installed after the current
>> action finished. Generating the complete list with accurate dependency
>> info by executing the Portfiles would take ~15 minutes and is thus
>> out of the question.
>>
>> We do have a cache of the information, which we call the port index.
>> However, since Portfiles can execute arbitrary code, some Portfiles
>> may currently depend on the behavior that they are being evaluated
>> during dependency calculation (e.g. by changing their depends_lib
>> depending on the variant selection the user has given). We should
>> figure out a solution for that, for example by
>>  (a) somehow adding the effect of a variant on dependency relations
>>      to the portindex, so we could tell what the dependencies of
>>      foobar +bar +baz were without actually running the Portfile.
> By variant do you mean the packages that are required by the package
> that we want to install.

No, variants in MacPorts world are used for disabling or enabling
features within a package.

Variants can add or remove dependencies, but need not.

Please see https://guide.macports.org/chunked/using.variants.html for a
general description of variants and
https://trac.macports.org/browser/trunk/dports/multimedia/mplayer-devel/Portfile
for an example Portfile with a lot of variants.

The problem with these is that currently, like for so many other things,
Portfiles must be completely evaluated (that is, the TCL code executed)
in order to have a complete list of dependencies -- and this "complete
list" isn't static but depends on the deactivated or activated variants,
the platform(!!! -- like darwin, linux, freebsd) ... and subplatform
(like macosx) and their specific versions.


> Do you mean like in Portfile we have:

Did you mean PortIndex instead of Portfile?


> packagename somenumber (I don't know what this number means) and
> followed by description.
>
> So here itself we add/cache the dependencies when we fetch the
> portfiles (or maybe when we are trying to install to keep reference
> for future).

The problem is that this is not static. See above. There are 2^n*
different states for n variants. For a port like mplayer-devel (or my
own ports, mpv and audacious-plugins), the amount of variant
combinations is *very* high and for each combination the dependencies
differ.

You do NOT want to record this in PortIndex, which is supposed to be
fast cache.



Mihai

*) Math is not my strength, this statement may be off by one or a million...

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL: <https://lists.macosforge.org/pipermail/macports-dev/attachments/20150319/7405fb1c/attachment.sig>