Speed up trace mode (GSoC Project)

[Clemens Lang]

Hi Mihir,

On Thu, Mar 28, 2019 at 12:07:20AM +0530, Mihir Luthra wrote:
> On Wed, Mar 27, 2019 at 10:22 PM Mojca Miklavec <mojca at macports.org> wrote:
> > What I miss a bit is some clear definition of deliverables, what
> > pieces of code would be suitable enough for merging them into base
> > and when.
> >
> > Background: One of the problems with many earlier projects with
> > macports base was that a student was working all summer in his own
> > branch, waiting for the code to be "complete and well-tested", and
> > the code was subsequently never merged into the trunk / master, or
> > maybe it's still considered useful, just not 100% polished, and some
> > members still plan to clean up and merge the code as old as 10
> > years. On the other hand, if the code ends up in the master branch
> > early and some issues are discovered, they would still be fixed. If
> > the code lies in its own branch, it doesn't get nearly as much
> > testing and might get forgotten.
> >
> > Of course you cannot always merge immediately, as some pieces might
> > need a bigger pile of code at once to produce something useful
> > without breaking stuff. But it would be very helpful if some code
> > could be merged into master at least once per week. If you could
> > split it into smaller reasonable chunks, it would be even better to
> > do this on an almost daily basis when possible (it's still ok to
> > have two weeks of some bigger feature rework in the meantime).

I'd like to second Mojca's feedback on this. There should be clear
milestones with dates and you should have a rough plan on what you're
going to implement over the summer. In my experience, projects that
start out designing the solution during the implementation period tend
to run into time problems later on and don't finish.

In terms of your proposal, that likely means you should look up
literature on one or two, at most three data structures that you would
deem suitable for the task (like the arXiv link I sent you) and maybe
have a clear plan on how to compare them and choose one as an action
item in your plan, but not a period of "look up potential data
structures", because that task would have a time boundary.


On Thu, Mar 28, 2019 at 11:13:35PM +0530, Mihir Luthra wrote:
> Before the build, dependencies are checked. Enabling trace mode hides
> incompatible versions of the current software being installed or the
> versions installed by other package managers and more such.

Stricly speaking what gets hidden are files that MacPorts considers to
part of other ports or installed manually or by other package managers.

> The injected darwintrace.dylib will replace file operations if needed
> to be hidden.
> But I don’t exactly understand which “processes” call those file
> operations and exactly when. Like some examples of such processes?

Basically, darwintrace.dylib gets injected into the build process of the
software we're packaging. What exactly that software is depends on the
port. The most common cases are probably cmake, GNU make, clang,
clang++, gcc, g++ and shell scripts (such as the ones generated by GNU
autotools).

> And how is this lookup being done that arises the need for trace mode?
> Is it just because of autoconf of gnu? And if yes, can’t that autoconf
> be changed? I am not very sure about this.

While we could in theory change all build systems of our ports to ignore
files from /usr/local and ignore files installed by ports that are not
in the dependency tree of a port, doing so would require us to modify
each build system, and such modifications are often non-trivial.

Trace mode allows us to achieve the same thing without modifying the
build systems. As such, it's much easier to use and thus much more
likely to be used.

Attempting to "fix" each build system to match our expectations
(remember, other users may well want the behavior that the build systems
show by default) is an uphill battle that we cannot realistically
achieve with the manpower we have. Trace mode is achievable.


HTH,
-- 
Clemens