GSoC Proposal: Rewrite key parts of MacPorts in Python

[Jackson Isaac]

Hi Ryan,

On Thu, Apr 2, 2020 at 3:44 AM Ryan Schmidt <ryandesign at macports.org> wrote:
>
> On Apr 1, 2020, at 18:20, Ryan Schmidt wrote:
>
> > On Mar 31, 2020, at 16:14, Alex Ionkov wrote:
> >
> >> I submitted a proposal this year for rewriting parts of MacPorts in Python. The eventual goal is to rewrite all of MacPorts in Python to increase modularity and make integration of other APIs with MacPorts easier.
> >
> > Is this an idea you just came up with or has there been some discussion of this before? I didn't see this idea on our GSoC page [1] and it comes as a surprise to me.
> >
> > [1] https://trac.macports.org/wiki/SummerOfCode#Projects
>
> I see now that it is actually on the list, as just a one-line item:
>
> https://trac.macports.org/wiki/SummerOfCode?action=diff&version=337&old_version=336
>
> So then my comments are not so much about your proposal, Alex, as it is about the fact that this item got onto the list at all.
>
>
> MacPorts has 18 years worth of Tcl code which not even us long-time MacPorts developers fully understand. It works as is, and in the spirit of not modifying a working system, my inclination would be not to change it. I may not know where everything is or how everything works in MacPorts base, but I know where many things are and how many things work, and I fear that knowledge would be lost if it were rewritten.  Certainly we should fix bugs. Certainly we should make incremental improvements like whatever is necessary to allow us to update to Tcl 8.6 (i.e. fix the "catch" problem). But rewriting MacPorts wholesale in another programming language seems completely infeasible to me.
>
> I'm sure that rewriting MacPorts base would introduce new bugs, and I'm not confident that our existing test suite would be adequate to catch them. In fact, our test suite is written in Tcl, so there's double the possibility of missing bugs: either because the test suite didn't cover a particular scenario or because rewriting the test suite in Python introduced new bugs into it.
>
> It's not just rewriting MacPorts base either. The more than 10,000 portfiles and the portgroups are also written in Tcl. Should they all be rewritten in Python as well? Doing that manually would take colossal effort, and doing it automatically would require writing a fairly full-featured Tcl-to-Python converter. Or if you propose that the portfiles should be kept in Tcl, then that means Python would have to launch Tcl to interpret each portfile, which seems unnecessarily more complex than what we have now. Using yet another programming language in MacPorts makes it that much harder for new developers to get up to speed.
>
> Note that Tcl was deliberately chosen because of its simple syntax. If we convert portfiles to Python, would that make basic everyday usage more verbose and if so do we really want that? As one specific example, setting the name of a port to "foo" is currently as simple as writing:
>
> name foo
>
> If portfiles become Python files, I am guessing that would have to be rewritten as:
>
> name = 'foo'
>
> and that kind of superfluous punctuational verbosity is what the original designers of MacPorts were trying to avoid.
>
> There are also a lot of little utilities in the macports-contrib repository that hook into MacPorts via Tcl. Our buildbot setup uses several more such scripts developed just for that purpose. All of these would then need to be rewritten in Python.
>
> Needless to say, this also requires a thorough understanding of Python, both to perform this massive one-time conversion and then on an ongoing basis by all MacPorts developers and maintainers. Maybe many people are already familiar with Python, but some MacPorts developers like myself are more familiar with Tcl at this point. Perhaps we could all retrain ourselves to learn Python. But I can't help but recall how four years ago we undertook the massive conversion of our source code from Subversion to GitHub, from which I still haven't fully recovered; I still can't contribute to MacPorts as effectively anymore as I did when we used Subversion. I'm probably in the minority there as other developers were already familiar with git at the time, and if everybody else wants MacPorts rewritten in Python and it can be done then I won't stand in the way.
>
> I don't recall any of this discussion occurring on the mailing list before. I usually stay away from Google Summer of Code issues, but I would have hoped that GSoC would be used as an opportunity for us to finally implement changes that we have wanted to do for years but never got around to doing, rather than to propose new projects whose ramifications have never been discussed before.

Thanks for your valuable insights and I really appreciate the detailed
feedback. I didn't realize these things before and the efforts that
went into creating and maintaining such a great project.

I sincerely apologize to the community and also to the student for my
ignorance and the fact that I didn't discuss this before in detail on
the mailing list and understand the implications that it would make
before putting anything on the ideas page.

It was a pre-mature action from my side, and my intention was not to
hurt anyone. This was also a learning for me personally to not put
things in place without much thought, discussion and understanding.

-- 
Jackson Isaac