Issues with oudated ports / GitHub

Fri Oct 7 09:02:19 PDT 2016

> On Oct 7, 2016, at 10:23 AM, Marcel Bischoff <marcel at herrbischoff.com> wrote:
> 

>>> Just today I commented on a ticket that is six weeks old, about an
>>> update to nodejs4. Version 4.5.0 was released on 16-Aug-2016, version
>>> 4.6.0 on 27-Sep-2016. Version 4 is considered the stable LTS variant,
>>> only fixing security issues without introducing new features. This makes
>>> timely updates all the more important. If installing software by hand
>>> results in more current and more secure software for my development
>>> machine, I don't get the point of using MacPorts in the first place.
>> 
>> Our ports do have maintainers, which are people who expressed interest
>> in a port and are interested in keeping them up-to-date. This is meant
>> to assign issues with ports to certain people that care about this port
>> and would have the most knowledge how to fix it.
>> 
>> However, we also have a maintainer timeout policy [1] by which updates
>> to ports can be committed by any project member after 72 hours of
>> submission if the maintainer does not show any reaction. In case a port
>> is in a broken state (does not compile at all, typo, etc.) or
>> vulnerabilities require a security patch, any project member can
>> immediately commit an update.
>> 
>> In this particular case [2], a ticket was filed with an update request,
>> but no patch was attached. With a patch, it would be much easier to
>> verify that the build still works and then commit the update.
>> 
>> The backlog of patches already approved by the maintainer is not that
>> long [3] right now. Those still in the queue often already went through
>> review, but issues with the update were found that need to be addressed.
>> 
>> If you think a contribution has stalled, ping the macports-dev list
>> about it and request a review. This has proved to work quite well in
>> recent time.
> 
> I see. My understanding was that a port maintainer would also be
> responsible for updating the port in a timely manner, regardless of an
> already available patch. So this is not required to be a maintainer?

That would be nice. However, we can't force volunteers to do anything. If a volunteer appears over time to be unresponsive to requests, we can remove that volunteer.

>>> It pains me to say that Homebrew is running circles around MacPorts in
>>> the department of current available packages.
>>> 
>>> If time and manpower is the problem, wouldn't it be better to move to a
>>> GitHub-based approach like Homebrew does? This way far more people would
>>> contribute. It would lower the bar to contributing significantly. I like
>>> MacPorts' clean implementation far more than Homebrews'. But if I still
>>> cannot install (for example) pandoc because ghc still requires llwm-3.5
>>> which does not compile on Sierra: what choice do I have? I need to get
>>> stuff done, not tinker with the infrastructure of my working environment
>>> for hours on end, just to get it to work. A package manager's sole
>>> reason for being is to make the routine task of installing software and
>>> updates easier, more reliable and trustworthy.
>> 
>> After all, MacPorts maintainers are just some people devoting their free
>> time to writing Portfiles. Sometimes other stuff in life stops them from
>> responding to tickets or addressing certain issues.
> 
> This is a given. Open source projects are mostly staffed by unpaid
> volunteers. I don't feel entitled to recieve updates, as it's all free.
> However, a well-run project does inspire constant commitment to keep it
> running smoothly. MacPorts does not *feel* like being very active, quite
> inaccessible to new users and there does not appear to be current
> information easily accessible. A good example for this would be the move
> to GitHub. Some status messages outside of the mailing list or even a
> dedicated FAQ would go a long way. Without this mailing list, an
> outsider easily gets the impression MacPorts is outdated and running on
> fumes.

What gives the impression that we aren't active?

There are tons of posts on the mailing lists every week.

There are tons of commits to the repository each day.

There are tons of tickets filed each week.

>> I can understand your frustration about broken ports and missing
>> updates. I have that as well.
>> 
>> If your work depends on these ports, you benefit from the donation of
>> time from the people keeping the port up-to-date. The best you can do if
>> you encounter a problem is to check if it is a known issue in our Trac.
>> Sometimes others already proposed a solution but did not have the time
>> to get through with updating the Portfile in multiple iterations testing
>> the change. Yes, this is a time-consuming process, but others would
>> benefit from your work as well as you benefit form theirs.
> 
> Absolutely. The process itself of posting a diff just feels archaic and
> like an uphill battle.

And we will switch to GitHub, and use pull requests. But sending a pull request, or attaching a diff to a ticket, is not the hard part.

>> Just switching to GitHub would not change the way port updates are put
>> together. In my experience, preparing the patch and attaching it to a
>> ticket is only a small fraction of what needs to be done for a port
>> update. You will still need to update the Portfile, test the build, etc.
> 
> This is obviously the part of the process that I have not seen yet.
> Until now I have only submitted Portfile diffs, hoping to get them
> included.

If you have submitted a diff, then you have done the hard part: you've figured out what needed to be changed.

>>> The way it is now, I repeatedly find myself in need to modify the
>>> Portfiles manually, test, check, test again and so on. Or I simply run
>>> `brew install pandoc` and — be done. Plus, doing all that manual diffing
>>> where just a minor mistake will frustrate me and requires me to attach a
>>> completely new diff file quite honestly keeps me from contributing
>>> regularly. In 2016 this feels like pulling teeth, not satisfaction or a
>>> sense of accomplishment that should accompany working together on free
>>> software.
>> 
>> Even as a non-committer you can easily work from a Subversion checkout
>> with local modifications and create the patch from there. Or you keep a
>> separate local ports tree that you diff against.
> 
> I actually keep a local ports tree with modifications for a couple of
> ports that are not updated in a timely manner. Given a smoother
> workflow, I'm very interested in updating those for the benefit of all
> users. My experience so far is that things move very slow if at all.

If a ticket does not move in 72 hours, write a note to this mailing list requesting someone look at it.

>> Where exactly do you see the benefit of Git/GitHub for this? I am
>> honestly interested in this for the upcoming workflow with GitHub.
> 
> I see the benefit mainly in branching and pull requests. The usual
> workflow I got used to is the following:
> 
> - Fork the project on GitHub.
> - Clone the whole project (in this case the ports tree) from a Git
> repository.
> - Create a branch in place.
> - Make the edits, test them, then commit them.
> - Push the branch to my own fork.
> - Create a pull request from this branch against the original
> repository.

Which, from my perspective, is more difficult than our current process which is:

- Check out a working copy of the Subversion repository.
- Make the edits, test them, create a diff.
- Attach the diff to a Trac ticket.

But in either case, the hard part is "make the edits, test them"; that doesn't change when we move to GitHub. We current receive many low-quality submissions that require further editing. I don't anticipate the quality of submissions will magically improve just by moving to GitHub.

Something we could do to improve the quality of submissions would be to improve the quality of our portfile-writing documentation. Someday I would like to rewrite all of our documentation in a new cohesive format, but that is a big effort.

> If changes are needed, they can be coordinated on a line-by-line basis
> directly within the pull request until everything is in order.

I will admit that this process is completely foreign to me. We will all have to learn how to work with GitHub once we switch.

> Optionally, an automatic build process can be triggered with any single
> commit so that immediate feedback about the state of the commit can be
> viewed directly from within said pull request.
> 
> Obviously, the feasibility of automatic builds relies heavily of the
> available infrastructure. With hundreds of commits per day this could
> make automatic build very slow. However, a service like Travis offers
> free CI infrastructure for open source projects and provides several Mac
> OS X/macOS environments.

Travis is not appropriate for us. (For one thing, Travis doesn't give root access, which MacPorts needs.) Therefore we have our own Buildbot system:

https://build.macports.org

But we cannot use the existing system for untested pull requests; among other problems, that's a security risk. We will need to develop a new system for that, likely involving spinning up new pristine virtual machines for each build attempt and destroying them afterward. I imagine this will take a long time to develop and I don't see this happening anytime soon.

>>> I'd really like to hear from several sides about the why's and how's
>>> regarding MacPorts' current state and what the plan is for the
>>> forseeable future. I was told that there is a current discussion
>>> underway about a possible move to GitHub. I would throw whatever weight
>>> I have behind that move. If you need another helping hand for clearly
>>> defined work, let me know. I'll be more than happy to involve myself.
>> 
>> This not only a discussion, we already decided and announced this [4].
>> 
>> This is an ongoing process for months now. The port managers worked on
>> this before the announcement and evaluated various options for the
>> future of the project. We also gathered input from macports-dev from our
>> maintainers and eventually reached consensus.
>> 
>> In the last months, lots of time of some core contributors to MacPorts
>> was already invested into the transition to git, to GitHub and of
>> everything else away from the closing Mac OS Forge, including our
>> buildbots. This is time that they would have otherwise been able to put
>> into the ports they maintain, into ticket reviews or base development.
> 
> This may explain why things feel like they are moving very slowly. With
> major changes behind the scenes it is quite understandable that an
> already limited supply of time is stretched even thinner. This is also a
> perfect example of why better communication would go a long way. A
> prominent note on the website with further information about the current
> state of the project, the switch to GitHub and insight into why things
> are the way they are would keep everyone in the loop.
> 
> If you think, like me, that this kind of information would benefit
> users, I'll gladly volunteer to keep those entries current, freeing
> resources for the core developers and infrastructure engineers.

We announce things on the announcement list and on the news section of our web site as it happens. I don't have any further status updates that I wish to communicate at this time. I don't know what kind of prominent note you would like to see on the web site. A note that we are still alive?

I recognize that our web presence is not up to modern standards. I indent to modernize it, after we move to GitHub.

>> Personally, I am looking forward to switching to Git/GitHub. When we are
>> finally through with the move, I hope to be able to focus on ports and
>> base again instead of the infrastructure.
>> 
>> Rainer
>> 
>> [1] https://guide.macports.org/#project.update-policies.nonmaintainer
>> [2] https://trac.macports.org/ticket/52110
>> [3]
>> https://trac.macports.org/query?status=assigned&status=new&status=reopened&keywords=~haspatch+maintainer&col=id&col=summary&col=keywords&col=status&col=owner&col=type&col=priority&col=time&col=changetime&order=priority
>> [4]
>> https://lists.macosforge.org/pipermail/macports-dev/2016-August/033405.html
> 
> Thanks again for taking the time from your no doubt busy schedule to
> clear things up for me.