Semi-automating updates

[Ryan Schmidt]

On Mar 24, 2018, at 16:06, Perry E. Metzger wrote:

> It occurs to me that
> 
> 1) Some ports have "livecheck", which is automated detection that an
>   updated source file is available.
> 2) Some ports have "port test" automated testing available (though not
>   many.)
> 
> It might therefore be interesting (though not a small amount of work)
> to set something up so that, for ports with both livecheck and tests,
> and where the maintainer consents, a service would look for updates,
> and if one happened, it would create a Portfile with a properly bumped
> revision number and updated checksums,

The "update the version and checksums" part is https://trac.macports.org/ticket/53851

> run the automated tests, and if
> they passed, submit a pull request on the maintainer's behalf. The
> maintainer would be assigned the request so they could quickly check
> why the update happened and decide if they wanted to proceed or not,
> but that would be a few moments of work instead of a much more
> significant amount of tedium.
> 
> Again, creating such a service would be no small amount of work, but
> it would get rid of a great deal of tedium and make it much easier to
> keep a large fraction of ports up to date with only a minute or two of
> work once set up.
> 
> Presumably a port maintainer could indicate that they're happy with
> this arrangement by setting a variable in their Portfile, which would
> not work unless livecheck and test were also turned on. Such a scheme
> really does require that tests be present, because without them,
> there's no way to know that the robot is not requesting a pull for a
> completely broken update.
> 
> (If the update doesn't build or test correctly on its own, presumably
> the bot could inform the maintainer so they could clean up any
> breakage manually.)
> 
> Anyway, do people think this is an interesting idea, or is it
> nutty?


It sounds like yet another build service, distinct yet again from our existing Buildbot and the Travis setup which is already different from the Buildbot setup. I already dislike that we have two separate services.

While many ports update trivially, and such a service could help with that, I do not find it inconvenient to manually run "port livecheck maintainer:ryandesign" every so often, and then run my update script for those ports that need it, verify it builds, and commit it.

More time is spent on the nontrivial updates, and I feel that that comprises a nontrivial portion of the updates. A new version of a program may require a new library. A new library may have a new major library version, necessitating identifying and revbumping all ports linking with it. A port may have had a stealth update on the previous version, necessitating removing the lines of code dealing with that when updating. And there are countless other little bits and pieces that tend to get updated at the same time as version updates. If the port doesn't indicate its license, I try to find it and indicate it. If I haven't added size to the checksums yet, I'll do that. If I haven't added my GitHub handle to the maintainers line yet, I'll do that. If a patchfile needs to be refreshed, I'll do that. If we've agreed that we want to defaulting to a newer version of Python, and this port uses an older version, I'll update it. If I notice something in the port that does not conform to our current best practices, I'll try to fix it. If the port's whitespace is wrong, I'll fix it.

These fixes may be in several commits but none of them would've happened if I hadn't looked into the portfile. Many times, a version update is the only time when I'll actually look at a portfile. If the basics were fully automated for me, I might never look at the portfiles, and never see all the horrible outdated methods it's using, which, were another developer to look at the portfile, might give them an incorrect impression of what our best practices are. I'm still constantly seeing developers submit portfiles that do something in a convoluted way, because they copied the code out of an old portfile, instead of using new simpler methods that had been developed since then.

If your suggestion would only apply to ports that have test phases, then that will apply to few ports indeed. And it presumes that the test suite should pass. I seldom run my ports' test suites, but I know some of the test suites don't pass. I don't let that stop me from updating the ports. If, for example, glib2's collation test doesn't pass, and it's been that way for years and the developers don't address the upstream ticket, not updating the port won't help anyone.

Perhaps automatically running the test suite sometime would help remind me and other maintainers that they're there, remind us of the test failures, encourage us to work with the developers to finally resolve the problems. Perhaps we should be more vigilant about looking for test suites and setting the port up to run them if available. Automatically running the test suite is something we could possibly add to the existing mpbb program used by the Buildbot and the Travis setup. If we want maintainers to notice the results and do something about them, we'd have to fix the problem where the emails the Buildbot sends don't always get delivered, and we'd have to add the test results into the email.

I guess I feel that test suites are not really our problem. It's nice that we have facilities for running them in MacPorts, for interested parties, but the responsibility for ensuring that a program's tests pass is the developers', not ours.