Volunteer for a workshop on "setting up your own buildbot/buildslave"?

[Ryan Schmidt]

On Nov 12, 2015, at 5:20 AM, Mojca Miklavec wrote:

> On Wed, Nov 11, 2015 at 12:15 PM, Ryan Schmidt wrote:
> 
>> I like the idea of only adding a non-default cxxstdlib to the filename, since that would allow all the existing archives to continue to be valid. However if we were to switch to xz compression for archives, we might want to repackage everything that way, in which case it wouldn't matter.
> 
> Yes, it would be helpful to repackage everything. But despite that I
> don't see the added value of adding the default stdlib to the package
> name. Or in particular, I would not be in favour of adding libc++ to
> the filenames on >= 10.9. (First of all, we don't add "-variant" to
> the names either and it's nicer to keep names shorter. Second of all,
> this is mostly relevant for < 10.9, I don't believe that anyone is
> trying to support libstdc++ on >= 10.9, so in the long run we'll end
> up with shorter names overall.)

These are all good points.


>> I agree that since we don't have a mechanism for detecting C++ software, it would be simplest to add the tag to all archives, even though that will result in a rather large increase in disk usage on the packages servers. If the packages server were using some sort of advanced automatic deduplicating filesystem the impact might not be so large, since many ports would hopefully build identically, but I don't know what filesystems do that or if we're using one of them.
> 
> If that is going to be a problem, we could at some point in future
> start thinking about "small" and "big" mirrors. Small mirrors (those
> that don't have sufficient disk space) could include just the latest
> versions of software and/or just for just the latest OSes, while the
> "big" mirrors could include everything. Old versions of binaries
> (older than one year and older than the latest successfully built
> version of that software) are seldom needed and even then fetching
> them might be highly problematic if one doesn't take extra care to
> also fetch the appropriate version of all the dependencies.

Good point: old outdated archives are probably an even bigger space hog on the packages server. We have a script to clean up old unneeded packages, but no process in place to run it automatically. If we set up a job to run that regularly, that would probably more than make up for any extra disk space the libc++ packages would use.


> Personally I don't believe that we would experience so much
> duplication that would be worth caring about. First of all, there's no
> need to use libc++ for noarch ports (except that we would need to put
> a bit of extra effort in making sure that versions of binaries
> compiled on two systems would not compete with each other). Second, we
> would probably want to use "delete_la_files yes" as well which also
> affects the builds of ports that don't need libc++. And finally it
> might be that we would at some point retire the builds for stdlibc++
> anyway.

True, we could use this opportunity to enable delete_la_files on older systems too, provided this switch to libc++ would require the user to uninstall and reinstall all ports, which might be the case.


>> I worry about increasing the disk space used, both for our current host Mac OS Forge as well as for the various voluntary mirror servers around the world, and it's also a consideration in the event that we might one day need to leave Mac OS Forge and have to find another place to house these mountains of data we're contemplating creating.
>> 
>> Another option: It should be possible to write code to detect whether any files in a destroot use a C++ library, and if so, MacPorts could include the C++ library name in the archive filename, otherwise don't. At archive fetch time, MacPorts wouldn't know whether a port uses C++ or not, but it could try to fetch both filenames and use whichever one exists.
> 
> Well, yes, that's also an option, even if a slightly more complicated
> one to implement. But what about delete_la_files? Should/could we
> attempt to rebuild all packages for < 10.9 in that case, making sure
> that all of them use delete_la_files or would this cause any problems?
> Given how long 10.6 and 10.7 have been offline, I don't think it would
> make much of a difference in terms of processor time if we would
> attempt a completely new rebuilds anyway, so we could just as well
> rebuild the latest versions of all ports with delete_la_files and
> store the resulting files into an xz file, all at the same time. (Or
> maybe that's simply too much a time after all.)

delete_la_files cannot be changed if any ports are installed, so if we want to change it from off to on on systems older than 10.9, that will require users to essentially follow the migration instructions.

MacPorts works fine with or without .la files, so if there is no other reason why we need to force everything to be reinstalled, then we could spare the user that inconvenience by leaving delete_la_files alone. Switching from libstdc++ to libc++ surely means the user must reinstall all C++ software, but that's probably a small portion of installed ports. But we currently have no means in MacPorts to identify C++ software, so I'm not sure yet how we could facilitate upgrading the C++ ports but not having to upgrade the others.


>>> Archive type is implicit (and I would still vote for trying to switch
>>> to xz one day).
>> 
>> I agree xz would be better because it is smaller.
>> 
>> I've previously assumed that since OS X does not include an xz executable, this would require us to bundle a copy of the xz executable with MacPorts, like we currently bundle Tcl 8.5. Since the xz executable is GPL-licensed, this might be a problem. If so, we could perhaps instead have MacPorts make direct use of the lzma library, which is in the public domain. That would be a change in how MacPorts works, however.
>> 
>> But I've now learned (or maybe I knew this before and forgot) that GNU tar and BSD tar both have built-in support for xz. The El Capitan version of tar supports this. We would need to see how far back this support goes. It was Mavericks when Apple removed GNU tar but I don't remember if they also had BSD tar before that. Wikipedia says BSD tar gained xz support in April 2009.
>> 
>> Looks like GNU tar does this by running the xz executable, but BSD tar uses libarchive, which uses liblzma. The libarchive port installs the bsdtar executable too. We might be able to have MacPorts use a bundled bsdtar/bzip2/libarchive/liblzma/zlib instead of the system's tar and compression utilities.
> 
> Should we collect this information and brainstorming about a switch to
> xz in a separate trac ticket?

Sure! You want to make one?