Volunteer for a workshop on "setting up your own buildbot/buildslave"?

[Mojca Miklavec]

On Wed, Nov 11, 2015 at 12:15 PM, Ryan Schmidt wrote:
> On Nov 10, 2015, at 7:26 AM, Mojca Miklavec wrote:
>
>> I assume that
>> should be easy enough to add.
>
> It would be easy enough to do, once we reach a decision on what, specifically, to do.

So I suggest we finish this discussion and meet the decision soon,
before we all forget about this and everyone who might potentially
still be interested in fixing bugs moves on and looses interest in
supporting libc++ on < 10.9.

>> A while back we were discussing about
>> how to implement this. My suggestion would be to only add that on <
>> 10.9 if libc++ is used. And to make things easier: unless it's a
>> noarch port, always add it, even if the port doesn't link against
>> libc++ explicitly. At the moment we don't have any procedure in place
>> to aid with decision about whether or not "libc++" is needed in the
>> filename, but we would like to start somewhere. For completeness one
>> could add "libstdc++" if libstdc++ is needed on >= 10.9. This approach
>> isn't 100% correct in all cases (in particular not in cases where g++
>> from macports-gcc is used), but it will allow us to proceed and to fix
>> more subtle problems later.
>
> I like the idea of only adding a non-default cxxstdlib to the filename, since that would allow all the existing archives to continue to be valid. However if we were to switch to xz compression for archives, we might want to repackage everything that way, in which case it wouldn't matter.

Yes, it would be helpful to repackage everything. But despite that I
don't see the added value of adding the default stdlib to the package
name. Or in particular, I would not be in favour of adding libc++ to
the filenames on >= 10.9. (First of all, we don't add "-variant" to
the names either and it's nicer to keep names shorter. Second of all,
this is mostly relevant for < 10.9, I don't believe that anyone is
trying to support libstdc++ on >= 10.9, so in the long run we'll end
up with shorter names overall.)

> I agree that since we don't have a mechanism for detecting C++ software, it would be simplest to add the tag to all archives, even though that will result in a rather large increase in disk usage on the packages servers. If the packages server were using some sort of advanced automatic deduplicating filesystem the impact might not be so large, since many ports would hopefully build identically, but I don't know what filesystems do that or if we're using one of them.

If that is going to be a problem, we could at some point in future
start thinking about "small" and "big" mirrors. Small mirrors (those
that don't have sufficient disk space) could include just the latest
versions of software and/or just for just the latest OSes, while the
"big" mirrors could include everything. Old versions of binaries
(older than one year and older than the latest successfully built
version of that software) are seldom needed and even then fetching
them might be highly problematic if one doesn't take extra care to
also fetch the appropriate version of all the dependencies.

Personally I don't believe that we would experience so much
duplication that would be worth caring about. First of all, there's no
need to use libc++ for noarch ports (except that we would need to put
a bit of extra effort in making sure that versions of binaries
compiled on two systems would not compete with each other). Second, we
would probably want to use "delete_la_files yes" as well which also
affects the builds of ports that don't need libc++. And finally it
might be that we would at some point retire the builds for stdlibc++
anyway.

> I worry about increasing the disk space used, both for our current host Mac OS Forge as well as for the various voluntary mirror servers around the world, and it's also a consideration in the event that we might one day need to leave Mac OS Forge and have to find another place to house these mountains of data we're contemplating creating.
>
> Another option: It should be possible to write code to detect whether any files in a destroot use a C++ library, and if so, MacPorts could include the C++ library name in the archive filename, otherwise don't. At archive fetch time, MacPorts wouldn't know whether a port uses C++ or not, but it could try to fetch both filenames and use whichever one exists.

Well, yes, that's also an option, even if a slightly more complicated
one to implement. But what about delete_la_files? Should/could we
attempt to rebuild all packages for < 10.9 in that case, making sure
that all of them use delete_la_files or would this cause any problems?
Given how long 10.6 and 10.7 have been offline, I don't think it would
make much of a difference in terms of processor time if we would
attempt a completely new rebuilds anyway, so we could just as well
rebuild the latest versions of all ports with delete_la_files and
store the resulting files into an xz file, all at the same time. (Or
maybe that's simply too much a time after all.)

>> Archive type is implicit (and I would still vote for trying to switch
>> to xz one day).
>
> I agree xz would be better because it is smaller.
>
> I've previously assumed that since OS X does not include an xz executable, this would require us to bundle a copy of the xz executable with MacPorts, like we currently bundle Tcl 8.5. Since the xz executable is GPL-licensed, this might be a problem. If so, we could perhaps instead have MacPorts make direct use of the lzma library, which is in the public domain. That would be a change in how MacPorts works, however.
>
> But I've now learned (or maybe I knew this before and forgot) that GNU tar and BSD tar both have built-in support for xz. The El Capitan version of tar supports this. We would need to see how far back this support goes. It was Mavericks when Apple removed GNU tar but I don't remember if they also had BSD tar before that. Wikipedia says BSD tar gained xz support in April 2009.
>
> Looks like GNU tar does this by running the xz executable, but BSD tar uses libarchive, which uses liblzma. The libarchive port installs the bsdtar executable too. We might be able to have MacPorts use a bundled bsdtar/bzip2/libarchive/liblzma/zlib instead of the system's tar and compression utilities.

Should we collect this information and brainstorming about a switch to
xz in a separate trac ticket?

Mojca