A separate PortIndex for libc++ on older systems

Thu Feb 9 14:53:05 UTC 2017

Hi,

I'm sorry for top-posting, but I wanted to keep some relevant context
(after 6 months of "silence").

The ticket (with a lot of useful discussion from this old thread missing):
     https://trac.macports.org/ticket/50448

Can we PLEASE PLEASE move forward with this? Let's not spend another
three years discussing what the optimal solution should be. Let's just
pick something and implement it. A better solution can still be
implemented later if needed.

>From what I understand we need the following:

(a) decide how to call archives and where to store them on the server
(+ where to put metadata like frameworks_dir etc.)

(b) implement the changes in the base and make a new release (ideally
including a binary that's pre-configured to use libc++)

(c) create and distribute a new PortIndex

(d) set up the build slaves

I expect the last step to be the most tricky one (we'll be fighting
with unexpected problems like having to activate the whole compiler
toolchain for compiling each individual port), but please let's not be
stuck at (a).

Related issues that deserve a higher priority:

- Support fetching from https (+ get the distfile mirroring job running again)
  https://trac.macports.org/ticket/51516

- Add support for .tar.xz
  https://trac.macports.org/ticket/52000

- Support VCS fetching (already partially implemented for GIT by
Rainer / raimue)
  https://trac.macports.org/ticket/16373

Mojca

On 12 August 2016 at 08:29, Ryan Schmidt wrote:
> On Aug 11, 2016, at 3:13 PM, David Evans wrote:
>> On 8/10/16 1:37 AM, Ryan Schmidt wrote:
>>>
>>> Currently, there is only one PortIndex file generated on the server, and published to the rsync server, for each OS name/version/endianness combination. So for example, there is one PortIndex for Darwin 12 Intel. There is not currently a separate PortIndex for Darwin 12 Intel with libc++, so anyone with e.g. OS X 10.8 with libc++ syncing from the rsync server will see the version of this port as 3.20.2 not 3.18.3 when querying the index, for example when running port info or port outdated. If we're going to change Portfile attributes such as version that get stored in the index based on the C++ library, and I agree this is a situation that would arise more and more as newer versions of software require libc++, should we make a separate libc++ PortIndex for 10.6/10.7/10.8?
>>>
>>
>> OK, I just tumbled to what you're saying here.  As you say, the indexed version of this port will be determined by the
>> value of configure.cxx_stdlib on the machine that is doing the indexing and, for those that use the rsynced indexes,
>> that machine may not be running the same C++ configuration (or even the same OS version) as the user's machine.
>
> We generate a portindex for each macOS version. And for 10.4 and 10.5 which run on Intel and PowerPC, we generate separate indexes for Intel and PowerPC. So although the portindex is not generated individually on each macOS version (in fact, for several years, it has been generated on a Linux machine), it contains information specific to that macOS version. However, the portindexing process does not currently know about libc++ and libstdc++ and that things in the index, such as a port's version, might differ based on that variable. I'm proposing we teach the indexing process about that and create a second index for macOS < 10.9 with libc++.
>
>
>> This
>> works for me because, as a maintainer, I use subversion to fetch Portfiles and regenerate the PortIndex myself.  In that
>> context this works fine.
>
> That's right. Those using svn generate their own indexes and don't see the ones on the rsync server.
>
>
>> As you say, the world is quickly moving on. Even projects like Inkscape, which has been slow to update to C++11 and gtk3
>> has announced that the upcoming 0.92 release will be the last to support C++98 (libstdc++) and gtk2.  Trunk development
>> has already moved to requiring C++11 and libc++ to build.
>>
>> Similarly, glibmm, gtkmm3 and friends have required C++11 (libc++) to build for two stable release cycles now and are
>> proposing requiring C++14 in the next based on the fact that that is the default standard for gcc6.  So, while I have
>> been tracking the latest releases in my C++11 respositories,  I have been holding back on some of these releases for a
>> while pondering how to handle this libstdc++/libc++ dichotomy.
>>
>> Personally, given time and resouces, I would rather dispense with maintaining outdated versions of GNOME ports
>> altogether and just use the cxx11 PortGroup where applicable.  It makes sense to me that we should put pressure on
>> users with older platforms to configure themselves to use libc++ where they possibly can rather than giving them further
>> excuses to stay with libstdc++ and C++98.  Maybe put the onus on those who want to stay with the older systems to curate
>> a separate repository(ies) altogether of ports whose versions are cherry picked to be compatible with their systems.
>> You could even provide separate buildbots for these systems.
>>
>> So, before coming up with more complex indexing systems, etc, maybe we need to take a clear look at what we can really
>> afford to support.  At the very least, the newer standards should have a higher priority for support over the older.
>> Right now I feel like it's the other way round.
>
> Yes. Getting a portindex for macOS < 10.9 that was made with configure.cxx_stdlib set to libc++ is a step toward getting users of macOS < 10.9 to use libc++.
>
>
>> END OF DAVE'S (mini) RANT FOR TODAY :-)
>>
>> GETTING BACK TO REALITY:
>>
>> So, I wonder, rather than trying to design and disseminate a single PortIndex that will work for all circumstances,
>
> We don't.
>
>> have
>> you considered modifying port selfupdate to regenerate the PortIndex on the user's system after rsyncing ports rather
>> than including it prebuilt?  Speaking as one who does this on a regular basis, the additional overhead isn't much if
>> people keep relatively up-to-date.
>
> Generating a new index from scratch takes a rather long time, with our 10,000+ Portfiles. New users installing MacPorts for the first time would wonder why the MacPorts installer was just sitting there on the "finishing up" screen for an hour, or however long it takes on their system. My old PowerBook G4 indexes about 1-2 ports per second. (Not to say we should base our decision solely on machines that old; just to give you one datapoint.)
>
>
>> Or would it really be a disaster if we took a hard stand on only supporting 10.9+?
>
> I don't like dropping support for older macOS versions. Just a few weeks ago I was asked to compile a binary of a port for 10.5 ppc, and with MacPorts port mdmg, I was easily able to. I like that. There were a hundred dependencies I would have had to compile by hand otherwise, but thanks to MacPorts still running on 10.5, I was spared that work. I don't expect all ports to work on old systems, but I want to arbitrarily prevent people from installing software old systems that still works on those systems.
>
>
>> I see the new buildbots already
>> label 10.8 and earlier as legacy so maybe we're half way there already.
>
> The new buildbot has "legacy" 10.6-10.8 builders (for libstdc++) and non-legacy 10.6-10.11 builders (for libc++). The non-legacy 10.6-10.8 builders are not yet doing anything, because we have not yet solved the problem of where they should upload their packages.
>
>
>>> We also still need to decide how to differentiate the URLs for libc++ packages from the URLs of the existing libstdc++ packages.
>>> One suggestion was to add cxx_stdlib as a variable in archive_sites.conf, and upload the libc++ archives with the same
>>> names as the libstdc++ archives
>>> but in a new subdirectory, e.g. https://packages.macports.org/libc++/.
>>> Once we decide that, then do we adopt the same strategy for naming and placing the PortIndex files?
>>
>> Perhaps, but you probably also need a way to specify whether a given port is able to build with either libc++ or
>> libstdc++ or both to guide/facilitate this process.  Possibly something like supported archs.
>>
>> For example, as mentioned above, Inkscape will shortly be a libc++ only port and latest glibmm and friends already are
>> while others like boost support either while yet others only support libstdc++.
>
> The cxx11 1.0 portgroup is sufficient for that for now, isn't it?
>
>
>> If doing subdirectories for packages, I would submit that the default should be libc++ as that will quickly be the
>> dominant mode if it is not already. Reserve the special case subdirectory for the diminishing libstdc++
>
> In hindsight, yes. But that's not the set of files we have. We have
>
> https://packages.macports.org/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_13.x86_64.tbz2
>
> which uses libc++ and
>
> https://packages.macports.org/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_12.x86_64.tbz2
>
> which uses libstdc++. No subdirectories. Changing this now so that in future
>
> https://packages.macports.org/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_12.x86_64.tbz2
>
> refers to a file built with libc++ would be problematic. First, it would involve re-syncing hundreds of gigabytes of data between our mirrors. We would have to flush the MaxCDN cache and get it to re-cache those files. And it would require changing MacPorts base so that systems using libstdc++ know to look in the new location. What if a user doesn't update MacPorts base for awhile and continues installing and upgrading ports? They'll get libc++ versions added to their libstdc++ ports tree, which will cause crashes.
>
>
> I'd be willing to change the URL scheme if we change all the URLs. For example, we could use:
>
> https://packages.macports.org/darwin_13/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_13.x86_64.tbz2
>
> https://packages.macports.org/darwin_12/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_12.x86_64.tbz2
>
> https://packages.macports.org/darwin_12-legacy/GraphicsMagick/GraphicsMagick-1.3.24_0%2Bq8.darwin_12.x86_64.tbz2
>
> Any requests that come in for the old URLs can be redirected to the new ones with Apache mod_rewrite rules. MaxCDN would have to cache the files at the new locations, but we would not have to purge the cache of the old files. We would want to coordinate with our mirror providers and set a time for the transition, so that they can install the Apache rules and move their files and not have to re-sync it all. After the files are moved, we can release a new MacPorts version using the new URLs.
>
>
> Or we could avoid changing the URLs by just putting the new 10.6-10.8 libc++ archives in a single new subdirectory.
>
> Or we could avoid it by using a different identifier in the archive name.
>
> Or we could avoid it by taking the opportunity to simultaneously change our compression format from tbz2 to txz (see #52000). On 10.6-10.8, tbz2 archives would continue to use libstdc++ while txz archives would use libc++. This is a hack but if we want to switch to txz anyway this would be a way to solve two problems at once.
>
>
>> Does adding this structure mean that we (maintainers) need to provide older versions of ports to support libstdc++
>> compatibility or is it sufficient to just support the latest libc++ versions.
>
> The more steps we complete for supporting libc++ properly on 10.6-10.8, the sooner we can forget about libstdc++.
>
>
> _______________________________________________
> macports-dev mailing list
> macports-dev at lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/macports-dev