portconfigure.tcl - why does appending to an ENV variable (like CFLAGS) not also append to configure.cflags ?

[Ryan Schmidt]

On Nov 7, 2018, at 11:00, Ken Cunningham wrote:

> On Nov 6, 2018, at 1:54 PM, Ryan Schmidt wrote:
> 
>> On Nov 6, 2018, at 14:48, Ken Cunningham wrote:
>> 
>>> I noticed this recently "fixing" some ports that don't use a configure step.
>>> 
>>> During the run of portconfigure.tcl, various things (sdkroot) might be tested, and the appropriate values appended to the ENV variables.
>>> 
>>> But these things don't seem to come out in the configure.variables, like I would have expected them to.
>>> 
>>> Is this desired behaviour? 
>>> 
>>> Or should portconfigure flesh out the configure.variables at the same time as it's appending to the ENV variables (like I would have expected)?
>>> 
>>> (Please tell me I'm missing some well-known step that everyone else knows about but I don't :> )
>> 
>> Can you give a specific example?
>> 
> 
> 
> OK. Let’s say to you want to build lz4 against an SDK. I defined the SDK to 10.13. To make the Portfile show the configure environment, I had to give it a sham configure phase by doing this tiny change:
> 
> #use_configure       no
> configure.cmd       /usr/bin/true
> 
> 
> 
> When you go to build the port, sudo port -d build lz4, you see the configure environment variables are set up right, wth the SDK specified in several places, as it should be:
> 
> DEBUG: Preferred compilers: clang macports-clang-5.0 macports-clang-4.0
> DEBUG: Using compiler 'Xcode Clang'
> DEBUG: Executing org.macports.configure (lz4)
> DEBUG: Environment: 
> CC='/usr/bin/clang'
> CC_PRINT_OPTIONS='YES'
> CC_PRINT_OPTIONS_FILE='/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/.CC_PRINT_OPTIONS'
> CFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> CPATH='/opt/universal/include'
> CPPFLAGS='-I/opt/universal/include -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk'
> CXX='/usr/bin/clang++'
> CXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> F77FLAGS='-m64'
> F90FLAGS='-pipe -Os -m64'
> FCFLAGS='-pipe -Os -m64'
> FFLAGS='-pipe -Os'
> INSTALL='/usr/bin/install -c'
> LDFLAGS='-L/opt/universal/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> LIBRARY_PATH='/opt/universal/lib'
> MACOSX_DEPLOYMENT_TARGET='10.13'
> OBJC='/usr/bin/clang'
> OBJCFLAGS='-pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> OBJCXX='/usr/bin/clang++'
> OBJCXXFLAGS='-pipe -Os -stdlib=libc++ -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -arch x86_64'
> Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal 
> DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/true --prefix=/opt/universal 
> DEBUG: Privilege de-escalation not attempted as not running as root.
> DEBUG: build phase started at Wed Nov  7 08:51:33 PST 2018
> 
> 
> 
> However, when the port is actually being built, the build args, which should be setup correctly by this:
> 
>     build.args-append       CFLAGS="${configure.cflags} [get_canonical_archflags cc]" \
>                             CXXFLAGS="${configure.cxxflags} [get_canonical_archflags cxx]"
> 
> 
> don’t actually show up on the build line with all the ${configure.cxxflags} as above. Instead, you just get the “default” values from the top of portconfigure.tcl:
> 
> Executing:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64" 
> DEBUG: system:  cd "/opt/universal/var/macports/build/_opt_universal_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_archivers_lz4/lz4/work/lz4-1.8.3" && /usr/bin/make -j16 -w all CC=/usr/bin/clang CXX=/usr/bin/clang++ PREFIX=/opt/universal CFLAGS="-Os -arch x86_64" CXXFLAGS="-Os -stdlib=libc++ -arch x86_64" 
> 
> 
> 
> The lz4 portfile has manually added the -stdlib=llibc++ in the Portfile, but it should not need to do that as it is already being done by portconfigure.tcl — but the modifications made by portconfigure.tcl to the CXXFLAGS are not making it into ${configure.cxxflags}, at least at the point where the build arguments are being set up.
> 
> 
> 
> 
> I hope this is a sufficiently clean example.

Yes. And the reason why I asked for an example is that the answer would be different for different options!

For the -stdlib=libc++ flag, portconfigure.tcl does in fact add that into configure.cxxflags for you when needed (i.e. when the compiler is clang++) and goes to some lengths (using traces) to ensure that it cannot be removed, except via the approved mechanism of changing configure.cxx_stdlib. There's no need for the lz4 port to add this manually so I've removed that code. If a port needs the flag in configure.ldflags, then the port needs to add it there manually, but that's not the case for lz4.

For SDK flags, you're right, they're not added to the configure.*flags variables; instead, they're added directly to the environment by portconfigure::configure_main when they're needed.

The reason for this probably has to do with the fact that originally the decision about whether or not to use SDK flags was based on whether or not we were building universal. To build universal on Tiger PowerPC, you need to use the 10.4u SDK, otherwise you don't.

The way that the default universal variant gets added is as follows: After the entire portfile has been processed, MacPorts base defines an empty default universal variant if ${universal_variant} is "yes". The default value of universal_variant is ${use_configure}, and the default value of use_configure is "yes".

The addition of -arch flags is handled the same way, for the same reason -- we need to add different -arch flags depending on whether we're universal or not, and we won't know for sure if the port will have a universal variant until after the entire portfile has been processed.

This way of defining the default universal variant certainly causes some problems which have been pointed out before. For example, if you want to add universal support to a port that uses "use_configure no", you have to manually add the -arch flags using [get_canonical_archflags], but you have to make sure the universal variant exists before then. You might think you could just "universal_variant yes" prior to calling it, but that doesn't work because the default universal variant won't get created until after the portfile is processed. Instead you have to manually create an empty universal variant ("variant universal {}"). Alternately, you could restrict yourself to using [get_canonical_archflags] in a phase (for example in a pre-build block), since the phases run after the portfile is processed, but that makes the Portfile code uglier and seems to make it impossible to copy values between phases, such as assign destroot.args the value of build.args, which I've done in many portfiles and is very convenient to be able to do.

How could we improve this? What if MacPorts defined the empty default universal variant before processing the portfile (on systems where we can build universal)? We would then need a way of deleting the variant in the event that "universal_variant no" (or "use_configure no") is encountered in the portfile. I don't think MacPorts base currently has a way to delete a variant. But maybe that wouldn't be too hard to add.

If we could do that, then we could modify the way the -arch flags and SDK flags are added. We could handle them similar to how configure.cxx_stdlib is handled. That should do what you want.

But there are a *lot* of ports out there already that handle this manually. There's a good chance that by fixing this in base, we'll end up breaking ports or portgroups. Which could then be fixed. But we should be very careful.

I see for example that the cmake portgroup specifies the SDK root using -DCMAKE_OSX_SYSROOT and the architectures in -DCMAKE_OSX_ARCHITECTURES. It doesn't want them mixed in with the CXXFLAGS. We should consider that when making changes. For example, we may want to have separate variables like configure.archflags, configure.sdk_ldflags, configure.sdk_cflags, which we put in configure.cflags, configure.ldflags, etc. by default but which the cmake portgroup could clear out.