#55884: mandoc @1.14.3: not using the right compiler

Sun Mar 4 08:50:46 UTC 2018

On Mar 3, 2018, at 15:14, Jan Stary wrote:

> On Feb 28 08:03:31, Ken Cunningham wrote:
>> The basic principle is that a build of a port on any darwin_N system
>> anywhere will be identical to every other build on that darwin version
>> on all other machines.
> 
> Does that mean that MP will set the same configure.cc for mandoc
> on e.g. every Darwin 17.4.0?

It will set it to /usr/bin/clang.

> What of I have two of those, with different Xcode on each?

Then the version of clang may differ slightly. If a port requires a specific version of clang, it can so indicate using the compiler_blacklist_versions portgroup.

> Can you please indly point me to the place in MP base
> where the value of configure.cc get decided?

portconfigure.tcl

>> To do that, but build environment (and the runtime environment)
>> need to be fully pre-defined.
> 
> As long as it's possible to have different Xcode versions on one darwin_N,
> in what sense can the build environment be "fully pre-defined"? For example,
> https://trac.macports.org/wiki/UsingTheRightCompiler says:
> 
> 	 For example, with Xcode 4.2 and later, CC is set to
> 	 "/usr/bin/clang" so that there is no ambiguity.
> 
> "/usr/bin/clang" will be a different compiler with Xcode 4.2
> and with Xcode 4.N

That's true. We currently do not completely lock down the compiler version. If we want 100% reproducible builds, we may have to change our strategy for this and many other things. But for now, this is what we do. Reproducible builds are a long-term goal that we will work towards over the coming years.

In Xcode 3 and earlier, gcc was used. When clang was introduced as an optional compiler in Xcode 3.2, and made the default in Xcode 4.2 and later, we had *many* ports that would not build with clang. It was necessary for the port to be able to blacklist clang, so that MacPorts base could pick the next best compiler to use. For this to work, the build system needs to use the compiler MacPorts tells it to.

>> See here: <https://reproducible-builds.org/> for more info,
>> and why this is what we need. I hope you can come to accept
>> some of these MacPorts policies
> 
> This makes sense, unlike what's being said at
> https://trac.macports.org/wiki/UsingTheRightCompiler
> 
> Are these policies clearly described somewhere?

Apologies if the wording of that wiki page is unclear. I wrote it a long time ago, before reproducible builds were a goal. I remember struggling to write that page, and it getting much longer than I had originally intended, as I tried to communicate the problem clearly, and I guess I still didn't succeed. Improvements to wiki pages are welcome.

> On Feb 28 10:20:56, Ryan Schmidt wrote:
>>>> I had my MacPorts configured to error out if `cc` is used
>>>> using the method described at  UsingTheRightCompiler#testing.
>>> 
>>> Why would you do that?
>> 
>> To detect ports that do not use a specific compiler.
>> MacPorts policy is that all ports shall use a specific compiler.
> 
> Is this policy described somewhere?

I don't know. We've just described it to you...

>>> BTW, this is how configure.cc is described in the guide:
>>> 
>>> configure.cc
>>>   Set CC compiler flags for selecting a C compiler.
>>>   Default: ???
>>>   Example:
>>> 
>>> "compiler flags for selecting a compiler"?
>> 
>> Rainer has now adjusted these descriptions in the guide.
> 
> https://guide.macports.org/chunked/reference.phases.html still says
> "Set CC compiler flags for selecting a C compiler."

There is currently no automated process to deploy guide updates. I need to add those tasks to the buildbot. Until I do, Rainer or Clemens must manually run a task on the server to update the guide.

> On Feb 28 17:36:38, Rainer Mueller wrote:
>> On 2018-02-28 08:52, Jan Stary wrote:
>>> The ticket is closed, so I will continue here.
>>> 
>>>> Actually it's not specific to the universal variant. Rather, the problem
>>>> is that the port is not UsingTheRightCompiler.
>>> 
>>> mandoc's ./configure sets CC to whatever make(1) thinks CC is:
>>> CC=`printf "all:\\n\\t at echo \\\$(CC)\\n" | env -i make -sf -`
>>> Why exactly is it "not the right compiler"?
>> 
>> As env -i clears the environment,
>> how is this supposed to work at all?
>> This does not respect CC in the environment, which is what MacPorts
>> usually does.
>> 
>> $ export CC=foo
>> $ printf "all:\\n\\t at echo \\\$(CC)\\n" | env -i make -sf -
>> cc
> 
> That's on purpose: compile mandoc with
> whatever the local make(1) thinks $(CC) is.
> 
>  # You can manually override the compiler to be used.
>  # But that's rarely useful because ./configure asks your make(1)
>  # which compiler to use, and that answer will hardly be wrong.
> 
> mandoc's ./configure does not take CC from the environment.
> It determines it as above. You can override it of course,
> by setting CC=... in configure.local
> 
> That seems to be the issue here: MP _wants_to_ tell mandoc what compiler
> to use, but setting CC in the environment does not do it. Hence the CC
> dance that we have in the Portfile now. Is this understanding correct?

Yes.

> 
>>> It's true that /usr/bin/cc is a symlink to some suitable default compiler.
>>> So call that, thus calling a suitable default compiler, _by_default_.
>> 
>> You are already assuming that 'cc' is /usr/bin/cc. If anyone has a
>> /opt/local/bin/cc, it might be a different compiler.
> 
> If it is actually a C compiler, it will compile mandoc just fine.

Good.

> If it is instead a shell script that errors out with "you must not run
> this as your compiler", then it will not of course.

Of course.

>> In general, we do not want to use what the user selected as their
>> preferred compiler, but what MacPorts selected to compile this port.
> 
> Is this strict policy clearly described somewhere, including the
> rationale? Not in https://trac.macports.org/wiki/UsingTheRightCompiler

I apologize if what I wrote in that page is unclear. The rationale is that some ports do not build with any random compiler. Suppose that tomorrow, Apple releases a new version of Xcode, and its new version of clang is stricter about some code construction than previous versions. This causes a port to fail to compile. We need to be able to specify that that version of clang cannot be used to compile the port. We do that using the compiler_blacklist_versions portgroup. That can't work if the port's build system does not use the compiler MacPorts told it to use.

Or suppose that the user has used "sudo port select gcc" to make "cc" and "gcc" point to some compiler that the port author has not tested. Perhaps an early version of gcc 4.x that does not support -arch flags. The port will fail to build, because MacPorts supplies -arch flags to the build. We do not want to waste time handling bug reports from users who have configured their systems in this manner. We want ports to build with a predictable known-good compiler.

It is possible for a developer to override the MacPorts compiler choice at the command line when installing a port. A developer may wish to do this for testing purposes. This too does not work if the build system does not respect the compiler MacPorts told it to use.

>>> Yes, there are ports that do need to call some particular compiler,
>>> but it's those ports' job then to find and use that.
> 
>>>> I had my MacPorts configured to error out if `cc` is used
>>>> using the method described at  UsingTheRightCompiler#testing.
>>> 
>>> Why would you do that?
>>> 
>>> Mandoc is extremely portable. I have yet to see a UNIX system where
>>> it does not build. Except here: of course it "breaks" if you explicitly
>>> make it impossible to run 'cc', which is the right compiler.
>> 
>> This setup is a good idea for developers to catch ports that are not
>> respecting the environment variables for compiler selection.
> 
> This seems to confirm that this is what's "wrong" with mandoc in this
> respect - it does not listen to MP's CC setting and has to be told
> explicitly.

Yes.

>> Looks like an old copy & paste mistake to me. I committed a fix. [1]
>> Ryan was even faster to notice than me typing this mail. :-)
>> 
>>>> Perhaps the portfile code that set `CC`
>>>> that was removed in [14f7de9e9ad521cce6389678675706e8593b9834/macports-
>>>> ports] was needed after all.
>>> 
>>> It is only "needed" if you make it artificially impossible
>>> to call 'cc' (or whatever the port ./configures to be CC).
>>> 
>>> So: because ports must not call 'cc' on your system,
>>> let's complicate the mandoc Portfile for everyone.
>> 
>> Build systems are supposed to follow CC in the environment.
> 
> This would be much easier to get if this was
> written somewhere in the guide. Can we please have
> https://trac.macports.org/wiki/UsingTheRightCompiler
> incorporated in the Guide, with carefuly worded rationale,
> explicitly stating those policies? Maybe somewhere in
> https://guide.macports.org/index.html#development.practices ?
> That seems to be my whole misunderstanding.

Improvements to the guide are welcome. I would love to overhaul/rewrite the entire guide and website, since they have been neglected for years, but I have not found time to do so.

> (Maybe start with a better title. An excelent piece of extremely portable
> software such as mandoc, that compiles just anywhere - and brings its own
> compat_strtonum() in case the system doesn't have it - is being accused
> right off of 'not using the right compiler'. Like, uh, 'cc').

Apologies if the title and article were unclear. "The right compiler" means "the compiler MacPorts told it to use". The software is not being accused; the port is. It is the port's job to communicate MacPort settings to the build system by whatever means necessary.

>> The whole
>> point of options like configure.cc is that they are respected when
>> building the port.
> 
> There is a similar setting in mandoc's Portfile now for INSTALL.
> Is there also sucha thing as 'the right installer'? As in,
> a specific install(1)? Is that needed?

No, there is no such rule. Most software can use any install(1), or the install-sh compatibility script included with autotools software. MacPorts base by default tells ports to use /usr/bin/install, for historical reasons which may no longer be relevant (see https://trac.macports.org/changeset/30216), but which nobody has bothered sufficiently to investigate or change. I investigated it slightly when I discovered that some ports, specifically wine, are not compatible with /usr/bin/install, so wine overrides this choice; see https://bugs.winehq.org/show_bug.cgi?id=35310.

mandoc makes use of the MacPorts default install program, but specifies what permissions to use. This has been in place for INSTALL_PROGRAM since the port was initially created:

https://github.com/macports/macports-ports/commit/f7092bfe28646fadb8d6610fe8358dc3b1a3ad9b

I can only assume the author of the port determined that it was necessary to do so. I would presume that the programs were not being installed with execute permission.

You added INSTALL_LIB, INSTALL_MAN and INSTALL_DATA later; see https://trac.macports.org/ticket/53887; presumably you did so because you found it was necessary.

>> As an alternative to pass compiler in environment variables would be
>> generating wrapper scripts that use the right compiler as selected by
>> MacPorts. This is the way other projects like Fink do it. Then the usual
>> names cc/gcc/clang/... will always end up at the intended binary by
>> putting them to the front of PATH. However, this is a long-standing
>> issue and I am not aware anyone was working on this. Therefore, build
>> systems still need to be patched to honor CC from the environment.
> 
> OK, I think I get it now. Thank you.