[MacPorts] #63415: p5.28-net-ssleay failed to compile under Monterey (21A5304g)

[MacPorts]

#63415: p5.28-net-ssleay failed to compile under Monterey (21A5304g)
----------------------------+--------------------------
  Reporter:  ilTofa         |      Owner:  markmentovai
      Type:  defect         |     Status:  reopened
  Priority:  Normal         |  Milestone:
 Component:  ports          |    Version:  2.7.1
Resolution:                 |   Keywords:  monterey
      Port:  p5-net-ssleay  |
----------------------------+--------------------------

Comment (by markmentovai):

 This is getting off into the weeds a little bit, but I can’t think of a
 better place to discuss.

 [comment:36 @breiter comment:36]:

  If you try to `dlopen` a /path/to/libcrypto.dylib that doesn't exist on
 disk `dlopen` will not simply return ENOENT but will instead `dlopen`
 libcrypto.dylib from the shared cache -- if it exists. Any code that tries
 to find a different version of a library that Apple includes in the shared
 cache by probing a set of paths until there is no ENOENT will open the
 Apple shared dylib version instead of the one it was looking for unless
 the first path it probed was correct. What happened here is the build
 script tried to `dlopen` /opt/local/libcrypto.dylib and if it had gotten
 ENOENT would have correctly loaded /opt/local/lib/libcrypto.dylib. Instead
 `dlopen` /opt/local/libcrypto.dylib which is set up to `abort()` unless it
 is opened by Apple's `/usr/bin/python2` (and maybe some other things) but
 print out a warning:
 {{{
 WARNING: Executing a script that is loading libcrypto in an unsafe way.
 This will fail in a future version of macOS. Set the
 LIBRESSL_REDIRECT_STUB_ABORT=1 in the environment to force this into an
 error.
 }}}
  /usr/bin/python3 just aborts. Presumably there is some cohort of
 important stuff that relies upon /usr/bin/python to load
 /usr/lib/libcrypto.dylib that Apple doesn't want to totally break, yet.

 This is essentially correct. `dlopen` doesn’t promise to expose `errno` in
 any particular way, but `dlopen`, upon encountering `ENOENT` internally,
 ''should'' fail and return `NULL` (and `dlerror` would indicate the
 problem as “image not found” (macOS 11) or, now, “no such file”). Instead,
 we’re seeing this new fallback behavior in macOS 12 that causes it to
 search for the library based on the path’s leaf name. You can see the
 search path by inspecting what `dlerror` has to say when you try to
 `dlopen` a library that doesn’t exist ''and'' for which the fallback
 search fails (so, ''not'' `libcrypto.dylib`):

 {{{
 mark at sweet16 zsh% ~/t_dlopen /enoent/libenoent.dylib
 t_dlopen: dlopen: dlopen(/enoent/libenoent.dylib, 0x0005): tried:
 '/enoent/libenoent.dylib' (no such file), '/usr/local/lib/libenoent.dylib'
 (no such file), '/usr/lib/libenoent.dylib' (no such file)
 }}}

 You can also, perhaps more directly, set `DYLD_PRINT_SEARCHING=1`, which
 indicates that this is called “default fallback”:

 {{{
 mark at sweet16 zsh% DYLD_PRINT_SEARCHING=1 ~/t_dlopen
 /enoent/libenoent.dylib
 dyld[59641]: find path "/usr/lib/libSystem.B.dylib"
 dyld[59641]:   possible path(original path): "/usr/lib/libSystem.B.dylib"
 dyld[59641]:   found: dylib-from-cache: (0x00AA)
 "/usr/lib/libSystem.B.dylib"
 dyld[59641]: find path "/enoent/libenoent.dylib"
 dyld[59641]:   possible path(original path): "/enoent/libenoent.dylib"
 dyld[59641]:   possible path(default fallback):
 "/usr/local/lib/libenoent.dylib"
 dyld[59641]:   possible path(default fallback): "/usr/lib/libenoent.dylib"
 dyld[59641]:   not found: "/enoent/libenoent.dylib"
 t_dlopen: dlopen: dlopen(/enoent/libenoent.dylib, 0x0005): tried:
 '/enoent/libenoent.dylib' (no such file), '/usr/local/lib/libenoent.dylib'
 (no such file), '/usr/lib/libenoent.dylib' (no such file)
 }}}

 The “soft” warning discussing `LIBRESSL_REDIRECT_STUB_ABORT` comes in when
 the main executable is something explicitly allow-listed by the poisoned
 `libcrypto.dylib` stub. The list is:

 {{{
 /usr/bin/perl
 /usr/bin/python
 /usr/bin/ruby
 /usr/bin/tcl
 /usr/bin/tk
 /System/Library/Frameworks/Python.framework
 /System/Library/Frameworks/Ruby.framework
 /System/Library/Frameworks/Tcl.framework
 /usr/local/bin/astris
 }}}

 If your main executable, as identified by `NSGetExecutablePath`, begins
 with any of those paths, then you get the warning message but no
 `abort()`, unless you set the environment variable as the message
 describes. This is a substring search, so you might assume that
 `/usr/bin/python` ought to catch both Apple-shipped `python2` and
 `python3`. `python2`, however, actually runs out of
 `/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7`,
 and you can see how that comes into play in the allow-list. `python3` is
 unbundled from the OS, and what you see at `/usr/bin/python3` is actually
 a stub to delegate to `python3` embedded within the active developer tools
 installation in accordance with `xcode-select` rules (so, respecting
 `DEVELOPER_DIR`, etc. This ultimately winds up at a path like
 `/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python`
 which, as you can see, is ''not'' in the allow-list, so Apple’s
 `libcrypto.dylib` is poison even for that interpreter.

 The shared cache issue is ''somewhat'' irrelevant here. Internally, dyld
 is attempting a fallback by searching /usr/local/lib and /usr/lib for the
 leaf name of the module you’re attempting to load. It just so happens that
 `libcrypto.dylib` can be serviced via the cache and not the filesystem.
 You can even “beat” the poisoned `libcrypto.dylib` in /usr/lib (virtually,
 via the shared cache) by sticking your own `libcrypto.dylib` in
 /usr/local/lib and have it do whatever you want:

 {{{
 mark at sweet16 zsh% rm -f /usr/local/lib/libcrypto.dylib
 mark at sweet16 zsh% ./t_dlopen /opt/local/libcrypto.dylib
 WARNING: /Users/mark/t_dlopen is loading libcrypto in an unsafe way
 zsh: abort      ./t_dlopen /opt/local/libcrypto.dylib
 mark at sweet16 zsh% clang -Wall -Werror -x c -dynamiclib -o
 /usr/local/lib/libcrypto.dylib - << __EOF__
 #include <stdio.h>

 __attribute__((constructor))
 static void Initialize() {
   fprintf(stderr, "I may not be the module you intended.\n");
 }
 __EOF__
 mark at sweet16 zsh% ./t_dlopen /opt/local/libcrypto.dylib
 I may not be the module you intended.
 }}}

  The path /usr/lib/libcrypto.dylib also doesn't exist.
 {{{
 $ ls /usr/lib/libcrypto.dylib
 ls: cannot access '/usr/lib/libcrypto.dylib': No such file or directory
 }}}

 This is normal. Since macOS 11, Apple no longer ships any dylibs in the
 filesystem for things that are contained within the dyld shared cache.

  Is this true for '''any''' dylib? If you try to `dlopen` something that
 doesn't exist on disk but does exist in the shared cache, you just
 magically get the shared cache version (that seems bad)? Or a special case
 bug libcrypto.dylib (less bad)?

 There’s no special-casing here. This is how dyld behaves now on macOS 12,
 although it reflects new additions to the search path and not strictly the
 shared cache per se—but libraries can certainly be provided by the shared
 cache, as in the `libcrypto.dylib` case. It’s contrary to the `dlopen`
 documentation and, in my opinion, it’s a bug.

-- 
Ticket URL: <https://trac.macports.org/ticket/63415#comment:41>
MacPorts <https://www.macports.org/>
Ports system for macOS