[MacPorts] #41124: graphviz uses a fuzzy check whether alloca(3) uses malloc(3), which fails in some cases

[MacPorts]

#41124: graphviz uses a fuzzy check whether alloca(3) uses malloc(3), which fails
in some cases
-----------------------+--------------------------
  Reporter:  cal@…     |      Owner:  ryandesign@…
      Type:  defect    |     Status:  new
  Priority:  Normal    |  Milestone:
 Component:  ports     |    Version:  2.2.1
Resolution:            |   Keywords:
      Port:  graphviz  |
-----------------------+--------------------------
Description changed by cal@…:

Old description:

> Graphviz uses a check in `lib/vmalloc/features/vmalloc` that is supposed
> to find out whether `alloca(3)` is internally using `malloc(3)`. To do
> this, this check provides a custom implementation of `malloc(3)` and
> assumes that if this implementation is going to be called it was called
> by `alloca(3)`.
>
> However, this assumption doesn't hold in all cases, e.g. it breaks when
> having `DYLD_INSERT_LIBRARIES` is set, as is the case for trace mode,
> because the Mac OS loader `dyld` internally calls `malloc(3)`, causing
> the check to fail.
>
> The check should instead use a global volatile variable to check whether
> the current call to `malloc(3)` happened while trying to use `alloca(3)`.
> The overriden `malloc(3)` should behave like the standard one in all
> other cases.
>
> The attached patch fixes this. Additional checks for `dlsym(3)` and
> `dlfcn.h` might be required. Can you take this upstream? I assume the
> same applies to `graphviz-devel`, too.

New description:

 Graphviz uses a check in `lib/vmalloc/features/vmalloc` that is supposed
 to find out whether `alloca(3)` is internally using `malloc(3)`. To do
 this, this check provides a custom implementation of `malloc(3)` and
 assumes that if this implementation is going to be called it was called by
 `alloca(3)`.

 However, this assumption doesn't hold in all cases, e.g. it breaks when
 having `DYLD_INSERT_LIBRARIES` set, as is the case for trace mode, because
 the Mac OS loader `dyld` internally calls `malloc(3)`, causing the check
 to fail.

 The check should instead use a global volatile variable to check whether
 the current call to `malloc(3)` happened while trying to use `alloca(3)`.
 The overriden `malloc(3)` should behave like the standard one in all other
 cases.

 The attached patch fixes this. Additional checks for `dlsym(3)` and
 `dlfcn.h` might be required. Can you take this upstream? I assume the same
 applies to `graphviz-devel`, too.

--

-- 
Ticket URL: <https://trac.macports.org/ticket/41124#comment:1>
MacPorts <http://www.macports.org/>
Ports system for OS X