The static library discussion

[Jeremy Huddleston]

On Dec 21, 2011, at 16:33, Marin Saric wrote:

> Hi everyone,
> 
> An issue has been raised in a ticket recently of whether the static
> libraries should be (if possible) dealt away with entirely in
> MacPorts.
> 
> Here's some examples of cases against doing away with them, in no
> particular order of importance:
> 
> Example 1: ABI incompatibilities
> 
> x264 is a great example of where having it in a static form removes
> just the rebuilding problem that occurred now. x264 does not have a
> very stable ABI. It's source level API is consistent, but depending on
> what the version of x264 you download (new "snapshots" on something
> like a daily basis), your program is going to link against different
> symbols. In the newest version of x264 the ABI broke again.

Ok, so x264's development model is broken upstream.  Go yell at them to do better.  This is not a "MacPorts" issue.  It's an x264 issue, and that impacts all distributions, not just us.

> Assume x264 is dynamic. Now assume a set of dynamic libraries H export
> symbols against an older ABI version of x264. Binaries depending on H
> and binaries dynamically linking against libraries depending on H all
> break once x264 is replaced with a new version.

AIUI, they do at least bump the dylib version, right?  So libx264.10.dylib becomes libx264.11.dylib when there is ABI incompatibility.  That, while sub-ideal, is still tolerable.  We dealt with this on a larger scale with libpng going from 1.2 to 1.4 and then again to 1.5.  All dependent 

> The old symbols are
> not found or, even worse, they are found but their semantics or
> interfaces have changed in a subtle way causing malfuction.

That SHOULDN'T ever happen. If they change their ABI in this way, they need to bump the library version.  We should not do something like ship static libraries to work around their bad development practices.  If they ever do this, we should not ship their code until they fix it.

> Hence the
> "bump_dev_revs.sh" script, forcing a recompile of everything that
> depends on x264. If MacPorts were a binary-only distribution, we'd be
> in trouble.

No, if we were a binary-only distribution, we'd be set because we could have multiple instances of libx264 installed at the same time (one for each needed ABI version) ... just like every other binary distribution out there ... even source distros like Gentoo use SLOTs to deal with this.

> This can be circumvented with "properly" versioning x264, but this is
> something outside of MacPorts control (IMHO too much involvement
> upstream).

This issue does not impact just us.  Any distribution is impacted by this and if they really ship two releases where the library versions match but the ABI don't, that's a huge bug on their end.  If they don't see it as such, they need to be educated.

> One "bad apple" can wreak havoc on the whole system. Note that this is
> different than the symbol-collision problem fixed by two-level
> namespaces.

Yeah, so fix the bad apple.  Don't wreck the system to lessen the impact of the bad apple.


> Example 2: "Feature freeze"
> If you want your binary to just depend on a specific version of a
> library that you have validated against instead of the newest latest
> and greatest, you will link against a static version of it.

Or if you're trying to support a distribution and someone reports a bug occurring in some media player which is crashing in its x264 code, and that was provided by a static library, there's no way to know what version of the static library is present.

Another way of looking at that is that if x264 is bumped multiple times, the binary package of VLC or mplayer or ffmpeg won't be consistent across those releases.  We'll need to bump the revision of those ports to rebuild against the newer x264 to pick up the bug fix, whereas they would "just work" (modulo ABI changes) if you just used dynamic libraries.

>  On MacOS X
> you can put stuff in your own framework or have it live in an
> application bundle. Outside the world of MacOS X, linking against a
> static library is a way to do it.

Uhm... what?

"Outside the world of Mac OS X" this is exactly the same issue.  A Framework is just a dylib and a bunch of headers.  Strip away the packaging, and it's exactly the same thing.

> In a perfect world, you'd trust the library developer not to break the
> desired behavior by adhering to a consistent versioning. You might
> also expect the newer versions of the library never to introduce a
> regression.

If there's a regression, report it and fix it.  This is no different than anywhere else.

> Or you might just say, OK, I am happy with the
> functionality provided by the current version of the library and I am
> OK with statically linking against it.

If you're happy with that version, then stick with that version by providing a local overlay of that port which is stuck at that version.  Using a static library just makes it exceedingly difficult to manage, leads to wasted memory due to duplication of this library across processes in a way that can't be shared, wasted disk space due to additional duplication, and is generally BAD software engineering.


> Example 3: Building plugins - avoiding versioning conflict.
> This is a test case I am dealing with right now.  I have binaries that
> open my plugin through the dlopen API. These binaries are outside of
> MacPorts and they depend on their own private versions of freetype and
> some other libraries. Unfortunately they work in a flat namespace
> model and thus things break.

Why?  Fix their misuse and reliance on the flat namespace.

> Static library is a simple fix.

I think you mean "messy workaround" rather than "simple fix"

> I am not
> dependent on the application developer to "fix" their program or to
> resolve the conflict in any way.

Report the bug to them and get it fixed anyway.  

> My plugin is a dylib (or on Linux a
> .so), but it links statically against all the dependencies, so loads
> and runs without a problem in the application that uses different
> versions of dependent libraries.

If you didn't flatten your namespace, this wouldn't be an issue.  THAT is the real bug.

> Other than standard system depencies
> on stdc++ library and the usual MacOS frameworks), no symbols are
> exported, so there's no collisions or user-unfriendly load-time plugin
> failures.
> 
> Example 4: Buidling libraries and plugins - deployment
> You can use the MacPorts provided libraries, yet provide a dynamic
> library or a plugin that does not have extra dependencies. You will
> find examples already of software inside MacPorts building "private"
> versions of library for the very same reasons.

What do you mean by this?

> Example 4.2: "Convenience" libraries
> This is a well-known subcase of Example 4. Static "tool" libraries
> that are linked into dynamic libraries. This is pretty common across
> open-source projects.
> http://www.freesoftwaremagazine.com/articles/building_shared_libraries_once_using_autotools

That article seems to completely argue against your point.


> Example 5: Binary deployment.
> Statically built binaries with MacPorts become trivially deployable
> outside of MacPorts. No extra libraries to bundle with or ask the user
> to compile, etc., no need for an installer, etc.

I don't consider that something that we care about.


> Example 6: Load time
> Statically built binaries load very quickly. This comes at a price of
> an increased cost in RAM. This used to be an issue when machines came
> with an order of magnitude less RAM then they do today. The incurred
> cost is small in absolute terms nowadays (end of 2011)

1) The common use case will have load times reduced by using dylibs instead because the dylib can be shared across different processes.  If you have ffmpeg and mencoder both running, they can share libx264.dylib, but they can't share libx264.a.  Thus, only one copy of libx264 needed to be loaded.  This saves time and memory pressure.

2) Not every device is a powerhouse workstation connected to the grid.  Even if you have excessive amounts of memory available, that extra disk/memory I/O will cost you power.  In the case where you don't have excessive amounts of memory, well... all arguments points to reducing your memory footprint, and dylibs are the clear winner.

> --
> 
> MacOS X offers some more advanced solutions to remedy some problems
> pointed out by the examples above, but since MacPorts focuses a lot on
> bringing projects from the UNIX world into MacOS X, supporting static
> libraries means supporting cross-platform solutions to the above
> problems. A developer of an open-source package does not need to
> special case the MacOS X deployment, but can rely on the standrad
> static linking solution.

Yes, I agree that they should not need to special case Mac OS X deployment.  That has nothing to do with using static vs dynamic libraries.  That is a consideration that exists on pretty much every platform and is an argument that has nothing to do with Mac OS X.