The static library discussion

[Jeremy Huddleston]

On Dec 22, 2011, at 07:38, Marin Saric wrote:

> Hi Jeremy,
> 
> Just a general remark before answering some of the interesting
> subpoints you make. A case for supporting static library deployment on
> a Unix ports project does not need to collide with supporting dynamic
> libraries. It sounds in your arguments as if you are imagining
> everyone linking against only static libraries. I can definitely
> imagine a mess if this were to happen, which is what dynamic libraries
> were designed to avoid in the first place.

Right.

> But I believe there's cases when it's very useful. What remains open
> is whether MacPorts wants to support static deployments or not and
> hopefully some more discussion in this thread will resolve this.

> If MacPorts decides it doesn't support static libraries, they could be
> disabled for a ton of ports and it would cut down the compile times
> for all the ports that were made to provide both.

I think you need to define what you mean by "support" here.  By supporting static libraries, do you mean
a) Shipping them and supporting our users use of them?
b) Shipping our ports linked against such static libraries.

Option (a) makes me squirm, but I understand there may be some reason for it, so I would want to restrict that to a +static variant.

Option (b) is something that we should definitely not do under any circumstance, mainly for the support reasons.  My VLC @1.2.0 and your VLC @1.2.0 might not be the same because they linked against different static libx264.  That is a nightmare from a support perspective.

Another consideration is if your application (say ffmpeg) links against libx264 and libmyencoder, and libmyencoder links against libx264.  Aside from wasting memory space due to this code duplication, we can run into problems if the two versions of libx264 are not the same in both and data is passed between the two.  This is a case that may not be obvious at first, but I've had enough headaches over similar issues to avoid this whenever possible.  One such issue could be a case where ffmpeg passes an x264 object to libmyencoder, but the layout of that object changed between the two libx264 versions used, so when libmyencoder passes it along to its libx264, it mishandles the data.


> I also would like to understand more the negative impact of supporting
> static libraries better.
> 
> So far I have this:
> - If MacPorts allows a port A to link against a static library B and
> that library B later gets updated, the ports dependent on B won't
> update, and it's complicated to determine whether they're outdated or
> not.

Right.  In a perfect world, we have full knowledge of what ports linked against the static lib, and we could reversion bump every dependent port after every version bump of that library.  That is error prone, incredibly messy, and wastes time rebuilding ports.

To get such knowledge, we could use darwintrace to verify that any static libraries linked against during the build are provided directly by ports in depends_lib.  In fact, that would be useful beyond the static lib case and would probably be a nice GSOC project.  I'm sure there are plenty of ports whose dependencies are incomplete and would benefit from this.  I'd argue that static libs should be in depends_build directly rather than inherited by the dependency tree.

> If there are outdated ports, they will all need to be recompiled,
> which after some point will become unmanageable.

I'm not sure why this sentence is contingent on "outdated ports" ... can you please elaborate on that?

>  - Static libraries don't advertise versions. If a user reports a
> crash in B, we will not know which version of B caused the crash. This
> is something easily fixable by MacPorts. I believe that having
> MacPorts creating a small log file of what port versions were used to
> produce a target would be useful both for static libraries and dynamic
> libraries and, of course, for binaries. As such, MacPorts specific
> dependency versioning information, such as revision numbers could be
> preserved for dynamic libraries as well.

To do this, it sounds like every port's receipt will need to have a list of all active ports and their respective versions at the time of compilation.  We'd also need to request this information from users when they file reports.  This does solve the issue of having the information available, but I don't know if it's necessarily worth the overhead.

> On Thu, Dec 22, 2011 at 7:00 AM, Jeremy Huddleston
> <jeremyhu at macports.org> wrote:
>> On Dec 21, 2011, at 16:33, Marin Saric wrote:
>>> It's source level API is consistent, but depending on
>>> what the version of x264 you download (new "snapshots" on something
>>> like a daily basis), your program is going to link against different
>>> symbols. In the newest version of x264 the ABI broke again.
>> 
>> Ok, so x264's development model is broken upstream.  Go yell at them to do better.  This is not a "MacPorts" issue.  It's an x264 issue, and that impacts all distributions, not just us.
> 
> I won't go and yell at anyone about this, not my style ;)

HAHA.  Fair enough.  I know plenty of others who have already, so that's fine. =)

> I agree that the above behavior definitely shouldn't be encouraged, I
> just didn't know MacPorts was in the business of enforcing an
> engineering style/philosophy on other open-source projects, but this
> thread will hopefully clarify this.

I don't think we're in the business of "enforcing an engineering style/philosophy on other open-source projects" ... We're a distribution, and it's our job to package up other OSS projects in a way that makes it easy for our users to use.  We can best do that when said projects follow well established "rules" that the community at large has developed.  Many build systems make this easy.  auto*/glibtool projects have their (IMO complicated) -version-info scheme (http://www.gnu.org/software/libtool/manual/html_node/Updating-version-info.html) or -version-number to handle this.  CMake, qmake, and other have similar schemes.  They should know by now that this is a problem for any software distributor, and they should correct their practices.  They can put out a version with a stable API that receives bug fixes only, and we can ship that.  Then they can do whatever API churn they want on their development branch and reduce ABI incompatible changes in a new version down the road.  This is what pretty much every other project goes through, so why do they need to be special cased?

> 
>>> Assume x264 is dynamic. Now assume a set of dynamic libraries H export
>>> symbols against an older ABI version of x264. Binaries depending on H
>>> and binaries dynamically linking against libraries depending on H all
>>> break once x264 is replaced with a new version.
>> 
>> AIUI, they do at least bump the dylib version, right?  So libx264.10.dylib becomes libx264.11.dylib when there is ABI incompatibility.  That, while sub-ideal, is still tolerable.  We dealt with this on a larger scale with libpng going from 1.2 to 1.4 and then again to 1.5.  All dependent
> 
> OK, good to know this has showed up before. This is ticket 28029, right?

Right.  That was the libpng case.

> What I see is that it was dealt with was the same like suggested for
> x264 right now, by bumping the version number and recompiling.

Yep.  In either situation (static/dynamic), dependent ports will need to update when there are ABI incompatible changes made.  With the dynamic case, it's at least easier to spot the cases where we missed it (apps won't load because the file they link against is missing).  In the static case, the apps will still load, but we'll be in a situation where we don't know what version of the static lib is in the executable.

> Was there going and yelling at libpng developers for this? :-)
> Or was it OK because the version bump was from 1.2 to 1.4?

Correct.  They did it right.  They bumped their library version.  In fact, it's possible for both versions to be installed on the system at the same time to allow existing applications to work.  MacPorts has issue with this because we can't have multiple versions of the same port installed at the same time, but that is an artifact of our system, not libpng.

To take the libpng example a step further, you'll see this on Lion:

$ ls -l /usr/X11/lib/libpng*dylib
-rwxr-xr-x 1 root wheel 296864 Oct  6 21:56 /usr/X11/lib/libpng.3.dylib
lrwxr-xr-x 1 root wheel     14 Jul 10 12:42 /usr/X11/lib/libpng.dylib -> libpng15.dylib
-rwxr-xr-x 1 root wheel 294160 Oct  6 21:56 /usr/X11/lib/libpng12.0.dylib
-rwxr-xr-x 1 root wheel 318160 Oct  6 21:56 /usr/X11/lib/libpng15.15.dylib
lrwxr-xr-x 1 root wheel     17 Jul 10 12:42 /usr/X11/lib/libpng15.dylib -> libpng15.15.dylib

Lion's X11's SDK provides libpng at version 1.5.x, but for compatibility reasons, we also continue to ship the libpng-1.2.x binaries, so existing applications which used the old ABI continue to run.



> Does anyone know any explicit standards for ABI compatibility that are
> based on version numbers?

There are slight differences between what we can do on Mac OS X and what we can do in UNIX generally, so these rules are the OSS/UNIX rules:
If you change or remove an interface, bump the major version number (changes the dylib id/soname).
If you add an interface, bump the minor version number and keep the dylib id the same.

On some systems, including Mac OS X, it's possible to mark "new" symbols as weak, so you can check for their existence at run time in case you want to support older versions of the library that didn't have that.  We're getting a tad off topic here, but if you look in /usr/include/xpc/xpc.h, you'll see __OSX_AVAILABLE_STARTING(__MAC_10_7, __IPHONE_5_0).  This causes the symbols to be weak when you set your deployment target to Snow Leopard or prior.  Your code can use XPC on Lion and go down a different route on Snow Leopard by checking for the symbol's existence.  As I said, this is a tad off topic, but I thought you might be interested in how we're able to do this with something like libSystem which never has a major version bump.

> I thought that generally it's up to the library developer to come up
> with their own policy of when to break the ABI and when not to.

Yes, they can break it whenever they want.  They just need to follow these simple rules of conduct when shipping their code.

> 
>>> The old symbols are
>>> not found or, even worse, they are found but their semantics or
>>> interfaces have changed in a subtle way causing malfuction.
>> 
>> That SHOULDN'T ever happen. If they change their ABI in this way, they need to bump the library version.  We should not do something like ship static libraries to work around their bad development practices.  If they ever do this, we should not ship their code until they fix it.
> 
> So there is two aspects to this:
> - Am I suggesting MacPorts should ship a static library as a workaround?
> No, I think MacPorts should ship both and let the other open source
> developers choose which one to use. Because of this issue, I'd always
> link statically against x264 on my project but others can choose at
> will.

And by "your project" I'm assuming something which you ship externally and not as a port.  If that's the case I'm fine with it.  How you ship your products is entirely up to you.

> The developers offer a static library along with the dynamic library.
> If I, as an open source developer knowing the style of these guys want
> to not worry about the version changes, I just link my library or
> binary against the static version of they library, which they also
> provided for. Of course I can't do that if MacPorts does not want to
> support static deployments.

I'd be fine with having a +static variant.  I've been slightly annoyed at the static libs installed and just rm /opt/local/lib/lib*.a every once in a while.  I should probably just add --disable-static to the default configure.args and see if anything breaks.

I understand that there are certain cases where it actually makes sense to have the static libs, but we should try not to promote it.

>> This issue does not impact just us.  Any distribution is impacted by this and if they really ship two releases where the library versions match but the ABI don't, that's a huge bug on their end.  If they don't see it as such, they need to be educated.
> 
> It's the opposite, the x264 library version never really matches. Each
> one is essentially marked by the date of their snapshot and there are
> a ton of them on their website. I think internally they have some sort
> of revision number for ABI that they keep bumping up, but I am not
> familiar enough with their strategy.

I'm not concerned with the version on their tarball.  I'm concerned with the version in their library (the dylib id on Mac / soname on Linux, etc).  On my system, I currently have /opt/local/lib/libx264.118.dylib. This "118" is what I'm talking about here.  If they ever break ABI, they need to bump that version number.

> This is exactly what I wanted more feedback on from this thread.
> Whether MacPorts is in the business of educating others on how to do
> things "the-right-way(TM)" or provided multiple ways to just get a
> port done quickly, provide new versions to users and call it a day.

It sounds like they do do the right thing (or at least they try to) considering that they have such a high version number.

>>> One "bad apple" can wreak havoc on the whole system. Note that this is
>>> different than the symbol-collision problem fixed by two-level
>>> namespaces.
>> 
>> Yeah, so fix the bad apple.  Don't wreck the system to lessen the impact of the bad apple.
> 
> I want to understand more about why is providing a developer a chance
> to link against your library statically wrecking the system? I am not
> arguing for switching everything to static, that will not work.

I think that as long as we do not ship our ports linking against the static libs, we're fine.  External developers can do whatever they want with static libs that we ship (although I'd prefer a +static variant for that).

> But will MacPorts allow other projects to link statically?
> It's really convenient to rely on MacPorts as a build system.

Like I said before, I'm more concerned about what ports do within MacPorts and external developers can do whatever they like with what we ship.  I just don't want to be in the position where 3 libraries that my port link against are static and it makes it nearly impossible for me to reproduce a bug that a user is having.

>>> Example 2: "Feature freeze"
>>> If you want your binary to just depend on a specific version of a
>>> library that you have validated against instead of the newest latest
>>> and greatest, you will link against a static version of it.
>> 
>> Or if you're trying to support a distribution and someone reports a bug occurring in some media player which is crashing in its x264 code, and that was provided by a static library, there's no way to know what version of the static library is present.
> 
> That's definitely a valid point.
> However, MacPorts could keep a log during compile time about the
> versions of all of the port dependencies and store that with a port. I
> don't think this would be very difficult to script at all. Even
> dynamic libraries wont tell you the MacPorts revision number. Some
> dynamic libraries won't report the version anyway, for example tbb and
> eigen.

The difference for dylibs is that it doesn't matter what dylib was present at build time.  It just matters what is present at runtime.

> I think this would be a useful debugging/development feature in general.

Yeah, possibly.  The data's there.  It would just need to be stored.

>> Another way of looking at that is that if x264 is bumped multiple times, the binary package of VLC or mplayer or ffmpeg won't be consistent across those releases.  We'll need to bump the revision of those ports to rebuild against the newer x264 to pick up the bug fix, whereas they would "just work" (modulo ABI changes) if you just used dynamic libraries.
> 
> I think this is a legitimate issue and it can quickly become a
> maintenance headache.
> 
>>>  On MacOS X
>>> you can put stuff in your own framework or have it live in an
>>> application bundle. Outside the world of MacOS X, linking against a
>>> static library is a way to do it.
>> 
>> Uhm... what?
> 
> Does the "uhm... what"  mean that I wrote something unclear or that
> you are expressing a disagreement with something I wrote?

I think I'm a bit unclear on what you mean because I don't see a difference between the mac vs non-mac cases.  Your statement here confused me, but from below, I think I understand what you mean.

>> "Outside the world of Mac OS X" this is exactly the same issue.  A Framework is just a dylib and a bunch of headers.  Strip away the packaging, and it's exactly the same thing.
> 
> I was referring to the fairly common practice of deploying your own
> versions of dependent libraries inside an application bundle on MacOS
> X, as opposed to expecting to pick it up from a system directory.

Ok, so I'm going to pick on VLC again here as an example.  I haven't looked at how they package everything, so this probably doesn't reflect their reality but will be useful for my use case.

I think you're saying that VLC might want to use its own zlib, so it packages up VLC.app to have Frameworks/Zlib.framework and links against it.  Similarly, you're saying that they might package it up for Fedora as a standalone package without using Fedora's zlib by linking against a static zlib.

It's quite possible to ship VLC.app with the static zlib, and it's possible to ship your own dynamic zlib in Fedora which would be picked up instead of the system one.  In fact, my experience with some 3rd party linux applications (mainly older versions of Maya and Shake here) were that they shipped their own versions of system libraries as dynamic libraries and used LD_LIBRARY_PATH (or maybe it was LD_RUN_PATH... it's been a while since I dealt with that on Linux) to pick up their copies instead of system ones.

>>> Or you might just say, OK, I am happy with the
>>> functionality provided by the current version of the library and I am
>>> OK with statically linking against it.
>> 
>> If you're happy with that version, then stick with that version by providing a local overlay of that port which is stuck at that version.  Using a static library just makes it exceedingly difficult to manage, leads to wasted memory due to duplication of this library across processes in a way that can't be shared, wasted disk space due to additional duplication, and is generally BAD software engineering.
> 
> What exactly do you mean by a local overlay?

mkdir -p ~/dports_overlay/multimedia/x264
cp /path/to/your/dports/multimedia/x264/Portfile ~/dports_overlay/multimedia/x264

Edit /opt/local/etc/macports/sources.conf and add file:///Users/<my username>/dports_overlay to that file.

That will cause you to have control over what version of x264 is installed.  I forget what the ordering needs to be in the file, but that should be easy enough to test.

> I am not convinced deploying static binaries is bad software
> engineering. But, it might be bad for a source-level distribution.
> There are worse sins to make than spend a few hundred kilobytes or
> even megabytes of memory when there are gigabytes available and thus
> speed up the application load time and reduce the number of
> dependencies on other people's code.

Again, not every system has gigabytes of memory.  Perhaps your target audience does, but I'm more concerned about memory footprint and power usage for portable devices.

>>> Example 3: Building plugins - avoiding versioning conflict.
>>> This is a test case I am dealing with right now.  I have binaries that
>>> open my plugin through the dlopen API. These binaries are outside of
>>> MacPorts and they depend on their own private versions of freetype and
>>> some other libraries. Unfortunately they work in a flat namespace
>>> model and thus things break.
>> 
>> Why?  Fix their misuse and reliance on the flat namespace.
> 
> They're not open-source.

I assume they have a bug tracker though.. :/

> Even they were, sometimes making these fixes is much more tedious than
> providing a static build.

True, but it doesn't mean the real issue shouldn't be addressed.

>>> Static library is a simple fix.
>> 
>> I think you mean "messy workaround" rather than "simple fix"
> 
> I don't see it as a workaround, but as a feature freeze. Often, in a
> deployment I do not want to theorize about every single of the billion
> libraries my big application will depend on and just say, OK, this is
> what I have, this is what works and this is how it will stay until I
> release a new binary. This way I don't make my users volunteer to test
> all possible interactions of all possible dependent libraries.

Right, but you could do that with dylibs as well, assuming the flat namespace issue were addressed.

>>> My plugin is a dylib (or on Linux a
>>> .so), but it links statically against all the dependencies, so loads
>>> and runs without a problem in the application that uses different
>>> versions of dependent libraries.
>> 
>> If you didn't flatten your namespace, this wouldn't be an issue.  THAT is the real bug.
> 
> I usually target MacOS and Linux, sometimes also Windows.
> I thought only MacOS provides two-level namespaces.
> Am I wrong about this? I believe you agreed that this is should be a
> non-MacOS specific issue.

It's been about 7 years since I tortured myself with details of glibc, so I'm not sure what has happened since then ... especially with uclibc and eglibc, but I believe they still use a flat namespace across the flavors of Linux.  I have no idea about Windows.

>>> Example 4: Buidling libraries and plugins - deployment
>>> You can use the MacPorts provided libraries, yet provide a dynamic
>>> library or a plugin that does not have extra dependencies. You will
>>> find examples already of software inside MacPorts building "private"
>>> versions of library for the very same reasons.
>> 
>> What do you mean by this?
> 
> I can provide a single dylib (or .so in Linux) as a plugin file. No
> other extra libraries/files to deploy on the system.

Ah ok.  That makes sense then.  I agree that using static libs in this case would be helpful.

>>> Example 5: Binary deployment.
>>> Statically built binaries with MacPorts become trivially deployable
>>> outside of MacPorts. No extra libraries to bundle with or ask the user
>>> to compile, etc., no need for an installer, etc.
>> 
>> I don't consider that something that we care about.
> 
> OK, I just want to make sure this is the general sentiment.

By saying that it's not something that we care about, I'm saying that it's not something that I'm concerned with since it's out of the scope of the project (or at-least what I care about getting from the project).  I don't think we should go out of our way to remove functionality that some, like yourself, have grown dependent on, but I don't think we should sacrifice the general use case to support your use case if it ever came to that.

>>> Example 6: Load time
>>> Statically built binaries load very quickly. This comes at a price of
>>> an increased cost in RAM. This used to be an issue when machines came
>>> with an order of magnitude less RAM then they do today. The incurred
>>> cost is small in absolute terms nowadays (end of 2011)
>> 
>> 1) The common use case will have load times reduced by using dylibs instead because the dylib can be shared across different processes.  If you have ffmpeg and mencoder both running, they can share libx264.dylib, but they can't share libx264.a.  Thus, only one copy of libx264 needed to be loaded.  This saves time and memory pressure.
> 
> Maybe I am not up to date, but how does it result in a faster load
> time? Doesn't dyld have a ton more work to do to resolve all the
> symbols than in the statically linked target that offers only a few
> symbols? With devices operating in hundreds of megabytes per second
> (SSD) and tens of megabytes per second (platter hard drives), the time
> to actually read from disk is negligible in the whole process, it's
> the symbol lookup stuff that takes time.

This really just comes down to use case.  If libx264.118.dylib is already in memory, then we don't need to do any disk I/O to load it.

>> 2) Not every device is a powerhouse workstation connected to the grid.  Even if you have excessive amounts of memory available, that extra disk/memory I/O will cost you power.  In the case where you don't have excessive amounts of memory, well... all arguments points to reducing your memory footprint, and dylibs are the clear winner.
> 
> The biggest energy waster is the CPU, which would work a lot more to
> do all the symbol resolution for a dynamically linked target. I don't
> think extra memory I/O will meaningfully impact power consumption. The
> extra disk I/O is negligible, especially in terms of power
> consumption.

That all depends.  Every system has its tradeoffs.

> On which platforms is this a concern? How much wasted memory and disk
> are we talking about?

Not much.  I was just trying to provide a counterpoint to what seemed like a blanket statement.

> Is MacPorts targeting embedded platforms?

I believe someone started working on using MacPorts for the iPhone  I'd have to dig through macports-dev to find it.