snapd

[Ryan Schmidt]

On Jun 17, 2016, at 12:34 PM, Alexey Kuznetsov wrote:
> 
> On Fri, Jun 17, 2016 at 8:26 PM, Ryan Schmidt wrote:
>> Again: why? What problem are you trying to solve?
> 
> I think snapd is a good point to try to start to be together. May be because no one tried three separate build system exists on Mac. Why waste people energy by supporting all of them instead of good collaboration?
> 
> Maybe 1000 years later while we well sit on Mars and get back to this discussion. Why people use different formats to store the same data? See you later.

Ok, so the problem you're trying to solve is: there are three package managers for OS X -- Fink, MacPorts and Homebrew -- and you would like for there to be only one package manager, or for the three package managers to interoperate with one another and/or install software from each other's repositories? Realistically, that's probably never going to happen. These are not three projects that were forked from a common code base; they are three independent software products, written in different programming languages, by different developers, with different design philosophies. They are not compatible with one another; in fact, they conflict with one another. I don't think it was ever anyone's intention for that to be the case, but it is the case.

Fink was written in 2000 shortly after the release of Mac OS X Public Beta. MacPorts was started in 2002. Why didn't its creators instead contribute to Fink? I don't know; I wasn't part of the project back then. A note on the Fink web site says the MacPorts creators wanted to try a different approach; it's written in Tcl and C and based conceptually on FreeBSD's ports system, though it doesn't share any of its code. Wanting to try a different approach is probably also what led to Homebrew being created in 2009; it's written in Ruby. Today, each of the three projects have a rich history of finding different ways of solving problems. It may be that multiple solutions are valid, or that some solutions work better than others. But the point is they are different solutions, and they do not interoperate with one another. I cannot take a MacPorts portfile written in Tcl and convert it into a format that can be understood by Fink or Homebrew.

I can understand the thought that it should not matter whether you install a package using Fink or using MacPorts or using Homebrew or by downloading the source and compiling it yourself -- that you should get the "same data" either way. But the reality is that there can be important differences. Let's take the boost package as an example. Fink currently has boost 1.58; MacPorts currently has boost 1.59; Homebrew currently has boost 1.60; and the latest version on the boost web site is 1.61. Boost is notorious for breaking backward compatibility, so it's entirely possible that a program that works with boost 1.58 does not work with boost 1.61, for example. The reason why I have not updated boost in MacPorts past 1.59 is that we did discover during testing that updating to 1.60 would break some other packages; see #50671. Ignoring all the other reasons why this would not work today, if it were possible to install Homebrew's boost 1.60 into MacPorts, this would break those ports that are not compatible with boost 1.60.

In addition, boost is written in C++. That means it uses a C++ standard library. There are two C++ standard libraries on OS X: the old libstdc++ and the new libc++. Software compiled for one C++ standard library cannot interoperate with software compiled for a different C++ standard library, so you must decide which C++ standard library you're going to use before compiling anything, and then compile everything using that C++ standard library. Which C++ standard library is used by default varies by OS X version. MacPorts offers the user the ability to choose which C++ standard library to use, so that, for example, users of older OS X versions like Lion where libstdc++ is the default can nevertheless compile software that requires features offered by the newer libc++ library, such as C++11 support. I don't know whether C++ packages built with Fink or Homebrew use libc++ or libstdc++. If we imagine that a user of MacPorts on Lion who wants to use libc++ installs a library from Fink that was built with libstdc++, programs using that library will crash.

I already mentioned the differing prefix. Homebrew defaults to installing software in /usr/local; software compiled by hand defaults to that location as well. Fink installs in /sw, and MacPorts defaults to /opt/local. If I understand correctly, unlike on Linux, binaries built on OS X contain the absolute paths of libraries they link with. This means it's nontrivial to relocate OS X software to a different location after it's been compiled. (Doing so involves running multiple install_name_tool commands.) It is possible to compile software on OS X in a relocatable fashion, but MacPorts does not do so because it is not how packages compile by default and there has not previously been a need to do so. Changing this now would be a laborious process of likely requiring changes in each portfile. As far as I know, Fink and Homebrew don't build relocatable software either.

Individual MacPorts maintainers may be responsible only for their ports, but we try to keep the MacPorts collection functioning as a whole, for example by not updating a port if it will break other ports, or by fixing the ports that would break. But we can't really be expected to also now coordinate those matters with two other package collections that may have different goals. As far as I know there has not been any attempt at collaboration between the three OS X package managers before. We have a bit of a friendly rivalry, and we share solutions as the need arises. I've certainly looked to Homebrew for patches to particular problems, and I've seen MacPorts patches turn up in Homebrew. We might also adopt features from each other over time, if we see something another package manager is doing that we like. Much as it does in other situations, I think this type of competition spurs innovation and keeps development from going stagnant.

It still wasn't clear to me what the "snap" format entails: whether it refers to the recipe for how to build software, which we in MacPorts call Portfiles, which are written in Tcl; or whether it refers to the compiled software that results from following that recipe, which we in MacPorts call archives, which are compressed tarballs. We've discussed years ago whether we should look into replacing Tcl Portfiles with a more modern programming language, due to the Tcl language being considered somewhat esoteric. But the Tcl language was chosen for reasons which may still be valid, and we have a growing collection of currently over 10,000 Portfiles. Converting them all by hand would take too long, and I doubt they could be converted automatically, given how many of them do use features of Tcl that may be difficult to replicate in other languages. As for the binary format we use, a tarball containing the payload plus some informational files has been sufficient so far.