Use HFS compression on installed ports...

[Eric A. Borisch]

On Mon, Nov 6, 2017 at 9:24 PM, Ryan Schmidt <ryandesign at macports.org>
wrote:

>
> On Nov 6, 2017, at 21:10, Eric A. Borisch wrote:
>
> > Just wondering what everyone would think of applying this patch?
>
> Are you aware of prior discussion in this ticket?
>
> https://trac.macports.org/ticket/36560


Yes. I thought I'd start here to cast a wider net.

I actually had been running with my own patch to run afsctool (in parallel)
over extracted files, and then discovered that bsdtar has the
--hfsCompression option; we can use _a_ tar, just like we usually do, with
one extra flag, and it's done. Unfortunately even on High Sierra the native
bsdtar is 2.8.3, and it looks [1] like this went in in 3.1.0.

As you can see from the patch there are only four new actions: two tests
(bsdtar in path? bsdtar accepts --hfsCompression?) and two assignments if
those are both true (use tar=bsdtar, and use --hfsCompression in the flags).

It's a slam dunk from my perspective. Compared with five years ago, the
things that make it beneficial (trading CPU cores sitting idle for disk
space) are just more in abundance, and with more systems with soldered-on
storage, drive capacity is still at a premium. The only question (to me) is
if we should bundle (~2MB installed, ~900k packaged) libarchive, or test
for it at runtime similar to lbzip2. (Which is another
everyone-should-use-it tool in the multicore age...) I can see arguments
for both, but the test-at-activation-time is certainly lower impact to
begin with, and reduces the required footprint.

The (by far) biggest downside is that it negatively impacts activation
time. This as due to the extra (expensive) compression step performed done
during activation, but it doesn't have to write the file and then
re-process it (afsctool route) -- bsdtar does it in-line. (If a file
doesn't compress well, it ends up writing the compressed one, and then
re-writing the decompressed one, from my quick read of the code.)

Also of (new) note, it appears to still work on APFS volumes. There were
mixed messages in early reviews of APFS if this would be the case, but at
least the finder is reporting a file extracted via bsdtar with
--hfsCompression clearly shows the file as compressed and reports both the
size and space used on disk -- and they're vastly different for a trivially
compressible (repetitive) text file.

Some quick sizes (High Sierra)

boost: 290MB w/o compression; 190MB with
qt5-qtbase: 56MB / 30MB
llvm-4.0: 135MB / 60MB

I'm sure there will be those that will question how it will impact
performance. I'm sure there are tests one could devise where it is not
beneficial, but I'll exit with one final note: Apple HFS+ compresses just
about everything I've checked (again High Sierra) in the system / default
applications. Take a look for yourself; 'open /usr/include' and then use
the finder to get info on a header file. curses.h as a nice large one to
see.

 - Eric

(I'll post a link to this thread in the ticket for good measure.)

[1] https://github.com/libarchive/libarchive/commit/fb67a295
4f6ad067a272373eca797278e8232223

[Sorry for the duplicate Ryan; didn't hit reply-all; edited third paragraph
from previous email that just went to you, FWIW]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20171106/d3accae7/attachment.html>