Using xz by default for compression

[Eric A. Borisch]

FYI, lbzip2 is what you probably want, as it *can* decompress a 'stock'
archive faster. From the man page:

"The lbzip2 utility employs multiple threads and an input-bound splitter
even when decompressing .bz2 files created by standard bzip2."

That said, I'm in favor of having a list of valid suffixes that it can
search for archives against. Doesn't seem too outlandish. A simple script
could be provided for users who would like to recompress their local
archives.

As I posted earlier the benefits of xz (especally xz -9) for clang, llvm,
gcc, and boost (some of the biggest archives in my install) were fairly
significant: clang-3.5 (50% reduction) llvm-3.5 (49% reduction) and gcc48
and 49 (both ~50% the tbz2 size; only saved 20% with "xz" (no -9)).


 - Eric

On Sun, Jan 25, 2015 at 4:31 PM, Ryan Schmidt <ryandesign at macports.org>
wrote:

>
> On Jan 25, 2015, at 3:27 PM, René J.V. Bertin wrote:
>
> > On Sunday January 25 2015 14:29:01 Daniel J. Luke wrote:
> >
> >> and how much more time does it take to compress/decompress?
> >
> > Decompression is not noticeably slower or faster, compression is. I'd
> say that shouldn't be an issue for the build bots, and every user can
> decide for him/herself whether it is locally.
> >
> > I'm not working on patches because the support is already there, and
> requires a point edit in macports.conf to be activated. I'm not familiar
> with installer development, otherwise I'd propose a patch presenting the
> user with the choice between xz (slower but more space-efficient
> compression) and bzip2.
>
> I'm in favor of making xz the default (there is no need to ask the user
> about this), the trick would be handling existing installations, which have
> bz2 archives, without problems, and figuring out how best to handle the
> pre-built binary packages situation (either write a script to recompress
> everything as xz, or find a way to have MacPorts check for both).
>
> One thing to consider is that we somewhat recently made MacPorts base
> capable of using pbzip2 if it is installed, a parallel bzip2 implementation
> which uses more than 1 processor core for compression of bz2 files, and
> uses more than 1 processor core for decompression of bz2 files created with
> pbzip2. I worked with the developer of pbzip2 recently to improve its build
> system and to report an intermittent crashing bug, which got fixed, and
> this now seems to work stably on my system, where I don't use the binaries
> from the packages server. However since our buildbot builders do not have
> pbzip2 active, the archives created by the buildbot builders are not able
> to be decompressed more quickly by pbzip2 than by bzip2, so it is of
> limited value. I had considered bundling a copy of pbzip2 with MacPorts to
> solve this. There seem to be some parallel xz implementations; we should
> see if any of them is stable enough, and using a suitably compatible
> license, to consider being included in MacPorts.
>
>
> _______________________________________________
> macports-dev mailing list
> macports-dev at lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/macports-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.macosforge.org/pipermail/macports-dev/attachments/20150125/2cfc47bd/attachment-0001.html>