Thoughts on switching our archive compression method

[Ryan Schmidt]

On Sep 22, 2020, at 07:58, Vincent wrote:

>> It would be nice if we could easily switch our precompiled archives from bzip2-compressed tarballs (tbz2) to better compression methods as they become available. For example, xz-compressed tarballs (txz) would be better today. OS X 10.9 Mavericks and later has built-in support for xz compression (https://trac.macports.org/ticket/56237) so we could use that to avoid needing to use the xz port or needing to bundle a copy of xz (https://trac.macports.org/ticket/52000).
> 
> How much is gained by switching from tbz2 to txz?

It varies of course but here are a few examples:


$ bzcat /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2 | xz -9 > texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz
$ ls -l /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2 texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz
-rw-r--r--  1 macports  wheel  565981279 Aug 23 23:16 /opt/local/var/macports/software/texlive-latex-extra/texlive-latex-extra-54584_0+doc.darwin_17.noarch.tbz2
-rw-r--r--  1 rschmidt  wheel  481850300 Sep 22 08:56 texlive-latex-extra-54584_0+doc.darwin_17.noarch.txz

(xz is 85% of bz2 size)


$ bzcat /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2 | xz -9 > go-1.15.2_0.darwin_17.x86_64.txz
$ ls -l /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2 go-1.15.2_0.darwin_17.x86_64.txz
-rw-r--r--  1 macports  wheel  124756642 Sep 14 09:09 /opt/local/var/macports/software/go/go-1.15.2_0.darwin_17.x86_64.tbz2
-rw-r--r--  1 rschmidt  wheel   86893084 Sep 22 09:32 go-1.15.2_0.darwin_17.x86_64.txz

(xz is 70% of bz2 size)


$ bzcat /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2 | xz -9 > python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz
$ ls -l /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2 python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz
-rw-r--r--  1 root      wheel  28496603 Sep  6 01:16 /opt/local/var/macports/software/python38/python38-3.8.5_1+universal.darwin_17.i386-x86_64.tbz2
-rw-r--r--  1 rschmidt  wheel  16136624 Sep 22 09:36 python38-3.8.5_1+universal.darwin_17.i386-x86_64.txz

(xz is 57% of bz2 size)


So I think we could save ourselves and our mirror providers, CDN, and users some disk space and bandwidth by switching to xz. bz2 was the best available built-in compression on Mac OS X 10.6 when we started doing binary archives but there are better options now.