When wget and curl know better than you do: unwanted automatic decompression

[Mojca Miklavec]

On Wed, Jan 21, 2015 at 11:05 AM, Joshua Root wrote:
> On 2015-1-21 20:19 , Mojca Miklavec wrote:
>> Hi,
>>
>> I have troubles testing the new version of CMake. Reason?
>>
>> Here's what happens on a linux box:
>>
>>> curl http://www.cmake.org/files/v3.1/cmake-3.1.0.tar.gz -o cmake-3.1.0.tar.gz
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>                                  Dload  Upload   Total   Spent    Left  Speed
>> 100 5816k  100 5816k    0     0   737k      0  0:00:07  0:00:07 --:--:--  791k
>>
>>> file cmake-3.1.0.tar.gz
>> cmake-3.1.0.tar.gz: gzip compressed data, last modified: Mon Dec 15
>> 21:19:37 2014, from Unix
>>
>>> md5sum cmake-3.1.0.tar.gz
>> 188eb7dc9b1b82b363bc51c0d3f1d461  cmake-3.1.0.tar.gz
>>
>>> gzip -d cmake-3.1.0.tar.gz
>>> md5sum cmake-3.1.0.tar
>> 6da9011024a215073c3b390e9ed6ee21  cmake-3.1.0.tar
>>
>>
>> And here's what happens on my mac:
>>
>>> curl http://www.cmake.org/files/v3.1/cmake-3.1.0.tar.gz -o cmake-3.1.0.tar.gz
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>                                  Dload  Upload   Total   Spent    Left  Speed
>> 100 29.8M  100 29.8M    0     0  22.5M      0  0:00:01  0:00:01 --:--:-- 22.5M
>>
>>> file cmake-3.1.0.tar.gz
>> cmake-3.1.0.tar.gz: POSIX tar archive
>>
>>> md5 cmake-3.1.0.tar.gz
>> MD5 (cmake-3.1.0.tar.gz) = 6da9011024a215073c3b390e9ed6ee21
>>
>> Can anyone understand why both curl and wget happily automatically
>> decompress the file on my Mac? This happens with both the original
>> version in /usr and the version shipped by MacPorts.
>>
>>
>> (At first I was questioning the weird behaviour of MacPorts that
>> refused to extract the file, while "tar xvzf cmake-3.1.0.tar.gz"
>> worked just fine. But it seems that "tar xvzf" simply ignored the fact
>> that the file wasn't compressed.)
>
> Adding the -i option to curl in both cases so it prints the headers may
> be enlightening?

On Linux (the properly working one, outside of firewall):

HTTP/1.1 200 OK
Date: Wed, 21 Jan 2015 10:10:00 GMT
Server: Apache
Last-Modified: Wed, 17 Dec 2014 18:10:04 GMT
ETag: "1f9ae58-5ae0fc-50a6d6282b700"
Accept-Ranges: bytes
Content-Length: 5955836
Connection: close
Content-Type: application/x-gzip
Content-Encoding: x-gzip

On Mac (getting decompressed file, inside the firewall):

HTTP/1.1 200 OK
Via: 1.1 ZID
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 31261184
Date: Wed, 21 Jan 2015 08:56:38 GMT
Age: 4327
Content-Type: application/x-gzip
ETag: "1f9ae58-5ae0fc-50a6d6282b700"
Server: Apache
Accept-Ranges: bytes
Last-Modified: Wed, 17 Dec 2014 18:10:04 GMT


I think I understand now. It must be our "smart" firewall (running on
Windows 2003!) that's modifying files. I'll talk to our sysadmin (who
will most likely have no clue how to solve the problem).

Thanks a lot for the hint.

Mojca