plans for 64bit support

Tue Dec 11 16:08:40 PST 2007

On 11.12.2007, at 20:20, Ryan Schmidt wrote:

> Please Reply To All so that your reply goes to the mailing list  
> too, not just to me.
>
> On Dec 11, 2007, at 13:00, Daniel Oberhoff wrote:
>
>> Am 11.12.2007 um 09:50 schrieb Ryan Schmidt:
>>
>>> On Dec 11, 2007, at 02:35, Daniel Oberhoff wrote:
>>>
>>>> Am 10.12.2007 um 01:33 schrieb Ryan Schmidt:
>>>>
>>>>> On Dec 9, 2007, at 14:43, Daniel Oberhoff wrote:
>>>>>
>>>>>> Now that Leopard is out and already at 10.5.1 will macports be  
>>>>>> supporting 64bit libraries? It's just I need 64bit in my  
>>>>>> octave installation. I pull my octave from octave.org's cvs,  
>>>>>> but it needs quite a lot of support libraries. From what I  
>>>>>> gather it should be possible on Leopard to build fat  
>>>>>> libraries, i.e. ones that contain 64 and 32 bit code (i think  
>>>>>> it works using -arch x84_64 -arch i686 as gcc flags). Or will  
>>>>>> this be left to the separate ports?
>>>>>
>>>>> MacPorts is supposed to build libraries for whatever system  
>>>>> it's running on. So I would have thought that if you're on a 64- 
>>>>> bit Intel system, it should build 64-bit Intel libraries. Is it  
>>>>> building 32-bit Intel libraries for you?
>>>>>
>>>>> We have the +universal variant for building 2-way (32-bit,  
>>>>> Intel and PowerPC) universal binaries. We are still in the  
>>>>> process of getting this to work with many of the ports. It  
>>>>> could be changed to build 4-way (32-bit and 64-bit, Intel and  
>>>>> PowerPC) universal binaries. This should be possible on Tiger  
>>>>> too, as far as I know. It won't fix any ports that are having  
>>>>> trouble building 2-way universal binaries. Not sure if it would  
>>>>> mess up any ports that are already working. Are all the ports  
>>>>> that you need already working as 2-way universal binaries?
>>>>>
>>>>> I haven't heard anyone suggest building libraries that contain  
>>>>> 32-bit and 64-bit code for just one processor family before (in  
>>>>> relation to MacPorts). It would of course be possible, but I  
>>>>> think it would make most sense to continue along our current  
>>>>> path: software should be default install for the architecture  
>>>>> you're on, and if you need multiple architectures, then you  
>>>>> need the +universal variant.
>>>>>
>>>>> It has been said before that maybe 64-bit binaries aren't all  
>>>>> that helpful, but the Ars Technica review of Leopard explains  
>>>>> that while 64-bit binaries aren't that helpful on the PowerPC  
>>>>> architecture, they really are quite good for secondary reasons  
>>>>> on the Intel architecture. The 32-bit Intel architecture has  
>>>>> often been called inferior to the 32-bit PowerPC architecture,  
>>>>> but the 64-bit Intel architecture seems to fix many of the  
>>>>> issues. Also, maybe Leopard being a full 64-bit system makes 64- 
>>>>> bit binaries more relevant.
>>>>>
>>>>> Perhaps we could do 4-way universal binaries only when MacPorts  
>>>>> is running on Leopard.... but that might be a bad idea, since  
>>>>> only people running Leopard could  then develop and test this.
>>>>>
>>>>> We could introduce a new automatic variant... +universal4?  
>>>>> +universal64? People could test with this new variant and if  
>>>>> any problems are encountered it would not prevent anyone from  
>>>>> using the existing 2-way 32-bit +universal variant. I'm wary of  
>>>>> this though... I wouldn't want, say, "universal64" directives  
>>>>> to start appearing in portfiles, if we want to eventually fold  
>>>>> +universal64 into +universal.
>>>>>
>>>>> Just some thoughts off the top of my head.
>>>>
>>>> Hmm, ok. That makes sense, since I upgraded with an installed  
>>>> macports base. I suppose I should really reinstall all then,  
>>>> since otherwise I will end up with a crazy mix?
>>>
>>> Yes, if you're moving from one architecture to another (G4->G5,  
>>> PPC->Intel, 32-bit->64-bit, etc.) you should really throw away  
>>> your old MacPorts (saving any conf files or databases or other  
>>> data) and rebuild all your ports.
>>>
>>>> Though, per default on Leopard compilation seems to be 32 bit.  
>>>> When you just compile something without further options pointer  
>>>> size is 32bit. Only when you specify -arch x86_64 or -m64 do you  
>>>> get 64bit.
>>>
>>> Ah, I see. I didn't know. I haven't had a 64-bit machine.
>>>
>>>> So it would really make sense to have both (but not ppc). So I  
>>>> would vote for an autamatic variant to get intel only 32/64bit.
>>>
>>> So by default we build 32-bit binaries for the host architecture.  
>>> And we have a +universal variant for building 32-bit binaries for  
>>> both architectures. You want an additional variant for building  
>>> fat 32-bit and 64-bit Intel binaries. Do we then also need a  
>>> separate variant for building fat 32-bit and 64-bit PowerPC  
>>> binaries? And what about a fat universal variant with all 4  
>>> options? This proliferation of variants is what I want to avoid.  
>>> Most users will not need it and it will confuse some of them.  
>>> It's much extra work for some portfile authors, because some  
>>> software (rather a lot, actually) does not build successful  
>>> universal binaries using the simple tricks, and much more work is  
>>> required to build universal. See for example the openssl port  
>>> (which, to build universal, you'll have to back-date to version  
>>> 0.9.8e because of bug #12911) which goes through crazy hoops to  
>>> build first for one architecture, then the other, then lipo them  
>>> together. Similar (though perhaps not identical) hoops would have  
>>> to be gone through for fat binaries or 4-way binaries. Well, that  
>>> is until MacPorts base grows some universal-build helpers. (I  
>>> still really think MacPorts base should have an easy way to build  
>>> a port multiple times, once for each architecture, and lipo it  
>>> together, automatically, selected with just a single line in the  
>>> portfile).
>>
>> Yah, that doesn't sound fun. Would it be feasible to make these  
>> incremental? After all the mechanism to combine architectures is  
>> always the same (use -arch flags, lipo...). I.e. make it possible  
>> to say +64 +32 +ppc +intel to get the 4way build or just +ppc  
>> +intel to get a two way build. Or at least have something like  
>> that behind the scenes, and just offer specializations if some of  
>> the variants need some special care.
>
> I don't want users to have to think about what architecture(s)  
> they're on. Macs are supposed to be easy. It's easier if the  
> +universal variant includes all architectures, and if one day  
> +universal becomes the default. "Software just works," as it says on
>
> http://www.apple.com/universal/
>
> The mechanism to combine architectures is not always the same. For  
> example, it's different if it's a configure-based project vs. an  
> Xcode-based project. Also, while many configure-based projects  
> accept multiple -arch flags to build for multiple architectures at  
> once, some need you to build for just one arch, then later lipo  
> these together. MacPorts does not have any special support for the  
> latter at this time, though I think it should.
>

we put some research into this during GSOC this year: It's quite hard  
to come up with a magic lipo for everything as it is necessary to  
combine not only binaries (thats the easy part) but also stuff like  
header files (also easy), heap images of interpreters, scripting  
language programs, etc. -- which can get practically impossible.

We could enable four-way builds for (probably) all ports that we can  
build universal right now, but there is currently no way of saying  
"no ppc64 please" (etc.) in a port (and no tests/experience on what  
exactly will be necessary).

>> Alternatively maybe have two macports trees, one with 64bit, one  
>> with 32bit. Kinda like those lib64 dirs on some linux flavours. Is  
>> that possible right now?
>
> You're talking about two collections of port binaries, presumably?  
> Since we don't have any binaries at this point, or any 64-bit  
> builds, anything's possible, of course. Is it wise to have 64-bit- 
> only binaries? I wouldn't have thought so but I really have no idea.
>
> If you mean, is it possible to have 64-bit builds go into lib64  
> instead of lib, then I would say if that's not the default for the  
> software already, it would be a rather large hassle to fix every  
> portfile to do this, wouldn't you say? And why is this better than  
> a single all-encompassing superuniversal library installed in the  
> normal expected location?
>

you could always pass a "-arch x86_64" (or "-arch ppc64") to the  
cflags/cxxflags/.. manually if you desperately need 64 bit support,  
e.g.:
$> port -d install readline configure.cflags="-O2 -arch x86_64"
(no idea if that one compiles though)

If you want to make some tests with four-way universal builds,  
replace each occurrence (there are 3) of "-arch i386 -arch ppc" in / 
opt/local/share/macports/Tcl/port1.0/portconfigure.tcl with "-arch  
i386 -arch x86_64 -arch ppc -arch ppc64".
We are of course interested in your results... ;)

For octave you're out of luck though as it is build with the vanilla  
gcc 4.2 which cannot produce universal binaries.
Perhaps you can test the Apple gcc 4.2 beta preview for octave to  
achieve this (if you're on 10.5) -- though the Fortran part may be a  
problem here as octave seems to use it (the Apple compilers don't do  
Fortan while the vanilla ones cannot create universal binaries).

>>> Why, by the way, do you need 64-bit octave? Just curious.
>>
>> Various reasons:
>>
>> 1) I do get data clumps in the gigabyte range sometimes. Yes, it  
>> does lead to some paging, but that's better than a crash with out  
>> of memory :).
>> 2) this is more of a hack: I use Judy for some sparse double  
>> precision structs, and since Judy constrains data size to sizeof 
>> (void*) I need 64 bit.
>> 3) afaik 64bit has more registers than 32bit on intel  
>> architecture, avoiding register spillage, which sometimes snails  
>> some of my code.
>>
>> all in all, it is not critical right now. But every now and then  
>> 64bit would help a lot. Maybe not always though. Also due to the  
>> policy on Leopard to default to 32bit would probably make it hard  
>> to make do with an exclusive 64bit set of libraries.
>
> Ok.
>
>> Hmm, thinking of that: is it possible to convince port _not_ to  
>> switch over to 64bit now? I'd rather have a clean set of 32 bit  
>> libs :).
>

As you said: Even if a port can be build four way universal (with 64  
bit support), the 64 bit version doesn't necessarily get executed...
Of course you could always strip the unwanted code out of the binary  
manually afterwards (with lipo).

Regards,

-Markus

---
Dipl. Inf. (FH) Markus W. Weissmann
http://www.mweissmann.de/
http://www.macports.org/