Request for help with port organisation: how to install several versions of data files

Tue Jul 2 06:27:04 PDT 2013

On Tue, Jul 2, 2013 at 1:08 PM, Ryan Schmidt wrote:
> On Jul 2, 2013, at 05:42, Mojca Miklavec wrote:
>>
>> So what is the difference between the two scenarios then? (Your
>> suggestion was to do (b).)
>> a) geant4.9.5 depends on geant-something-1.1 and
>> geant-somethingelse-3.0; geant4.9.6 depends on geant-something-1.2 and
>> geant-somethingelse-3.0
>> b) geant4.9.5 depends on geant4.9.5-data and geant4.9.6 depends on
>> geant4.9.6-data
>
> The difference is: what happens when you now want geant4.9.6 to depend on geant-something-1.3? First of all, you have to increase the revision of geant4.9.6,

No, see below. This would only happen when geant4.9.6 goes from, say,
beta 1 (geant4.9.6.b01) to beta 2 (geant4.9.6.b02) or to official
release. So it would need a rebuild anyway.

> thereby causing a rebuild. And after the upgrade, the user will have both geant-something-1.2 and geant-something-1.3 on their system, even if the user had upgraded with "sudo port -u upgrade"; the -u flag will uninstall the previous version of a port being upgraded, but MacPorts has no way of knowing that the geant-something-1.2 and geant-something-1.3 ports are in any way related so MacPorts will not uninstall geant-something-1.2 automatically.

This is indeed the downside. But it is also true that when user will
uninstall geant4.9.5 (with be it with "-u" or not) and install
geant4.9.6, all the geant-something-1.2 (independent of packaging)
from geant4.9.5 would be left on the system.

I was trying to argue that many data files can be shared between
different versions of Geant4, but looking at the statistics below,
it's usually just the small files that don't get updated. The huge 450
M data sets always get updated between major releases.

> If upgrading the version of any data files happens often, the user will end up with many of these older ports still installed and active, and will have to go through many unnecessary geant4.x.y rebuilds on the way.

The first is true, the second is not.

>> When the user will switch from 4.9.5 to 4.9.6, the "something-1.1"
>> will be left in either case. Except that in (b) also another copy of
>> "somethingelse-3.0" will stay on the hard drive, so one will get a
>> leftover of 600 MB instead of a leftover of 100 MB.
>
> I'm not familiar with the frequency of geant4 releases. Which do you think will occur more frequently: that a user upgrades from the geant4.9.5 port to the geant4.9.6 port, or that you increase the version of one of the data files?

Here's the schedule of the latest releases (b01 = beta 1, p01 = patch
1, ...) and for some of them the required versions of data files (and
their sizes at the end). It seems that major releases happen once per
year and each major release publishes approximately four to six betas
& patch releases. You also need to consider internal (MacPorts)
revisions related to bugs in packaging or upgrades of dependencies
(that doesn't apply to data files however).

Once the official release is out, the data files don't change any
more, they only change during the beta testing.

geant4 9.4
    - b01 2010-06-25 3.13 6.13 2.0 3.2 3.0 1.0 --- 1.0
    - 000 2010-12-17 3.14 6.19 2.1 3.3 3.0 1.0 1.2 1.0
    - p01 2011-02-18
    - p02 2011-06-24
    - p03 2011-12-09
    - p04 2012-04-13 3.14 6.19 2.1 3.3 3.0 1.0 1.2 1.0

geant4 9.5
    - b01 2011-03-30 3.15 6.22 2.2 3.3 3.0 1.0 1.3 1.0
    - 000 2011-12-02 4.0  6.23 2.2 3.4 3.0 1.1 1.3 1.0
    - p01 2012-03-20 4.0  6.23 2.2 3.4 3.0 1.1 1.3 1.0
    - p02 2012-10-22 4.0  6.23 2.2 3.4 3.0 1.1 1.3 1.0

geant4 9.6
    - b01 2012-06-29 4.1  6.27 2.2 3.4 3.0 1.1 1.3 1.0 ---
    - 000 2012-11-30 4.2  6.32 2.3 3.6 --- 1.2 1.3 1.0 1.1
    - p01 2013-02-01 4.2  6.32 2.3 3.6 --- 1.2 1.3 1.0 1.1
    - p02 2013-05-17 4.2  6.32 2.3 3.6 --- 1.2 1.3 1.0 1.1

geant4 10.0
    - b01 2013-06-28 4.3  6.33 2.3 3.7 --- 1.3 1.3 1.0 1.1

444M  G4NDL4.3
 75M  G4EMLOW6.33
 33M  PhotonEvaporation2.3
 11M  RadioactiveDecay3.7
5,8M  G4NEUTRONXS1.3
 24M  G4PII1.3
 21M  RealSurface1.0
164K  G4SAIDDATA1.1

(I also noticed that one of the data files was released twice [once
with a change in filename] because the first one had some wrong
permissions. This is a trivial fix that could be applied inside
MacPorts, but in case of my suggestion, it would be a bit difficult to
fix without increasing the revision - which is about the only slightly
problematic scenario which comes to mind.)

>> Do you want to say that changing
>>    name                geant4-data
>>    version             1.0
>> to
>>    name                geant4-data
>>    version             1.1
>> or "revision 1/2/3/..."
>>
>> will update all files?
>
> Yes, my suggestion was that you have a geant4.9.5-data port and a geant4.9.6-data port, and whenever the version of any of the data files therein needs to be changed, you increase e.g. the geant4.9.5-data port's version or revision. That way the user rebuilds the geant4.9.5-data port, but geant4.9.5 itself does not need to be rebuilt, or even touched at all.

This would (or should) almost never be the case. The geant data
changes less frequently than geant4 itself. The data only changes when
geant4 is updated (or if the packager makes a weird mistake - which is
not to be excluded).

(Maybe the core Geant4 developers and/or scientists in CERN change the
data files more often, but that's not something to worry about here.)

> * you write a portfile for the new version, fetching the file from a compromised mirror; you verify the md5 checksum with what upstream published and it matches; you generate new rmd160 and sha256 checksums of the compromised file and put them in the portfile

Yes, I understand this risk (but still believe that in case of a
compromised server, I wouldn't be immune myself). I don't think they
have any mirrors, or at least the files seem to be served from the
server that also displays the main website, but I will ask about the
other checksums.

Mojca