[MacPorts] #60509: Upgrading ports sometimes breaks internet connection

[MacPorts]

#60509: Upgrading ports sometimes breaks internet connection
---------------------+--------------------
  Reporter:  iEFdev  |      Owner:  (none)
      Type:  defect  |     Status:  new
  Priority:  Normal  |  Milestone:
 Component:  base    |    Version:  2.6.2
Resolution:          |   Keywords:
      Port:          |
---------------------+--------------------

Comment (by ryandesign):

 Replying to [comment:3 iEFdev]:
 > ''32/64… MacPorts sometimes says I'm i386, but `uname` says x86_64.''

 There is an unfortunate terminology clash with the term "arch" in
 MacPorts.

 The MacPorts variable `os.arch` has one of two possible values at this
 time: `i386` means any computer with an Intel processor regardless whether
 32-bit or 64-bit and `powerpc` means any computer with a PowerPC processor
 regardless whether 32-bit or 64-bit. MacPorts prints the value of
 `os.arch` in several places in the log.

 The second and completely separate meaning of "arch" in MacPorts is that
 it can use four possible values for `-arch` flags: `i386` (32-bit Intel),
 `x86_64` (64-bit Intel), `ppc` (32-bit PowerPC), and `ppc64` (64-bit
 PowerPC). These values appear in `-arch` flags in the log and also in
 archive filenames.

 Since you are on Lion, we know you have a 64-bit Intel processor; Lion and
 later require that. So I recognize that your situation is not identical to
 the one I mentioned on our 32-bit Snow Leopard build machine. But it had
 similarities that made me want to mention it anyway.


 Replying to [comment:5 iEFdev]:
 > One ''could'' do the same line with `awk` instead – just to see if
 there's a difference? …if `grep|cut` is the problem.

 I don't think `grep` or `cut` specifically are the problem. I think part
 of the problem is we are spawning too many processes regardless what those
 processes are. Therefore one of my suggestions is to remove the use of all
 but the `ping` process and do the text processing in Tcl.


 > Where is this recheck/update done? During the port syncing or when
 actually installing upgrading a port?
 >
 > For me it broke was when upgrading the port it self, and not when
 checking for outdated ports.

 During the fetch phase MacPorts checks the ping times of the servers from
 which it might download during that phase.


 > When working with mirrors… Isn't location more imporant than `ping`
 times?

 Maybe but we do not know the locations of the servers.


 > Thinking… If location is automated from where I am - perhaps `ping`
 times could be centralized, and updated (just like Portindex) on a weekly
 basis?

 We update the Portindexes on our servers because the calculation should be
 the same for all users and by centralizing it we can save our users' CPU
 time.

 Ping checks are not the same for all users. They depend on where in the
 world each user is. The point is to determine which servers have the best
 performance for that user so that they download files from servers that
 would be fastest for them. So we cannot do ping checks centrally on our
 servers.

 There's also no point repeatedly (daily or weekly or even monthly)
 checking ping times for hosts that are not being used. We do have many old
 ports in our collection that still work but have not been updated for
 years. The current implementation of checking ping times right before
 deciding which of a set of hosts to download from seems fine.

 If you are suggesting that we should use a geoip database to determine the
 approximate location of the user and create a central database with geoip-
 derived locations of all of our hosts and use that data to choose a server
 close to the user, then those would be major changes to how MacPorts and
 its infrastructure works and it could certainly be discussed elsewhere
 (bring it up on the macports-dev mailing list if you are strongly
 interested in this) but it would be a lot of work and I'm not sure the end
 result would be so much more accurate than our current ping time check
 that it would be worth the effort.


 > Or what's the reason behind the 24h, except being daily updated?

 We don't want to check the ping times every single time we fetch; that
 would be overkill. So we cache the values for 24 hours. I guess Joshua
 thought that was a reasonable length of time when he implemented it. We
 don't want the values to get too stale. A user's location in the world (or
 a server's) might change which would invalidate the ping results. Or an
 Internet service provider might add additional links between their network
 and others which might make connectivity between the user and some servers
 faster. So we need to recheck ping times periodically.


 > Unless there are daily updates of perl ports, it will check that every
 time.

 Not necessarily. Many of the hosts used in the perl cpan fetch group also
 store non-perl files and might be used by other ports, so the ping times
 of some of the hosts might already be known due to other non-perl port
 updates.


 > Just a thougth… but could it be time related? I dont know how the `x
 -man-page://ping` looks like on newer versions of OS X/macOS, but can it
 be tweaked with other options to be more generous, timely. Like `-m`,
 `-T`, `-t` or  `-W`.

 As shown above, we already use `-t3` to limit the total time of each ping
 process to 3 seconds. Since we spawn all pings simultaneously, the total
 time taken should be no more than 3 seconds plus however long it takes the
 OS to spawn the processes. I don't know whether adding any other ping
 flags would be useful.


 > Ie. what happens if I'm an a slow connection …which I am, if my
 connection can't keep up the speed with the script?

 You can read the code I linked to earlier if you like, but if MacPorts
 cannot determine the ping time of a host, it assigns it a fake high ping
 time. If your network is not up to the task of processing 207 pings in 3
 seconds, then the list of ping times will have some inaccuracies, which
 will have the consequence that for 24 hours MacPorts might try to download
 files from hosts in an order which is not necessarily the fastest for you.

 So far I see no evidence that this is a slow network problem. It seems
 more likely to me that MacPorts is overtaxing the OS by launching so many
 simultaneous processes and thereby using up so many file descriptors
 simultaneously.

-- 
Ticket URL: <https://trac.macports.org/ticket/60509#comment:6>
MacPorts <https://www.macports.org/>
Ports system for macOS