[MacPorts] #68192: Archives can remain cached by the CDN without corresponding signature

[MacPorts]

#68192: Archives can remain cached by the CDN without corresponding signature
------------------------------+---------------------
  Reporter:  ulange-eso-org   |      Owner:  admin@…
      Type:  defect           |     Status:  new
  Priority:  Normal           |  Milestone:
 Component:  server/hosting   |    Version:  2.8.1
Resolution:                   |   Keywords:
      Port:  py39-markupsafe  |
------------------------------+---------------------

Comment (by ryandesign):

 Replying to [comment:7 ulange-eso-org]:
 > perhaps the ports command
 > should detect that and fall back to build from source for the older
 version?

 I think that would be a good idea. MacPorts base should not assume both
 files will exist; if either one or the other doesn't exist, it should try
 the next packages mirror or build from source. It should try fetching the
 rmd160 file first, since it's smaller.


 > In this
 > case only a few hours after the rebuild finished we now have to restart
 from the beginning because
 > py39-markupsafe 2.1.1 -> 2.1.3.

 You might not have seen this problem much before because I didn't used to
 clean up old archives on the server until the server's disk had nearly
 filled up, at which point I would try to remember how to run the scripts
 that do the cleanup. But since a few months, as we've wanted to do for
 years, the [ticket:56181 cleanup happens automatically] every Sunday.

 The cleanup script uses a number of criteria in weighting each outdated
 archive to decide which ones to delete first. Age and size are some of the
 considerations. py-markupsafe is small and was only updated to 2.1.3 on
 September 10, 2023, however the 2.1.1 archives dated back to March 16,
 2022, which is probably why they got deleted this past Sunday. We expect
 users to keep their ports reasonably up to date; MacPorts warns you if
 your ports tree is more than two weeks old. In this case, the script
 decided to delete an old archive less than two weeks after the new version
 was available, which is not ideal, and it would be nice if we didn't do
 that, but I'm not sure how we should modify the scripts to achieve that.


 Replying to [comment:8 jmroot]:
 > We should figure out a way to immediately expire files from the CDN
 cache when we remove them from the main server.

 We control when we run the deletion script on the private server but we do
 not control when the main public mirror synchronizes with the private
 server so doing the CDN purge when we run the deletion script wouldn't
 work. Someone could still request the file between the time that we purged
 it and the time that the public mirror synchronizes, which would cause the
 CDN to cache it again.

 We could write a script to monitor the private server's rsync log and then
 purge those items from the CDN as we see in the log that their deletion
 was synchronized. It might still not happen at exactly the same time (I'm
 not sure if rsync needs additional time to commit changes to disk for
 example).

 We'd also have to look into the CDN API for purging specific files.
 Hopefully there is an API that allows specifying an arbitrary number of
 paths at once. I would not want, for example, to have to send thousands of
 API requests, one per deleted file.

-- 
Ticket URL: <https://trac.macports.org/ticket/68192#comment:9>
MacPorts <https://www.macports.org/>
Ports system for macOS