[mpbb] branch master updated: Add smarter distfile mirroring script

[Ryan Schmidt]

On Mar 9, 2018, at 20:56, Ryan Schmidt wrote:

> On Mar 9, 2018, at 20:47, Joshua Root wrote:
> 
>> On 2018-3-10 13:39 , Joshua Root wrote:
>>> On 2018-3-10 13:07 , Ryan Schmidt wrote:
>>>> 
>>>> On Mar 9, 2018, at 20:03, Ryan Schmidt wrote:
>>>> 
>>>>> The first run of this script on the buildbot hasn't completed yet after an hour, and I don't know how to check on its progress...
>>>> 
>>>> Oh, I did check the mirrorcache folder, and the only file it has created so far is L/llvm_select.
>>> 
>>> Fixed the variant thing. It's probably seeing a lot of failures because
>>> of that, and hence not creating mirrorcache files, but it should still
>>> more or less work.
>> 
>> The distfile for caffe has indeed shown up on distfiles.macports.org for
>> example.
> 
> And the one for libiconv's +cp932fix variant. So it is working, but not updating the mirrorcache for ports that have variants.
> 
> I think I should cancel the current mirror job, and see how the next one goes, with the fixes you committed. Ok?

Well I did cancel it... Looking at the log, the list of ports isn't being deduplicated. Mirroring ncurses was attempted 506 times, gperf was attempted 687 times, etc.

https://build.macports.org/builders/jobs-mirror/builds/1212/steps/mirror/logs/stdio

The next run looked better, with 21 more ports being marked completed in the mirrorcache within the first 3 minutes. But then nothing further for 20 minutes. I cancelled that too so I could look at the log:

https://build.macports.org/builders/jobs-mirror/builds/1213/steps/mirror/logs/stdio

It's still mirroring the same ports multiple times, even ports like libiconv that are marked in the mirrorcache.

I'm going to cancel the remaining mirroring jobs for this commit. We'll be able to debug this better looking at logs from commits of fewer ports.