Feedback request regarding speed optimisations in trace mode

[Joshua Root]

On 2019-6-13 16:40 , Mihir Luthra wrote:
> Hi,
>  
> 
>     I would like to see the time for each run (if 10 runs, then 10 columns
>     i.e., xx_run1, xx_run2, ...), rather than only average of them.
> 
>     Collect as much as insights we could, maybe we find some pattern or
>     something that might help us (not sure what though). Since I have a
>     feeling, we might get better performance in one run and not so good in
>     another.
> 
> 
> Thats true, that’s what troubling me the most. Before testing I read on
> stack overflow that sys+user time is always correct,
> but the most disturbing factor was that same port gives results
> significant differences on multiple runs. Like in case of gettext +
> reps, sometimes I get time as high as 13 mins and sometimes 11:30.
> Differences are also noticeable in sys and user times.
> Also I always made these tests on a completely new installation of the base.

With modern macOS and modern hardware there are performance effects that
you can't control, perhaps most significantly thermal throttling, but
also background jobs like Time Machine backups and iCloud sync.

You need to apply a little bit of statistical theory. Standard error
increases with variance, but can be decreased by increasing the sample
size. Calculate what sample size is needed to bring the standard error
down to an acceptable level.

>     In an ideal scenario, I would prefer a clean environment (can be a
>     fresh install of OS, bare minimum apps running, reboot before each
>     run, etc.) to run each test (containers maybe? or some CI like travis)
>     which might not be affected by other processes running in background.
> 
> I didn’t reboot the system last time when I tested, but I had all my
> apps closed and only 2 tabs open in terminal, 1 for modified base and 1
> for original. Also I didn’t make any runs simultaneously. This time I
> will reboot after each run, but testing like that for sure Is gonna take
> some days for a good number of ports.
> Fresh install of OS probably would be hard for me, can’t it be like a
> guest user.

I don't think there is much value in going to so much effort to ensure a
completely cold cache before starting. Close other apps yes, but don't
reboot or reinstall everything. Since we don't have control over disk
cache misses, and a warm cache is more likely in the real world anyway,
it's common practice to pre-warm the cache by doing one run before you
start recording results.

- Josh