Feedback request regarding speed optimisations in trace mode

[Jackson Isaac]

Hi Mihir,

On Thu, Jun 13, 2019 at 8:40 AM Mihir Luthra <1999mihir.luthra at gmail.com> wrote:
>
> Hi,
>
>>
>> I would like to see the time for each run (if 10 runs, then 10 columns
>> i.e., xx_run1, xx_run2, ...), rather than only average of them.
>>
>> Collect as much as insights we could, maybe we find some pattern or
>> something that might help us (not sure what though). Since I have a
>> feeling, we might get better performance in one run and not so good in
>> another.
>
>
> Thats true, that’s what troubling me the most. Before testing I read on stack overflow that sys+user time is always correct,
> but the most disturbing factor was that same port gives results significant differences on multiple runs. Like in case of gettext + reps, sometimes I get time as high as 13 mins and sometimes 11:30. Differences are also noticeable in sys and user times.
> Also I always made these tests on a completely new installation of the base.
>>
>>
>> I notice that some ports take less time with modification and some
>> ports take more time. Just want to make sure it happens in multiple
>> runs.
>
>
> I have made some more improvements since the last time, This time will make tests on every port for 10 times and upload on spreadsheet.

Not necessary to stick to 10 runs, just because I said so. It can be
even 3 or 5 runs. Whichever you feel seem to be good enough to give
some insights. E.g., you might feel that after 3 runs you are getting
constant time and not much difference, in another case you might get
all different values even on running it more than 3-5 times.

>>
>>
>> In an ideal scenario, I would prefer a clean environment (can be a
>> fresh install of OS, bare minimum apps running, reboot before each
>> run, etc.) to run each test (containers maybe? or some CI like travis)
>> which might not be affected by other processes running in background.
>>
> I didn’t reboot the system last time when I tested, but I had all my apps closed and only 2 tabs open in terminal, 1 for modified base and 1 for original. Also I didn’t make any runs simultaneously. This time I will reboot after each run, but testing like that for sure Is gonna take some days for a good number of ports.
> Fresh install of OS probably would be hard for me, can’t it be like a guest user.
>

These were just some random thoughts I had. Not necessary to stick to
these or follow each and every one. Just try different ways. Guest
user might be a good option (not sure though).

I would say if these things (i.e., reboot/fresh install of OS) impact
the timeline if you end up running tests for several days or weeks, I
would say then just run acceptable number of tests and when you feel
'Ya, this is the sweet spot', you may skip extensive tests (like
rebooting system for each and every port 10 times or so).

Also, I would rather not do fresh install of OS on personal laptop
each time I want to test a port. I was thinking more on terms of a
Sandboxed environment or a VM(?) which has same setup and resources on
each run, and which can be easily spinned up and deleted (in matter or
minutes, in parallel, or whatever). Like a cloud platform as a service
kind of thing, if it helps. It's easier if you can run these tests on
Linux, since cloud has linux servers, but I don't believe major cloud
providers provide MacOS VM's or anything of that sorts.

-- 
Jackson Isaac