[fpc-devel] Are x86 optimizations across various platforms shared?

[Mark Morgan Lloyd]

Graeme Geldenhuys wrote:
> On 2013-02-11 19:03, Mark Morgan Lloyd wrote:
>>> No of tests  |  Type of Tests  |  Linux     |   FreeBSD
>>> -------------+-----------------+------------+----------------
>>>   151        | CSV persistence |    0:22    |    0:27
>>> -------------+-----------------+------------+----------------
>>>   151        | TAB persistence |    0:22    |    0:27
>>> -------------+-----------------+------------+----------------
>>>   151        |    XMLLight     |    0:23    |    0:26
>>> -------------+-----------------+------------+----------------
>>>   151        | SqlDB-Firebird  |    3:14    |    3:38
>>> -------------+-----------------+------------+----------------
>>>   682        | Non-Persistent  |    1:09    |    1:30
>>> -------------+-----------------+------------+----------------
>>>
>>> As you can see, consistently the FreeBSD tests take longer than the
>>> Linux ones. The test project on each platform was compiled with exactly
>>> the same compiler settings.
>> What exactly are we looking at there: 151 iterations inside a single 
>> program, or 151 programs?
> 
> The unit testing project is a single executable running all the above
> tests. 151 is 151 different unit tests to test the various persistence
> layers. The test suite is based on a hierarchy of classes, that is why
> all persistence tests have 151. The exact same persistence tests are run
> for each persistence layer. The 682 is again different tests for
> anything non persisting - testing various classes, and pretty much all
> functionality of those classes.
> 
> I did not setup the testing framework to run multiple iterations, only
> one run was completed with a total of around 1200+ unit tests taking
> around 4:30-5:00 to complete, from start to finish.
> 
> So memory cache etc should really have an effect. Because each test case
> starts from scratch, does the test, then cleans up. Then the next test
> etc etc.

There's a vast number of factors that you're underreporting. For 
example, the underlying VM system could detect which OS is running as a 
guest, and behave differently. Or the OS could reconfigure the CPU and 
chipset (specifically, the cache architecture) to a different extent. Or 
Linux's habit of using all available memory as disc buffering could be 
accelerating startup and/or execution. Or BSD could be making heavy 
going of fixing up dynamic library references. Or the different guests 
could be running their (simulated) disc controllers differently.

The only way that you're going to get anywhere is by tweaking the 
programs to loop, so that you can factor startup time out. Then separate 
groups of tests that don't make heavy OS use from those that do. All 
basic engineering stuff.

-- 
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]