[fpc-devel] Feature request/discussion - SetLengthNoInit

[RORDEN, CHRIS]

The dynamic arrays in Lazarus have so many terrific properties. Giving power users the ability to not initialize them would give clear performance benefits. This is useful in real world sitautions where one has long term arrays, but also can use transient arrays of the same type. Another feature that could help power users is the ability to set the alignment of setlength() - see issue 0034031. From my view, dynamic arrays are one of the features that makes Pascal code so much nicer than C. Web searches for "Speed "problem" with SetLength” and "Faster way of initializing arrays in Delphi” suggest others encounter these issues as well. I would advocate for a couple of advanced and optional features (alignment and initialization control) would allow this elegant property of Pascal to scale into these niches. It certainly makes code moree readable and consistent to keep one type of structure, rather than having to switch to GetMem for doing some tasks (SIMD, GPU, transient large arrays). Giving users control can obviously deviate from guaranteed behavior (e.g. one could choose a poor alignment, or not initialize strong arrays and leave random data in elements). However, allowing advanced users to elect to use these features would really help dynamic arrays scale to problems we face in the real world.

Despite these preferences, the experiment below shows that even for very large arrays, the initialization penalty may not be big so long as long you fill the array immediately after setting the length of the array. Note that SetLength() is very slow, but the subsequent filling of the array is fast (presumably due to cache), while getmem returns quickly, but the intiailization is substantially slower. I would be grateful if anyone can suggest any issues here or any method to accelerate either set of routines.

Milliseconds SetLength: 214..221 mean 218
Milliseconds fill array: 56..103 mean 95
Milliseconds GetMem: 0..0 mean 0
Milliseconds fill array: 251..270 mean 259

--------------------

program test3;
//fpc -O3 test3.pas; ./test3
{$IFDEF FPC}{$mode delphi} {$H+}{$ENDIF}

  uses Math, SysUtils,DateUtils;

type
 TUInt32s = array of uint32;
 TInt32s = array of int32;
 TRGBA = packed record //red,green,blue,alpha
   R,G,B,A : byte;
 end;
 TRGBAs = array of TRGBA;
 TRGBA0 = array [0..MAXINT] of TRGBA;
 TRGBAp = ^TRGBA0; //pointer to RGBA array
 TUInt320 = array [0..MAXINT] of uint32;
 TUInt32p = ^TUInt320; //pointer to RGBA array


procedure SetLengthP(var S: TRGBAp; Len: SizeInt); overload;
begin
ReAllocMem(S, Len *sizeof(TRGBA));
end;

procedure setlengthTest();
const
nVox = (512 * 512 * 512); //typical CT scan as RGBA array
nTest = 10;
var
startTime: TDateTime;
i, j, ms, mn,mx,tot, mn2, mx2, tot2: int64;
ArrayDyn: TRGBAs;
Arrayp: TRGBAp;
asUint32s: TUInt32s;
asUint32p: TUInt32p;
begin
//setlength() is slow, presumably because it intializes array (fillchar())
//https://alt.comp.lang.borland-delphi.narkive.com/O0jRrNgS/speed-problem-with-setlength
//test setlength
mn := maxint;
mx := 0;
tot :=  0;
mn2 := maxint;
mx2 := 0;
tot2 :=  0;
for i := 1 to nTest do begin
startTime := Now;
SetLength(ArrayDyn, nVox);
ms := MilliSecondsBetween(Now,startTime);
mn := min(ms,mn);
mx := max(ms,mx);
tot += ms;
//fill arrays
startTime := Now;
asUint32s := TUInt32s(ArrayDyn);
for j := 0 to nVox-1 do
asUint32s[j] := (j );
SetLength(ArrayDyn,0);
ms := MilliSecondsBetween(Now,startTime);
mn2 := min(ms,mn2);
mx2 := max(ms,mx2);
tot2 += ms;
end;
writeln(format('Milliseconds SetLength: %d..%d mean %d', [mn,mx, round(tot/nTest)]));
writeln(format('Milliseconds fill array: %d..%d mean %d', [mn2,mx2, round(tot2/nTest)]));
//test GetMem
mn := maxint;
mx := 0;
tot :=  0;
mn2 := maxint;
mx2 := 0;
tot2 :=  0;
Arrayp := nil; //must initialize!
for i := 1 to nTest do begin
startTime := Now;
SetLengthP(Arrayp, nVox);
ms := MilliSecondsBetween(Now,startTime);
mn := min(ms,mn);
mx := max(ms,mx);
tot += ms;
//fill arrays
startTime := Now;
{$IFDEF X}
asUint32s := TUInt32s(Arrayp);
for j := 0 to nVox-1 do
asUint32s[j] := (j );
{$ELSE}
asUint32p := TUInt32p(Arrayp);
for j := 0 to nVox-1 do
asUint32p[j] := (j );
{$ENDIF}
SetLengthP(Arrayp,0);
ms := MilliSecondsBetween(Now,startTime);
mn2 := min(ms,mn2);
mx2 := max(ms,mx2);
tot2 += ms;
end;
writeln(format('Milliseconds GetMem: %d..%d mean %d', [mn,mx, round(tot/nTest)]));
writeln(format('Milliseconds fill array: %d..%d mean %d', [mn2,mx2, round(tot2/nTest)]));
end;

begin
setlengthTest();
    Exit;
end.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20200915/9e532d94/attachment-0001.htm>