[fpc-devel] Why/how does the compiler have a non-trivial number of memory leaks after over two decades of development?
Ben Grasset
operator97 at gmail.com
Mon Jul 30 02:58:43 CEST 2018
I was feeling inquisitive today, so I added
'SetHeapTraceOutput('log.trc');" to the first line of pp.pas and built
myself a debug copy of the compiler using Lazarus. After building the
following very simple program:
program Test2;
{$mode objfpc}{$H+}
procedure TestProc(const S: String);
begin
WriteLn('The test string is: ', S);
end;
begin
TestProc('Hello, world!');
end.
I was honestly surprised to find that there was already an unfreed block:
140145 memory blocks allocated : 23669524/23770688
140144 memory blocks freed : 23669468/23770632
1 unfreed memory blocks : 56
True heap size : 1114112 (192 used in System startup)
True free heap : 1113696
Should be : 1113736
Call trace for block $000000005E642220 size 56
$000000010000DB6B
$00000001000A35ED TNODE__ALLOCOPTINFO, line 973 of node.pas
$0000000100164889 SETEXECUTIONWEIGHT, line 366 of optutils.pas
$00000001000C2873 FOREACHNODESTATIC, line 321 of nutils.pas
$00000001000C2DAA PROCESS_CHILDREN, line 308 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
$00000001000C2DF1 PROCESS_CHILDREN, line 309 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
So then I tried something just a little bit more complex:
program Test;
procedure TestProc(const S: String);
begin
WriteLn('The test string is: ', S);
end;
generic procedure GenericTestProc<T>(const S: T);
begin
WriteLn('The test value is: ', S);
end;
begin
TestProc('Hello, world!');
specialize GenericTestProc<LongInt>(15);
end.
And unfortunately things got a lot worse very quickly:
141240 memory blocks allocated : 23794575/23896072
141219 memory blocks freed : 23791692/23893176
21 unfreed memory blocks : 2883
True heap size : 2195456 (192 used in System startup)
True free heap : 2188992
Should be : 2189680
Call trace for block $000000007CECC6A0 size 56
$000000010000DB6B
$00000001000A35ED TNODE__ALLOCOPTINFO, line 973 of node.pas
$0000000100164889 SETEXECUTIONWEIGHT, line 366 of optutils.pas
$00000001000C2873 FOREACHNODESTATIC, line 321 of nutils.pas
$00000001000C2DAA PROCESS_CHILDREN, line 308 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
$00000001000C2DF1 PROCESS_CHILDREN, line 309 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
Call trace for block $000000007CF57310 size 39
$000000010000DA72
$00000001000080C3
$0000000100008FAB
$00000001000088B4
$00000001000D3950 PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 394
of pgenutil.pas
$00000001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
Call trace for block $000000007CF57490 size 46
$000000010000DA72
$00000001000080C3
$0000000100009052
$000000010000842D
$00000001000D38A0 PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 391
of pgenutil.pas
$00000001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
Call trace for block $000000007CEE93F0 size 128
$0000000100015F5B
$000000010000DB4B
$00000001000282D5 TFPLIST__SETCAPACITY, line 757 of cclasses.pas
$00000001000287C3 TFPLIST__EXPAND, line 846 of cclasses.pas
$000000010002845C TFPLIST__ADD, line 784 of cclasses.pas
$00000001000D368C PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 374
of pgenutil.pas
$00000001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
Call trace for block $000000007CEEA950 size 128
$0000000100015F5B
$000000010000DB4B
$00000001000282D5 TFPLIST__SETCAPACITY, line 757 of cclasses.pas
$00000001000287C3 TFPLIST__EXPAND, line 846 of cclasses.pas
$000000010002845C TFPLIST__ADD, line 784 of cclasses.pas
$00000001000D35FF PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 366
of pgenutil.pas
$00000001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
Call trace for block $000000007CF34A60 size 12
$000000010000DB6B
$00000001000D35DD PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 364
of pgenutil.pas
$00000001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
$000000010018BA4A SUB_EXPR, line 4184 of pexpr.pas
$000000010018BA72 SUB_EXPR, line 4189 of pexpr.pas
$000000010018BA72 SUB_EXPR, line 4189 of pexpr.pas
Call trace for block $000000007CF57550 size 24
$000000010000DA72
$000000010000BF6A
$000000010000BE05
$000000010014E0A5 TSPECIALIZATIONCONTEXT__CREATE, line 62 of
pgentype.pas
$00000001000D3EBB GENERATE_SPECIALIZATION_PHASE1, line 578 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
$000000010018BA4A SUB_EXPR, line 4184 of pexpr.pas
Call trace for block $000000007CF57790 size 24
$000000010000DA72
$000000010000BF6A
$000000010000BE05
$0000000100029063 TFPOBJECTLIST__CREATE, line 1052 of cclasses.pas
$0000000100028E74 TFPOBJECTLIST__CREATE, line 1024 of cclasses.pas
$000000010014E08A TSPECIALIZATIONCONTEXT__CREATE, line 61 of
pgentype.pas
$00000001000D3EBB GENERATE_SPECIALIZATION_PHASE1, line 578 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
Call trace for block $000000007CF576D0 size 24
$000000010000DA72
$000000010000BF6A
$0000000100028E4C TFPOBJECTLIST__CREATE, line 1023 of cclasses.pas
$000000010014E08A TSPECIALIZATIONCONTEXT__CREATE, line 61 of
pgentype.pas
$00000001000D3EBB GENERATE_SPECIALIZATION_PHASE1, line 578 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
$000000010018BA4A SUB_EXPR, line 4184 of pexpr.pas
Call trace for block $000000007C9BAF50 size 312
$000000010000DA72
$000000010000BF6A
$000000010014E058 TSPECIALIZATIONCONTEXT__CREATE, line 60 of
pgentype.pas
$00000001000D3EBB GENERATE_SPECIALIZATION_PHASE1, line 578 of
pgenutil.pas
$000000010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$00000001001887ED FACTOR, line 3390 of pexpr.pas
$000000010018BA4A SUB_EXPR, line 4184 of pexpr.pas
$000000010018BA72 SUB_EXPR, line 4189 of pexpr.pas
Call trace for block $000000007CDCDD60 size 416
$000000010000DA72
$000000010000BF6A
$0000000100217BB0 TCGCALLPARANODE__CREATE, line 146 of ncgcal.pas
$0000000100138E6D TINLINENODE__HANDLE_READ_WRITE, line 1364 of ninl.pas
$000000010013F731 TINLINENODE__PASS_TYPECHECK, line 3268 of ninl.pas
$000000010014603C TYPECHECKPASS_INTERNAL, line 81 of pass_1.pas
$0000000100146188 DO_TYPECHECKPASS_CHANGED, line 124 of pass_1.pas
$00000001001857D0 POSTFIXOPERATORS, line 1994 of pexpr.pas
Call trace for block $000000007CCE9EA0 size 256
$000000010000DA72
$000000010000BF6A
$00000001001035B0 TTYPECONVNODE__CREATE_INTERNAL, line 966 of ncnv.pas
$0000000100138E48 TINLINENODE__HANDLE_READ_WRITE, line 1365 of ninl.pas
$000000010013F731 TINLINENODE__PASS_TYPECHECK, line 3268 of ninl.pas
$000000010014603C TYPECHECKPASS_INTERNAL, line 81 of pass_1.pas
$0000000100146188 DO_TYPECHECKPASS_CHANGED, line 124 of pass_1.pas
$00000001001857D0 POSTFIXOPERATORS, line 1994 of pexpr.pas
Call trace for block $000000007CD23DC0 size 224
$000000010000DA72
$000000010000BF6A
$000000010013341C TDEREFNODE__CREATE, line 755 of nmem.pas
$0000000100138E2A TINLINENODE__HANDLE_READ_WRITE, line 1365 of ninl.pas
$000000010013F731 TINLINENODE__PASS_TYPECHECK, line 3268 of ninl.pas
$000000010014603C TYPECHECKPASS_INTERNAL, line 81 of pass_1.pas
$0000000100146188 DO_TYPECHECKPASS_CHANGED, line 124 of pass_1.pas
$00000001001857D0 POSTFIXOPERATORS, line 1994 of pexpr.pas
Call trace for block $000000007CCEA040 size 256
$000000010000DA72
$000000010000BF6A
$00000001000FCFCC TTEMPREFNODE__CREATE, line 1103 of nbas.pas
$0000000100138E0C TINLINENODE__HANDLE_READ_WRITE, line 1365 of ninl.pas
$000000010013F731 TINLINENODE__PASS_TYPECHECK, line 3268 of ninl.pas
$000000010014603C TYPECHECKPASS_INTERNAL, line 81 of pass_1.pas
$0000000100146188 DO_TYPECHECKPASS_CHANGED, line 124 of pass_1.pas
$00000001001857D0 POSTFIXOPERATORS, line 1994 of pexpr.pas
Call trace for block $000000007CEEA050 size 120
$000000010000DB6B
$00000001000FC14D TTEMPCREATENODE__CREATE, line 906 of nbas.pas
$0000000100138CFA TINLINENODE__HANDLE_READ_WRITE, line 1341 of ninl.pas
$000000010013F731 TINLINENODE__PASS_TYPECHECK, line 3268 of ninl.pas
$000000010014603C TYPECHECKPASS_INTERNAL, line 81 of pass_1.pas
$0000000100146188 DO_TYPECHECKPASS_CHANGED, line 124 of pass_1.pas
$00000001001857D0 POSTFIXOPERATORS, line 1994 of pexpr.pas
$0000000100188A4A FACTOR, line 3431 of pexpr.pas
Call trace for block $000000007CEC2680 size 66
$0000000100015F5B
$000000010000DB4B
$0000000100029649 TFPHASHLIST__SETSTRCAPACITY, line 1369 of cclasses.pas
$0000000100029DA2 TFPHASHLIST__STREXPAND, line 1538 of cclasses.pas
$000000010002991A TFPHASHLIST__ADDSTR, line 1433 of cclasses.pas
$0000000100029A5A TFPHASHLIST__ADD, line 1464 of cclasses.pas
$00000001000D5BBA PARSE_GENERIC_PARAMETERS, line 1220 of pgenutil.pas
$00000001001691C6 CONSUME_PROC_NAME, line 714 of pdecsub.pas
Call trace for block $000000007CF525D0 size 32
$00000001000160C5
$000000010000DB4B
$00000001000296FB TFPHASHLIST__SETHASHCAPACITY, line 1387 of
cclasses.pas
$00000001000295E7 TFPHASHLIST__SETCAPACITY, line 1327 of cclasses.pas
$0000000100029D1A TFPHASHLIST__EXPAND, line 1526 of cclasses.pas
$0000000100029A04 TFPHASHLIST__ADD, line 1459 of cclasses.pas
$00000001000D5BBA PARSE_GENERIC_PARAMETERS, line 1220 of pgenutil.pas
$00000001001691C6 CONSUME_PROC_NAME, line 714 of pdecsub.pas
Call trace for block $000000007CDCC860 size 576
$0000000100015F5B
$000000010000DB4B
$000000010002959D TFPHASHLIST__SETCAPACITY, line 1323 of cclasses.pas
$0000000100029D1A TFPHASHLIST__EXPAND, line 1526 of cclasses.pas
$0000000100029A04 TFPHASHLIST__ADD, line 1459 of cclasses.pas
$00000001000D5BBA PARSE_GENERIC_PARAMETERS, line 1220 of pgenutil.pas
$00000001001691C6 CONSUME_PROC_NAME, line 714 of pdecsub.pas
$00000001001671AA PARSE_PROC_HEAD, line 881 of pdecsub.pas
Call trace for block $000000007CEBF580 size 64
$000000010000DA72
$000000010000BF6A
$00000001000297D8 TFPHASHLIST__CREATE, line 1404 of cclasses.pas
$000000010002A73B TFPHASHOBJECTLIST__CREATE, line 1840 of cclasses.pas
$00000001000D5B30 PARSE_GENERIC_PARAMETERS, line 1209 of pgenutil.pas
$00000001001691C6 CONSUME_PROC_NAME, line 714 of pdecsub.pas
$00000001001671AA PARSE_PROC_HEAD, line 881 of pdecsub.pas
$000000010016A841 PARSE_PROC_DEC, line 1597 of pdecsub.pas
Call trace for block $000000007CF52690 size 24
$000000010000DA72
$000000010000BF6A
$000000010002A6FC TFPHASHOBJECTLIST__CREATE, line 1838 of cclasses.pas
$00000001000D5B30 PARSE_GENERIC_PARAMETERS, line 1209 of pgenutil.pas
$00000001001691C6 CONSUME_PROC_NAME, line 714 of pdecsub.pas
$00000001001671AA PARSE_PROC_HEAD, line 881 of pdecsub.pas
$000000010016A841 PARSE_PROC_DEC, line 1597 of pdecsub.pas
$000000010017FD47 READ_PROC, line 2157 of psub.pas
Call trace for block $000000007CEC2220 size 56
$000000010000DB6B
$00000001000A35ED TNODE__ALLOCOPTINFO, line 973 of node.pas
$0000000100164889 SETEXECUTIONWEIGHT, line 366 of optutils.pas
$00000001000C2873 FOREACHNODESTATIC, line 321 of nutils.pas
$00000001000C2DAA PROCESS_CHILDREN, line 308 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
$00000001000C2DF1 PROCESS_CHILDREN, line 309 of nutils.pas
$00000001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
I tried other, more complex things after this, some of which are simply too
long to even be reasonably posted on the mailing list.
This raises a couple of big questions:
Considering that FPC provides optional built-in memory tracking that will
take you directly to the exact source line number of a leak, the fact that
there's so many either means people have simply not been doing leak checks
on the compiler at all, or have just been actively ignoring them. Neither
of those are "good news", but which is it?
Secondly, are there any plans to try to clean up and refactor the compiler
codebase in general? After diving in to take a look at where some of these
leaks originated, it's not at all surprising they exist. The entire
compiler essentially seems to revolve around "nodes" and various subclasses
of them, that are created in too many different places and ways to count,
and cannot be tracked in any meaningful way. This kind of tree design is
perfectly valid and relatively common, but the way FPC does it leaves more
than a few things to be desired.
I was even able to successfully actually fix some of the leaks, but others
boiled down to a "node" being created as a local variable in the middle of
some 1000+ line method, getting assigned to the "left" or "right" property
of some other globally visible node, and then never being freed.
To add to that, some of then actually *could not** *be freed even when I
tried to do so without raising an untraceable access violation later in the
compilation process, also, which means the compiler is basically knowingly
relying on undebuggable undefined behaviour in multiple places that could
easily be affected or altered by anything else anywhere else in the
codebase at any time.
The way-too-short, highly undescriptive naming of variables doesn't exactly
help, either! "hp" and "p" are not good variable names. They really aren't!
Overall, I'm not trying to put blame on anyone in particular here, but as
someone who loves using FPC and would like to see it continue to grow, it
would be nice if we could collectively address some of these basic,
fundamental issues (which I'd be happy to help with myself, of course.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20180729/4cf8fb5f/attachment.html>
More information about the fpc-devel
mailing list