[fpc-pascal] FPC 3.3.x breaks the Firebird Project's Firebird.pas

[Tony Whyman]

Michael,

I believe that the diplomatic answer to your response is that it is 
"disappointing". This is a serious problem and needs to be discussed.

Interfaces between modules written in different programming languages or 
even built with different compilers are never easy and depend on having 
published and stable definitions for each data type and structure. This 
is a common problem for anyone using Pascal given how so many code 
libraries are written 'C' or C++. Change can imply breaking many user 
programs and should not be done lightly and for purely internal reasons.

Both object and class type data structures are published in the Free 
Pascal Documentation and I do not believe that it is unreasonable to 
expect that the FPC developers will do their best to maintain stability 
for these structures as they would for any other data type. You seem to 
think that assuming that the object data structure is a stable and well 
known data structure that can be relied upon to be the same in each 
successive compiler version is "an invalid assumption". If that is the 
case then can we rely upon the stability of any data type and hence of 
any interface between FPC and other language environments?

In the case of monitor data, you have said that Delphi have implemented 
this as a "hidden" field. I presume that you mean that this uses a 
technique such as a negative offset to the object instance pointer. This 
would certainly be a good way to add system information to an object 
without destabilising the data structure for the user. Surely the right 
way to go.

The Firebird OO API may be the only interface library that uses the 
existing object layout in order to implement the interface. That would 
surprise me and there may well be more interfaces that are broken by  
FPC 3.3.x.

In the case of the Firebird OO API a complete redesign will be needed to 
work around your change - requiring a separation of the data blocks 
exchanged via the interface, and the classes supporting them. Even 
though a code generator is used, there are still about a hundred classes 
to consider with some used as callbacks. The Firebird.pas unit itself 
comprises about 14,000 lines of code and comments. You are asking 
voluntary programmers to devote hundreds, perhaps thousands of hours of 
time, to build and test a new interface design purely for the benefit of 
FPC, when the existing design works with both Delphi and FPC and has 
done so since the early part of the last decade. Until someone puts in 
this effort Pascal users of Firebird will be restricted to using the old 
legacy interface (which has not been updated since Firebird 2.5 and 
misses many important features), keeping with the old version of the 
compiler, or just moving to Delphi.

And, before going down this path, perhaps you might like to give a heads 
up of what other changes are in the pipeline that may knowing break 
interfaces to other programming environments.

Tony Whyman


On 02/05/2024 13:55, Michael Van Canneyt via fpc-pascal wrote:
>
>
> On Thu, 2 May 2024, Tony Whyman via fpc-pascal wrote:
>
>> This is a problem reported to me by an IBX user. I have not yet 
>> confirmed it (need to find the time to set up the latest development 
>> branch of FPC), but the issue looks solid, and is due to the 
>> following addition to TObject (copied from the GitLab master branch)
>>
>> |TObject = class|{$IFDEF SYSTEM_HAS_FEATURE_MONITOR}
>> strictprivate
>> _MonitorData : Pointer;
>> private
>> functionSetMonitorData(aData,aCheckOld : Pointer):Pointer; inline;
>> functionGetMonitorData:Pointer; inline;
>> {$ENDIF}
>>       ...
>>
>> Since Firebird 3.0, the Firebird project has provided a file called 
>> "Firebird.pas" in order to provide an interface to Pascal (both 
>> Delphi and FPC) from a C++ environment. There is an underlying 
>> assumption in the interface. That is TObject contains no fields.
>
> This assumption is not valid in Delphi either. It also has this field, 
> but
> it is 'hidden' at a not-fixed location.
>
>>
>> The idea is simple enough. There is a single entry point to the 
>> Firebird client library with the Pascal signature
>>
>> function fb_get_master_interface : IMaster; cdecl;
>>
>> This returns a pointer to a structure that may contain fields but is 
>> primarily a list of pointers to cdecl functions. These functions can 
>> perform various tasks. In the case of IMaster, they can also return 
>> similar structures representing other "interfaces".
>>
>> On the Pascal side, IMaster is defined as a class with no virtual 
>> methods and a single field called "vTable" and this is set to the 
>> pointer to the structure returned by fb_get_master_interface. The 
>> rest of the class comprises methods used to call functions from the 
>> vTable. This makes IMaster look like a normal class and hides the use 
>> of function pointers. The code is generated by a program know as 
>> "cloop". All Firebird interfaces declared in Firebird.pas follow the 
>> same approach.
>>
>> Given the assumption that TObject defines no fields, it is possible 
>> to coerce the pointer returned by fb_get_master_interface to appear 
>> as an object "instance" of the IMaster class, and similarly for every 
>> other Firebird interface class. This coercion ceases to be valid as 
>> soon as a field, such as _MonitorData is added to TObject.
>>
>> It is probably a valid argument to say that this is Firebird's 
>> problem as the TObject definition is an internal data structure. 
>> However, this ignores a serious real world problem.
>
> The real problem is the invalid assumption.
>
> This assumption is invalid for Delphi as well. TObject is not empty 
> there either. The memory layout is simply different.
>
>> A fix or workaround is needed and that has to be in Firebird.pas. It 
>> would be a logistic nightmare to try and change the C++ Firebird 
>> client DLL/SO.
>
> It would be a logistic nightmare in FPC as well.
>
>>
>> Also, does anyone know whether a similar problem will arise with 
>> Delphi, or is thus just an FPC issue?
>>
>> One of the following seems to be needed:
>>
>> a) Rewriting Firebird.pas to set the vTable field explicitly rather 
>> than implicitly through coercion - ideally without breaking any user 
>> code... (note: no current need to free IMaster etc)
>>
>> b) Taking a different approach to introducing _MonitorData e.g.
>
> This field was introduced for Delphi compatibility:
> The implementation of TMonitor requires this data.
>
> Delphi has this field as well but "hides" the field behind all other 
> actually declared fields.
>
> If you examine the actual memory size allocated for TObject, you'll 
> see that an extra field is actually present.
>
> In FPC this turned out to be not so easy to do without modifying the 
> compiler, so it was decided that the cleanest approach is to actually 
> introduce the field as part of TObject.
>
> The firebase people use a trick to save you some work, based on an 
> invalid assumption.
> I don't think this is sufficient reason to expect the FPC team to 
> adapt their compiler.
>
> The correct solution is to change the cloop program to produce a 
> pascal record definition with procedural type fields at the correct 
> offsets. It should be perfectly usable without too many changes.
>
> So something like (adapt to actual procedure types)
>
> Type
>   RMaster = Record
>     init : procedure(xyz:sometype);
>     // etc
>   end;
>   IMaster = ^RMaster;
>
>   // optionally; although it seems this already exists?
>   TMaster = class
>     intf : IMaster;
>     procedure Init(xyz : sometype);
>     constructor create;
>   end;
>
>
> function fb_get_master_interface : IMaster; cdecl;
>
> constructor TMaster.Create;
>
> begin
>   intf:=fb_get_master_interface;
> end;
>
> Procedure TMaster.Init(xyz : sometype);
> begin
>   intf^.init(xyz);
> end;
>
> if you cannot change cloop, you can use fcl-passrc or some other tool 
> to actually parse the definition generated by cloop and change it to a 
> record, possibly with a wrapper class for easier access as in the above.
>
> That should not be difficult to do, and it will work in Delphi as well.
> The advantage is that it will be more robust against future changes as it
> makes no assumptions about the layout of objects.
>
> Michael.
> _______________________________________________
> fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
> https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
>