[fpc-devel] TClientDataset (was: Dnamic packages support)

[Michael Van Canneyt]

On Mon, 5 Nov 2007, Joost van der Sluis wrote:

> Op maandag 05-11-2007 om 10:44 uur [tijdzone +0100], schreef Michael Van
> Canneyt:
> > 
> > On Sun, 4 Nov 2007, Joost van der Sluis wrote:
> > 
> > > Op zondag 04-11-2007 om 00:34 uur [tijdzone +0100], schreef Michael Van
> > > Canneyt:
> > > > 
> > > > On Sat, 3 Nov 2007, Joost van der Sluis wrote:
> > > > 
> > > > > Op zaterdag 03-11-2007 om 19:57 uur [tijdzone +0100], schreef Michael
> > > > > Van Canneyt:
> > > > > > 
> > > > > > Packages and the missing TClientDataset are the only 2 remaining
> > > > > > things 
> > > > > > stopping me from using FPC/Lazarus for my daytime job. All else is 
> > > > > > covered or can be covered with a minimal amount of work. 
> > > > > 
> > > > > You'll have to explain the TClientDataset to me. Do you need a
> > > > > symantical identical equivalent of TClientDataset?
> > > > 
> > > > I need all features offered by TClientDataset:
> > > > 
> > > > - 3-tier database handling, TProvider and all.
> > > > - Local filtering
> > > > - Local sorting/indexing
> > > > - Locate()
> > > > - Maintained Aggregates
> > > > - Load/Save to file (briefcase model)
> > > > 
> > > > And all features are critical; We have about 3200 instances in 1 application,
> > > > almost all of them using all of the features mentioned, maybe with exception
> > > > of the aggregates.
> > > > 
> > > > > Or do you only miss some features in sqldb?
> > > > 
> > > > SQLDB will be good for our server tier, so I can drop the IBX.
> > > > But it's not suitable for the client layer.
> > > > 
> > > > As I see it, TClientDataset should be built on top of TBufDataset.
> > > > Local filtering is available as fas as I know (or did you put it 
> > > > in TSQLQuery ?), and the indexing probably also can somehow be 
> > > > managed. The locate can be done with a filter. The maintained
> > > > aggregates would need to be implemented from scratch, as well
> > > > as the packet handling and file handling...
> > > 
> > > Local filtering is in TBufDataset, locate also. The packet-handling is
> > > also there?
> > 
> > So you can create a delta packet, it can be sent/applied/whatever, and
> > you can integrate whatever changes were incorporated after the delta
> > was applied ? I must have missed that ?
> 
> Yes. But you can only apply the 'delta-packet'. Or cancel it, offcourse
> (ie: remove it). But that's because saving to disk and sending to
> another TDataser or over the network isn't implemented at all.

Aha. So we're partly there.

> > TProvider is needed to
> > 1. Supply initial data.
> > 2. Apply the deltas and send back reconciling data.
> 
> You don't need a TProvider for that. Take for example sqldb: TSQLQuery
> does nothing more (or less) then a TProvider would do. TBufDataset does
> the rest. (btw: TSQLQuert doesn't send back reconciling data, although
> TBufdataset can handle it)

I understand all this, but the idea of TProvider is that the source
of the data is opaque to the TClientDataset. 

> > > So we can add a TProviderDataset or something, that can connect to data
> > > on an application-server. Is this ok to you? That way you always heve to
> > > re-create your descendent of TBufDataset for every connection?
> > 
> > Hm. I'm not sure I understand this ?
> 
> Read the example above. See TSQLCustomQuery as the provider, TBufDataset
> as the TClientDataset.
>
> 
> > > Or do you like the TClientDataset model better? So that you can connect
> > > it to any TProvider you want - to get access to some data?
> > 
> > This is essential; Without this it is useless to me.
> 
> I need to know if you can also use the model above.

If I understand your proposal correct, the answer is no.

Which doesn't mean your proposal has no worth by itself, I think it does.
 
> > > You're right that we could make a TClientDataset which inherits from
> > > TBufDataset. But that kind of defeats the purpose of TBufDataset.
> > 
> > Nono. You should split out functionalities:
> > 
> > The TBufDataset should handle what it handles now, plus the local indexes
> > and saving to file. In short: a complete in-memory dataset. It doesn't care
> > where the actual data comes from; This is the job of a descendent. 
> > (TSQLQuery, TClientDataset)
> > 
> > What TClientDataset adds to this is the ability to work with a TProvider:
> > - Get records from a remote source (IAppProvider interface).
> > - Send and Apply delta, integrate result back in buffer.
> 
> TBufDataset can already do this. A descendent has to:
> 
> Override Fetch, LoadField, LoadBlobintoBuffer and ApplyrecUpdate.
> (integrating the results in the buffer is done by TBufDataset, just as
> TClientDataset does)
> 
> So TClientDataset could override these functions simply in such a way
> that they call the same functions, but then form an external provider.
> 
> ie:
> 
> function TClientDataset.Fetch : boolean;
> begin
>   Result := AProvider.Fetch;
> end;
> 
> And all the like...
> 
> If you really need compatibility with the IAppProvider interface (why?)

Why ? 
Because we make use of it in 3200 places, and recoding this is
not an option :-)

> then there's more work to do. But that was what I meant with my first
> question. Do you need the functionality ot TClientDataset, or a
> symantical equivalent?

I need IAppProvider support, symantically equivalent.

With what you say, this should not be hard to implement, TClientDataset
just needs a 'IAppServer' functionality.

Implementing TProvider would be more work.

> 
> > > Because if you do so, it would also be better to base TSQLQuery on
> > > TDataset directly. Or else you're double-buffering your data when you
> > > connect a TSQLQuery (TBufDataset) with a TProvider to a TClientDataset
> > > (another TBufDataset).
> > 
> > You'll always have double buffering. Don't forget that TSQLQuery and TProvider
> > are on the application server tier, and TClientDataset is on the client
> > tier. The client tier doesn't have direct access to the database so
> > TSQLQuery would be useless.
> 
> On the client side you're right. But on the server side... woudn't you
> need both a TSQLQuery and a TClientDataset on the application-server?

That depends on how you want to apply the delta...

> 
> > That's why a 'unidirectional' property in TSQLQuery/TBufDataset is needed: 
> > in that case, only 2 buffers are needed: one for the current record and one 
> > for edit/update/filter. Makes a HUGE difference on the server. IBX implemented 
> > it especially for that.
> 
> What does TBufDataset do more then TDataset when that property is set to
> True?

? It actually keeps a buffer with the current data. 
TDataset has no data buffer by itself, it only 'knows' about buffers.
 
> If you need a unidirectional dataset, don't use a TBufDataset. That's
> also why I once made a TSQLQuery based on TDataset. 

I understand what you're getting at. The problem is mainly that we don't
know in advance whether a particular instance will be unidirectional or not.
We still set it at runtime, based on some other properties. 

If I would do as you propose, I would have to change one instance of
TDataset with another... I need 1 component that supports both models.

If I would start from zero, I would not have problems with setting up the
whole thing to something like you describe. The main problem is that there 
is almost 10 years of code which should work as-is, as much as possible.
Since we heavily rely on almost all aspects of TClientDataset/TProvider 
stuff, that means we need it almost 100% equivalently...

We could try to compile Borlands' VCL code with FPC, of course, but then
there is the problem that the midas library is a black box, available only
for windows i386, which kind of defeats the purpose of the whole exercise..

Michael.