[fpc-devel] TClientDataset (was: Dnamic packages support)

[Joost van der Sluis]

Op maandag 05-11-2007 om 12:37 uur [tijdzone +0100], schreef Michael Van
Canneyt:
> 
> On Mon, 5 Nov 2007, Joost van der Sluis wrote:
> 
> > Op maandag 05-11-2007 om 10:44 uur [tijdzone +0100], schreef Michael Van
> > Canneyt:
> > > 
> > > On Sun, 4 Nov 2007, Joost van der Sluis wrote:
> > > 
> > > > Op zondag 04-11-2007 om 00:34 uur [tijdzone +0100], schreef Michael Van
> > > > Canneyt:
> > > > > 
> > > > > On Sat, 3 Nov 2007, Joost van der Sluis wrote:
> > > > > 
> > > > > > Op zaterdag 03-11-2007 om 19:57 uur [tijdzone +0100], schreef Michael
> > > > > > Van Canneyt:
> > > > > > > 
> > > > > > > Packages and the missing TClientDataset are the only 2 remaining
> > > > > > > things 
> > > > > > > stopping me from using FPC/Lazarus for my daytime job. All else is 
> > > > > > > covered or can be covered with a minimal amount of work. 
> > > > > > 
> > > > > > You'll have to explain the TClientDataset to me. Do you need a
> > > > > > symantical identical equivalent of TClientDataset?
> > > > > 
> > > > > I need all features offered by TClientDataset:
> > > > > 
> > > > > - 3-tier database handling, TProvider and all.
> > > > > - Local filtering
> > > > > - Local sorting/indexing
> > > > > - Locate()
> > > > > - Maintained Aggregates
> > > > > - Load/Save to file (briefcase model)
> > > > > 
> > > > > And all features are critical; We have about 3200 instances in 1 application,
> > > > > almost all of them using all of the features mentioned, maybe with exception
> > > > > of the aggregates.
> > > > > 
> > > > > > Or do you only miss some features in sqldb?
> > > > > 
> > > > > SQLDB will be good for our server tier, so I can drop the IBX.
> > > > > But it's not suitable for the client layer.
> > > > > 
> > > > > As I see it, TClientDataset should be built on top of TBufDataset.
> > > > > Local filtering is available as fas as I know (or did you put it 
> > > > > in TSQLQuery ?), and the indexing probably also can somehow be 
> > > > > managed. The locate can be done with a filter. The maintained
> > > > > aggregates would need to be implemented from scratch, as well
> > > > > as the packet handling and file handling...
> > > > 
> > > > Local filtering is in TBufDataset, locate also. The packet-handling is
> > > > also there?
> > > 
> > > So you can create a delta packet, it can be sent/applied/whatever, and
> > > you can integrate whatever changes were incorporated after the delta
> > > was applied ? I must have missed that ?
> > 
> > Yes. But you can only apply the 'delta-packet'. Or cancel it, offcourse
> > (ie: remove it). But that's because saving to disk and sending to
> > another TDataser or over the network isn't implemented at all.
> 
> Aha. So we're partly there.
> 
> > > TProvider is needed to
> > > 1. Supply initial data.
> > > 2. Apply the deltas and send back reconciling data.
> > 
> > You don't need a TProvider for that. Take for example sqldb: TSQLQuery
> > does nothing more (or less) then a TProvider would do. TBufDataset does
> > the rest. (btw: TSQLQuert doesn't send back reconciling data, although
> > TBufdataset can handle it)
> 
> I understand all this, but the idea of TProvider is that the source
> of the data is opaque to the TClientDataset. 
> 
> > > > So we can add a TProviderDataset or something, that can connect to data
> > > > on an application-server. Is this ok to you? That way you always heve to
> > > > re-create your descendent of TBufDataset for every connection?
> > > 
> > > Hm. I'm not sure I understand this ?
> > 
> > Read the example above. See TSQLCustomQuery as the provider, TBufDataset
> > as the TClientDataset.
> >
> > 
> > > > Or do you like the TClientDataset model better? So that you can connect
> > > > it to any TProvider you want - to get access to some data?
> > > 
> > > This is essential; Without this it is useless to me.
> > 
> > I need to know if you can also use the model above.
> 
> If I understand your proposal correct, the answer is no.
> 
> Which doesn't mean your proposal has no worth by itself, I think it does.
>  
> > > > You're right that we could make a TClientDataset which inherits from
> > > > TBufDataset. But that kind of defeats the purpose of TBufDataset.
> > > 
> > > Nono. You should split out functionalities:
> > > 
> > > The TBufDataset should handle what it handles now, plus the local indexes
> > > and saving to file. In short: a complete in-memory dataset. It doesn't care
> > > where the actual data comes from; This is the job of a descendent. 
> > > (TSQLQuery, TClientDataset)
> > > 
> > > What TClientDataset adds to this is the ability to work with a TProvider:
> > > - Get records from a remote source (IAppProvider interface).
> > > - Send and Apply delta, integrate result back in buffer.
> > 
> > TBufDataset can already do this. A descendent has to:
> > 
> > Override Fetch, LoadField, LoadBlobintoBuffer and ApplyrecUpdate.
> > (integrating the results in the buffer is done by TBufDataset, just as
> > TClientDataset does)
> > 
> > So TClientDataset could override these functions simply in such a way
> > that they call the same functions, but then form an external provider.
> > 
> > ie:
> > 
> > function TClientDataset.Fetch : boolean;
> > begin
> >   Result := AProvider.Fetch;
> > end;
> > 
> > And all the like...
> > 
> > If you really need compatibility with the IAppProvider interface (why?)
> 
> Why ? 
> Because we make use of it in 3200 places, and recoding this is
> not an option :-)

Ok, then there's the question: Do we want to provide the ability to use
existing DB-code with fpc?

It's a serious question, because if decide to do so, that will cost
resources which could be otherwise used for something else.
(Implementing a better db-model? Is the Delphi-model perfect?)

> > then there's more work to do. But that was what I meant with my first
> > question. Do you need the functionality ot TClientDataset, or a
> > symantical equivalent?
> 
> I need IAppProvider support, symantically equivalent.
> 
> With what you say, this should not be hard to implement, TClientDataset
> just needs a 'IAppServer' functionality.
> 
> Implementing TProvider would be more work.
> 
> > 
> > > > Because if you do so, it would also be better to base TSQLQuery on
> > > > TDataset directly. Or else you're double-buffering your data when you
> > > > connect a TSQLQuery (TBufDataset) with a TProvider to a TClientDataset
> > > > (another TBufDataset).
> > > 
> > > You'll always have double buffering. Don't forget that TSQLQuery and TProvider
> > > are on the application server tier, and TClientDataset is on the client
> > > tier. The client tier doesn't have direct access to the database so
> > > TSQLQuery would be useless.
> > 
> > On the client side you're right. But on the server side... woudn't you
> > need both a TSQLQuery and a TClientDataset on the application-server?
> 
> That depends on how you want to apply the delta...
> 
> > 
> > > That's why a 'unidirectional' property in TSQLQuery/TBufDataset is needed: 
> > > in that case, only 2 buffers are needed: one for the current record and one 
> > > for edit/update/filter. Makes a HUGE difference on the server. IBX implemented 
> > > it especially for that.
> > 
> > What does TBufDataset do more then TDataset when that property is set to
> > True?
> 
> ? It actually keeps a buffer with the current data. 
> TDataset has no data buffer by itself, it only 'knows' about buffers.
>  
> > If you need a unidirectional dataset, don't use a TBufDataset. That's
> > also why I once made a TSQLQuery based on TDataset. 
> 
> I understand what you're getting at. The problem is mainly that we don't
> know in advance whether a particular instance will be unidirectional or not.
> We still set it at runtime, based on some other properties. 
> 
> If I would do as you propose, I would have to change one instance of
> TDataset with another... I need 1 component that supports both models.
> 
> If I would start from zero, I would not have problems with setting up the
> whole thing to something like you describe. The main problem is that there 
> is almost 10 years of code which should work as-is, as much as possible.
> Since we heavily rely on almost all aspects of TClientDataset/TProvider 
> stuff, that means we need it almost 100% equivalently...
> 
> We could try to compile Borlands' VCL code with FPC, of course, but then
> there is the problem that the midas library is a black box, available only
> for windows i386, which kind of defeats the purpose of the whole exercise..

I think I understand your point of view now. At first I thought that
TBufDataset wouldn't be needed if we make a TClientDataset. But if you
want to use a property to switch unidirectional on and off, it makes
sense.

Joost