[fpc-pascal] Need help to fix a bug

[Joost van der Sluis]

On Wed, 2010-01-13 at 16:13 +0100, JoshyFun wrote:
> I wish to fix this bug http://bugs.freepascal.org/view.php?id=15460
> but I had found serious problems to understand how data is structured
> in the TBufIndex and descendant classes, specially the
> TDoubleLinkedBufIndex.

Oohh... nice... Someone to help me. ;)

Too bad you used such a general topic-name, I almost overlooked it.

> Can somebody help me ? From TBufDataset.BuildIndex I think that:

Note that TBufDataset supports more then one index-system. (index-based
and double-linked-list based) Only the second form supports indexing
atm, so that TBufDataset.BuildIndex assumes that the double-linked-list
us used. I'll also assume this further on in this mail.

> PBufRecLinkItem.Next points to next record.
> PBufRecLinkItem.Prior points to previous record.
> 
> Maybe PBufRecLinkItem points to the dataset item data ?

Somewhat. The item-data and the linked list are stored within one memory
block. The PBufRecLinkItem points at the start of that memory block. The
block starts with the several indexes, then the real data. Now assume
that RecPtr is a PBufRecLinkItem which points at address 0000.

Assume a pointer is 32 bit/4 bytes, one record is stored into memory as
follows:

Address  What is stored inside it   Pointer to this address

0000     PBufRecLinkItem.prior (1)  RecPtr[0].prior
0004     PBufRecLinkItem.next  (1)  RecPtr[0].next   
0008     PBufRecLinkItem.prior (2)  RecPtr[1].prior
000c     PBufRecLinkItem.next  (2)  RecPtr[1].next   
0010     PBufRecLinkItem.prior (3)  RecPtr[2].prior
0014     PBufRecLinkItem.next  (3)  RecPtr[2].next   
0018     Start of real data

So in one record-memory block, the indexes and the data are stored. As
you can see the block above can contain three indexes. Not more, because
in that case the memory for all records should be re-allocated. This is
no problem when you define your indexes before opening the dataset.
(This is forced by TBufDataset when using this double-linked index) By
default there are always two indexes defined: the primary index which
contains the records in the order they are read. And a second index
which is initially blank. This index is used when you use the
IndexFieldNames property. Now you also know why it's not necessary to
allocate memory when building this second index. It's already allocated
during the initial allocation for the data for the records. It's also
why this merge-sort algorithm is used. It doesn't need any _additional_
space.

> Which is supossed to be pointed by:
> 
> (AIndex as TDoubleLinkedBufIndex).FFirstRecBuf
> (AIndex as TDoubleLinkedBufIndex).FLastRecBuf
> The first and last item in the dataset ?

Well, the TDoubleLinkedBufIndex contains the additional
index-information and functions to work with it. FFirstRecBuf points to
the first record. (It's address 0000 as in the memory-layout above)
FLastRecBuf points to the record _after_ the last record. This record is
used as the 'spare' record. It is used for all kinds of tasks, like the
record which you use when you set a filter. Or in which data is read

> Also which is the meaning of this line:
> 
> (AIndex as TDoubleLinkedBufIndex).FLastRecBuf[(AIndex as TDoubleLinkedBufIndex).IndNr].next:=(AIndex as TDoubleLinkedBufIndex).FFirstRecBuf;

Well, AIndex is the used index. FLastRecBuf is it's spare record.
AIndex.IndNr is the number of the index. Since FLastRecBuf is defined as
a record containing a prior and a next pointer, FLastRecBuf[0].next
points to the .prior element of the first (zeroth) index.
FLastRecBuf[1].next points to the second index. Which is hard-coded the
index used for the IndexFieldNames-index.

So the line above let the next record of the spare-record point to the
first record of the index. (Creating a closed loop. Seems to be a bad
idea, but there is probably some logic behind it)

> I'm completly lost in this piece of code, and I really need it as my
> SQL fetches could take around 1 minute and reorder data is a must. I
> know I can reissue the query with a different order but this will take
> around 20 seconds again while doing it in local memory is a breeze.

It should work. Maybe you can try if you can get the tests in
packages/fcl-db/tests running. There is also a test for IndexFieldNames,
iirc. I can't help you with the bug report as long as I don't have an
example.

It would be completely perfect if you can create a test for the
db-testsuite that fails, so that I can fix that.

> PS: To check changes in FPC I'm launching a complete build (without
> clean) but this process will take 2 minutes anyway. Is there a faster
> way to correctly build only the affected package (in this case fcl-db)

Yes, go to the packages/fcl-db directory and do a 'make all install'.
Then compile your program. You only have to recompile the LCL (nothing
more) if you changed something in the interface-section of one of the
db-units.

You can use the 'before compilation' settings of Lazarus to do the 'make
all install' You can even let Lazarus parse it's output, so that in case
of an error, Lazarus will show you the offending line.

This works in 90% of the cases. So you almost never have to clean and
rebuild everything.

You can also simply add packages/fcl-db/src, packages/fcl-db/src/base,
packages/fcl-db/src/sqldb, packages/fcl-db/src/sqldb/interbase and all
other database-paths to the compiler-search-path in Lazarus. But cleanup
the created .ppu's and .o's afterwards, or you'll run into trouble when
you forget them.

Joost.