[fpc-devel]Re: question about tokens.pas

[Peter Vreman]

> > 
> > >Can I declare a TOKEN of length 1 such as "y" or "o"?
> > 
> >   Currently, the scanner knows that there are no single chars that
> > are tokens :
> > 
> > See this text extracted from
> > tscannerfile.readtoken (line 1227 of current CVS state)
> > 
> >           { keyword or any other known token,
> >             pattern is always uppercased }
> >             if (pattern[1]<>'_') and (length(pattern) in [2..tokenidlen]) then
> >              begin
> >                low:=ord(tokenidx^[length(pattern),pattern[1]].first);
> >                high:=ord(tokenidx^[length(pattern),pattern[1]].last);
> > 
> > If you have one char tokens, you need to change the [2..tokenidlen]
> > into a [1..tokenidlen].
> > 
> > The following lines also explain why you need to have
> > alphabetically ordered tokens.
> 
> << Can you tell me in which file you have found it? I'm browsing CVS
> in "http://www.freepascal.org/cgi-bin/cvsweb/fpc/compiler/" I've searched 
> tscannerfile, or readtoken or state (there weren't such files) and in 
> tokens.pas, in the last version, I didn't find (only in tokens.pas) that
> the procedure/function you have told me about exists.

In scanner.pas at line 1241.

> 
> << Another question in tokens.pas just before const arraytokeninfo:
> 
> ttokenidx=array[2..tokenidlen,'A'..'Z'] of tokenidxrec;
> 
> Do I have to change 2 and put a 1, to admit tokens of only 1 char?
> And what is ttokenidx because it's like a matrix of tokenidxrec which is a 
> record with first and last which are tokens. 

Yes, you also need to adjust this table

> 
> It's like a chart or table, in which square there are two tokens, quite strange,
> could you please explain it to me?


It's to limit the binary search. When a token is read from the source the length of the
token and the first character of the token are used to get the lowest and highest value
from this table. These values are then used to limit the binary search in the token list.

> 
> <<Why tokenideln value is: 14? Any special reason?

Just to limit, the largest english keyword is 14 characters, if longer keywords are
available it'll be increased.

> 
> << Well, I've noticed tokens.pas declares tokens, but the only important 
> procedure is create_tokenidx, Which only process NORMAL WORDS cause their 
> special boolean value is false. And nothing more.
> 
> So could you tell me please where the other tokens(operators mainly) are 
> proccesed (in which other files) to make an index or whatever? 

They are assigned in scanner.pas when parsing things like a bracket or plus character. And
for tokens like 'in' they are assigned using the op field from the token array.