[fpc-devel]Re: question about tokens.pas

[adrian15 at europe.com]

> 
> >Can I declare a TOKEN of length 1 such as "y" or "o"?
> 
>   Currently, the scanner knows that there are no single chars that
> are tokens :
> 
> See this text extracted from
> tscannerfile.readtoken (line 1227 of current CVS state)
> 
>           { keyword or any other known token,
>             pattern is always uppercased }
>             if (pattern[1]<>'_') and (length(pattern) in [2..tokenidlen]) then
>              begin
>                low:=ord(tokenidx^[length(pattern),pattern[1]].first);
>                high:=ord(tokenidx^[length(pattern),pattern[1]].last);
> 
> If you have one char tokens, you need to change the [2..tokenidlen]
> into a [1..tokenidlen].
> 
> The following lines also explain why you need to have
> alphabetically ordered tokens.

<< Can you tell me in which file you have found it? I'm browsing CVS
in "http://www.freepascal.org/cgi-bin/cvsweb/fpc/compiler/" I've searched 
tscannerfile, or readtoken or state (there weren't such files) and in 
tokens.pas, in the last version, I didn't find (only in tokens.pas) that
the procedure/function you have told me about exists.

<< Another question in tokens.pas just before const arraytokeninfo:

ttokenidx=array[2..tokenidlen,'A'..'Z'] of tokenidxrec;

Do I have to change 2 and put a 1, to admit tokens of only 1 char?
And what is ttokenidx because it's like a matrix of tokenidxrec which is a 
record with first and last which are tokens. 

It's like a chart or table, in which square there are two tokens, quite strange,
could you please explain it to me?

<<Why tokenideln value is: 14? Any special reason?

<< Well, I've noticed tokens.pas declares tokens, but the only important 
procedure is create_tokenidx, Which only process NORMAL WORDS cause their 
special boolean value is false. And nothing more.

So could you tell me please where the other tokens(operators mainly) are 
proccesed (in which other files) to make an index or whatever? 



---------------------------------------------
This message was sent using Endymion MailMan.
http://www.endymion.com/products/mailman/