[fpc-devel]Re: question about tokens.pas

[Pierre Muller]

>Can I declare a TOKEN of length 1 such as "y" or "o"?

  Currently, the scanner knows that there are no single chars that
are tokens :

See this text extracted from
tscannerfile.readtoken (line 1227 of current CVS state)

          { keyword or any other known token,
            pattern is always uppercased }
            if (pattern[1]<>'_') and (length(pattern) in [2..tokenidlen]) then
             begin
               low:=ord(tokenidx^[length(pattern),pattern[1]].first);
               high:=ord(tokenidx^[length(pattern),pattern[1]].last);

If you have one char tokens, you need to change the [2..tokenidlen]
into a [1..tokenidlen].

The following lines also explain why you need to have
alphabetically ordered tokens.