[fpc-devel]Hi for first time, and a question about tokens.pas

[Peter Vreman]

> Well, as most of you perhaps know I am trying to translate freepascal to 
> spanish. Well, I have many problems with tokens.pas file.
> 
> And the boards, aren't as much as I wanted, so I came to this emaillist.
> 
> The one that is in pp\source\compiler
> 
> What I try is to recompile the compiler, with a changed tokens.pas file
> 
> I've noticed that in tokens.pas there's a procedure that orders the normal 
> words, which are treated by this procedure because they are not special 
> (boolean special is false). But it needs the token str to be ordered as they 
> are... you know first by length and when the same length, you order it 
> alphabetacally.
> 
> I've seen many files that treat tokens defined in tokens.pas, that means that 
> PERHAPS they have to be ordered in tokens.pas BUT I'M NOT SURE. 
> 
> And I'm also not sure of these files being the unique to access tokens.pas and 
> need something ordered.

The order in should be by length and then by alphabet. The only thing you need to take
care off is to change the order accordingly in the enum declaration. And also all
operators should be after eachother so that they can be checked by a range.

> In pexpr.pas I found this:
> 
>   const
>      { Warning these stay be ordered !! }
>      operator_levels:array[Toperator_precedence] of set of Ttoken=
>         ([_LT,_LTE,_GT,_GTE,_EQUAL,_UNEQUAL,_OP_IN,_OP_IS],
>          [_PLUS,_MINUS,_OP_OR,_OP_XOR],
>          [_CARET,_SYMDIF,_STARSTAR,_STAR,_SLASH,
>           _OP_AS,_OP_AND,_OP_DIV,_OP_MOD,_OP_SHL,_OP_SHR],
>          [_STARSTAR] );
> 
> in line 2027 if you want to check.
> 
> WHAT DOES MEAN "Warning these stay be ordered "?
> They have to be ordered as they are in tokens.pas
> like the original pexpr.pas or what?

The predence order. The order of the tokens itself doesn't matter here as it's a set.

> 
> 
> psub.pas
> in line 425 I have this:
> 
>   Message(parser_e_overload_operator_failed);
> optoken:=token;
> consume(Token);
> 
> What means that goes from _PLUS (I think it is because
> it is supposed to be always the first token )
> 
> and to last_overloaded, which is defined somewhere else:

This is a thing that has been forgotten to update, the constants can easily be moved to
tokens.pas and fixed in the psub unit

> And another thing about the procedure that orders (well ok, in fact
> it creates an index) I don't know if it works ok.

It only determines at which placed in the array a word with length X and starting with
letter Y can be found so it reduces the place where it needs to search for the word.

> 
> With this tokens.pas (www.geocities.com/beatlesnap/tokens.pas) "Repite" token 
> (I change repit token and i put repite) is not detected as a known identifier.
> 
> Is it that I don't order it ok, or it's a bug?
> 
> Thank you for all, "repite" not known as a token is driving me
> mad.

Here it's working correct. Did you delete the .ppu files first and then recompiled the
compiler?