[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace

[Michael Van Canneyt]

On Mon, 23 Aug 2010, Graeme Geldenhuys wrote:

> Op 2010-08-23 01:17, Michael Van Canneyt het geskryf:
>>> Well, does it hurt having them tokenized?
>>
>> Of course.
>> It slows the code. Marginally, I agree, but nevertheless.
>
> Sorry, but I disagree here. Before it got tokenized as a tkWhitespace; now
> it gets tokenized as tkLineEnding or tkTab. It's still being assigned a
> token, just one that describes it better. As for the parsers NextToken
> method now having two extra tokens it needs to ignore. I profiled that, and
> the difference was too small to notice.

That's why I said 'marginally' :-)

>
>
>> I never said it is bad; I just don't see why you need it.
>>
>> You could perfectly do it differently, e.g. insert a newline after certain tokens:
>> then
>> begin
>> ;
>
> Vincent hit the nail on the head. Now the compiled code with profiling has
> very different line numbers (now much more than before) to the real code.
> So line number info reported by LineInfo unit is now even more useless. And
> considering that under 64-bit systems, method names are not reported by
> LineInfo for some reason, the line number info is rather important.

This is IMHO a fallacy: the line numbers will change anyway if you insert code
lines, so you must check the generated code with profiling in each case, not
the original code. Therefor the information about the original lines is not
needed.

Anyway, the point is moot; I'll apply the patch.

Michael.