[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace

[Michael Van Canneyt]

On Sun, 22 Aug 2010, Graeme Geldenhuys wrote:


> Well, does it hurt having them tokenized?

Of course. 
It slows the code. Marginally, I agree, but nevertheless.

>
> If anybody other than me comes up with a new idea/way of using the
> fcl-passrc parser on tokenizer - maybe they would like to rebuild
> source code at some point. Without tokens telling you where EOL used
> to be or TAB characters used to be (though the latter is not to
> important), it does limit it's functionality.
>
> Attached is a file processed with the profiler and fcl-passrc that
> doesn't use tkLineEnding or tkTab. It's rather hard to read and fix
> compiler errors introduced by the profile inserting code.
>
> I don't see why introducing such new tokens is bad? It can only make
> the tokenizer and parser more useful in the long term.

I never said it is bad; I just don't see why you need it.

You could perfectly do it differently, e.g. insert a newline after certain tokens:
then
begin
;
end
else
var
type
const

That would perfectly be enough for your purposes, which is, to debug your profiler.
And once that is done, it is no longer needed at all.

Don't worry, I'll apply the patch, but you have not convinced me it is a
needed patch, even for your purposes. I'm surprised you didn't think of 
the above ways to do what you need.

Michael