[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace
Michael Van Canneyt
michael at freepascal.org
Mon Aug 23 01:17:58 CEST 2010
On Sun, 22 Aug 2010, Graeme Geldenhuys wrote:
> Well, does it hurt having them tokenized?
It slows the code. Marginally, I agree, but nevertheless.
> If anybody other than me comes up with a new idea/way of using the
> fcl-passrc parser on tokenizer - maybe they would like to rebuild
> source code at some point. Without tokens telling you where EOL used
> to be or TAB characters used to be (though the latter is not to
> important), it does limit it's functionality.
> Attached is a file processed with the profiler and fcl-passrc that
> doesn't use tkLineEnding or tkTab. It's rather hard to read and fix
> compiler errors introduced by the profile inserting code.
> I don't see why introducing such new tokens is bad? It can only make
> the tokenizer and parser more useful in the long term.
I never said it is bad; I just don't see why you need it.
You could perfectly do it differently, e.g. insert a newline after certain tokens:
That would perfectly be enough for your purposes, which is, to debug your profiler.
And once that is done, it is no longer needed at all.
Don't worry, I'll apply the patch, but you have not convinced me it is a
needed patch, even for your purposes. I'm surprised you didn't think of
the above ways to do what you need.
More information about the fpc-devel