[fpc-devel] [patch] pscanner: differentiate between EOL & Tab characters from general Whitespace

[Graeme Geldenhuys]

On 22 August 2010 18:25, Michael Van Canneyt wrote:
>
> Why ? To the compiler there is no difference between EOL and whitespace.
>
> If you want to generate it with the exact layout, then, yes.
> But for profiling I don't see the need.

Maybe to the compiler it doesn't matter, but to a human trying to
debug their profiler to see if they inserted code in the correct
place, or to see why code doesn't compile any more after the profiler
modified the code - such "human readable code" is rather important.
:)
If no NewLine characters are inserted into the newly generated units,
a compiler error on line 1 doesn't help the developer much. Line 1
could be 50k bytes or more long.

Lazarus can't open the file (error message says it doesn't look like a
text file), other editor show one line, but clips text because they
can't display such a long single line of text. The profiler becomes a
near perfect obfuscater. :-)


>
> Well, I still fail to see why you need the tab and EOL.

Well, does it hurt having them tokenized?

If anybody other than me comes up with a new idea/way of using the
fcl-passrc parser on tokenizer - maybe they would like to rebuild
source code at some point. Without tokens telling you where EOL used
to be or TAB characters used to be (though the latter is not to
important), it does limit it's functionality.

Attached is a file processed with the profiler and fcl-passrc that
doesn't use tkLineEnding or tkTab. It's rather hard to read and fix
compiler errors introduced by the profile inserting code.

I don't see why introducing such new tokens is bad? It can only make
the tokenizer and parser more useful in the long term.



-- 
Regards,
  - Graeme -


_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fpputils.pas
Type: text/x-pascal
Size: 6915 bytes
Desc: not available
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20100822/c0e7bbf7/attachment.pas>